How to Build an Internal Developer Platform: A Complete Guide to Backstage, ArgoCD, and Crossplane

<p>Every fast-growing engineering team eventually hits the same wall.</p> <p>A developer needs a new staging environment, so they file a ticket. The platform team queues it.</p> <p>Two weeks later, the environment exists. It's configured slightly differently from the last one, with a naming convention that doesn't match the production setup, missing the observability stack the previous environment had. The developer deploys. Something breaks. Nobody knows why.</p> <p>The problem isn't the ticket queue. The problem is the absence of a platform: a paved road where developers can self-serve infrastructure, deployments, and environments that are consistent, auditable, and safe without requiring a platform engineer for every request.</p> <p>An Internal Developer Platform (IDP) solves this. Not by removing platform engineers from the picture, but by shifting their work from executing individual requests to building the systems that execute those requests automatically.</p> <p>This handbook builds a production-grade IDP from the three CNCF tools that form its core in 2026: Backstage as the developer portal and software catalog, ArgoCD as the GitOps continuous delivery engine, and Crossplane as the Kubernetes-native infrastructure control plane.</p> <p>By the end, developers on your platform will be able to provision a cloud database, deploy an application to staging, and register a new service in the catalog — all without filing a single ticket.</p> <h2 id="heading-table-of-contents">Table of Contents</h2> <ul> <li><p><a href="#heading-what-youll-learn">What You'll Learn</a></p> </li> <li><p><a href="#heading-prerequisites">Prerequisites</a></p> </li> <li><p><a href="#heading-part-1-idp-architecture-the-three-layer-model">Part 1: IDP Architecture — The Three-Layer Model</a></p> </li> <li><p><a href="#heading-part-2-argocd-the-gitops-foundation">Part 2: ArgoCD — The GitOps Foundation</a></p> </li> <li><p><a href="#heading-part-3-crossplane-infrastructure-as-kubernetes-resources">Part 3: Crossplane — Infrastructure as Kubernetes Resources</a></p> </li> <li><p><a href="#heading-part-4-backstage-the-developer-portal">Part 4: Backstage — The Developer Portal</a></p> </li> <li><p><a href="#heading-part-5-wiring-it-together-the-golden-path">Part 5: Wiring It Together — The Golden Path</a></p> </li> <li><p><a href="#heading-part-6-finops-integration-cost-attribution-on-the-idp">Part 6: FinOps Integration — Cost Attribution on the IDP</a></p> </li> <li><p><a href="#heading-part-7-the-platform-maturity-model-measuring-what-youve-built">Part 7: The Platform Maturity Model — Measuring What You've Built</a></p> </li> <li><p><a href="#heading-best-practices-summary">Best Practices Summary</a></p> </li> <li><p><a href="#heading-resources">Resources</a></p> </li> </ul> <h2 id="heading-what-youll-learn">What You'll Learn</h2> <ul> <li><p>The three-layer IDP architecture and why each layer must be implemented in a specific order</p> </li> <li><p>How to install and configure ArgoCD with ApplicationSets for multi-environment GitOps delivery</p> </li> <li><p>How to define cloud infrastructure as Kubernetes custom resources using Crossplane Compositions</p> </li> <li><p>How to deploy and configure Backstage with a software catalog and Software Templates</p> </li> <li><p>How to wire Backstage, ArgoCD, and Crossplane together into a single self-service golden path</p> </li> <li><p>How to implement cost attribution on your IDP so every resource provisioned through it carries team and cost center metadata</p> </li> <li><p>How to measure your IDP's maturity using the CNCF Platform Engineering Maturity Model</p> </li> </ul> <p>Let's build it.</p> <h2 id="heading-prerequisites">Prerequisites</h2> <p>Before following along, you should have:</p> <p><strong>Knowledge:</strong></p> <ul> <li><p>Working familiarity with Kubernetes: you can deploy applications, write YAML manifests, and understand namespaces and RBAC</p> </li> <li><p>Basic GitOps understanding: you know what "Git as source of truth" means in practice</p> </li> <li><p>Comfort with Helm, Terraform HCL, and TypeScript at a reading level</p> </li> <li><p>Understanding of AWS services: EKS, RDS, S3, IAM</p> </li> </ul> <p><strong>Tools and access:</strong></p> <ul> <li><p>An EKS cluster running Kubernetes 1.28 or later with at least 3 nodes (m5.xlarge or equivalent)</p> </li> <li><p><code>kubectl</code> configured and pointing at your cluster</p> </li> <li><p><code>helm</code> 3.12 or later installed</p> </li> <li><p>AWS CLI v2 configured with admin-level permissions for the provisioning steps</p> </li> <li><p>Node.js 18 or later and Yarn (for Backstage)</p> </li> <li><p>A GitHub organisation you control (for the GitOps repositories and Backstage GitHub integration)</p> </li> </ul> <p><strong>Companion repository:</strong></p> <pre><code class="language-bash">git clone https://github.com/aayostem/platform-toolkit cd platform-toolkit </code></pre> <p>The repository contains all manifests, Helm values files, Crossplane Compositions, and Backstage templates referenced in this guide. Each part maps to a directory in the repo.</p> <p><strong>Estimated time:</strong> The full implementation takes one to two days for an experienced platform engineer. Parts 1–3 can be completed in the morning and produce a working GitOps delivery layer.</p> <h2 id="heading-part-1-idp-architecture-the-three-layer-model">Part 1: IDP Architecture — The Three-Layer Model</h2> <h3 id="heading-11-what-an-idp-actually-is">1.1 What an IDP Actually Is</h3> <p>An Internal Developer Platform isn't a tool. It's a product: a collection of tools, workflows, and abstractions that platform teams build and maintain so that application developers can move fast without managing infrastructure directly.</p> <p>The distinction matters because it shapes every architectural decision. A tool is installed and configured. A product is designed for users, iterated based on feedback, and measured by whether those users actually adopt it. The platform teams that build the IDPs that developers love think like product managers, not system administrators.</p> <p><a href="https://cloud.google.com/resources/content/2025-dora-ai-capabilities-model-report">The DORA 2025 report</a> found that nearly 90% of enterprises now have some form of internal platform. But having a platform and having a platform that developers actually use are different things.</p> <p>The survey found that developer satisfaction with internal platforms varied dramatically. And the gap between satisfied and unsatisfied teams correlated directly with whether the platform team treated the IDP as a product with a roadmap and user research, or as an infrastructure project with a ticket queue.</p> <p>The three tools in this guide — Backstage, ArgoCD, and Crossplane — are the most widely adopted open-source stack for production IDPs in 2026. But the architecture that connects them matters as much as the tools themselves.</p> <h3 id="heading-12-the-three-layer-architecture">1.2 The Three-Layer Architecture</h3> <p>A production IDP has three distinct layers, each with a single responsibility:</p> <pre><code class="language-plaintext">Layer 1: Developer Interface (Backstage) ├── Software catalog — inventory of all services, APIs, and resources ├── Software Templates — self-service forms that trigger provisioning workflows ├── TechDocs — documentation co-located with each catalog entity └── Plugins — integrations with ArgoCD, Kubernetes, PagerDuty, Grafana Layer 2: Delivery Layer (ArgoCD) ├── GitOps sync — continuous reconciliation of cluster state to Git ├── ApplicationSets — multi-environment deployment from a single definition ├── Rollout management — progressive delivery with health checks └── Audit trail — every deployment change linked to a Git commit Layer 3: Infrastructure Layer (Crossplane) ├── Composite Resources — cloud resources defined as Kubernetes CRDs ├── Compositions — templates that expand a simple claim into full AWS infrastructure ├── ProviderConfigs — credentials and region configuration for each cloud provider └── Usage tracking — every provisioned resource tagged with team and cost centre </code></pre> <p>The critical architectural rule: Backstage never talks directly to Kubernetes or cloud APIs. When a developer submits a Software Template in Backstage, the output is a Git commit — a YAML file representing a Crossplane claim or an ArgoCD Application manifest. ArgoCD picks up that commit and applies it to the cluster. Crossplane translates the cluster resource into actual cloud infrastructure.</p> <p>This indirect path isn't complexity for complexity's sake. It means every infrastructure change is a Git commit, with an author, a timestamp, a pull request, and a review. The audit trail is automatic. The rollback mechanism is <code>git revert</code>.</p> <pre><code class="language-plaintext">Developer → Backstage Template → Git commit → ArgoCD → Crossplane → AWS ↑ Single source of truth Full audit trail Rollback = git revert </code></pre> <p>Here's what the incorrect alternative looks like — Backstage calling cloud APIs directly:</p> <pre><code class="language-typescript">// Bad: Backstage template calling AWS SDK directly // No audit trail, no rollback, no reconciliation loop // If the call fails halfway, you have partial infrastructure with no record import { S3Client, CreateBucketCommand } from "@aws-sdk/client-s3"; const client = new S3Client({ region: "us-east-1" }); await client.send(new CreateBucketCommand({ Bucket: bucketName })); </code></pre> <p>And the correct approach — Backstage outputting a Crossplane claim to Git:</p> <pre><code class="language-yaml"># Good: Backstage template output — a Crossplane claim committed to Git # ArgoCD applies it, Crossplane reconciles it, AWS creates the bucket # Every step is tracked, auditable, and reversible apiVersion: platform.cloudfrugal.com/v1alpha1 kind: S3Bucket metadata: name: ${{ values.bucket_name }} namespace: ${{ values.team_namespace }} labels: team: ${{ values.team_name }} cost-centre: ${{ values.cost_centre }} environment: ${{ values.environment }} spec: versioning: true encryption: AES256 region: us-east-1 </code></pre> <h3 id="heading-13-implementation-order">1.3 Implementation Order</h3> <p>Build in this order. Deviating from it creates integration problems that are difficult to debug:</p> <pre><code class="language-plaintext">Step 1: ArgoCD — the delivery foundation everything else depends on Step 2: Crossplane — infrastructure control plane, delivered by ArgoCD Step 3: Backstage — the portal, pointing at ArgoCD and Crossplane as backends Step 4: Wire together — Software Templates that produce GitOps manifests Step 5: FinOps layer — cost attribution metadata in every provisioned resource </code></pre> <h2 id="heading-part-2-argocd-the-gitops-foundation">Part 2: ArgoCD — The GitOps Foundation</h2> <p>ArgoCD is a declarative continuous delivery tool for Kubernetes that implements the GitOps pattern. If you haven't used a GitOps tool before, the core idea is simple: your Git repository is the single source of truth for what should be running in your cluster, and ArgoCD continuously reconciles actual cluster state to match it.</p> <p>If a developer manually changes a resource in the cluster, ArgoCD detects the drift and resyncs from Git. If Git changes, ArgoCD applies the change to the cluster. Human intervention isn't required, and is actively discouraged — the goal is a cluster whose state is always fully explained by what's in Git.</p> <p>ArgoCD is a CNCF Graduated project, meaning it's production-ready and widely used. It runs as a set of pods in your cluster with a web UI, a CLI, and a REST API. Everything you need to manage deployments across multiple environments lives in one place.</p> <h3 id="heading-21-installing-argocd">2.1 Installing ArgoCD</h3> <pre><code class="language-bash"># Create the ArgoCD namespace kubectl create namespace argocd # Install ArgoCD using the official manifest kubectl apply -n argocd \ -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml # Wait for all pods to be running before proceeding kubectl wait --for=condition=Ready pods \ --all -n argocd --timeout=300s # Get the initial admin password argocd_password=$(kubectl -n argocd get secret argocd-initial-admin-secret \ -o jsonpath="{.data.password}" | base64 -d) echo "ArgoCD initial password: $argocd_password" echo "Save this somewhere secure before proceeding" # Port-forward to access the ArgoCD UI locally kubectl port-forward svc/argocd-server -n argocd 8080:443 &amp; # Login via CLI argocd login localhost:8080 \ --username admin \ --password "$argocd_password" \ --insecure # Change the password immediately argocd account update-password \ --current-password "$argocd_password" \ --new-password "your-secure-password" </code></pre> <h3 id="heading-22-repository-structure-for-gitops">2.2 Repository Structure for GitOps</h3> <p>The repository structure ArgoCD watches determines how you manage multiple environments. The pattern that scales best is environment-per-directory, with overlays managed by Kustomize.</p> <p>Kustomize is a Kubernetes-native configuration management tool that lets you define a base configuration once and layer environment-specific overrides on top of it. This means your staging and production configurations share the same YAML structure but differ in replica counts, image tags, and resource limits.</p> <pre><code class="language-plaintext">gitops-repo/ ├── apps/ │ ├── base/ # Shared configuration across all environments │ │ ├── payment-api/ │ │ │ ├── deployment.yaml │ │ │ ├── service.yaml │ │ │ └── kustomization.yaml │ │ └── user-api/ │ │ ├── deployment.yaml │ │ ├── service.yaml │ │ └── kustomization.yaml │ └── overlays/ │ ├── staging/ # Staging-specific overrides │ │ ├── payment-api/ │ │ │ └── kustomization.yaml # Override: 1 replica, staging image tag │ │ └── kustomization.yaml │ └── production/ # Production-specific overrides │ ├── payment-api/ │ │ └── kustomization.yaml # Override: 3 replicas, pinned image tag │ └── kustomization.yaml └── infrastructure/ ├── crossplane/ # Crossplane installation and providers ├── monitoring/ # Prometheus, Grafana └── ingress/ # NGINX or ALB ingress controller </code></pre> <h3 id="heading-23-applicationsets-managing-multiple-environments">2.3 ApplicationSets — Managing Multiple Environments</h3> <p>An ApplicationSet is an ArgoCD resource that generates multiple Application objects from a single template. Instead of creating one Application manifest per service per environment — which becomes unmanageable at scale — you define one ApplicationSet that covers all services across all environments. A matrix generator combines a list of environments with a Git directory scan to produce every combination automatically:</p> <pre><code class="language-yaml"># applicationset-apps.yaml # This single resource generates one ArgoCD Application # for each combination of environment and application directory apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: platform-apps namespace: argocd spec: generators: - matrix: generators: # Generator 1: environments - list: elements: - environment: staging cluster: https://staging.eks.cluster.local - environment: production cluster: https://production.eks.cluster.local # Generator 2: application directories in the overlay - git: repoURL: https://github.com/your-org/gitops-repo revision: HEAD directories: - path: apps/overlays/{{environment}}/* template: metadata: name: "{{environment}}-{{path.basename}}" labels: environment: "{{environment}}" app: "{{path.basename}}" spec: project: default source: repoURL: https://github.com/your-org/gitops-repo targetRevision: HEAD path: "apps/overlays/{{environment}}/{{path.basename}}" destination: server: "{{cluster}}" namespace: "{{path.basename}}" syncPolicy: automated: prune: true # Delete resources removed from Git selfHeal: true # Revert manual cluster changes syncOptions: - CreateNamespace=true - PrunePropagationPolicy=foreground </code></pre> <p>Verify the ApplicationSet is generating the expected Applications:</p> <pre><code class="language-bash"># List all generated Applications kubectl get applications -n argocd # Expected output: one Application per environment per app # staging-payment-api Synced Healthy # staging-user-api Synced Healthy # production-payment-api Synced Healthy # production-user-api Synced Healthy # Check sync status for a specific application argocd app get staging-payment-api </code></pre> <h3 id="heading-24-argocd-rbac-for-platform-teams">2.4 ArgoCD RBAC for Platform Teams</h3> <p>In a multi-team IDP, different teams need different levels of access to ArgoCD. Application teams should be able to view and sync their own applications. Platform teams should have broader access. Nobody should have unrestricted cluster admin through ArgoCD.</p> <p>The default policy is <code>readonly</code> — every authenticated user can see everything but change nothing:</p> <pre><code class="language-yaml"># argocd-rbac-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: argocd-rbac-cm namespace: argocd data: policy.default: role:readonly policy.csv: | # Platform team: full access to all applications p, role:platform-team, applications, *, */*, allow p, role:platform-team, clusters, get, *, allow p, role:platform-team, repositories, *, *, allow # Application teams: sync and get their own namespace only p, role:app-team, applications, get, */staging-*, allow p, role:app-team, applications, sync, */staging-*, allow # Bind roles to GitHub teams g, your-org:platform-engineers, role:platform-team g, your-org:developers, role:app-team scopes: '[groups]' </code></pre> <h2 id="heading-part-3-crossplane-infrastructure-as-kubernetes-resources">Part 3: Crossplane — Infrastructure as Kubernetes Resources</h2> <p>Crossplane is a CNCF Graduated open-source framework that extends Kubernetes into a universal infrastructure control plane.</p> <p>The core idea: instead of managing cloud resources with separate tools like Terraform or CloudFormation that live outside your cluster, you define cloud resources — RDS databases, S3 buckets, VPCs, IAM roles — as Kubernetes custom resource definitions.</p> <p>Once you apply a Crossplane resource to the cluster, Crossplane's controllers take over and reconcile the desired state to the actual AWS state, exactly the way Kubernetes reconciles a Deployment to a set of running pods.</p> <p>The key abstraction Crossplane adds on top of that is the Composite Resource. A platform team defines a high-level <code>PostgreSQLDatabase</code> type that abstracts over the thirty-plus configuration fields an actual RDS instance requires.</p> <p>Developers interact with the simple type. Crossplane expands it into the full AWS resource configuration behind the scenes, applying the platform team's security and operational standards automatically — standards that developers can't bypass because they never see the underlying fields.</p> <h3 id="heading-31-installing-crossplane">3.1 Installing Crossplane</h3> <p>Crossplane is delivered to your cluster by ArgoCD — the first integration between the two tools. By installing Crossplane through an ArgoCD Application rather than running <code>helm install</code> directly, you make Crossplane itself part of the GitOps-managed infrastructure. Any change to Crossplane's configuration goes through a Git commit and review:</p> <pre><code class="language-yaml"># infrastructure/crossplane/application.yaml # ArgoCD Application that installs Crossplane via Helm apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: crossplane namespace: argocd spec: project: default source: repoURL: https://charts.crossplane.io/stable chart: crossplane targetRevision: 1.15.0 helm: values: | provider: packages: # AWS provider — manages all AWS resources - xpkg.upbound.io/upbound/provider-aws-s3:v1.2.0 - xpkg.upbound.io/upbound/provider-aws-rds:v1.2.0 - xpkg.upbound.io/upbound/provider-aws-iam:v1.2.0 destination: server: https://kubernetes.default.svc namespace: crossplane-system syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true </code></pre> <pre><code class="language-bash"># Apply the ArgoCD Application — ArgoCD installs Crossplane kubectl apply -f infrastructure/crossplane/application.yaml # Watch Crossplane pods come up kubectl get pods -n crossplane-system -w # Verify providers are installed and healthy kubectl get providers # Expected: # NAME INSTALLED HEALTHY PACKAGE # upbound-provider-aws-s3 True True xpkg.upbound.io/... # upbound-provider-aws-rds True True xpkg.upbound.io/... </code></pre> <h3 id="heading-32-provider-credentials">3.2 Provider Credentials</h3> <p>Crossplane needs AWS credentials to provision resources. The recommended approach for EKS is IAM Roles for Service Accounts (IRSA) — a mechanism that lets Kubernetes pods assume IAM roles directly without storing any credentials in the cluster.</p> <p>The pod's Kubernetes service account is annotated with an IAM role ARN, and AWS automatically provides short-lived credentials when the pod makes API calls. No access keys, no secrets to rotate, and no credentials to accidentally expose:</p> <pre><code class="language-bash"># Create the IAM role for Crossplane with the necessary AWS permissions aws iam create-role \ --role-name CrossplaneProviderRole \ --assume-role-policy-document '{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/YOUR_OIDC_ID" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.us-east-1.amazonaws.com/id/YOUR_OIDC_ID:sub": "system:serviceaccount:crossplane-system:provider-aws" } } }] }' # Attach the permissions policy (scope this to minimum required in production) aws iam attach-role-policy \ --role-name CrossplaneProviderRole \ --policy-arn arn:aws:iam::aws:policy/AdministratorAccess </code></pre> <pre><code class="language-yaml"># provider-config.yaml # Configure the AWS provider with IRSA — no static credentials apiVersion: aws.upbound.io/v1beta1 kind: ProviderConfig metadata: name: default spec: credentials: source: IRSA # Use the IAM role attached to the provider service account </code></pre> <h3 id="heading-33-defining-a-composite-resource-postgresql-database">3.3 Defining a Composite Resource — PostgreSQL Database</h3> <p>This is where the IDP abstraction lives. The platform team defines two YAML files: the <code>CompositeResourceDefinition</code> (XRD), which specifies the shape of what developers can request, and the <code>Composition</code>, which specifies how that request expands into actual AWS resources with platform standards applied.</p> <p>The XRD is the API contract with developers. Keep it simple — only fields developers genuinely need to control should appear here:</p> <pre><code class="language-yaml"># xrd-postgresql.yaml # Defines the PostgreSQLDatabase type that developers can request # Developers never see the RDS-specific configuration below apiVersion: apiextensions.crossplane.io/v1 kind: CompositeResourceDefinition metadata: name: xpostgresqldatabases.platform.cloudfrugal.com spec: group: platform.cloudfrugal.com names: kind: XPostgreSQLDatabase plural: xpostgresqldatabases claimNames: kind: PostgreSQLDatabase # This is what developers create plural: postgresqldatabases versions: - name: v1alpha1 served: true referenceable: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: # Developer-facing fields only — simple and bounded storageGB: type: integer minimum: 20 maximum: 1000 description: "Storage in GB. Min 20, max 1000." instanceClass: type: string enum: ["small", "medium", "large"] description: "small=db.t4g.medium, medium=db.r7g.large, large=db.r7g.2xlarge" environment: type: string enum: ["staging", "production"] </code></pre> <p>The Composition is the platform team's implementation. It maps the simple developer fields to the full RDS configuration and enforces platform standards that developers can't override:</p> <pre><code class="language-yaml"># composition-postgresql.yaml # Defines what a PostgreSQLDatabase claim expands into # Platform standards (encryption, backup, deletion protection) are applied here # Developers cannot override them — the platform enforces them apiVersion: apiextensions.crossplane.io/v1 kind: Composition metadata: name: postgresql-aws-composition labels: provider: aws spec: compositeTypeRef: apiVersion: platform.cloudfrugal.com/v1alpha1 kind: XPostgreSQLDatabase resources: # The actual RDS instance — expanded from the simple developer claim - name: rds-instance base: apiVersion: rds.aws.upbound.io/v1beta1 kind: Instance spec: forProvider: region: us-east-1 engine: postgres engineVersion: "15.4" # Platform standards — always applied, not developer-configurable storageEncrypted: true # Always encrypted backupRetentionPeriod: 7 # Always 7-day backup deletionProtection: true # Always deletion-protected multiAZ: false # Overridden to true for production (see patches) dbSubnetGroupNameSelector: matchLabels: platform.cloudfrugal.com/subnet-group: private patches: # Map the developer's simple instanceClass to the actual RDS instance type - type: CombineFromComposite combine: variables: - fromFieldPath: spec.instanceClass strategy: string string: fmt: | %s toFieldPath: spec.forProvider.dbInstanceClass transforms: - type: map map: small: db.t4g.medium medium: db.r7g.large large: db.r7g.2xlarge # Enable Multi-AZ for production automatically - type: FromCompositeFieldPath fromFieldPath: spec.environment toFieldPath: spec.forProvider.multiAZ transforms: - type: map map: staging: "false" production: "true" # Copy team labels from the claim to the RDS instance for cost attribution - type: FromCompositeFieldPath fromFieldPath: metadata.labels toFieldPath: spec.forProvider.tags </code></pre> <p>A developer requesting a PostgreSQL database now writes this — nothing more:</p> <pre><code class="language-yaml"># Developer creates this in their team's namespace # No RDS knowledge required. No IAM configuration. No subnet group lookup. apiVersion: platform.cloudfrugal.com/v1alpha1 kind: PostgreSQLDatabase metadata: name: payment-service-db namespace: payments-team labels: team: payments cost-centre: payments-engineering environment: staging spec: storageGB: 100 instanceClass: medium environment: staging </code></pre> <p>Crossplane reconciles this claim to a full RDS instance within minutes, with encryption, backup, and all platform standards applied automatically.</p> <h3 id="heading-34-verifying-crossplane-resource-provisioning">3.4 Verifying Crossplane Resource Provisioning</h3> <pre><code class="language-bash"># Watch the claim status — it should transition to Ready=True kubectl get postgresqldatabases -n payments-team -w # Check the composite resource for detailed status kubectl describe xpostgresqldatabases.platform.cloudfrugal.com # Verify the actual AWS resource was created aws rds describe-db-instances \ --query 'DBInstances[?TagList[?Key==`team` &amp;&amp; Value==`payments`]].[DBInstanceIdentifier,DBInstanceStatus]' \ --output table </code></pre> <h2 id="heading-part-4-backstage-the-developer-portal">Part 4: Backstage — The Developer Portal</h2> <p>Backstage is a CNCF incubating open-source framework originally built by Spotify. It serves as the developer-facing interface of your IDP — the single place where developers discover services, request infrastructure, and find documentation, without needing to know which underlying system provides any of it.</p> <p>Backstage provides three core capabilities:</p> <ol> <li><p>A software catalog that inventories every service, API, library, and resource in your organisation</p> </li> <li><p>Software Templates that give developers self-service forms for provisioning infrastructure and scaffolding new services</p> </li> <li><p>TechDocs that co-locate documentation with the catalog entity it documents so that documentation is always findable from the same place as the service it covers.</p> </li> </ol> <p>Backstage is built in TypeScript with a React frontend and a Node.js backend. It's configured rather than installed: you create a Backstage app, configure it with your organisation's specifics, and deploy it to your cluster.</p> <h3 id="heading-41-creating-and-configuring-backstage">4.1 Creating and Configuring Backstage</h3> <pre><code class="language-bash"># Create a new Backstage app npx @backstage/create-app@latest # When prompted: # App name: platform-portal # Choose SQLite for local development, PostgreSQL for production cd platform-portal </code></pre> <p>Configure Backstage to connect to your ArgoCD instance and GitHub:</p> <pre><code class="language-yaml"># app-config.production.yaml app: title: Cloudfrugal Platform Portal baseUrl: https://platform.your-company.com backend: baseUrl: https://platform.your-company.com database: client: pg connection: host: ${POSTGRES_HOST} port: 5432 user: ${POSTGRES_USER} password: ${POSTGRES_PASSWORD} database: backstage # GitHub integration for catalog discovery and template scaffolding integrations: github: - host: github.com apps: - appId: ${GITHUB_APP_ID} webhookSecret: ${GITHUB_WEBHOOK_SECRET} clientId: ${GITHUB_CLIENT_ID} clientSecret: ${GITHUB_CLIENT_SECRET} privateKey: ${GITHUB_PRIVATE_KEY} # ArgoCD plugin configuration argocd: username: ${ARGOCD_USERNAME} password: ${ARGOCD_PASSWORD} appLocatorMethods: - type: 'config' instances: - name: main url: https://argocd.your-company.com # Catalog auto-discovery — finds catalog-info.yaml files across your GitHub org catalog: providers: github: your-org: organization: 'your-github-org' catalogPath: '/catalog-info.yaml' filters: branch: 'main' </code></pre> <h3 id="heading-42-the-software-catalog-registering-services">4.2 The Software Catalog — Registering Services</h3> <p>Every service, API, library, and resource in your platform should be registered in the Backstage catalog via a <code>catalog-info.yaml</code> file committed to the service's repository. Backstage discovers these files automatically through the GitHub integration — no manual registration required once the file exists:</p> <pre><code class="language-yaml"># catalog-info.yaml — committed to each service's repository root # Backstage discovers this automatically via the GitHub integration apiVersion: backstage.io/v1alpha1 kind: Component metadata: name: payment-api title: Payment API description: "Core payment processing service. Handles transaction initiation, authorisation, and settlement." annotations: # Links ArgoCD to show deployment status in the Backstage UI argocd/app-name: production-payment-api # Links GitHub Actions workflow status github.com/project-slug: your-org/payment-api # Links Grafana dashboard for this service grafana/dashboard-selector: "title=Payment API" # Links PagerDuty on-call schedule pagerduty.com/service-id: P123456 tags: - payments - typescript - critical links: - url: https://payment-api.docs.your-company.com title: Documentation - url: https://grafana.your-company.com/d/payment-api title: Grafana Dashboard spec: type: service lifecycle: production owner: group:payments-team system: payment-platform dependsOn: - component:user-api - resource:payment-service-db providesApis: - payment-api-v2 </code></pre> <h3 id="heading-43-software-templates-self-service-infrastructure">4.3 Software Templates — Self-Service Infrastructure</h3> <p>A Software Template is a Backstage form that, when submitted, produces a Git commit. The commit contains whatever YAML, code, or configuration the template defines.</p> <p>For infrastructure provisioning, the output is a Crossplane claim. For new service scaffolding, the output is a complete service skeleton committed to a new repository.</p> <p>The key design decision: templates should create pull requests, not merge directly. The PR gives platform teams visibility, gives developers a review moment, and gives everyone an audit trail. Auto-merge policies can eliminate the review step for low-risk provisioning once you've built trust in the template's outputs:</p> <pre><code class="language-yaml"># templates/postgresql-database/template.yaml # This template gives developers a form to request a PostgreSQL database # The output is a Crossplane PostgreSQLDatabase claim committed to the GitOps repo apiVersion: scaffolder.backstage.io/v1beta3 kind: Template metadata: name: postgresql-database title: PostgreSQL Database description: Provision a managed PostgreSQL database on AWS RDS. Encryption, backups, and deletion protection are configured automatically by the platform. tags: - database - postgresql - aws spec: owner: group:platform-team type: infrastructure # The form developers fill out in the Backstage UI parameters: - title: Database Configuration required: [name, team, environment, storageGB, instanceClass] properties: name: title: Database Name type: string description: "Lowercase, hyphens only. E.g. payment-service-db" pattern: '^[a-z][a-z0-9-]*$' team: title: Owning Team type: string description: "Your team name. Used for cost attribution and ownership." ui:field: OwnerPicker ui:options: catalogFilter: kind: Group environment: title: Environment type: string enum: [staging, production] default: staging storageGB: title: Storage (GB) type: integer minimum: 20 maximum: 1000 default: 50 instanceClass: title: Instance Size type: string enum: [small, medium, large] enumNames: - "Small (db.t4g.medium) — dev/staging workloads" - "Medium (db.r7g.large) — moderate production traffic" - "Large (db.r7g.2xlarge) — high-throughput production" default: small # What the template does when submitted steps: - id: generate-claim name: Generate Crossplane Claim action: fetch:template input: url: ./skeleton # Contains the Crossplane claim YAML template values: name: ${{ parameters.name }} team: ${{ parameters.team | parseEntityRef | pick('name') }} environment: ${{ parameters.environment }} storageGB: ${{ parameters.storageGB }} instanceClass: ${{ parameters.instanceClass }} - id: create-pr name: Create Pull Request to GitOps Repo action: publish:github:pull-request input: repoUrl: github.com?repo=gitops-repo&amp;owner=your-org title: "Platform: Provision PostgreSQL database ${{ parameters.name }} for ${{ parameters.team }}" branchName: "provision-db-${{ parameters.name }}-${{ '' | now }}" description: | Requesting PostgreSQL database provisioned by Crossplane. - **Name:** ${{ parameters.name }} - **Team:** ${{ parameters.team }} - **Environment:** ${{ parameters.environment }} - **Storage:** ${{ parameters.storageGB }}GB - **Instance:** ${{ parameters.instanceClass }} Approve this PR to trigger provisioning. ArgoCD will pick up the change and Crossplane will create the RDS instance within ~5 minutes of merge. sourcePath: ./skeleton output: links: - title: View Pull Request url: ${{ steps['create-pr'].output.remoteUrl }} - title: Track Provisioning in ArgoCD url: https://argocd.your-company.com/applications </code></pre> <p>The template skeleton directory contains the Crossplane claim with template variable placeholders:</p> <pre><code class="language-yaml"># templates/postgresql-database/skeleton/databases/${{ values.name }}.yaml apiVersion: platform.cloudfrugal.com/v1alpha1 kind: PostgreSQLDatabase metadata: name: ${{ values.name }} namespace: ${{ values.team }}-platform labels: team: ${{ values.team }} cost-centre: ${{ values.team }}-engineering environment: ${{ values.environment }} managed-by: backstage-scaffolder spec: storageGB: ${{ values.storageGB }} instanceClass: ${{ values.instanceClass }} environment: ${{ values.environment }} </code></pre> <h2 id="heading-part-5-wiring-it-together-the-golden-path">Part 5: Wiring It Together — The Golden Path</h2> <p>The Golden Path is the complete end-to-end workflow: a developer uses Backstage to request infrastructure, that request becomes a Git commit, ArgoCD applies the commit to the cluster, Crossplane provisions the actual AWS resource, and the result appears in both the Backstage catalog and the ArgoCD dashboard.</p> <h3 id="heading-51-the-complete-flow">5.1 The Complete Flow</h3> <pre><code class="language-plaintext">Developer fills form in Backstage ↓ Backstage Software Template renders the Crossplane claim YAML ↓ Backstage creates a Pull Request in the GitOps repository ↓ Platform engineer (or auto-merge policy) approves and merges the PR ↓ ArgoCD detects the new file in the GitOps repository ↓ ArgoCD applies the Crossplane claim to the cluster ↓ Crossplane reconciles the claim to an actual AWS RDS instance ↓ Developer receives the database endpoint via Kubernetes Secret ↓ Backstage catalog shows the new resource, owned by the requesting team </code></pre> <h3 id="heading-52-surfacing-resource-status-back-in-backstage">5.2 Surfacing Resource Status Back in Backstage</h3> <p>The Backstage Kubernetes plugin pulls live pod and resource status from your clusters and displays it on each catalog entity page. Developers can see whether their service is running, how many replicas are healthy, and whether the last deployment synced — without leaving Backstage or learning <code>kubectl</code>:</p> <pre><code class="language-bash"># Install the Kubernetes plugin packages cd platform-portal yarn --cwd packages/app add @backstage/plugin-kubernetes yarn --cwd packages/backend add @backstage/plugin-kubernetes-backend </code></pre> <pre><code class="language-yaml"># app-config.production.yaml — add Kubernetes cluster configuration kubernetes: serviceLocatorMethod: type: 'multiTenant' clusterLocatorMethods: - type: 'config' clusters: - name: production-eks url: ${PRODUCTION_CLUSTER_URL} authProvider: serviceAccount serviceAccountToken: ${PRODUCTION_SA_TOKEN} caData: ${PRODUCTION_CA_DATA} - name: staging-eks url: ${STAGING_CLUSTER_URL} authProvider: serviceAccount serviceAccountToken: ${STAGING_SA_TOKEN} caData: ${STAGING_CA_DATA} </code></pre> <p>Annotate each catalog entity to link it to its Kubernetes resources:</p> <pre><code class="language-yaml"># In each service's catalog-info.yaml annotations: backstage.io/kubernetes-label-selector: 'app=payment-api' backstage.io/kubernetes-namespace: payments-team </code></pre> <h3 id="heading-53-installing-the-argocd-plugin">5.3 Installing the ArgoCD Plugin</h3> <p>The ArgoCD plugin shows deployment history and sync status directly in the Backstage entity page. When a developer opens the payment-api page in the catalog, they can see the last 10 deployments, the current sync state, and whether the application is healthy — all without opening the ArgoCD UI:</p> <pre><code class="language-bash">yarn --cwd packages/app add @roadiehq/backstage-plugin-argo-cd </code></pre> <pre><code class="language-typescript">// packages/app/src/components/catalog/EntityPage.tsx import { EntityArgoCDOverviewCard } from '@roadiehq/backstage-plugin-argo-cd'; // Add to the service entity page layout const serviceEntityPage = ( &lt;EntityLayout&gt; &lt;EntityLayout.Route path="/" title="Overview"&gt; &lt;Grid container spacing={3}&gt; &lt;Grid item md={6}&gt; &lt;EntityAboutCard variant="gridItem" /&gt; &lt;/Grid&gt; &lt;Grid item md={6}&gt; {/* ArgoCD deployment status — shows sync state and recent deployments */} &lt;EntityArgoCDOverviewCard /&gt; &lt;/Grid&gt; &lt;/Grid&gt; &lt;/EntityLayout.Route&gt; &lt;/EntityLayout&gt; ); </code></pre> <h2 id="heading-part-6-finops-integration-cost-attribution-on-the-idp">Part 6: FinOps Integration — Cost Attribution on the IDP</h2> <p>An IDP that provisions resources without cost attribution creates a new problem: you now have automated infrastructure provisioning with no clear ownership of the bill it generates. Every resource created through the IDP must carry team and cost centre metadata from the moment it's provisioned.</p> <h3 id="heading-61-mandatory-labels-on-every-crossplane-composition">6.1 Mandatory Labels on Every Crossplane Composition</h3> <p>The Crossplane Compositions are where cost attribution is enforced — not in the developer-facing claim, but in the platform layer that the developer can't bypass. These labels flow through to the actual AWS resource as tags, which means they appear in AWS Cost Explorer and can be used to build team-level cost reports:</p> <pre><code class="language-yaml"># In every Composition, add mandatory cost attribution patches patches: # These labels flow to the actual AWS resource as tags # They can't be omitted or overridden by the developer claim - type: FromCompositeFieldPath fromFieldPath: metadata.labels[team] toFieldPath: spec.forProvider.tags[team] - type: FromCompositeFieldPath fromFieldPath: metadata.labels[cost-centre] toFieldPath: spec.forProvider.tags[cost-centre] - type: FromCompositeFieldPath fromFieldPath: metadata.labels[environment] toFieldPath: spec.forProvider.tags[environment] # Add a managed-by tag to identify all IDP-provisioned resources - type: FromCompositeFieldPath fromFieldPath: metadata.name toFieldPath: spec.forProvider.tags[managed-by] transforms: - type: string string: fmt: "idp-crossplane" </code></pre> <h3 id="heading-62-cost-attribution-query">6.2 Cost Attribution Query</h3> <p>With mandatory tags on every resource, you can query actual cost by team directly from AWS Cost Explorer:</p> <pre><code class="language-bash"># Monthly cost breakdown by team — all IDP-provisioned resources aws ce get-cost-and-usage \ --time-period Start=$(date -d 'last month' +%Y-%m-01),End=$(date +%Y-%m-01) \ --granularity MONTHLY \ --filter '{ "Tags": { "Key": "managed-by", "Values": ["idp-crossplane"] } }' \ --group-by Type=TAG,Key=team \ --metrics UnblendedCost \ --query 'ResultsByTime[0].Groups[*].{Team:Keys[0],Cost:Metrics.UnblendedCost.Amount}' \ --output table </code></pre> <p>Every team that provisions resources through the IDP now has a line on the cost report with their name on it. This is the chargeback model that makes FinOps sustainable at platform scale — attribution is automatic, not manual.</p> <h2 id="heading-part-7-the-platform-maturity-model-measuring-what-youve-built">Part 7: The Platform Maturity Model — Measuring What You've Built</h2> <p>The CNCF Platform Engineering Maturity Model defines five levels of platform maturity. Knowing where you sit helps you decide what to build next and communicate progress to engineering leadership.</p> <table> <thead> <tr> <th>Level</th> <th>Name</th> <th>Characteristics</th> </tr> </thead> <tbody><tr> <td>1</td> <td>Provisional</td> <td>Ad hoc scripts, manual provisioning, no standard tools</td> </tr> <tr> <td>2</td> <td>Operational</td> <td>Standardised tools, some automation, Kubernetes in use</td> </tr> <tr> <td>3</td> <td>Scalable</td> <td>Self-service portal, GitOps delivery, documented golden paths</td> </tr> <tr> <td>4</td> <td>Optimising</td> <td>Cost attribution, SLOs on the platform itself, user feedback loops</td> </tr> <tr> <td>5</td> <td>Optimised</td> <td>AI-assisted provisioning, predictive scaling, full FinOps integration</td> </tr> </tbody></table> <p>A complete Backstage + ArgoCD + Crossplane implementation, with cost attribution and Software Templates covering your most common developer requests, puts you at Level 3. Moving to Level 4 requires adding SLO alerting on the platform's own health, running quarterly developer experience surveys, and producing a monthly cost-by-team report from the attribution tags.</p> <p>The most common mistake at Level 3: building more features instead of measuring adoption. A platform that has 12 Software Templates but only 2 are regularly used hasn't reached Level 3 — it's reached Level 2 with more YAML. Measure which golden paths are used, interview developers who aren't using the portal, and fix the friction before adding capabilities.</p> <h2 id="heading-best-practices-summary">Best Practices Summary</h2> <p>✅ <strong>Do:</strong> Build in order — ArgoCD first, then Crossplane, then Backstage. Each layer depends on the previous one.</p> <p>✅ <strong>Do:</strong> Use Backstage as a Git commit generator, not as an infrastructure caller. All infrastructure changes must be auditable Git commits.</p> <p>✅ <strong>Do:</strong> Apply cost attribution tags in the Crossplane Composition layer, not in the developer claim. Attribution that developers can bypass will be bypassed.</p> <p>✅ <strong>Do:</strong> Start with two or three Software Templates and make them excellent before building more. Template adoption is your most important early metric.</p> <p>✅ <strong>Do:</strong> Register every service in the Backstage catalog from day one. The catalog's value is proportional to its coverage.</p> <p>✅ <strong>Do:</strong> Deliver Crossplane to your cluster via ArgoCD, not <code>helm install</code>. Everything the IDP manages should itself be managed by the IDP.</p> <p>❌ <strong>Don't:</strong> Connect Backstage directly to cloud APIs. No audit trail, no rollback, no reconciliation.</p> <p>❌ <strong>Don't:</strong> Give developers the Crossplane XRD directly. The Composition abstraction exists to hide RDS-specific configuration and enforce platform standards. Bypassing it defeats the purpose.</p> <p>❌ <strong>Don't:</strong> Build the IDP in isolation and announce it as done. Platform engineering is product engineering. Schedule user interviews after the first two templates are live.</p> <p>❌ <strong>Don't:</strong> Skip the ArgoCD RBAC configuration. An IDP that gives all developers cluster-admin through the delivery layer has created a security problem larger than the one it solved.</p> <h2 id="heading-resources">Resources</h2> <ul> <li><p><a href="https://backstage.io/docs"><strong>Backstage Documentation</strong></a> — Official reference for plugin development, Software Templates, and catalog configuration</p> </li> <li><p><a href="https://docs.crossplane.io"><strong>Crossplane Documentation</strong></a> — CompositeResourceDefinition and Composition reference, provider installation guides</p> </li> <li><p><a href="https://argo-cd.readthedocs.io"><strong>ArgoCD Documentation</strong></a> — ApplicationSet generator reference, RBAC configuration, and sync policy options</p> </li> <li><p><a href="https://tag-app-delivery.cncf.io/whitepapers/platform-eng-maturity-model/"><strong>CNCF Platform Engineering Maturity Model</strong></a> — The maturity framework referenced in Part 7</p> </li> <li><p><a href="https://marketplace.upbound.io/providers/upbound/provider-aws"><strong>AWS Provider for Crossplane</strong></a> — Complete reference for all AWS resource types available through Crossplane</p> </li> <li><p><a href="https://backstage.io/docs/features/kubernetes/"><strong>Backstage Kubernetes Plugin</strong></a> — Setup guide for the Kubernetes resource visibility integration in Part 5</p> </li> <li><p><a href="https://www.finops.org/framework/capabilities/"><strong>FinOps Foundation — FinOps for Platform Engineering</strong></a> — Framework reference for the cost attribution model in Part 6</p> </li> <li><p><a href="https://github.com/aayostem/platform-toolkit"><strong>Companion Repository</strong></a> — All manifests, Compositions, ApplicationSets, and Backstage templates from this guide</p> </li> <li><p><a href="https://cloud.google.com/resources/content/2025-dora-ai-capabilities-model-report"><strong>2025 DORA State of AI-assisted Software Development Report</strong></a></p> </li> </ul>