Course Remote

How to Build a Production-Ready DevSecOps Platform from Homelab to AWS [Full Book]

Osomudeya Zudonu · Remote

At a glance

Type: Course
Organisation: Osomudeya Zudonu
Location: Remote
Work mode: Remote
Deadline: Rolling / not stated
Posted: 27 Jul 2026

About this course

In this book, you'll build a fintech transaction ledger from scratch and progressively transform it into a production-ready DevSecOps platform. You'll also deploy it on AWS. The app processes credit and debit transactions, fires compliance alerts, and stores everything in a database. You'll build the infrastructure around it yourself: automation, scanning, policy enforcement, secrets management, threat detection, and observability. By the time you're finished, you'll be able to talk through every decision in an interview because you made each one. This guide doesn't hand you a pre-built solution. It makes you feel out. and understand each problem before introducing the tool that solves it. All the code, manifests, scripts, and stage-by-stage READMEs live in the companion repository. Clone it before you start: <pre><code class="language-bash">git clone https://github.com/Osomudeya/clearledger.git cd clearledger </code></pre> Everything in this book refers to files inside that repo. <h2 id="heading-prerequisites">Prerequisites</h2> You'll need these tools installed on your machine before Stage 0: <ul> <li>Multipass: creates a lightweight Ubuntu VM so Kubernetes has enough resources </li> <li>kubectl: talks to your Kubernetes cluster from your terminal </li> <li>Helm: installs apps into Kubernetes </li> <li>Docker Desktop: builds container images </li> <li>jq: formats JSON output so it's readable </li> </ul> You'll also need free accounts on GitHub and Docker Hub. And you should be comfortable with the following knowledge and skills: <ul> <li>Basic Linux command line: navigating directories, reading files, running scripts </li> <li>Git: clone, commit, push </li> <li>What a container is and roughly how Docker builds one </li> </ul> You don't need prior Kubernetes, security, or cloud experience. This guide builds that from Stage 0. Your machine needs at least 24 GB of RAM, 6 CPU cores, and 80 GB of free disk space. See <a href="#heading-how-to-set-up-your-machine">How to Set Up Your Machine</a> for the exact install commands. The companion repo is at <a href="https://github.com/Osomudeya/clearledger">github.com/Osomudeya/clearledger</a>. Star it, clone it, then continue. <h2 id="heading-table-of-contents">Table of Contents</h2> <ul> <li><a href="#heading-prerequisites">Prerequisites</a> </li> <li><a href="#heading-what-you-are-building">What You Are Building</a> </li> <li><a href="#heading-how-to-work-through-this-lab">How to Work Through This Lab</a> </li> <li><a href="#heading-tools-you-will-use">Tools You Will Use</a> </li> <li><a href="#heading-how-to-choose-your-path">How to Choose Your Path</a> </li> <li><a href="#heading-how-to-save-your-progress">How to Save Your Progress</a> </li> <li><a href="#heading-who-this-is-for">Who This Is For</a> </li> <li><a href="#heading-how-to-set-up-your-machine">How to Set Up Your Machine</a> </li> <li><a href="#heading-how-to-start-the-lab">How to Start the Lab</a> </li> <li><a href="#heading-how-to-manage-disk-space">How to Manage Disk Space</a> </li> <li><a href="#heading-how-to-try-the-app-without-kubernetes">How to Try the App Without Kubernetes</a> </li> <li><a href="#heading-how-to-configure-local-domain-names">How to Configure Local Domain Names</a> </li> <li><a href="#heading-stage-0-the-running-system">Stage 0 — The Running System</a> </li> <li><a href="#heading-stage-1-ci-pipeline-github-actions-self-hosted-runner">Stage 1 — CI Pipeline (GitHub Actions + Self-Hosted Runner)</a> </li> <li><a href="#heading-stage-2-gitops-with-argocd">Stage 2 — GitOps with ArgoCD</a> </li> <li><a href="#heading-stage-3-security-gates">Stage 3 — Security Gates</a> </li> <li><a href="#heading-stage-4-admission-control-kyverno">Stage 4 — Admission Control (Kyverno)</a> </li> <li><a href="#heading-stage-5-secrets-management-vault">Stage 5: Secrets Management (Vault)</a> </li> <li><a href="#heading-stage-6-runtime-security-falco">Stage 6 — Runtime Security (Falco)</a> </li> <li><a href="#heading-stage-65-chaos-engineering-optional">Stage 6.5 — Chaos Engineering (Optional)</a> </li> <li><a href="#heading-stage-7-security-observability">Stage 7 — Security Observability</a> </li> <li><a href="#heading-stage-75-opentelemetry-optional">Stage 7.5 — OpenTelemetry (Optional)</a> </li> <li><a href="#heading-stage-8-aws-migration">Stage 8 — AWS Migration</a> </li> <li><a href="#heading-troubleshooting-see-troubleshootingmd">Troubleshooting (see troubleshooting.md)</a> </li> <li><a href="#heading-compliance-reference">Compliance Reference</a> </li> <li><a href="#heading-interview-preparation">Interview Preparation</a> </li> <li><a href="#heading-aws-cost-reference">AWS Cost Reference</a> </li> <li><a href="#heading-conclusion">Conclusion</a> </li> </ul> <h2 id="heading-what-you-are-building">What You Are Building</h2> ClearLedger is a fintech transaction ledger built with three FastAPI microservices, PostgreSQL, Redis, and a web frontend. Users can register, sign in, record credit and debit transactions, view their account balance, and receive compliance alerts whenever a transaction exceeds a predefined threshold. The application is intentionally simple. Its purpose isn't to teach fintech. It gives you a realistic system that you'll secure and operate like a production platform. The project consists of four components: <ul> <li>auth-service: Handles user registration, login, and JWT authentication. </li> <li>ledger-service: Processes transactions, maintains account balances, and stores transaction history. </li> <li>notification-service: Listens for large transactions through Redis and generates compliance alerts. </li> <li>frontend: A web interface for logging in, viewing balances, submitting transactions, and reviewing alerts. </li> </ul> By the end of this book, every one of these services will still exist. What changes is how they are built, deployed, secured, and operated. The application is simply the vehicle. DevSecOps is the destination. <h3 id="heading-how-the-platform-evolves">How the Platform Evolves</h3> You won't install every tool on day one. Instead, the platform grows the same way production systems usually do: a problem appears first, then a solution is introduced. You'll begin with a manually deployed Kubernetes application. From there, each stage solves one real operational problem. Stage 0: Raw Kubernetes You'll deploy and run the application manually, which lets you understand the system before introducing automation. Stage 1: Continuous Integration Building container images becomes automatic whenever code is pushed, eliminating manual build steps. Stage 2: GitOps Deployments are no longer done with kubectl. Git becomes the single source of truth, preventing configuration drift. Stage 3: Security Gates Every commit passes through security scanning so vulnerable code, secrets, and misconfigurations are stopped before deployment. Stage 4: Admission Control Even if something bypasses the pipeline, Kubernetes policies prevent insecure workloads from entering the cluster. Stage 5: Secrets Management Application credentials move out of Kubernetes Secrets into Vault, removing sensitive data from Git and cluster storage. Stage 6: Runtime Security Falco continuously watches running containers and detects suspicious behavior after deployment. Stage 6.5 (Optional): Chaos Engineering Failures are introduced deliberately to verify that the platform can recover instead of simply detecting problems. Stage 7: Observability Metrics, logs, and dashboards provide visibility into the health, performance, and security of the platform. Stage 7.5 (Optional): OpenTelemetry Distributed tracing follows requests across every service, revealing how a single transaction moves through the system. Stage 8: AWS Migration The same architecture is deployed on AWS using EKS, ECR, RDS, and an Application Load Balancer without changing how the application itself works. If you simply want to explore the application before touching Kubernetes, an optional Docker Compose stack lets you run everything locally on your machine. The guiding principle of this book is simple: every stage makes you feel the problem before introducing the tool that solves it. <h2 id="heading-how-to-work-through-this-lab">How to Work Through This Lab</h2> Throughout this process of understanding each problem before introducing the tool that solves it, three habits will carry you through every stage. <ol> <li>Read first before you run: The paragraphs before each command explain why you're running it. Skipping them means you can reproduce the steps but not explain them, and explaining them is what gets you hired. The commands are proof you understand. </li> <li>Choose with a reason: Every tool here solves a specific problem. Why use Vault instead of Kubernetes Secrets? Why split code and manifests into two repos? Don't just follow the steps: ask what breaks if we skip this? If you understand the problem, you'll remember the solution. </li> <li>Go in order and verify every checkpoint: Each stage depends on the one before it. When you hit an issue, read the error. Getting stuck and debugging is part of the learning: employers want to hear "I hit X error and fixed it by doing Y." </li> </ol> At every ✋ Hands-on checkpoint: <ol> <li>Run the command. </li> <li>Compare your output with Expected. </li> <li>If it doesn't match, fix it before continuing. </li> <li>When <code>make check-N</code> passes: <code>make snapshot STAGE=N && make snapshots</code>. Only continue after you see <code>clearledger.stageN</code>. </li> </ol> Avoid these mistakes: <ul> <li>Don't skip a checkpoint because it passed before. </li> <li>Don't run <code>make restore</code> without checking available snapshots first. </li> <li>Replace <code>your-username</code> with your real Docker Hub or GitHub username everywhere it appears. </li> <li>Run runner commands inside the VM (prompt shows <code>ubuntu@clearledger</code>), not on your Mac. </li> </ul> Take screenshots at each portfolio checkpoint. These moments become your evidence: proof that the platform runs, detects, blocks, syncs, and observes real activity. <h2 id="heading-tools-you-will-use">Tools You Will Use</h2> Come back to the below table when a new name appears and you wonder why now. Each entry is one line: what it does and when it appears. On your laptop: <ul> <li>Multipass creates the Ubuntu VM. </li> <li>Docker builds images. </li> <li><code>make</code> wraps long commands into <code>make setup</code> / <code>make check-N</code>. </li> <li><code>/etc/hosts</code> entries like <code>clearledger.local</code> let your browser reach the cluster. </li> </ul> The app: <ul> <li>Three Python APIs (auth, ledger, notifications) + a web frontend. </li> <li>Postgres stores data </li> <li>Redis lets ledger publish alerts without calling notification directly </li> <li>nginx ingress routes browser traffic to the right service. </li> </ul> <table> <thead> <tr> <th>Tool</th> <th>One-line role</th> <th>Stage</th> </tr> </thead> <tbody><tr> <td>MicroK8s / kubectl</td> <td>Kubernetes cluster inside the VM: <code>kubectl</code> talks to it.</td> <td>0</td> </tr> <tr> <td><code>clearledger</code> (repo)</td> <td>App code + CI workflow: what you build.</td> <td>1</td> </tr> <tr> <td><code>clearledger-infra</code> (repo)</td> <td>Kubernetes YAML only: what the cluster should run. CI updates it, ArgoCD deploys it.</td> <td>1</td> </tr> <tr> <td>GitHub Actions + self-hosted runner</td> <td>Builds images and updates infra repo on every push. Runner lives in the VM to reach the local cluster.</td> <td>1</td> </tr> <tr> <td>ArgoCD</td> <td>Watches <code>clearledger-infra</code>, syncs the cluster to match Git, reverts unauthorized changes.</td> <td>2</td> </tr> <tr> <td>Gitleaks</td> <td>Blocks commits that contain secrets (API keys, tokens).</td> <td>3</td> </tr> <tr> <td>Semgrep</td> <td>SAST: catches unsafe Python patterns (injection, hardcoded credentials).</td> <td>3</td> </tr> <tr> <td>Checkov</td> <td>IaC scanning: misconfigs in Dockerfiles and Kubernetes YAML.</td> <td>3</td> </tr> <tr> <td>Trivy</td> <td>Image scanning: known CVEs in pip/npm packages and the built container.</td> <td>3</td> </tr> <tr> <td>Syft + Grype</td> <td>SBOM generation and vulnerability check on the artifact itself.</td> <td>3</td> </tr> <tr> <td>Cosign</td> <td>Signs container images: Stage 4 rejects unsigned ones at deploy time.</td> <td>3</td> </tr> <tr> <td>Kyverno</td> <td>Admission control: blocks non-compliant pods at the cluster gate (root containers, missing limits, unsigned images).</td> <td>4</td> </tr> <tr> <td>Vault</td> <td>Stores credentials outside Git and etcd: injects them into pods via a sidecar at startup.</td> <td>5</td> </tr> <tr> <td>Falco</td> <td>eBPF runtime detection: alerts when a shell starts or a sensitive file is read inside a running container.</td> <td>6</td> </tr> <tr> <td>Network policies</td> <td>Kubernetes firewall between pods: limits blast radius if one service is compromised.</td> <td>6</td> </tr> <tr> <td>LitmusChaos</td> <td>Kills pods deliberately to prove the app recovers (optional).</td> <td>6.5</td> </tr> <tr> <td>Prometheus / Grafana / Loki</td> <td>Metrics, dashboards, and log search: turns security events into evidence.</td> <td>7</td> </tr> <tr> <td>OpenTelemetry + Tempo</td> <td>Distributed traces: shows where one request spent its time across services (optional).</td> <td>7.5</td> </tr> <tr> <td>Terraform / EKS / ECR / RDS</td> <td>Infrastructure as code for the AWS migration: same app, cloud-managed backing services.</td> <td>8</td> </tr> </tbody></table> Each stage adds a new security layer. The tools aren't interchangeable: scanners check your code and images before deployment, ArgoCD keeps the cluster synced to Git, Vault handles secrets, Kyverno blocks unsafe workloads before they run, and Falco watches for suspicious behavior after they're running. That's why the order matters: you're building defense in depth, one layer at a time. <h2 id="heading-how-to-choose-your-path">How to Choose Your Path</h2> Pick one path from your host RAM before you provision a cluster. Switching mid-lab after OOM kills or disk pressure wastes a day, so choose upfront. <table> <thead> <tr> <th>Your situation</th> <th>Path</th> <th>What you get</th> </tr> </thead> <tbody><tr> <td>8 GB RAM, or unsure this laptop can carry the lab</td> <td>Docker Compose first</td> <td>The real app: register, post a transaction, see the compliance alert fire. Then decide on a cluster. <code>make integration-up</code> · <a href="#heading-how-to-try-the-app-without-kubernetes">Local integration stack</a></td> </tr> <tr> <td>16 GB RAM on the host</td> <td>Lite local cluster (Stages 0–5)</td> <td>Running on one VM: This setup includes Kubernetes, CI/CD, GitOps, security checks, admission control, and Vault. To use fewer resources, edit <code>scripts/setup-cluster.local.env</code> before running <code>make setup</code>.</td> </tr> <tr> <td>Under 16 GB host RAM and you need Kubernetes, or you want all 8 stages</td> <td>Cloud VM</td> <td>Provision a remote machine (4–8 vCPU, 16–32 GB RAM), clone the repo, run the lab there, <code>make teardown</code> when done. Stages 6.5 / 7 / 7.5 (chaos + full observability) need 24 GB on the host, use this path if your laptop cannot spare that.</td> </tr> </tbody></table> The default path in this guide assumes 24 GB+ RAM and the full local VM (Before You Start). If that's not you, start from the row that matches your machine. <h2 id="heading-how-to-save-your-progress">How to Save Your Progress</h2> Mac + Multipass only: <code>make snapshot</code> and <code>make restore</code> require Multipass. If you're using Linux without Multipass, skip snapshots and use Path B if something goes wrong. This lab takes several days to complete. Your source code lives on your computer, so rebuilding or deleting the VM doesn't delete your Git repository, commits, manifests, or configuration files. The VM stores your running environment, including deployed pods, Vault secrets, Postgres data, and Grafana dashboards. <h3 id="heading-save-your-progress">Save Your Progress</h3> After completing each stage, create a snapshot before moving on. For example: <pre><code class="language-bash">make snapshot STAGE=7 make snapshots </code></pre> Always run <code>make snapshots</code> to confirm the snapshot was created. <h3 id="heading-restore-your-progress">Restore Your Progress</h3> If the VM becomes unusable after a while, restore the latest working snapshot: <pre><code class="language-bash">make snapshots make restore STAGE=7 export KUBECONFIG=~/.kube/clearledger-config make check-7 </code></pre> <h3 id="heading-what-happens-if-the-vm-breaks">What Happens If the VM Breaks?</h3> You keep: <ul> <li>Your Git repository </li> <li>Your commits </li> <li><code>.env</code> </li> <li><code>setup-cluster.local.env</code> </li> <li><code>clearledger-infra</code> on GitHub </li> </ul> You lose anything stored inside the VM after your last snapshot, including: <ul> <li>Running pods </li> <li>Vault secrets </li> <li>Postgres data </li> <li>Grafana and Loki data </li> </ul> That's why it's a good idea to create a snapshot after every completed stage. <h4 id="heading-path-a-you-have-a-snapshot-recommended">Path A: You Have a Snapshot (Recommended)</h4> Restore the latest working snapshot and continue from that stage. <pre><code class="language-bash">make snapshots make restore STAGE=6 export KUBECONFIG=~/.kube/clearledger-config make check-6 </code></pre> <h4 id="heading-path-b-no-snapshot">Path B: No Snapshot</h4> Rebuild the lab. <pre><code class="language-bash">make teardown make setup export KUBECONFIG=~/.kube/clearledger-config </code></pre> Your Git repositories are still intact, but the Kubernetes cluster starts empty. Continue the book from the stage you had reached and rebuild the platform from there. If you run into problems such as disk space issues, failed snapshots, Mac sleep or restart problems, Vault authentication errors, or pods stuck in CrashLoopBackOff, see <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md</a> for detailed recovery steps. <h2 id="heading-who-this-is-for">Who This Is For</h2> Junior DevOps (0–2 yrs): do every stage in order. Don't skip the pain point sections. Expect Stage 0–2 to take a full day each, Stages 3–7 half a day each, Stage 8 a few hours. That's normal, so don't rush. Mid-level DevOps (2–4 yrs): skim Stages 0–2 to understand the app, focus time on Stages 3–7 where the security layers are. Interview preparation: complete through Stage 4, then read <code>docs/interview-prep.md</code>. The questions are based on exactly what's in this lab. <h2 id="heading-how-to-set-up-your-machine">How to Set Up Your Machine</h2> Requirements are in <a href="#heading-prerequisites">Prerequisites</a> above. Confirm 24 GB RAM, 6 CPU cores, and 80 GB free disk before installing. <h3 id="heading-install-the-required-tools">Install the Required Tools</h3> <table> <thead> <tr> <th>Tool</th> <th>What it does</th> <th>macOS</th> <th>Linux</th> <th>Windows</th> </tr> </thead> <tbody><tr> <td>Multipass</td> <td>Creates lightweight Ubuntu VMs on your laptop</td> <td><code>brew install --cask multipass</code></td> <td><code>sudo snap install multipass</code></td> <td><a href="https://multipass.run/install">multipass.run/install</a></td> </tr> <tr> <td>kubectl</td> <td>Talks to your Kubernetes cluster from your terminal</td> <td><code>brew install kubectl</code></td> <td><code>sudo snap install kubectl --classic</code></td> <td><code>winget install Kubernetes.kubectl</code></td> </tr> <tr> <td>Helm</td> <td>Package manager for Kubernetes (like apt/brew but for cluster apps)</td> <td><code>brew install helm</code></td> <td><code>sudo snap install helm --classic</code></td> <td><code>winget install Helm.Helm</code></td> </tr> <tr> <td>Docker Desktop</td> <td>Builds container images on your machine</td> <td><a href="https://docs.docker.com/desktop/">docker.com</a></td> <td><a href="https://docs.docker.com/engine/install/">docker.com</a></td> <td><a href="https://docs.docker.com/desktop/">docker.com</a></td> </tr> <tr> <td>jq</td> <td>Formats JSON output so you can read it</td> <td><code>brew install jq</code></td> <td><code>sudo apt install jq</code></td> <td><code>winget install jqlang.jq</code></td> </tr> </tbody></table> Windows users: Run all commands inside WSL2 Ubuntu. Don't use PowerShell for this lab because the setup uses <code>make</code> and Bash scripts. Verify everything before continuing: <pre><code class="language-bash">multipass --version kubectl version --client helm version docker --version jq --version </code></pre> If any command fails, install the missing tool before continuing. <h2 id="heading-how-to-start-the-lab">How to Start the Lab</h2> The main lab path starts at <a href="#heading-stage-0-the-running-system">Stage 0: the Running System</a>. After you have run the setup once step by step, you can use this shortcut next time: <pre><code class="language-bash">make setup export KUBECONFIG=~/.kube/clearledger-config kubectl get nodes </code></pre> Expected: one node named <code>clearledger</code> with STATUS <code>Ready</code>. <code>make setup</code> provisions the Multipass VM, installs MicroK8s, applies disk-safety caps, and updates <code>/etc/hosts</code>. Takes 3–5 minutes. <h2 id="heading-how-to-manage-disk-space">How to Manage Disk Space</h2> The lab runs on a single-node MicroK8s VM with a fixed disk (80 GB by default). Over days or weeks (especially after CI builds, Helm upgrades, and Stage 7 observability) container images, logs, and journald can fill the root filesystem. Pods then fail with <code>Evicted</code>, <code>ImagePullBackOff</code>, or mysterious <code>Pending</code> states. <code>make setup</code> applies preventive caps automatically (log rotation, image GC thresholds, journald cap). See <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">Disk health in troubleshooting.md</a> for the full table and recovery steps. Check disk health: <pre><code class="language-bash">make doctor # PASS / WARN / FAIL + PVC and Prometheus TSDB sizes </code></pre> Clean up unused files inside the VM without deleting app data: <pre><code class="language-bash">make reclaim </code></pre> If <code>make doctor</code> still reports FAIL after reclaim, you may need <code>make teardown && make setup</code> and restore from a snapshot. Full guidance: <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md. VM disk full</a>. <h2 id="heading-how-to-try-the-app-without-kubernetes">How to Try the App Without Kubernetes</h2> If your machine doesn't have enough resources for Kubernetes, you can run ClearLedger with Docker Compose. <pre><code class="language-bash">docker compose -f docker-compose.integration.yml up --build -d </code></pre> Open <a href="http://localhost:3000">http://localhost:3000</a>. When you're ready, stop the stack and continue with Stage 0. <pre><code class="language-bash">docker compose -f docker-compose.integration.yml down </code></pre> <h3 id="heading-how-to-sign-in-for-the-first-time">How to Sign In for the First Time</h3> First, you'll need to register. The database starts empty after each fresh <code>up</code> (or <code>down -v</code>). Use a real-looking email (Pydantic rejects <code>@*.local</code>), for example <code>test@clearledger.io</code> for an email and <code>SecurePass123</code> for a password. Then sign in with the same credentials. Wrong password shows Incorrect email or password. If you see a stale error, hard-refresh or run <code>localStorage.removeItem('cl_token')</code> in the browser console. <h3 id="heading-how-to-run-the-demo-flow">How to Run the Demo Flow</h3> First, register and sign in at <a href="http://localhost:3000">http://localhost:3000</a>. Submit a few credits and debits (for example, Salary +$5000, Rent −$1200). Then confirm the balance updates and history lists entries. Now submit a transaction ≥ $10,000: the Alerts panel should show <code>LARGE_TRANSACTION</code>. Here's an optional smoke test against the same base URL: <pre><code class="language-bash">BASE_URL=http://localhost:3000 bash scripts/dast/smoke.sh </code></pre> <h2 id="heading-how-to-configure-local-domain-names">How to Configure Local Domain Names</h2> Add the ClearLedger hostnames to your hosts file. <h3 id="heading-macos-or-linux-with-multipass">macOS or Linux with Multipass</h3> Run: <pre><code class="language-bash">sudo bash scripts/setup-hosts.sh </code></pre> Or do it manually: <pre><code class="language-bash">VMIP=$(multipass info clearledger | grep IPv4 | awk '{print $2}') echo "$VMIP clearledger.local argocd.local grafana.local vault.local falco.local litmus.local" | sudo tee -a /etc/hosts </code></pre> Verify after Stage 0: <pre><code class="language-bash">curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/auth/health </code></pre> Expected: <code>200</code>. <h3 id="heading-wsl2">WSL2</h3> Find your WSL IP: <pre><code class="language-bash">ip -4 addr show eth0 | grep inet </code></pre> Use the IP shown (or <code>127.0.0.1</code> if it works on your machine), then add it to <code>/etc/hosts</code>: <pre><code class="language-bash">LAB_IP=<YOUR_IP> echo "$LAB_IP clearledger.local argocd.local grafana.local vault.local falco.local litmus.local" | sudo tee -a /etc/hosts </code></pre> If you use Chrome or Edge on Windows instead of inside WSL, add the same line to: <code>C:\Windows\System32\drivers\etc\hosts</code> Verify: <pre><code class="language-bash">curl http://clearledger.local/auth/health </code></pre> <h2 id="heading-stage-0-the-running-system">Stage 0 — The Running System</h2> Starting point: Nothing is deployed yet, so you're about to build a Kubernetes cluster and deploy ClearLedger manually. Goal: By the end of this stage, ClearLedger will be running on Kubernetes. You'll be able to register a user, submit transactions, and see compliance alerts, all deployed by hand, with no automation. Every deployment, update, and fix is manual. That's intentional. Before automating a platform, you need to understand how it works without automation. <h3 id="heading-01-provision-the-cluster">0.1: Provision the Cluster</h3> Next you'll be creating a virtual machine on your laptop that runs its own Kubernetes cluster. Think of it as a miniature data center inside your computer. Multipass creates lightweight Ubuntu VMs. MicroK8s is a minimal Kubernetes distribution that runs inside that VM. Together they give you a real cluster without needing cloud resources. Recommended: one command (do this): <pre><code class="language-bash">make setup export KUBECONFIG=~/.kube/clearledger-config kubectl get nodes </code></pre> Expected: <pre><code class="language-plaintext">NAME STATUS ROLES AGE VERSION clearledger Ready <none> 2m v1.29.x </code></pre> <code>make setup</code> runs <code>scripts/setup-cluster.sh</code> (VM + MicroK8s + disk-safety caps) and <code>scripts/setup-hosts.sh</code> (<code>/etc/hosts</code> entries). It takes 3–5 minutes. Disk-safety (log rotation, image GC thresholds, journald cap) is configured automatically. See Disk health in <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md</a> for more info. If STATUS is <code>NotReady</code>, wait 60 seconds and try again. Here's the manual setup (only if <code>make setup</code> failed and you need to debug step by step): <pre><code class="language-bash">multipass launch \ --name clearledger \ --cpus 6 --memory 12G --disk 80G \ 22.04 </code></pre> Get the VM IP (needed for <code>/etc/hosts</code>): <pre><code class="language-bash">multipass info clearledger | grep IPv4 </code></pre> Add hosts entries. See the <a href="#heading-how-to-configure-local-domain-names">Domain Names</a> section above, or run <code>sudo bash scripts/setup-hosts.sh</code>. <pre><code class="language-bash">multipass shell clearledger </code></pre> Inside the VM: <pre><code class="language-bash">sudo snap install microk8s --classic --channel=1.29/stable sudo usermod -aG microk8s ubuntu && newgrp microk8s microk8s enable dns ingress storage helm3 rbac echo "alias kubectl='microk8s kubectl'" >> ~/.bashrc echo "alias helm='microk8s helm3'" >> ~/.bashrc source ~/.bashrc kubectl get nodes exit # back to your host machine </code></pre> Connect kubectl from your host: <pre><code class="language-bash">multipass exec clearledger -- microk8s config > ~/.kube/clearledger-config export KUBECONFIG=~/.kube/clearledger-config kubectl get nodes </code></pre> <h3 id="heading-02-understand-the-application-before-deploying-it">0.2: Understand the Application Before Deploying it</h3> Open these files before running a single <code>kubectl</code> command. Reading the code first builds context that makes everything else make sense. <table> <thead> <tr> <th>File</th> <th>What it does</th> </tr> </thead> <tbody><tr> <td><a href="../app/auth-service/main.py"><code>app/auth-service/main.py</code></a></td> <td>Register, login, verify JWT</td> </tr> <tr> <td><a href="../app/ledger-service/main.py"><code>app/ledger-service/main.py</code></a></td> <td>Transactions, balance, calls auth-service to verify every request</td> </tr> <tr> <td><a href="../app/notification-service/main.py"><code>app/notification-service/main.py</code></a></td> <td>Subscribes to Redis, fires alerts when amount ≥ $10,000</td> </tr> <tr> <td><a href="../app/frontend/src/app.js"><code>app/frontend/src/app.js</code></a></td> <td>SPA: calls the same API as the curl commands</td> </tr> <tr> <td><a href="../app/auth-service/Dockerfile"><code>app/auth-service/Dockerfile</code></a></td> <td>Non-root user, pinned base image, HEALTHCHECK</td> </tr> </tbody></table> Notice this line in every Dockerfile: <code>USER appuser</code>. It means the image is designed to run as a normal user instead of root. The Kubernetes manifests also set <code>runAsNonRoot: true</code>. Later, in Stage 4, Kyverno enforces that rule and rejects pods that don't declare they run as non-root. Your app is prepared early so it passes that policy later. Also look at <a href="../infra/manifests/auth-service/secret.yaml"><code>infra/manifests/auth-service/secret.yaml</code></a>. The database password is <code>changeme-stage0</code> encoded in base64. Decode it: <pre><code class="language-bash">echo "Y2hhbmdlbWUtc3RhZ2Uw" | base64 -d # changeme-stage0 </code></pre> That password is sitting in a YAML file anyone with repo access can read. base64 is encoding, not encryption. It's trivially reversible. Remember this moment. It's why Stage 5 exists. <h3 id="heading-03-docker-hub-setup">0.3: Docker Hub Setup</h3> You need a container registry: a place to store the built images so the cluster can pull them. Docker Hub is the simplest option. You'll replace it with a private registry (ECR) in Stage 8. Create four public repositories on Docker Hub (free account, hub.docker.com): <ol> <li>Go to <a href="http://hub.docker.com"><code>hub.docker.com</code></a> </li> <li>Click Create repository </li> <li>Choose your Docker Hub username as the namespace </li> <li>Enter one repository name from the list below </li> <li>Set visibility to Public </li> <li>Click Create </li> <li>Repeat for all four services </li> </ol> <pre><code class="language-plaintext">YOUR_USERNAME/clearledger-auth-service YOUR_USERNAME/clearledger-ledger-service YOUR_USERNAME/clearledger-notification-service YOUR_USERNAME/clearledger-frontend </code></pre> Next, generate an access token. Go to hub.docker.com, then Account Settings, Security, and New Access Token (Read/Write/Delete). Save it. You won't see it again. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/85562991-e4e7-4d17-8d91-4c4ec2f60114.png" alt="image screenshot guide describing where and how to create access token" style="display:block;margin:0 auto" width="302" height="888" loading="lazy"> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/ad8a99a9-c6e4-41ac-a1c0-6799f731782c.png" alt="Image screenshot showing how to create access token" style="display:block;margin:0 auto" width="1039" height="593" loading="lazy"> <pre><code class="language-bash">docker login # Username: your Docker Hub username # Password: the access token (NOT your account password) </code></pre> Build and push all four services: <pre><code class="language-bash"># Replace your-username with your Docker Hub username, the same string everywhere in this lab export DOCKER_USERNAME=your-username echo "Using DOCKER_USERNAME=$DOCKER_USERNAME" </code></pre> ✋ Hands-on checkpoint: Docker Hub username <pre><code class="language-bash"># Must print your real username, not the literal text "your-username" echo "$DOCKER_USERNAME" </code></pre> Expected: one line with your Docker Hub name (for example, <code>veeno-demo</code>). If you see <code>your-username</code> instead, stop and fix <code>export</code> before building. Build and push all four services: <pre><code class="language-bash">docker build -t $DOCKER_USERNAME/clearledger-auth-service:v0.1.0 ./app/auth-service docker build -t $DOCKER_USERNAME/clearledger-ledger-service:v0.1.0 ./app/ledger-service docker build -t $DOCKER_USERNAME/clearledger-notification-service:v0.1.0 ./app/notification-service docker build -t $DOCKER_USERNAME/clearledger-frontend:v0.1.0 ./app/frontend # Push docker push $DOCKER_USERNAME/clearledger-auth-service:v0.1.0 docker push $DOCKER_USERNAME/clearledger-ledger-service:v0.1.0 docker push $DOCKER_USERNAME/clearledger-notification-service:v0.1.0 docker push $DOCKER_USERNAME/clearledger-frontend:v0.1.0 </code></pre> ✋ Hands-on checkpoint: images on Docker Hub Open hub.docker.com and go to your profile, then Repositories. Then confirm that all four <code>clearledger-*</code> repos exist and each shows tag <code>v0.1.0</code>. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/fb07dc5f-9db2-4819-810c-89b76b01a0e1.png" alt="screenshot image confirming what docker image repo looks like when done" style="display:block;margin:0 auto" width="776" height="397" loading="lazy"> On your laptop, run: <pre><code class="language-bash">docker pull $DOCKER_USERNAME/clearledger-auth-service:v0.1.0 </code></pre> Expected: <code>Status: Downloaded newer image</code> or <code>Image is up to date</code>, not <code>repository does not exist</code> or <code>denied</code>. <h3 id="heading-04-look-at-the-manifests-before-applying-them">0.4: Look at the Manifests Before Applying Them</h3> Kubernetes uses manifest files (YAML) to describe the resources it should create. Instead of clicking buttons, you declare the desired state, and Kubernetes creates it. Before deploying ClearLedger, take a quick look at these manifests: <ul> <li><code>infra/manifests/namespace.yaml</code>: Creates the <code>clearledger</code> namespace. </li> <li><code>infra/manifests/postgres/</code>: Deploys PostgreSQL. </li> <li><code>infra/manifests/redis/redis.yaml</code>: Deploys Redis. </li> <li><code>infra/manifests/auth-service/</code>: Deploys the authentication service. </li> <li><code>infra/manifests/ledger-service/</code>: Deploys the ledger service. </li> <li><code>infra/manifests/notification-service/</code>: Deploys the notification service. </li> <li><code>infra/manifests/frontend/</code>: Deploys the web application. </li> <li><code>infra/manifests/ingress.yaml</code>: Makes the application available at <code>clearledger.local</code>. </li> <li><code>infra/manifests/rbac/rbac.yaml</code>: Defines who may do what inside the cluster. </li> </ul> You don't need to understand every field yet. The goal is simply to see how the application is described before Kubernetes creates it. You'll understand how Ingress routing and RBAC work in the two optional sections after §0.6. For now, just see how the app is described before Kubernetes creates it. <h3 id="heading-05-deploy-clearledger-layer-by-layer">0.5: Deploy ClearLedger (Layer by Layer)</h3> Deploy in six layers. Finish each layer before starting the next. Run <code>kubectl get pods -n clearledger</code> after layers 2, 3, and 6 to confirm progress. Set a short path variable and confirm your username is still set: <pre><code class="language-bash">export DOCKER_USERNAME=your-username # skip if already set in §0.3 STAGE0=stages/stage-0-raw-kubernetes/infra/manifests </code></pre> <h4 id="heading-051-layer-1-namespace-and-rbac">0.5.1 — Layer 1: Namespace and RBAC</h4> Nothing else can be created until the namespace exists. RBAC also must exist before workloads reference ServiceAccounts. <pre><code class="language-bash">kubectl apply -f infra/manifests/namespace.yaml kubectl apply -f infra/manifests/rbac/rbac.yaml </code></pre> Verify: <pre><code class="language-bash">kubectl get namespace clearledger kubectl get serviceaccount -n clearledger # Expected: auth-service, ledger-service, notification-service, clearledger-viewer </code></pre> <h4 id="heading-052-layer-2-postgresql">0.5.2 — Layer 2: PostgreSQL</h4> Database must be running before auth-service or ledger-service start. Both services connect to Postgres on startup to run migrations and serve requests, and they'll crash-loop if the database isn't there yet. <pre><code class="language-bash">kubectl apply -f infra/manifests/postgres/postgres-secret.yaml kubectl apply -f infra/manifests/postgres/postgres.yaml kubectl wait --for=condition=ready pod -l app=postgres \ -n clearledger --timeout=120s </code></pre> Expected after <code>kubectl apply</code>: <pre><code class="language-plaintext">secret/postgres-secret created persistentvolumeclaim/postgres-pvc created statefulset.apps/postgres created service/postgres created </code></pre> Expected when <code>kubectl wait</code> succeeds: the command exits with no output (exit code 0). If it times out, see If Postgres stays Pending below before continuing. Verify: <pre><code class="language-bash">kubectl get pods -n clearledger -l app=postgres kubectl get pvc -n clearledger </code></pre> Expected: <pre><code class="language-plaintext">NAME READY STATUS RESTARTS AGE postgres-0 1/1 Running 0 45s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE postgres-pvc Bound pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx 5Gi RWO microk8s-hostpath 45s </code></pre> If Postgres stays Pending (<code>kubectl wait</code> times out, pod shows <code>0/1 Pending</code>, PVC shows <code>Pending</code>): Postgres needs a PersistentVolumeClaim: disk space on the cluster. MicroK8s provides that through the <code>hostpath-storage</code> addon. If <code>make setup</code> was interrupted or you used manual setup without <code>microk8s enable storage</code>, the PVC has nothing to bind to and the pod never schedules. Check the events: you'll usually see something like: <pre><code class="language-plaintext">Warning FailedScheduling ... pod has unbound immediate PersistentVolumeClaims Normal FailedBinding ... no persistent volumes available for this claim and no storage class is set </code></pre> Fix it on the VM, then restart the postgres pod. Run this from your host: the same command on macOS, Linux, or Windows PowerShell (Multipass is installed on the host. It executes inside the VM for you): <pre><code class="language-bash"># Enable storage (and ingress/rbac if make setup skipped them) multipass exec clearledger -- microk8s enable storage ingress rbac # Confirm a default StorageClass exists kubectl get storageclass # Expected: microk8s-hostpath (default) # Kick the pod so it reschedules against the new storage class kubectl delete pod postgres-0 -n clearledger kubectl wait --for=condition=ready pod -l app=postgres \ -n clearledger --timeout=120s kubectl get pods -n clearledger -l app=postgres # Expected: postgres-0 1/1 Running </code></pre> Don't continue to auth-service or ledger-service until Postgres is <code>Running</code>. They will crash-loop without a database. <h4 id="heading-053-layer-3-redis">0.5.3 — Layer 3: Redis</h4> Why Redis is here (a quick scenario): Imagine a customer posts a $15,000 debit. Ledger-service saves it to Postgres, then publishes a message to Redis: "large transaction, user X, amount 15000." Notification-service is listening on that channel. It picks up the message and records a compliance alert: the one you'll see in the UI later when you curl <code>/notifications/alerts</code>. Ledger-service and notification-service don't call each other directly. Redis sits in the middle as a message bus: ledger publishes, notification subscribes. That's why Redis must be running before you deploy notification-service (and why you deploy it now, alongside Postgres, before the app layer). <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/f099d734-8b73-483d-a28b-16cd7703703b.png" alt="flow daigram image explaining how redis works" style="display:block;margin:0 auto" width="1536" height="1024" loading="lazy"> <pre><code class="language-bash">kubectl apply -f infra/manifests/redis/redis.yaml </code></pre> Verify: <pre><code class="language-bash">kubectl get pods -n clearledger -l app=redis </code></pre> Expected: <pre><code class="language-plaintext">NAME READY STATUS RESTARTS AGE redis-xxxxxxxxxx-xxxxx 1/1 Running 0 30s </code></pre> <h4 id="heading-054-layer-4-application-secrets">0.5.4 — Layer 4: Application secrets</h4> Credentials live in Kubernetes Secrets for Stage 0 (Stage 5 moves them to Vault). <pre><code class="language-bash">kubectl apply -f infra/manifests/auth-service/secret.yaml kubectl apply -f infra/manifests/ledger-service/secret.yaml </code></pre> Verify: <pre><code class="language-bash">kubectl get secrets -n clearledger | grep -E 'auth-service|ledger-service' </code></pre> Expected (AGE will differ, DATA counts must match): <pre><code class="language-plaintext">auth-service-secret Opaque 2 64s ledger-service-secret Opaque 1 8s </code></pre> <code>auth-service-secret</code> holds two keys (<code>database_url</code>, <code>jwt_secret</code>). <code>ledger-service-secret</code> holds one (<code>database_url</code>). Stage 5 replaces these with Vault, for now they live in the cluster as Kubernetes Secrets. <h4 id="heading-055-layer-5-application-workloads">0.5.5 — Layer 5: Application workloads</h4> You're about to start the four app services: auth, ledger, notification, and frontend. Postgres, Redis, and the Secrets from the last two layers are already in place. Now Kubernetes needs to pull your Docker Hub images and run them as pods. Two files per service (mostly): A Deployment tells Kubernetes which container image to run and how many copies. A Service gives that app a stable name inside the cluster (for example, <code>auth-service</code> so ledger can find auth without knowing pod IP addresses). You apply the Deployment first, then the Service. So why are we using the <code>sed</code> command below? The deployment YAML files in Git contain a placeholder: literally the text <code>DOCKER_USERNAME</code>, because everyone's Docker Hub username is different. You already set yours in §0.3 (<code>export DOCKER_USERNAME=YOUR_DOCKERHUB_USERNAME</code>). The <code>sed</code> line swaps that placeholder for your real username on the fly, as the manifest is sent to Kubernetes. You never edit the file in Git. If you skip <code>sed</code> and apply the raw file, Kubernetes tries to pull an image called <code>DOCKER_USERNAME/clearledger-auth-service</code>, which doesn't exist. Why do we use the Stage 0 folder? This repo has more than one copy of the Kubernetes manifests. For this manual deployment, use <code>stages/stage-0-raw-kubernetes/infra/manifests/</code>. Those files are prepared for Stage 0 and contain the <code>DOCKER_USERNAME</code> placeholder that the commands below replace. Don't use <code>infra/manifests/</code> yet, as those files are for the GitOps stages later. Deploy each service in order. Run these from the repo root with <code>DOCKER_USERNAME</code> still exported: 1. auth-service: login and registration <pre><code class="language-bash">sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \ "$STAGE0/auth-service/deployment.yaml" | kubectl apply -f - kubectl apply -f infra/manifests/auth-service/service.yaml </code></pre> 2. ledger-service: transactions and balance (needs Postgres + the secret you created in §0.5.4) <pre><code class="language-bash">sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \ "$STAGE0/ledger-service/deployment.yaml" | kubectl apply -f - kubectl apply -f infra/manifests/ledger-service/service.yaml </code></pre> 3. notification-service: listens on Redis for large-transaction alerts (no database secret in this one) <pre><code class="language-bash">sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \ "$STAGE0/notification-service/deployment.yaml" | kubectl apply -f - kubectl apply -f infra/manifests/notification-service/service.yaml </code></pre> 4. frontend: the web UI (Deployment and Service are in one file here) <pre><code class="language-bash">sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \ "$STAGE0/frontend/deployment.yaml" | kubectl apply -f - </code></pre> Verify (all app pods should reach <code>Running</code>: auth and ledger may take ~30s while they connect to Postgres): <pre><code class="language-bash">kubectl get pods -n clearledger </code></pre> Expected. You should see Postgres and Redis from earlier layers plus new pods for each app (exact pod names vary): <pre><code class="language-plaintext">NAME READY STATUS RESTARTS AGE postgres-0 1/1 Running 0 15m redis-xxxxxxxxxx-xxxxx 1/1 Running 0 10m auth-service-xxxxxxxxxx-xxxxx 1/1 Running 0 45s auth-service-xxxxxxxxxx-xxxxx 1/1 Running 0 45s ledger-service-xxxxxxxxxx-xxxxx 1/1 Running 0 40s ledger-service-xxxxxxxxxx-xxxxx 1/1 Running 0 40s notification-service-xxxxxxxxxx-xxxxx 1/1 Running 0 35s frontend-xxxxxxxxxx-xxxxx 1/1 Running 0 30s </code></pre> If auth-service or ledger-service is <code>CrashLoopBackOff</code>, check the logs: <pre><code class="language-bash">kubectl logs -n clearledger deploy/auth-service --tail=20 </code></pre> Common cause: you applied <code>infra/manifests/*/deployment.yaml</code> instead of the Stage 0 files above: logs may show <code>DATABASE_URL is not set</code>. Re-run the <code>sed</code> + <code>kubectl apply</code> commands in this section. ✋ Hands-on checkpoint: workloads before ingress <pre><code class="language-bash">kubectl get deployment -n clearledger kubectl get pods -n clearledger --field-selector=status.phase!=Running </code></pre> Expected: four Deployments (<code>auth-service</code>, <code>ledger-service</code>, <code>notification-service</code>, <code>frontend</code>) with <code>READY</code> matching desired replicas (auth and ledger show <code>2/2</code>). The second command prints nothing: no pods stuck in Pending or CrashLoopBackOff. <h4 id="heading-056-layer-6-ingress">0.5.6 — Layer 6: Ingress</h4> Exposes the cluster to <code>http://clearledger.local</code>. <pre><code class="language-bash">kubectl apply -f infra/manifests/ingress.yaml </code></pre> Verify: <pre><code class="language-bash">kubectl get ingress -n clearledger curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/ # Expected: 200 </code></pre> <h4 id="heading-057-watch-until-stable">0.5.7: Watch until stable</h4> <pre><code class="language-bash">kubectl get pods -n clearledger -w </code></pre> Expected final state (press Ctrl+C to stop watching once all pods show <code>Running</code>): <pre><code class="language-plaintext">NAME READY STATUS RESTARTS auth-service-xxx 1/1 Running 0 auth-service-yyy 1/1 Running 0 frontend-xxx 1/1 Running 0 ledger-service-xxx 1/1 Running 0 ledger-service-yyy 1/1 Running 0 notification-service-xxx 1/1 Running 0 postgres-0 1/1 Running 0 redis-xxx 1/1 Running 0 </code></pre> Pod stuck in <code>Pending</code> or <code>CrashLoopBackOff</code>? These two commands show you what went wrong: <pre><code class="language-bash">kubectl describe pod POD_NAME -n clearledger kubectl logs POD_NAME -n clearledger --previous </code></pre> <h3 id="heading-06-verify-the-running-system">0.6: Verify the Running System</h3> Use one test account for both browser and curl so nothing conflicts: <table> <thead> <tr> <th>Field</th> <th>Value</th> </tr> </thead> <tbody><tr> <td>Email</td> <td><code>test@clearledger.io</code></td> </tr> <tr> <td>Password</td> <td><code>SecurePass123</code></td> </tr> </tbody></table> If you already registered in the browser with a different password, either sign in with that password or pick a new email: the curl commands below must use the same email and password you actually registered with. <h4 id="heading-browser-verification-recommended">Browser verification (recommended):</h4> Open <code>http://clearledger.local</code> in your browser. You should see the ClearLedger login screen. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/ba678edb-ed82-4063-b2e3-304bfe31e27c.png" alt="clearledger login screen UI screenshot" style="display:block;margin:0 auto" width="1140" height="1106" loading="lazy"> Click Register and create an account with <code>test@clearledger.io</code> / <code>SecurePass123</code> (same as the curl block below. Pydantic rejects obviously fake emails like <code>test@test.com</code>). Sign in with that email and password. On first login the dashboard auto-seeds demo transactions. Wait a few seconds for them to appear: <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/4bf4bd15-ad3b-466c-ba46-8539c2e1536c.png" alt="screenshot of clearledger UI after login" style="display:block;margin:0 auto" width="1141" height="933" loading="lazy"> Look at the Current Balance card. It should show a dollar amount with a sparkline chart. Look at Transaction History. You should see entries like "Salary (Acme Corp", "Rent) May 2026", and so on. And look at the Alerts panel at the bottom. You should see <code>LARGE_TRANSACTION</code> alerts with a red badge. Two of the demo transactions exceed $10,000, which triggers the compliance alert automatically. Then submit your own transaction over $10,000 and watch the alert count increase in real time. What to look for: <ul> <li>Balance updates immediately after each transaction </li> <li>Credits show as green <code>+$</code> amounts, debits show as red <code>−$</code> amounts </li> <li>The Alerts badge count increases when you submit a transaction ≥ $10,000 </li> <li>Each alert shows the amount, direction, and timestamp </li> </ul> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/05690dde-58de-4453-9a3f-8eee09b33293.png" alt="screenshot of clearledger UI after login and making transactions" style="display:block;margin:0 auto" width="1473" height="1269" loading="lazy"> Take a screenshot of the dashboard showing transactions and at least one alert. This is the first piece of your portfolio. Alternatively via curl (same account: useful if the browser is not cooperating): <pre><code class="language-bash"># Register (skip if you already registered in the browser with the same email) curl -s -X POST http://clearledger.local/auth/register \ -H "Content-Type: application/json" \ -d '{"email":"test@clearledger.io","password":"SecurePass123"}' | jq . </code></pre> Expected: <code>{"user_id":"...","email":"test@clearledger.io"}</code>, or an error that the email is already registered (fine if you used the browser first). <pre><code class="language-bash"># Login — save the token (must match the password you registered with) TOKEN=$(curl -s -X POST http://clearledger.local/auth/login \ -H "Content-Type: application/json" \ -d '{"email":"test@clearledger.io","password":"SecurePass123"}' \ | jq -r .access_token) echo "Token: ${TOKEN:0:30}..." </code></pre> If <code>TOKEN</code> is empty or login returns <code>401</code>, your browser password doesn't match: re-register with the table above or use your actual password in the <code>-d</code> JSON. <pre><code class="language-bash"># Create a large transaction (triggers notification alert) curl -s -X POST http://clearledger.local/ledger/transactions \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"amount":15000,"direction":"debit","description":"Property payment"}' | jq . </code></pre> Expected: a transaction object with <code>id</code>, <code>amount: 15000</code>, <code>direction: "debit"</code>: <pre><code class="language-bash"># Check balance curl -s http://clearledger.local/ledger/balance \ -H "Authorization: Bearer $TOKEN" | jq . </code></pre> <pre><code class="language-bash"># Confirm the notification alert fired curl -s http://clearledger.local/notifications/alerts | jq . </code></pre> Expected (curl-only path, no browser demo seed): at least one alert for the $15,000 transaction, for example, <code>{"total":1,"alerts":[{"type":"LARGE_TRANSACTION","amount":15000,...}]}</code>. If you already used the browser, <code>total</code> may be 3 or more (two demo alerts plus yours), that is also correct. If you see <code>{"detail":"Unauthorized"}</code>: your token has expired. JWTs are short-lived for security. This is intentional. Re-run the login command above to get a fresh token, then retry the failed command. This only affects the <code>$TOKEN</code> variable in your current terminal session. If you open a new terminal, you need to run the login command again because <code>$TOKEN</code> doesn't persist across sessions. <pre><code class="language-bash">make check-0 </code></pre> <h3 id="heading-understanding-ingress-optional">Understanding Ingress (Optional)</h3> Read this after §0.6 if you want to understand how <code>clearledger.local</code> reaches your pods. Your cluster runs four application services: frontend, auth-service, ledger-service, and notification-service. Each has an internal Service address inside the cluster, but none are reachable from your browser until an Ingress routes external traffic. The Ingress is the front door. When a request hits <code>clearledger.local</code>, Kubernetes looks at the URL path and forwards to the right service. Requests to <code>/auth</code> go to auth-service, <code>/ledger</code> to ledger-service, <code>/notifications</code> to notification-service, and <code>/</code> to the frontend. Open <a href="./infra/manifests/ingress.yaml"><code>infra/manifests/ingress.yaml</code></a> and read the comments. The API paths use a rewrite: <code>/auth/login</code> becomes <code>/login</code> before it reaches auth-service, so backend routes stay simple. You'll add more hostnames later (<code>grafana.local</code>, <code>argocd.local</code>, and so on): each gets its own Ingress manifest in a later stage. This file is only the ClearLedger app. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/3307ca55-75d4-41be-a161-e741d2349b5e.png" alt="flow diagram explaining ingress, how it works." style="display:block;margin:0 auto" width="1677" height="938" loading="lazy"> <h3 id="heading-understanding-rbac-optional">Understanding RBAC (Optional)</h3> Ingress controls traffic coming from outside the cluster. RBAC controls permissions inside the cluster. This file creates identities and permissions for the <code>clearledger</code> namespace. Open <a href="../infra/manifests/rbac/rbac.yaml"><code>infra/manifests/rbac/rbac.yaml</code></a>. The comments at the top mirror this walkthrough. A ServiceAccount is an identity for a pod. For example, <code>auth-service</code>, <code>ledger-service</code>, and <code>notification-service</code> each get their own identity. A Role says what that identity is allowed to do. In your repo, the app roles are very limited: they can only <code>get</code> and <code>list</code> Kubernetes Endpoints. They can't read Secrets, delete pods, create resources, or access other namespaces. A RoleBinding connects the identity to the permissions. Without the RoleBinding, the Role exists but no pod receives those permissions. The <code>clearledger-viewer</code> ServiceAccount is for read-only debugging. It can inspect pods, services, endpoints, events, and configmaps, but it can't read Secrets. The default ServiceAccount is bound to a role with zero permissions. That way, if a pod forgets to set <code>serviceAccountName</code>, it falls back to an identity that can do nothing. The point is least privilege: even if a pod is compromised, Kubernetes doesn't hand it broad cluster access. <h3 id="heading-07-why-manual-deploys-cant-be-trusted">0.7: Why Manual Deploys Can't Be Trusted</h3> In Stage 0, you built and deployed the app by hand. Now you'll make one small code change and deploy it again. This shows the problem with manual deployments: they're hard to track, hard to roll back, and hard to prove. Stages 1 and 2 fix that with CI and GitOps. <h4 id="heading-step-1-make-a-visible-change">Step 1: Make a visible change.</h4> Open <code>app/auth-service/main.py</code> and find the <code>/health</code> endpoint. Change the return value so you can tell the new version is running: <pre><code class="language-python"># Before return {"status": "ok", "service": settings.service_name} # After — add a version field return {"status": "ok", "service": settings.service_name, "version": "0.2.0"} </code></pre> Save the file. This simulates a developer shipping a small fix. <h4 id="heading-step-2-build-push-and-deploy-by-hand">Step 2: Build, push, and deploy by hand.</h4> <pre><code class="language-bash">docker build -t $DOCKER_USERNAME/clearledger-auth-service:v0.2.0 ./app/auth-service docker push $DOCKER_USERNAME/clearledger-auth-service:v0.2.0 kubectl set image deployment/auth-service \ auth-service=$DOCKER_USERNAME/clearledger-auth-service:v0.2.0 \ -n clearledger </code></pre> Wait about 30 seconds for Kubernetes to pull the new image and restart the pods: <pre><code class="language-bash">kubectl rollout status deployment/auth-service -n clearledger </code></pre> <h4 id="heading-step-3-verify-your-change-is-live">Step 3: Verify your change is live.</h4> <pre><code class="language-bash">curl -s http://clearledger.local/auth/health | jq . </code></pre> Expected: <code>{"status":"ok","service":"auth-service","version":"0.2.0"}</code> If you still see the old response without <code>"version"</code>, wait a few more seconds and retry. Kubernetes is still rolling out the new pods. <h4 id="heading-step-4-notice-what-manual-deploy-doesnt-give-you">Step 4: Notice what manual deploy doesn't give you.</h4> You deployed a change. It works. But think about what just happened: <ul> <li>Who deployed this? There's no record. You ran <code>kubectl</code> from your laptop. If three people have cluster access, no one knows who changed what. </li> <li>What changed? The only evidence is the Docker Hub tag <code>v0.2.0</code>. Nothing links that tag to a specific commit or code review. </li> <li>What if <code>v0.2.0</code> is broken? You would need to remember the previous tag, then run <code>kubectl set image</code> again to roll back. What if you don't remember the tag? What if the previous image was deleted? </li> <li>What if someone else runs <code>kubectl apply</code> with <code>v0.1.0</code> while you're pushing <code>v0.2.0</code>? The cluster silently reverts to the old version. No error. No notification. You think your fix is live, but it's not. </li> <li>Where is the audit trail? Nowhere. In a regulated environment (banking, healthcare, government), you need proof of who deployed what and when. Right now you have nothing. </li> </ul> Manual deploys can work for a demo. But they don't hold up for a team or a regulated environment. Keep these gaps in mind. They're why the next stages exist. <h4 id="heading-step-5-revert-your-change-before-continuing">Step 5: Revert your change before continuing.</h4> Undo the health endpoint change in <code>app/auth-service/main.py</code> (remove <code>"version": "0.2.0"</code>). Don't rebuild: the cluster will keep running <code>v0.2.0</code> for now, and Stage 1 will take over image management. Stage 1 automates the build. Stage 2 fixes the deployment. <h3 id="heading-what-you-learned-in-stage-0">What You Learned in Stage 0</h3> <ul> <li>How to provision a local Kubernetes cluster with Multipass and MicroK8s </li> <li>How Kubernetes manifests describe the desired state of your system </li> <li>How an Ingress routes external traffic to internal services </li> <li>How to build, push, and deploy container images manually </li> <li>Why manual deploys can't be trusted: no audit trail, no rollback, no consistency </li> </ul> What you can now put on your CV / say in an interview: <blockquote> Deployed a multi-service application to Kubernetes by hand: namespace, RBAC, a StatefulSet database, Deployments, Services, and path-based Ingress routing, and can explain why each layer deploys in that order. </blockquote> <code>make snapshot STAGE=0 && make snapshots</code>. Confirm <code>clearledger.stage0</code>. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>. <h2 id="heading-stage-1-ci-pipeline-github-actions-self-hosted-runner">Stage 1 — CI Pipeline (GitHub Actions + Self-Hosted Runner)</h2> In Stage 0 you built and deployed by hand. Stage 1 automates the build: a <code>git push</code> runs a pipeline that builds images, scans them, pushes to Docker Hub, and records the new tag in <code>clearledger-infra</code>. Goal: every push to GitHub automatically builds images, pushes them to Docker Hub, and updates image tags in <code>clearledger-infra</code>. Am I ready for Stage 1? Run these yourself before §1.1: <pre><code class="language-plaintext">make check-0 echo "$DOCKER_USERNAME" # must not be empty or "your-username" curl -s -o /dev/null -w "%{http_code}" http://clearledger.local/auth/health </code></pre> Expected: health check green, <code>echo</code> prints your Docker Hub user, and curl prints <code>200</code>. What you'll need for this section: <ul> <li>Docker Hub account with four clearledger- repositories (see QUICKSTART.md §1b) </li> <li>GitHub account: you can create repos and personal access tokens </li> <li>~2–4 hours for runner install + first green pipeline (this is the hardest stage for beginners) </li> <li>Done when: make check-1 passes and you manually confirmed the five items in §1.7 below. Then save: make snapshot STAGE=1 → make snapshots (confirm clearledger.stage1). </li> </ul> <h3 id="heading-what-you-need-to-know-first">What You Need to Know First</h3> In Stage 0, your laptop was the deployment system. You typed <code>docker build</code>, <code>docker push</code>, and <code>kubectl set image</code> yourself. That worked for a demo, but it's not how teams should ship software. Manual builds create too many unanswered questions: <ul> <li>Did this image come from the latest code? </li> <li>Did someone build it from a dirty working tree? </li> <li>Did the build work the same way on another machine? </li> <li>Which commit produced the image currently running? </li> <li>Who pushed the image, and when? </li> </ul> CI (Continuous Integration) fixes the build side of that problem. It means that every time code is pushed, an automated system builds, checks, and packages it the same way. Think of CI as a factory line: <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/5a9a4283-a77c-493e-85c4-e457b6ab00c9.png" alt="flow diagram explain gihub ci flow" style="display:block;margin:0 auto" width="1024" height="1536" loading="lazy"> <pre><code class="language-text">Developer pushes code ↓ GitHub detects the push ↓ GitHub Actions starts the pipeline ↓ Runner executes the jobs ↓ Docker images are built and pushed ↓ Infra manifests are updated with the new image tags (in clearledger-infra — §1.3) </code></pre> The important idea is that the build no longer depends on your laptop. Your laptop writes code and the pipeline produces the release artifact. A CI system has three parts: <ol> <li>Pipeline host: the control plane. It notices a push and decides which workflow to run. In this lab, that's GitHub Actions. </li> <li>Pipeline file: the instructions. It's a YAML file at <code>.github/workflows/ci.yaml</code> that says what jobs to run. </li> <li>Runner: the worker machine. It actually executes the commands in the pipeline. </li> </ol> GitHub Actions normally uses GitHub-hosted runners in the cloud. In this lab, that's not enough. Your Kubernetes cluster lives inside a local Multipass VM and GitHub's cloud runner can't reach it. You also need the runner inside the VM to build Docker images using the local Docker daemon. So you install a self-hosted runner inside the VM. It connects outbound to GitHub, waits for work, then executes pipeline jobs locally where it can reach everything. Two repos, <code>clearledger</code> (code + CI) and <code>clearledger-infra</code> (Kubernetes YAML only). You'll create the second in §1.3. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/f7c69215-31c8-4806-b899-15caeace9485.png" alt="flow chart demonstrating self hosted github flow" style="display:block;margin:0 auto" width="1024" height="1536" loading="lazy"> <pre><code class="language-text">GitHub — clearledger (app repo) stores your code starts the workflow on git push ↓ Self-hosted runner (inside Multipass VM) builds Docker images pushes images to Docker Hub updates image tags in clearledger-infra ← you create this in §1.3 ↓ GitHub — clearledger-infra (infra repo) stores Kubernetes YAML with the new image tags ArgoCD watches this repo in Stage 2 (not yet) </code></pre> For Stages 1–7, the lab uses <code>.github/workflows/ci.yaml</code> with your self-hosted runner. It builds images, pushes them to Docker Hub, and updates <code>clearledger-infra</code>. Stage 8 adds a separate AWS workflow, <code>.github/workflows/ci-aws.yaml</code>, which pushes to ECR instead. You don't need to configure the AWS workflow until you reach Stage 8. <h3 id="heading-11-push-the-app-repo-to-github-not-clearledger-infra-yet">1.1: Push the App Repo to GitHub (Not <code>clearledger-infra</code> Yet)</h3> This step is repo #1, <code>clearledger</code> (application code + CI workflow). You're pushing the clone on your laptop: the same folder where you ran Stage 0 (<code>make setup</code>, <code>kubectl apply</code>, and so on). <code>clearledger-infra</code> comes later in §1.3. That second repo holds Kubernetes manifests only. Don't create it here. First, put the application repo somewhere GitHub Actions can see it. Go to GitHub and then New Repository: <ul> <li>Repository name: <code>clearledger</code> (exact name, not <code>clearledger-infra</code>) </li> <li>Visibility: Public or Private. Both work with the self-hosted runner and GitHub Actions. ArgoCD never reads this repo (see <a href="#heading-private-repos-what-syncs-where">Private repos: what syncs where</a> in §1.3). </li> <li>Do not initialize with a README or <code>.gitignore</code> </li> </ul> The repo already has those files locally. If GitHub creates its own, your first push may fail because the histories don't match. Run from your local <code>clearledger</code> project root on your laptop (where <code>app/</code>, <code>infra/</code>, and <code>.github/workflows/ci.yaml</code> live): <pre><code class="language-bash">cd ~/Desktop/clearledger # your clone path git remote add origin https://github.com/YOUR_USERNAME/clearledger.git git branch -M main git push -u origin main </code></pre> If <code>git remote add</code> fails because <code>origin</code> already exists: <pre><code class="language-bash">git remote -v git remote set-url origin https://github.com/YOUR_USERNAME/clearledger.git git push -u origin main </code></pre> Verify in the browser: <code>https://github.com/YOUR_USERNAME/clearledger</code>. You should see <code>app/</code>, <code>infra/manifests/</code>, <code>docs/</code>, and <code>.github/workflows/ci.yaml</code>. That confirms GitHub can trigger the pipeline on your next push. What you proved: the app repo is on GitHub. CI will run from here. Deployment manifests for GitOps land in <code>clearledger-infra</code> in §1.3. <h3 id="heading-12-install-the-self-hosted-runner-inside-the-vm">1.2: Install the Self-Hosted Runner Inside the VM</h3> The workflow file tells GitHub what to run. The runner is where it runs. This lab uses a self-hosted runner because your infrastructure is local. GitHub's cloud servers can't reach your MicroK8s cluster or Docker daemon inside the Multipass VM. The runner solves that by living inside the VM. It connects to GitHub to pick up jobs, then executes everything locally. If the runner is missing or offline, the pipeline can't execute. The workflow may sit queued, or it may fail because no matching runner is available. <h4 id="heading-step-1-open-githubs-runner-setup-page-keep-this-tab-open">Step 1: Open GitHub’s runner setup page (keep this tab open)</h4> GitHub gives you a full copy-paste install guide on one page. Use it: don’t hunt for URLs or tokens elsewhere. <ol> <li>Open <code>https://github.com/YOUR_USERNAME/clearledger</code> </li> <li>Go to Settings, Actions, Runners, and New self-hosted runner </li> <li>Select Linux and x64 </li> </ol> The page title should look like: Add new self-hosted runner · YOUR_USERNAME/clearledger. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/e0aa537c-a2b2-4ed7-8b48-f686a012ea8d.png" alt="e0aa537c-a2b2-4ed7-8b48-f686a012ea8d" style="display:block;margin:0 auto" width="1251" height="1267" loading="lazy"> That page has three sections you'll use: <table> <thead> <tr> <th>Section on GitHub</th> <th>What to do with it</th> </tr> </thead> <tbody><tr> <td>Download</td> <td>Copy the <code>mkdir</code>, <code>curl</code>, and <code>tar</code> commands into the VM in Step 4 (same versions as below)</td> </tr> <tr> <td>Configure</td> <td>Copy the token from the <code>./config.sh ... --token ...</code> line: do not run GitHub’s <code>./config.sh</code> as-is</td> </tr> <tr> <td>Using your self-hosted runner</td> <td>Ignore for now, the lab workflow needs the <code>clearledger</code> label (Step 4)</td> </tr> </tbody></table> Scroll to Configure. You'll see something like: <pre><code class="language-bash">./config.sh --url https://github.com/YOUR_USERNAME/clearledger --token AXXXXXXXXXXXXXXXXXXXXXXXXX ./run.sh </code></pre> The token is the long string after <code>--token</code> (starts with <code>A</code>, about 26 characters). Copy only that string. Keep this tab open until Step 4 finishes: the token expires in about 1 hour. If it expires, click New self-hosted runner again for a fresh token. <h4 id="heading-step-2-enter-the-vm">Step 2: Enter the VM</h4> <code>multipass shell clearledger</code> After this command, your prompt should look like <code>ubuntu@clearledger:~$</code>. That means you are inside the Ubuntu VM. If your prompt still shows your Mac username or MacBook name, you're still on your host machine and the runner setup will fail. Continue only when your prompt shows <code>ubuntu@clearledger</code>. Everything from Step 3 onwards runs inside the VM, not on your Mac. <h4 id="heading-step-3-install-docker-inside-the-vm">Step 3: Install Docker inside the VM</h4> The runner will build Docker images. That means Docker must exist where the runner runs. <pre><code class="language-bash">curl -fsSL https://get.docker.com | sh sudo usermod -aG docker ubuntu newgrp docker docker --version </code></pre> Expected: Docker prints a version number (for example, <code>Docker version 29.x.x</code>). Verify Docker works for the <code>ubuntu</code> user now: the runner doesn't exist yet (Step 4 creates <code>~/actions-runner</code>): <pre><code class="language-bash">docker ps </code></pre> Expected: a table header (CONTAINER ID, IMAGE, …), even if no containers are listed. Not <code>permission denied while trying to connect to the Docker API</code>. If <code>docker ps</code> fails with permission denied, the <code>docker</code> group has not applied yet. Run <code>newgrp docker</code> again, or log out of the VM (<code>exit</code>) and <code>multipass shell clearledger</code> back in, then retry <code>docker ps</code>. What you proved: the VM can run Docker without Docker Desktop on your Mac. Continue to Step 4 to install the runner. <h4 id="heading-step-4-install-and-register-the-runner">Step 4: Install and register the runner</h4> Still inside the VM (<code>ubuntu@clearledger</code> prompt): Download: you can copy the commands from the Download section on GitHub’s runner page (Step 1), or run the block below. They should match. Paste into the VM, not your Mac. Configure: use the lab command below, not GitHub’s <code>./config.sh</code> line. Paste your token from Step 1 and replace <code>YOUR_USERNAME</code>. <pre><code class="language-bash">mkdir -p ~/actions-runner && cd ~/actions-runner curl -o actions-runner-linux-x64-2.335.1.tar.gz -L \ https://github.com/actions/runner/releases/download/v2.335.1/actions-runner-linux-x64-2.335.1.tar.gz tar xzf ./actions-runner-linux-x64-2.335.1.tar.gz ./config.sh \ --url https://github.com/YOUR_USERNAME/clearledger \ --token YOUR_RUNNER_TOKEN \ --name clearledger-runner \ --labels clearledger,self-hosted,linux \ --work _work \ --unattended sudo ./svc.sh install sudo ./svc.sh start </code></pre> Do not run GitHub’s <code>./run.sh</code> for day-to-day use: the lab uses <code>sudo ./svc.sh</code> so the runner survives VM reboots. GitHub shows <code>./run.sh</code> for a quick test only. Expected after <code>./config.sh</code>: <code>Runner successfully added</code> (or similar). If you see Invalid token or Expired token, go back to Step 1 in the browser and copy a fresh token. The <code>clearledger</code> label is required GitHub’s default <code>./config.sh</code> on the setup page doesn't add it. The workflow uses: <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/f8fa25ff-81da-4759-b021-293c244add7c.png" alt="image showing where to add the label in github ui for the runner" style="display:block;margin:0 auto" width="938" height="252" loading="lazy"> <pre><code class="language-yaml">runs-on: [self-hosted, clearledger] </code></pre> GitHub schedules jobs by runner labels, not by runner name. A runner named <code>clearledger</code> without the <code>clearledger</code> label will stay online but jobs will remain queued with <code>Waiting for a runner to pick up this job</code>. What those last two commands mean: <pre><code class="language-text">sudo ./svc.sh install Registers the runner with systemd inside the VM. Without this, `sudo ./svc.sh status` says: not installed. sudo ./svc.sh start Starts the runner service in the background. After this, it keeps running even when you close the terminal. </code></pre> Check it locally from the same folder, still inside the VM: <pre><code class="language-bash">cd ~/actions-runner sudo ./svc.sh status </code></pre> Expected: the service is installed and running. If <code>docker ps</code> worked in Step 3 but a CI job later fails with Docker socket permission denied, the runner probably started before the <code>docker</code> group applied. Restart it after Step 4 (only when <code>~/actions-runner</code> exists): <pre><code class="language-bash">cd ~/actions-runner sudo ./svc.sh stop sudo ./svc.sh start docker ps # must work without sudo </code></pre> Or, if you started the runner manually with <code>./run.sh</code> instead of systemd: <pre><code class="language-bash">cd ~/actions-runner pkill -f "Runner.Listener|Runner.Worker|./run.sh" || true nohup ./run.sh > _diag/manual-runner.log 2>&1 & docker ps </code></pre> If you see this: <pre><code class="language-text">not installed </code></pre> then <code>sudo ./svc.sh install</code> didn't run successfully. Run: <pre><code class="language-bash">cd ~/actions-runner sudo ./svc.sh install sudo ./svc.sh start sudo ./svc.sh status </code></pre> If <code>install</code> fails, rerun <code>./config.sh</code> with a fresh GitHub runner token, then run the install/start commands again. <h4 id="heading-step-5-exit-the-vm">Step 5: Exit the VM</h4> <pre><code class="language-bash">exit </code></pre> <h4 id="heading-step-6-verify-the-runner-is-connected">Step 6: Verify the runner is connected</h4> Go to github.com/YOUR_USERNAME/clearledger then to Settings, Actions, and Runners. You should see <code>clearledger-runner</code> with a green dot and status Idle. Open the runner details and confirm the labels include: <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/f47d23ff-fecf-4789-b8ee-3be4191c1c3a.png" alt="screenshot image of github ui shpwing runner status as "idle" green" style="display:block;margin:0 auto" width="816" height="589" loading="lazy"> <pre><code class="language-text">self-hosted Linux X64 clearledger </code></pre> If <code>clearledger</code> is missing, add it in the runner settings before rerunning the workflow. The runner name alone is not enough. ✋ Hands-on checkpoint: runner ready for jobs Still on GitHub, Settings, Actions, and Runners, confirm: <table> <thead> <tr> <th>Field</th> <th>Expected</th> </tr> </thead> <tbody><tr> <td>Status</td> <td>Idle (green)</td> </tr> <tr> <td>Labels</td> <td>includes <code>self-hosted</code> and <code>clearledger</code></td> </tr> <tr> <td>OS</td> <td>Linux</td> </tr> </tbody></table> Then trigger a dry run from your laptop: <pre><code class="language-bash">git commit --allow-empty -m "test: verify runner picks up jobs" git push </code></pre> Open <code>https://github.com/YOUR_USERNAME/clearledger/actions</code>. Within 30 seconds a workflow run should show Queued then In progress, not stuck on “Waiting for a runner.” If it waits more than 2 minutes, the labels are wrong. Edit the runner on GitHub and add <code>clearledger</code>. If it shows Offline: <pre><code class="language-bash">multipass exec clearledger -- sudo systemctl status actions.runner.*.service multipass exec clearledger -- journalctl -u actions.runner.*.service --lines=50 </code></pre> What you proved: GitHub can now send work into your local lab environment. <h3 id="heading-13-create-the-infra-repo-on-github">1.3: Create the Infra Repo on GitHub</h3> Now separate application code from deployment state. Stage 1 introduces a second GitHub repository alongside the <code>clearledger</code> app repo you pushed in §1.1. You'll use two repositories for the rest of the lab: <table> <thead> <tr> <th>Repo</th> <th>What lives there</th> <th>Who changes it</th> <th>Why it exists</th> </tr> </thead> <tbody><tr> <td><code>clearledger</code></td> <td>App source code, Dockerfiles, tests, <code>.github/workflows/ci.yaml</code>, lab docs</td> <td>You, the developer</td> <td>This is where code changes start</td> </tr> <tr> <td><code>clearledger-infra</code></td> <td>Kubernetes manifests only: <code>deployment.yaml</code>, <code>service.yaml</code>, ingress, secrets templates</td> <td>The CI pipeline, then ArgoCD reads it</td> <td>This is the desired state of the cluster</td> </tr> </tbody></table> Think of <code>clearledger</code> as the question “What is the application?”. Python services, Dockerfiles, tests, and the CI workflow. Think of <code>clearledger-infra</code> as “What exact version should be running in Kubernetes right now?”. Deployments, Services, ingress rules, and the image tags that point at Docker Hub. Teams split these on purpose. If you edit <code>README.md</code> in <code>clearledger</code>, that is a documentation change. It shouldn't trigger a deployment. If you change <code>auth-service</code> code, the pipeline builds a new image (for example tag <code>abc123</code>) and, only after scans pass, records that tag in <code>clearledger-infra</code>: <pre><code class="language-yaml">image: $DOCKER_USERNAME/clearledger-auth-service:abc123 </code></pre> That line is a deployment contract: Git now says the cluster should run <code>abc123</code>. In Stage 1, the cluster doesn't change yet (and you'll prove that in §1.6). In Stage 2, ArgoCD watches <code>clearledger-infra</code>, compares Git to what is running, and syncs the cluster when they differ. The app repo is where work begins. The infra repo is what production is supposed to look like. <h4 id="heading-private-repos-what-syncs-where">Private repos: what syncs where</h4> This lab uses two GitHub repos. <code>clearledger</code> is your main project repo: app code, CI pipeline, docs, policies, and lab files. This repo can be private. <code>clearledger-infra</code> contains only Kubernetes manifests. ArgoCD watches this repo and uses it to deploy the app. For beginners, make this repo public so ArgoCD can read it without extra authentication. The flow looks like this: <pre><code class="language-text">clearledger app code + infra/manifests/ ↓ CI copies infra/manifests/ ↓ clearledger-infra Kubernetes manifests only ↓ ArgoCD syncs from this repo ↓ Kubernetes cluster </code></pre> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/4aa15b2c-c746-4711-aa64-704a9d3eada2.png" alt="flow chart explain how both repos work" style="display:block;margin:0 auto" width="1165" height="1350" loading="lazy"> ArgoCD doesn't read the main <code>clearledger</code> repo. It only reads <code>clearledger-infra</code>. If <code>clearledger</code> is private, that is fine. If <code>clearledger-infra</code> is private, you must give ArgoCD GitHub credentials later. If you do not, ArgoCD may show <code>ComparisonError</code>. Create the infra repo on GitHub: <ol> <li>Go to GitHub and then New Repository </li> <li>Name it <code>clearledger-infra</code> </li> <li>Choose Public </li> <li>Don't add a README </li> <li>Click Create </li> </ol> Later, the CI pipeline will update <code>clearledger-infra</code> automatically. In Stage 1, the pipeline doesn't run <code>kubectl apply</code> – it updates Git. In Stage 2, ArgoCD reads that Git repo and applies it to the cluster. Before pushing: set your Docker Hub username in Kustomize (image tags are resolved here, not in deployment YAML): <pre><code class="language-bash"># Replace YOUR_DOCKERHUB_USERNAME with the same value as $DOCKER_USERNAME from §0.3 sed -i.bak "s/YOUR_DOCKERHUB_USERNAME/${DOCKER_USERNAME}/g" infra/manifests/kustomization.yaml rm -f infra/manifests/kustomization.yaml.bak </code></pre> Push only the Kubernetes manifests from <code>infra/manifests/</code> (not everything under <code>infra/</code>): <pre><code class="language-bash">mkdir -p /tmp/clearledger-infra cp -r infra/manifests /tmp/clearledger-infra/ cd /tmp/clearledger-infra git init git remote add origin https://github.com/YOUR_USERNAME/clearledger-infra.git git add . && git commit -m "feat: initial manifests" && git push -u origin main cd - </code></pre> ✋ Hands-on checkpoint: infra repo on GitHub (do this before §1.4) On your laptop: <pre><code class="language-bash">grep "docker.io/${DOCKER_USERNAME}/" infra/manifests/kustomization.yaml | wc -l grep YOUR_DOCKERHUB_USERNAME infra/manifests/kustomization.yaml || echo "OK: placeholder replaced" </code></pre> Expected: first command prints <code>4</code> (four image lines). Second prints <code>OK: placeholder replaced</code>, not four lines still saying <code>YOUR_DOCKERHUB_USERNAME</code>. In the browser, open <code>https://github.com/YOUR_USERNAME/clearledger-infra/tree/main/manifests</code> and confirm with your eyes: <table> <thead> <tr> <th>File / folder</th> <th>Must exist</th> </tr> </thead> <tbody><tr> <td><code>kustomization.yaml</code></td> <td>Yes. Open it: <code>newName:</code> lines use your Docker Hub user</td> </tr> <tr> <td><code>auth-service/secret.yaml</code></td> <td>Yes. Stages 2–4 need this until Stage 5</td> </tr> <tr> <td><code>ledger-service/secret.yaml</code></td> <td>Yes</td> </tr> <tr> <td><code>auth-service/deployment.yaml</code></td> <td>Yes. Open it: must contain <code>secretKeyRef</code>, not <code>vault.hashicorp.com</code></td> </tr> <tr> <td><code>netpol/</code></td> <td>No. If present, delete the folder on GitHub before Stage 2</td> </tr> <tr> <td><code>vault/</code></td> <td>No. Vault rotation is Stage 5 only</td> </tr> </tbody></table> Which folders matter? You only pushed <code>infra/manifests/</code> to GitHub, that's correct. Everything else in this repo stays local for now. Some manifests for later stages (network policies, Vault extras) live under <code>infra/deferred-by-stage/</code> in the <code>clearledger</code> repo. You'll apply those by hand when you reach that stage. Do not copy that folder into <code>clearledger-infra</code>, or ArgoCD will deploy things too early. You might notice <code>stages/stage-1-ci-pipeline/</code> has no copy of the manifests. That is normal: the lab doesn't duplicate YAML there. The canonical copy is <code>infra/manifests/</code> in this repo, and the live GitOps copy is <code>clearledger-infra</code> on GitHub. What you proved: Kubernetes config now has its own repo and Git history, separate from application code. CI will update <code>clearledger-infra</code> after each build, and your app repo stays for code and the pipeline file. <h3 id="heading-14-set-up-github-secrets">1.4: Set up GitHub Secrets</h3> Go to <code>github.com/YOUR_USERNAME/clearledger</code> and then Settings, Secrets and variables, Actions, and New repository secret. The workflow needs credentials for Docker Hub, GitHub, and image signing: <ul> <li>Docker Hub, so it can push images. </li> <li>GitHub, so it can push image tag updates into <code>clearledger-infra</code>. </li> <li>Cosign, so it can sign the images after pushing them. </li> </ul> Do not paste these values into YAML files. Store them as GitHub Actions secrets. <h4 id="heading-secret-1-dockerusername">Secret 1, <code>DOCKER_USERNAME</code></h4> This is just your Docker Hub username. Example: <pre><code class="language-text">veeno-demo </code></pre> Get it from Docker Hub: hub.docker.com, profile menu, Account Settings. <h4 id="heading-secret-2-dockerpassword">Secret 2, <code>DOCKER_PASSWORD</code></h4> This should be a Docker Hub access token, not your normal Docker Hub password. Create it here: <pre><code class="language-text">hub.docker.com → Account Settings → Security → New Access Token → Description: clearledger-github-actions → Access permissions: Read, Write, Delete or Read/Write → Generate </code></pre> Copy the token immediately. Docker Hub only shows it once. <h4 id="heading-secret-3-infrarepotoken">Secret 3, <code>INFRA_REPO_TOKEN</code></h4> This is a GitHub Personal Access Token (PAT). The pipeline uses it to push commits to the second repo, <code>clearledger-infra</code>. Create it here: <pre><code class="language-text">GitHub profile settings → Settings → Developer settings → Personal access tokens → Tokens (classic) → Click "Generate new token" → Choose "Generate new token (classic)" → If GitHub asks for your password or 2FA, complete it → Note: clearledger-infra-ci → Expiration: choose a lab-friendly value → Select scope: repo This allows the pipeline to push to clearledger-infra. → Generate token </code></pre> Copy the token immediately. GitHub only shows it once. For this lab, <code>repo</code> scope is the simplest option. In production, you would use tighter permissions, such as a fine-grained token limited to only <code>clearledger-infra</code>. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/5a0d06bf-570a-4360-aadb-038b7ae7ed4e.png" alt="screenshot of docker ui showing where to set up PAT" style="display:block;margin:0 auto" width="302" height="888" loading="lazy"> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/b9e7f0c8-882c-4d15-b67b-d2ff38289836.png" alt="screenshot of docker ui showing where to set up token scope" style="display:block;margin:0 auto" width="1039" height="593" loading="lazy"> <h4 id="heading-secrets-4-and-5-cosignprivatekey-and-cosignpassword">Secrets 4 and 5, <code>COSIGN_PRIVATE_KEY</code> and <code>COSIGN_PASSWORD</code></h4> Cosign signs container images after the pipeline pushes them to Docker Hub. Later, Stage 4 uses the public key with Kyverno so the cluster can verify that images came from your trusted pipeline. Generate the key pair on your host machine, not inside the Multipass VM: <pre><code class="language-bash"># macOS: brew install cosign # Linux/WSL2: curl -sSL -o cosign https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64 && chmod +x cosign && sudo mv cosign /usr/local/bin/ cosign generate-key-pair </code></pre> This creates: <pre><code class="language-text">cosign.key # private key — never commit this cosign.pub # public key — keep for later Kyverno verification </code></pre> When Cosign asks for a password, enter one and save it in your password manager. If you already generated a key without a password, regenerate it with a password for this lab. Add these five secrets to the <code>clearledger</code> repo, not <code>clearledger-infra</code>: <table> <thead> <tr> <th>Secret name</th> <th>Value</th> <th>Purpose</th> </tr> </thead> <tbody><tr> <td><code>DOCKER_USERNAME</code></td> <td>Your Docker Hub username</td> <td>Pipeline logs in to push images</td> </tr> <tr> <td><code>DOCKER_PASSWORD</code></td> <td>Your Docker Hub access token</td> <td>Pipeline authenticates with Docker Hub</td> </tr> <tr> <td><code>INFRA_REPO_TOKEN</code></td> <td>The GitHub PAT from above</td> <td>Pipeline pushes image tag updates to clearledger-infra</td> </tr> <tr> <td><code>COSIGN_PRIVATE_KEY</code></td> <td>Contents of <code>cosign.key</code></td> <td>Pipeline signs pushed container images</td> </tr> <tr> <td><code>COSIGN_PASSWORD</code></td> <td>Password used when creating the Cosign key</td> <td>Unlocks the private key during signing</td> </tr> </tbody></table> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/638009a9-844b-4bcb-9701-6312ec18d5c6.png" alt="screenshot of github UI showing my repository secrets" style="display:block;margin:0 auto" width="980" height="338" loading="lazy"> Repository variables (not secrets) (optional) toggles for later stages. Add under Settings, Secrets and variables, Actions, Variables: <table> <thead> <tr> <th>Variable</th> <th>Stage 1</th> <th>When to enable</th> </tr> </thead> <tbody><tr> <td><code>ENABLE_ARGOCD_SYNC</code></td> <td>Leave unset</td> <td>Stage 2 — after ArgoCD’s first sync is healthy (see <a href="#heading-how-to-enable-the-ci-to-argocd-handoff">Enable CI → ArgoCD handoff</a>)</td> </tr> <tr> <td><code>ENABLE_DAST</code></td> <td>Leave unset</td> <td>Stage 3 — after the app is live at <code>clearledger.local</code> (see <a href="#heading-enable-dast-optional-after-stage-2">Enable DAST</a>)</td> </tr> </tbody></table> Don't add either variable in Stage 1. If you set them now, CI will try to refresh ArgoCD or run ZAP before the cluster is ready, and the pipeline output gets harder to read. The guide calls out the exact moment to turn each one on – you only need to remember that both exist. What you proved: the pipeline can authenticate to external systems without hardcoding credentials in the repo. <h3 id="heading-15-understand-the-pipeline-before-activating-it">1.5: Understand the Pipeline Before Activating it</h3> Don't treat the workflow file as magic. Open <code>.github/workflows/ci.yaml</code> and read it before you run it. The pipeline has two responsibilities: <ol> <li>Prove the code and images are safe enough to publish. </li> <li>Update the infra repo with the new image tags. </li> </ol> Here's the security flow first: <pre><code class="language-text">Developer pushes code to GitHub ↓ GitHub Actions starts workflow ↓ Self-hosted runner inside the Multipass VM picks up the job ↓ 1. Scan secrets (Gitleaks) ↓ 2. Run code security scans (Semgrep) + IaC scan (Checkov) — parallel ↓ 3. Prepare scanners (install Trivy/Syft/Grype/Cosign once; refresh Trivy DB once) ↓ 4. BUILD: docker build all four services (local tags only; nothing hits Docker Hub yet) ↓ 5. SCAN: Trivy on all images; Syft + Grype SBOM on auth-service; upload evidence ↓ 6. PUBLISH: push to Docker Hub + Cosign sign (only if scan passed) ↓ 7. UPDATE MANIFESTS: commit new image tags to clearledger-infra </code></pre> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/0cea04bf-e166-4fd3-9eb5-1a664e427206.png" alt="visual image of the cicd security flow pattern" style="display:block;margin:0 auto" width="1024" height="1536" loading="lazy"> <h4 id="heading-build-scan-publish-prod-style-gates">Build, scan, publish (prod-style gates)</h4> Real teams never push first and scan later. The pipeline separates three concerns into three jobs in <code>.github/workflows/ci.yaml</code>: <table> <thead> <tr> <th>Job</th> <th>What it does</th> <th>If it fails…</th> </tr> </thead> <tbody><tr> <td><code>build-images</code></td> <td><code>docker build</code> all services with tag <code>${{ github.sha }}</code></td> <td>No registry pollution, images never left the runner</td> </tr> <tr> <td><code>scan-images</code></td> <td>Trivy (all 4 images); Syft + Grype (auth only)</td> <td>Publish is skipped: bad images never reach Docker Hub</td> </tr> <tr> <td><code>publish-images</code></td> <td>Runs <code>scripts/ci-publish-image.sh</code> tag, push, Cosign sign</td> <td>Only runs after scan passes</td> </tr> </tbody></table> You do not run <code>scripts/ci-publish-image.sh</code> yourself before pushing code. GitHub Actions checks out the repo and calls it inside <code>publish-images</code>. Why can <code>build-images</code> and <code>scan-images</code> be separate jobs? Each job is a fresh checkout on GitHub-hosted runners. They don't share a disk. On your self-hosted runner, all three jobs run on the same Multipass VM and use the same Docker engine. Job 1 runs <code>docker build</code> and leaves the images on that machine. Job 2 runs Trivy against those same local images: no upload, no download. Job 3 pushes to Docker Hub only if the scan passed. That's a practical lab setup: one persistent build machine with Docker installed, like a dedicated CI worker in a real office. In Stage 8 (AWS), the pipeline uses GitHub-hosted runners instead: there, <code>build-images</code> saves the images to a file (<code>images.tar</code>) and passes that file to the next job as a workflow artifact, because those runners are throwaway VMs with no shared Docker cache. Then comes the GitOps handoff: <pre><code class="language-text">Secure images now exist in Docker Hub ↓ Runner checks out clearledger-infra from GitHub ↓ Deployment YAML image tags are updated ↓ Runner commits and pushes back to clearledger-infra ↓ Stage 1 ends here </code></pre> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/3be226da-a7d5-4231-8110-c37f1b8bfdce.png" alt="visual image of the cicd security flow pattern and github handoff journey" style="display:block;margin:0 auto" width="1024" height="1536" loading="lazy"> Here's how the image tag ties to your code: every pipeline run is triggered by a git commit. GitHub gives that commit a unique ID called the SHA (a long hex string like <code>a1b2c3d4e5f6789…</code>). The workflow sets <code>IMAGE_TAG</code> to that SHA and uses it everywhere: <ol> <li>Build: <code>docker build -t clearledger-auth-service:a1b2c3d4…</code> </li> <li>Publish: push to Docker Hub as <code>YOUR_DOCKERHUB_USERNAME/clearledger-auth-service:a1b2c3d4…</code> </li> <li>Update manifests: <code>kustomize edit set image …:a1b2c3d4…</code> in <code>clearledger-infra</code> </li> <li>Commit message: <code>ci: deploy a1b2c3d4… — all gates passed</code> </li> </ol> If production is running <code>YOUR_DOCKERHUB_USERNAME/clearledger-auth-service:a1b2c3d4</code>, you can copy that <code>a1b2c3d4</code> tag, open GitHub, and instantly find the exact commit that built that image. There's no guessing and no wondering if <code>latest</code> changed. Every deployed image points back to one specific version of the code, making rollbacks and debugging much easier. <h4 id="heading-the-kustomize-placeholder">The Kustomize placeholder</h4> <code>auth-service/deployment.yaml</code> uses a label instead of a real image address: <pre><code class="language-yaml">image: clearledger/auth-service:gitops </code></pre> That label isn't on Docker Hub. It tells Kustomize where to substitute. The real address lives in <code>kustomization.yaml</code>: <pre><code class="language-yaml">images: - name: clearledger/auth-service # matches the label above newName: docker.io/YOUR_DOCKERHUB_USERNAME/clearledger-auth-service newTag: abc123def456… # real commit SHA — CI writes this </code></pre> When ArgoCD deploys, <code>kustomize build</code> swaps the label for the full address. You edit <code>kustomization.yaml</code> once in §1.3 to set your Docker Hub username in <code>newName:</code>. After that, CI writes <code>newTag:</code> automatically on every green push. You never touch it by hand. <h4 id="heading-stage-1-ci-updates-github-not-the-cluster">Stage 1: CI updates GitHub, not the cluster</h4> After a green pipeline run, three things are true: <ul> <li>New images exist on Docker Hub </li> <li><code>clearledger-infra</code> on GitHub has new SHAs in <code>kustomization.yaml</code> </li> <li>Your Kubernetes cluster is unchanged. Still running whatever Stage 0 left there </li> </ul> CI never runs <code>kubectl apply</code>. It only commits to <code>clearledger-infra</code>. That's the whole Stage 1 lesson: build and scan are automated, but deploy is not: yet. Stage 2 installs ArgoCD, which reads <code>clearledger-infra</code> and updates the cluster for you. Kubernetes Checkov runs in Stage 1 but does not block the pipeline. It uploads findings so you can see hardening work ahead. Stage 4 turns those kinds of rules into cluster enforcement with Kyverno. Jobs run on your self-hosted runner (<code>runs-on: [self-hosted, clearledger]</code>). Both <code>ENABLE_ARGOCD_SYNC</code> and <code>ENABLE_DAST</code> are unset in Stage 1. See §1.4 for when each gets flipped. If a job fails, start with <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md"><code>docs/troubleshooting.md</code></a> before editing the workflow. <h4 id="heading-stage-1-security-posture-what-blocks-vs-what-waits">Stage 1 security posture: what blocks vs what waits</h4> Note that stage 1 is not “security off.” Some gates stop the pipeline while others run for evidence and tighten in later stages. Blocks the pipeline today: <ul> <li>Gitleaks (secrets in Git) </li> <li>Semgrep (SAST on Python) </li> <li>Checkov on Dockerfiles </li> <li>Trivy (fixable HIGH/CRITICAL CVEs in images) </li> <li>Grype on auth-service SBOM (fixable HIGH+) </li> <li>Manifest update to <code>clearledger-infra</code> (must succeed) </li> </ul> Runs but doesn't block yet: <ul> <li>Checkov on Kubernetes manifests – enforced in Stage 4 (Kyverno) </li> <li>Cosign sign + SLSA attest – enforced in Stage 4 (unsigned images rejected) </li> <li>Syft SBOM generation – supply-chain evidence. You'll purposely break gates in Stage 3. </li> <li>ArgoCD refresh – Stage 2 (<code>ENABLE_ARGOCD_SYNC=true</code>) </li> <li>DAST / ZAP – Stage 3 (<code>ENABLE_DAST=true</code>) </li> </ul> If you forget which stage fixes what, search this guide for “Stage 1 security posture” or follow the stage order: Stage 3 breaks gates on purpose, Stage 4 connects Checkov findings to Kyverno, Stage 5 moves secrets off Git, Stage 6 adds runtime detection, Stage 7 adds monitoring dashboards. Run <code>make check-3</code> and <code>make check-4</code> after those stages to confirm hardening landed. Design intent: Stage 1 proves CI can build, scan, push, and update Git without you touching Docker manually. Later stages turn evidence into enforcement. The relaxations here are deliberate. <h3 id="heading-16-activate-the-pipeline">1.6: Activate the Pipeline</h3> Run this in the <code>clearledger</code> app repo, not <code>clearledger-infra</code>. §1.3 created <code>clearledger-infra</code> with only Kubernetes manifests. It has no <code>.github/workflows/</code> and no pipeline. If your shell prompt says <code>clearledger-infra</code>, or you used <code>/tmp/clearledger-infra</code>, you're in the wrong place. <pre><code class="language-bash">cd /path/to/clearledger # the app repo you pushed in §1.1 git remote -v # must show .../clearledger.git — NOT clearledger-infra ls .github/workflows/ci.yaml # must exist before you commit </code></pre> The pipeline file already lives at <code>.github/workflows/ci.yaml</code>. Push any small change to <code>clearledger</code> on <code>main</code>: <pre><code class="language-bash">echo "# Pipeline activated $(date)" >> README.md git add README.md git commit -m "ci: activate GitHub Actions pipeline" git push origin main </code></pre> Watch the run at: <code>https://github.com/YOUR_USERNAME/clearledger/actions</code> (app repo Actions tab, not the infra repo). When the pipeline succeeds, it updates <code>clearledger-infra</code> for you. You don't need to push anything to the infra repo by hand for this step. Expected Output: all jobs green in about 8 minutes. <pre><code class="language-plaintext">✓ Build + Scan auth-service ✓ Build + Scan ledger-service ✓ Build + Scan notification-service ✓ Build + Scan frontend ✓ Update manifests → GitHub </code></pre> DAST and the ArgoCD refresh step show as skipped: this is expected, because Both toggles are unset until later (see §1.4). Note: this lab includes <code>.gitleaksignore</code> because some intentional demo secrets are already present in Git history. Gitleaks still runs normally. The ignore file only suppresses known lab fingerprints. Don't add new findings to it unless you've confirmed they're intentional test data. Click into the job logs and look for the story. Don't just wait for green: <ul> <li>Docker login succeeded </li> <li>Each service image built and pushed to Docker Hub </li> <li><code>clearledger-infra</code> was checked out </li> <li>Deployment YAMLs were updated with the new SHA tag </li> <li>A commit was pushed back to <code>clearledger-infra</code> </li> </ul> After the pipeline succeeds, open <code>https://github.com/YOUR_USERNAME/clearledger-infra</code> and look at the deployment manifests. The image tags should now use the current commit SHA. Now check the cluster: <pre><code class="language-bash">kubectl get deployment auth-service -n clearledger \ -o jsonpath='{.spec.template.spec.containers[0].image}' && echo </code></pre> You may still see the old image. That's expected. This is the most important learning in Stage 1: <pre><code class="language-text">GitHub pipeline succeeded. Docker Hub has new images. clearledger-infra has new image tags. The Kubernetes cluster did not update automatically. </code></pre> That's not a failure. It's the deployment gap. Stage 1 automated the build, but no controller is watching the infra repo yet. Stage 2 installs ArgoCD to close that gap. <h3 id="heading-17-hands-on-checkpoint-prove-stage-1-is-really-done">1.7 — Hands-on Checkpoint: Prove Stage 1 is Really Done</h3> Don't rely on a green workflow badge alone. Run each check yourself: <h4 id="heading-1-infra-repo-still-has-app-secrets-critical-for-stage-2">1. Infra repo still has app secrets (critical for Stage 2)</h4> Open <code>https://github.com/YOUR_USERNAME/clearledger-infra/tree/main/manifests/auth-service</code>, <code>secret.yaml</code> must be visible. On your laptop: <pre><code class="language-bash">git clone --depth 1 https://github.com/YOUR_USERNAME/clearledger-infra.git /tmp/verify-infra grep secretKeyRef /tmp/verify-infra/manifests/auth-service/deployment.yaml grep secret.yaml /tmp/verify-infra/manifests/kustomization.yaml rm -rf /tmp/verify-infra </code></pre> Expected: <code>secretKeyRef</code> in deployment output. kustomization lists <code>auth-service/secret.yaml</code> and <code>ledger-service/secret.yaml</code>. If secrets are missing, re-push §1.3 manifests before Stage 2. <h4 id="heading-2-kustomize-image-tags-updated-by-ci">2. Kustomize image tags updated by CI</h4> <pre><code class="language-bash">git clone --depth 1 https://github.com/YOUR_USERNAME/clearledger-infra.git /tmp/verify-infra grep newTag /tmp/verify-infra/manifests/kustomization.yaml rm -rf /tmp/verify-infra </code></pre> Expected: <code>newTag</code> is a 40-character git SHA (or your commit hash), not still <code>v0.1.0</code> only: unless you haven't pushed since §0.3. <h4 id="heading-3-docker-hub-has-signed-images-from-this-pipeline">3. Docker Hub has signed images from this pipeline</h4> Open hub.docker.com then <code>clearledger-auth-service</code> then Tags. The latest tag should match the SHA from step 2. <h4 id="heading-4-cluster-unchanged-deployment-gap-intentional">4. (Cluster unchanged (deployment gap) intentional)</h4> <pre><code class="language-bash">kubectl get deployment auth-service -n clearledger \ -o jsonpath='{.spec.template.spec.containers[0].image}' && echo </code></pre> Expected: still your Stage 0 tag (for example, <code>veeno-demo/clearledger-auth-service:v0.1.0</code>), not the new SHA. That proves CI didn't touch the cluster. <h4 id="heading-5-runner-still-idle">5. Runner still idle</h4> GitHub, Settings, Actions, Runners, <code>clearledger-runner</code>, Idle. <pre><code class="language-bash">make check-1 </code></pre> All five pass, onto Stage 2. <h3 id="heading-what-you-learned-in-stage-1">What You Learned in Stage 1</h3> <ul> <li>CI removes your laptop from the build process. Builds become repeatable, visible, and tied to Git commits. </li> <li>A runner is the worker, not the pipeline itself. GitHub schedules the job, the self-hosted runner executes it inside your VM. </li> <li>Artifacts and desired state are different things. Docker Hub stores built images. <code>clearledger-infra</code> on GitHub stores the Kubernetes manifests that say which image should run. </li> <li>Good pipelines don't secretly mutate clusters. This pipeline updates Git instead of running <code>kubectl</code>. </li> <li>The gap that remains: the infra repo changed, but the cluster didn't. Someone still has to apply the change manually. Stage 2 fixes that with GitOps. </li> </ul> What you can now put on your CV / say in an interview: <blockquote> Built a CI pipeline on a self-hosted GitHub Actions runner that builds and pushes container images on every push, and can debug a workflow that fails before any job is created. </blockquote> <code>make snapshot STAGE=1 && make snapshots</code>. Confirm <code>clearledger.stage1</code>. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>. <h2 id="heading-stage-2-gitops-with-argocd">Stage 2 — GitOps with ArgoCD</h2> From this point on, Git is in charge. Whatever is written in the infrastructure repository is what should be running. If someone changes the cluster by hand, ArgoCD notices the difference and changes it back to match Git. Goal: Install ArgoCD so it watches <code>clearledger-infra</code> and deploys changes to the cluster. The CI pipeline only updates the Git repository, it never connects to Kubernetes or runs <code>kubectl</code> commands. Here's a more conversational, compressed version: <h3 id="heading-am-i-ready-for-stage-2">Am I ready for Stage 2?</h3> Before moving on, finish §1.6, then run: <pre><code class="language-bash">make check-1 grep secretKeyRef infra/manifests/auth-service/deployment.yaml grep vault.hashicorp infra/manifests/auth-service/deployment.yaml && echo "STOP: Vault annotations present" || echo "OK" </code></pre> You should see: <ul> <li><code>check-1</code> passes </li> <li><code>secretKeyRef</code> is present </li> <li><code>OK</code> (no Vault annotations yet) </li> </ul> Quick checklist: <ul> <li><code>clearledger-infra</code> contains <code>auth-service/secret.yaml</code> and <code>ledger-service/secret.yaml</code> </li> <li>Your self-hosted runner is Idle with the <code>clearledger</code> label </li> <li><code>ENABLE_ARGOCD_SYNC</code> isn't set yet (you'll enable it after installing ArgoCD) </li> </ul> You're done with Stage 2 when <code>make check-2</code> passes and <a href="http://argocd.local"><code>http://argocd.local</code></a> shows ArgoCD syncing <code>clearledger</code>. Finally, save your progress: <pre><code class="language-bash">make snapshot STAGE=2 make snapshots </code></pre> Confirm that <code>clearledger.stage2</code> appears in the snapshot list. <h3 id="heading-what-you-need-to-know-first">What You Need to Know First</h3> The gap from Stage 1: CI already builds images and updates <code>clearledger-infra</code>. The cluster didn't change until someone ran <code>kubectl</code>. This stage closes that last step. <table> <thead> <tr> <th>Who</th> <th>Job</th> </tr> </thead> <tbody><tr> <td>CI (Stage 1)</td> <td>Build → scan → push images → update image tags in <code>clearledger-infra</code></td> </tr> <tr> <td>ArgoCD (Stage 2)</td> <td>Watch <code>clearledger-infra</code> → apply manifests → cluster runs what Git says</td> </tr> </tbody></table> <pre><code class="language-text">push code → CI updates clearledger-infra → ArgoCD syncs cluster </code></pre> <h3 id="heading-pre-sync-checklist-run-before-argocd-app-sync">Pre-sync Checklist: Run Before <code>argocd app sync</code></h3> ArgoCD applies whatever is in <code>clearledger-infra</code>. Wrong content causes red pods. Re-run the §1.7 checkpoint table to confirm GitHub-side content is still correct, then verify the laptop side: <pre><code class="language-bash"># Application manifest must point at YOUR infra repo grep repoURL stages/stage-2-gitops/argocd/clearledger-app.yaml # Stage 0 workloads still healthy before ArgoCD takes over kubectl get pods -n clearledger curl -s -o /dev/null -w "%{http_code}" http://clearledger.local/auth/health </code></pre> Expected: <code>repoURL</code> contains your GitHub username, all app pods <code>Running</code>, curl <code>200</code>. Only when both pass should you install ArgoCD and sync below. <pre><code class="language-bash">kubectl create namespace argocd 2>/dev/null || true kubectl apply -n argocd --server-side --force-conflicts -f \ https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml kubectl wait --for=condition=ready pod \ -l app.kubernetes.io/name=argocd-server -n argocd --timeout=180s </code></pre> Why <code>--server-side --force-conflicts</code>? Argo CD ships a very large <code>applicationsets.argoproj.io</code> CRD. A normal <code>kubectl apply</code> tries to stash the whole thing in an annotation, hits a 256 KiB limit, and errors with <code>metadata.annotations: Too long</code>. Server-side apply avoids that. It's <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/installation/">how Argo CD expects you to install</a>. Get the admin password: <pre><code class="language-bash">kubectl -n argocd get secret argocd-initial-admin-secret \ -o jsonpath="{.data.password}" | base64 -d && echo </code></pre> <h4 id="heading-configure-argo-cd-for-your-nginx-ingress">Configure Argo CD for your NGINX ingress</h4> The browser talks HTTPS to ingress and ingress talks plain HTTP to the Argo CD server. Without this, the UI often breaks with <code>503</code> or <code>ERR_TOO_MANY_REDIRECTS</code> on live-update URLs (<code>/api/v1/stream/*</code>). <pre><code class="language-bash">kubectl apply -f stages/stage-2-gitops/infra/argocd-cmd-params.yaml kubectl apply -f stages/stage-2-gitops/infra/argocd-ingress.yaml kubectl rollout restart deployment/argocd-server -n argocd kubectl rollout status deployment/argocd-server -n argocd --timeout=180s </code></pre> Expected in <code>argocd-cmd-params-cm</code>: <code>server.insecure: "true"</code>, <code>server.grpc.web: "true"</code>, <code>server.url: https://argocd.local</code>. Open <code>https://argocd.local</code>. Login: <code>admin</code> and the password from above. Accept the self-signed certificate warning if the browser shows one. Expected: The Applications page loads. In the browser console (F12 Console), you shouldn't see <code>401</code> or <code>ERR_HTTP2_PROTOCOL_ERROR</code>. If the UI looks fine in a normal window, you're done: incognito isn't required. If login fails with <code>401 Unauthorized</code> (often after a config change or a bad earlier login), try a private/incognito window or clear site data for <code>argocd.local</code>, then log in again. Still stuck? See <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md. ArgoCD</a>. Connect ArgoCD to the infra repo and apply the Application manifest: <h4 id="heading-1-edit-stagesstage-2-gitopsargocdclearledger-appyaml">1. Edit <code>stages/stage-2-gitops/argocd/clearledger-app.yaml</code></h4> Set <code>spec.source.repoURL</code> to your infra repo (your GitHub username, not <code>git config user.name</code>). <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/56020fa9-77fa-4d60-bcc5-bbd06b6c809f.png" alt="photo of manifest file pointing to what to change." style="display:block;margin:0 auto" width="705" height="161" loading="lazy"> <h4 id="heading-2-connect-argocd-to-your-infrastructure-repository">2. Connect ArgoCD to your infrastructure repository:</h4> This gives ArgoCD permission to watch <code>clearledger-infra</code> for new commits. Whenever the deployment manifests change, ArgoCD will update the cluster automatically. <pre><code class="language-bash"># macOS: brew install argocd argocd login argocd.local --username admin --password YOUR_PASSWORD --insecure --grpc-web # Public repo argocd repo add https://github.com/YOUR_USERNAME/clearledger-infra.git --grpc-web # Private repo — PAT from Stage 1 §1.4 (you saved it as GitHub secret INFRA_REPO_TOKEN) export INFRA_REPO_TOKEN='ghp_...' # paste here; GitHub only shows it once at creation argocd repo add https://github.com/YOUR_USERNAME/clearledger-infra.git \ --username git --password "$INFRA_REPO_TOKEN" --grpc-web </code></pre> Verify that Argo CD can reach the repo (do this before applying the Application): <pre><code class="language-bash">argocd repo list --grpc-web </code></pre> Look for your <code>clearledger-infra</code> URL with TYPE <code>git</code> and connection Successful. If it shows Failed or the repo is missing, Argo CD can't sync. Fix credentials before Stage 4 or any stage that depends on GitOps. After a VM restore or Argo CD reinstall, you may need to run <code>argocd repo add</code> again (credentials are stored in the cluster, not in Git). <h4 id="heading-3-apply-and-sync">3. Apply and sync:</h4> <pre><code class="language-bash">kubectl apply -f stages/stage-2-gitops/argocd/clearledger-app.yaml argocd app sync clearledger --grpc-web </code></pre> <h3 id="heading-how-to-read-the-argo-cd-ui">How to Read the Argo CD UI</h3> After sync, open the clearledger application in the tree view. Three badges at the top tell you almost everything: <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/2db73788-ee70-450d-b5ed-00835d50d180.png" alt="screenshot shwoing argocd UI" style="display:block;margin:0 auto" width="1127" height="1275" loading="lazy"> APP HEALTH: Healthy. Kubernetes thinks the workloads are running. Pods are up (or still starting if it says Progressing). SYNC STATUS: Synced: the cluster matches <code>clearledger-infra</code> on GitHub at the commit shown (for example, <code>main (2c88aa1)</code>). Git is the source of truth and Argo CD applied it. LAST SYNC: Succeeded: the most recent apply from Git worked. If this failed, click it for the error. The resource tree below is the same app broken into pieces: namespace, secrets, services, deployments, ingress, and so on Green checkmarks = applied from Git. Click any box (for example, <code>deploy/auth-service</code>) then Live Manifest vs Desired to see what Argo CD thinks should run. Quick "is the app actually working?" test (outside Argo CD): <pre><code class="language-bash">curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/auth/health </code></pre> <code>200</code> = the app is reachable end-to-end, not only "green in Argo CD." When something is wrong: HEALTH goes Degraded or Progressing for a long time, SYNC goes OutOfSync, and a resource in the tree turns red. Click that resource and then Events or Logs. The kubectl checks below double-check the same thing from the terminal. Confirm ArgoCD is watching all workloads (not only ingress): <pre><code class="language-bash">argocd app resources clearledger --grpc-web | grep Deployment </code></pre> Pass looks like your output: <pre><code class="language-text">apps Deployment clearledger auth-service No apps Deployment clearledger frontend No apps Deployment clearledger ledger-service No apps Deployment clearledger notification-service No apps Deployment clearledger redis No </code></pre> This command shows the Deployments that ArgoCD is managing for ClearLedger. You should see <code>auth-service</code>, <code>ledger-service</code>, <code>notification-service</code>, <code>frontend</code>, and <code>redis</code>. That means ArgoCD reads the full <code>kustomization.yaml</code> from <code>clearledger-infra/manifests</code>, not just one file. The last column is <code>ORPHANED</code>. <code>No</code> is good. It means ArgoCD knows this resource belongs to the ClearLedger app. You only need to worry if one of the Deployments is missing, or if ArgoCD shows <code>OutOfSync</code>, <code>Degraded</code>, or red resources in the UI. ✋ Hands-on checkpoint: first sync healthy Run these four checks. Pass looks like this: <pre><code class="language-bash">kubectl get pods -n clearledger # Every app pod 1/1 Running (postgres/redis may show older RESTARTS from VM reboots — OK) kubectl get application clearledger -n argocd \ -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}' # sync=Synced health=Healthy curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/auth/health # 200 kubectl logs -n clearledger deploy/auth-service --tail=5 2>/dev/null | head -3 # Lines like: GET /health HTTP/1.1" 200 OK # Bad sign: DATABASE_URL is not set </code></pre> If all four pass then, Stage 2 first sync is done. Continue to Enable CI, and ArgoCD handoff below, then <code>make check-2</code> and <code>make snapshot STAGE=2</code>. <h3 id="heading-how-to-enable-the-ci-to-argocd-handoff">How to Enable the CI to ArgoCD Handoff</h3> In Stage 1, the pipeline updated <code>clearledger-infra</code>, but it didn't update the cluster. That was intentional. Now ArgoCD is installed, so you can let the pipeline tell ArgoCD to check for changes after each successful run. In GitHub, open your <code>clearledger</code> repo and go to Settings, Secrets and variables, Actions, Variables, and then New repository variable. Add: <table> <thead> <tr> <th>Name</th> <th>Value</th> </tr> </thead> <tbody><tr> <td><code>ENABLE_ARGOCD_SYNC</code></td> <td><code>true</code></td> </tr> </tbody></table> From now on, a green pipeline does two things: <ol> <li>Updates <code>clearledger-infra</code> with the new image tag </li> <li>Asks ArgoCD to sync the cluster </li> </ol> If the pipeline can't trigger ArgoCD immediately, that's usually okay. ArgoCD checks <code>clearledger-infra</code> on its own every few minutes, so it should still pick up the new Git change. Leave <code>ENABLE_DAST</code> unset for now. You enable that in Stage 3 after the app is stable at <code>clearledger.local</code>. <h3 id="heading-if-the-argocd-ui-shows-red-pods-or-progressing-read-this-before-the-screenshot">If the Argocd UI Shows Red Pods or "Progressing" (Read This Before the Screenshot)</h3> This is a common first-sync surprise, not a broken install. <h4 id="heading-why-it-happens-in-stage-2">Why it happens in Stage 2</h4> ArgoCD syncs whatever is in <code>clearledger-infra</code>. Deployments must use <code>secretKeyRef</code> (Stages 2–4), not Vault injection. If your infra repo has Vault annotations from an older lab copy, auth/ledger crash with <code>DATABASE_URL is not set</code> until Stage 5. Network policies belong to Stage 6. In the main <code>clearledger</code> repo, they live in <code>infra/deferred-by-stage/stage-6-runtime-security/netpol/</code>, not in <code>infra/manifests/</code>. Don't copy them into <code>clearledger-infra</code> during Stage 2. If <code>manifests/netpol/</code> is still in your <code>clearledger-infra</code> repo on GitHub (from an older copy of the lab), ArgoCD will keep applying it. Those policies use default-deny and break DNS for new pods, so you see red 0/1 pods and Progressing health. <h4 id="heading-fix-for-stage-2">Fix for Stage 2</h4> Do both steps. Deleting only in the cluster is not enough: ArgoCD recreates policies from Git on the next sync. Step 1: remove from <code>clearledger-infra</code> on GitHub Delete the folder <code>manifests/netpol/</code> and commit: <code>chore: defer network policies to Stage 6</code>. Step 2: sync and restart <pre><code class="language-bash">argocd app sync clearledger --grpc-web kubectl delete networkpolicy -n clearledger --all # safe once Git no longer has netpol kubectl rollout restart deployment/auth-service deployment/ledger-service -n clearledger argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status" </code></pre> Network policies stay in <code>clearledger</code> under <code>infra/deferred-by-stage/</code> until you apply them in Stage 6. When that looks good, continue below. When ArgoCD finishes syncing, open the <code>clearledger</code> app in the ArgoCD UI. You should see green <code>Healthy</code> and <code>Synced</code> badges. The app should point to your <code>clearledger-infra</code> repo, use the <code>manifests</code> path, and deploy into the <code>clearledger</code> namespace. Then open the app tile. The resource tree should show your deployments, services, and ingress with no red resources. You can confirm the same thing from the terminal: <code>argocd app get clearledger --grpc-web</code> Look for <code>Sync Status: Synced</code> and <code>Health Status: Healthy</code>. <h3 id="heading-argocd-stuck-outofsync">ArgoCD stuck OutOfSync</h3> Normal path: CI copies full manifests + updates Kustomize tags, ArgoCD auto-syncs within ~3 minutes. If still OutOfSync after 10+ minutes: <pre><code class="language-bash">make fix-argocd </code></pre> This re-syncs canonical manifests to <code>clearledger-infra</code> (Kustomize SHAs preserved), re-applies the Application, and triggers a hard refresh. Don't <code>kubectl apply</code> deployments: fix Git, let ArgoCD sync. <pre><code class="language-bash">kubectl annotate application clearledger -n argocd argocd.argoproj.io/refresh=hard --overwrite argocd app sync clearledger --grpc-web --prune kubectl get application clearledger -n argocd -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}' </code></pre> Take a screenshot of that view: the app tile or the resource tree is fine. That’s your portfolio proof that GitOps is actually running. <h3 id="heading-prove-argocd-self-healing">Prove ArgoCD Self-Healing</h3> Now prove that Git is the source of truth. In this demo, you'll change the running cluster by hand. You will not change Git. ArgoCD should notice that the cluster no longer matches <code>clearledger-infra</code>, then change it back. Before you start, make sure the app is healthy and ArgoCD is managing the deployments: <pre><code class="language-bash">argocd app resources clearledger --grpc-web | grep Deployment </code></pre> Manually change the auth-service image in the cluster: <pre><code class="language-bash"># Manually change the image in the cluster only (Git stays the same) kubectl set image deployment/auth-service \ auth-service=$DOCKER_USERNAME/clearledger-auth-service:fake-tag \ -n clearledger </code></pre> Check ArgoCD: <pre><code class="language-bash"># ArgoCD should flip to OutOfSync within a minute or two argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status" </code></pre> Wait for ArgoCD to fix the cluster. The fake image tag may briefly cause an image pull error. That's expected in this demo. <pre><code class="language-bash"># Wait for selfHeal (default sync interval is ~3 minutes) sleep 180 </code></pre> Confirm the image was changed back to the Git version: <pre><code class="language-bash"># Cluster image should match clearledger-infra again — Git was never edited kubectl get deployment auth-service -n clearledger \ -o jsonpath='{.spec.template.spec.containers[0].image}' </code></pre> If the image changed back, ArgoCD self-healing worked. You changed the cluster by hand, but ArgoCD restored it to match <code>clearledger-infra</code>. That's GitOps: Git says what should run, and ArgoCD keeps the cluster matching Git. <pre><code class="language-bash">make check-2 </code></pre> <h3 id="heading-how-to-roll-back-a-bad-deploy">How to Roll Back a Bad Deploy</h3> You just proved that ArgoCD reverts unauthorized cluster changes. Now flip it: what if you pushed a bad commit yourself? GitOps rollback isn't a button. It's a Git operation. This section explains why, shows you both methods, and has you practice each one before you need them under pressure. <h4 id="heading-how-you-know-you-need-to-roll-back">How you know you need to roll back</h4> These symptoms appearing within minutes of a push to <code>clearledger-infra</code> point at a bad commit: <ul> <li>Pods stuck in <code>CrashLoopBackOff</code> or <code>Error</code>. Check with <code>kubectl get pods -n clearledger</code>. </li> <li><code>kubectl logs <pod> -n clearledger --previous</code> shows startup errors that weren't there before. </li> <li>ArgoCD health flips from <code>Healthy</code> to <code>Degraded</code> or stays on <code>Progressing</code>. Check with <code>argocd app get clearledger --grpc-web</code>. </li> <li>The app returns 5xx errors or login stops working. Check with <code>curl -I http://clearledger.local/health</code>. </li> </ul> If this happens right after a push, roll back first. Once the app is stable again, investigate the bad commit. <h4 id="heading-why-argocd-rollback-isnt-just-a-button">Why ArgoCD rollback isn't just a button</h4> ArgoCD has a rollback button in the UI and an <code>argocd app rollback</code> command. Both work. But only if you understand the interaction with <code>selfHeal</code>. Your Application (<code>stages/stage-2-gitops/argocd/clearledger-app.yaml</code>) is configured with: <pre><code class="language-yaml">syncPolicy: automated: selfHeal: true </code></pre> ArgoCD keeps the cluster matched to Git. In this lab, Git means <code>clearledger-infra</code>. If someone changes the cluster by hand, ArgoCD treats that as drift and changes it back to match Git. This also affects rollback. The ArgoCD UI rollback changes the cluster, but it doesn't change Git. If <code>clearledger-infra</code> still points to the bad version, and self-heal can bring the bad version back. The safer GitOps rollback is to change Git with <code>git revert</code> in <code>clearledger-infra</code>. Then ArgoCD syncs the cluster to the reverted, good version. If you need an emergency UI rollback, turn off auto-sync first, roll back in ArgoCD, then fix Git afterward. <h4 id="heading-method-1-git-revert-preferred-always-try-this-first">Method 1: Git revert (preferred, always try this first)</h4> This is the GitOps way. You don't touch the cluster. You change Git, and ArgoCD syncs the fix. When to use: You have a few minutes and can identify the bad commit in <code>clearledger-infra</code>. How it works: <pre><code class="language-plaintext">Bad commit pushed to clearledger-infra ↓ ArgoCD auto-synced it (cluster is now broken) ↓ You run: git revert <bad-commit> && git push ↓ ArgoCD auto-syncs the revert (cluster is fixed, selfHeal works with you) ↓ Git history shows the bad deploy AND the revert, full audit trail </code></pre> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/2371fab2-b05d-4453-ac28-80465a95a88f.png" alt="demo showing how argocd works and the flow" style="display:block;margin:0 auto" width="1024" height="1536" loading="lazy"> Step-by-step: <pre><code class="language-bash"># 1. Go to your clearledger-infra repo (wherever you cloned it) cd ~/clearledger-infra # adjust path if you cloned elsewhere git pull # make sure you are up to date # 2. Find the bad commit git log --oneline -10 # Output looks like: # abc1234 update ledger-service image to v1.4.0 ← this broke prod # def5678 update auth-service image to v1.3.1 ← was fine # 9a1b2c3 add vault rotation cronjob # 3. Revert it — this creates a NEW commit, it does not delete history git revert abc1234 --no-edit # 4. Push — ArgoCD picks it up automatically within ~3 minutes git push # 5. Confirm the cluster recovered kubectl get pods -n clearledger argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status" # Expected: Sync Status: Synced, Health Status: Healthy </code></pre> This method is preferred because it fixes the source of truth: <code>clearledger-infra</code>. After you push the revert, ArgoCD sees the new Git state and syncs the cluster to it. Nothing fights you because Git and the cluster are supposed to match. It also leaves a clear history. Git shows the bad deploy, the revert, who made both changes, and when they happened. That's easier to debug, easier to review, and better for compliance. <h4 id="heading-method-2-emergency-argocd-rollback-when-the-cluster-is-on-fire">Method 2: Emergency ArgoCD rollback (when the cluster is on fire)</h4> Use this if the cluster is broken right now and you don't have time to push a Git fix. It pins the cluster to a previous known good deployment immediately. You'll still fix Git afterward. This isn't a permanent fix. When to use: Incident in progress. Pods are crashing, users are affected, and you need the cluster back to a known good state in under 30 seconds. Before you start: confirm your ArgoCD CLI session is still valid. If it expired, re-login first: an expired session will silently fail every command below. <blockquote> <pre><code class="language-bash">argocd account get-user-info --grpc-web # If you see "Unauthenticated", re-login: ARGOCD_PASSWORD=$(kubectl -n argocd get secret argocd-initial-admin-secret \ -o jsonpath="{.data.password}" | base64 -d) argocd login argocd.local --username admin --password "$ARGOCD_PASSWORD" \ --insecure --grpc-web </code></pre> </blockquote> Step 1: Disable auto-sync (critical). Skip this and selfHeal will undo your rollback within 3 minutes. <pre><code class="language-bash">argocd app set clearledger --sync-policy none --grpc-web # Confirm: automated sync is now off argocd app get clearledger --grpc-web | grep "Sync Policy" # Expected: Sync Policy: <none> </code></pre> Step 2: Find the last known-good deployment ID <pre><code class="language-bash">argocd app history clearledger --grpc-web # Output looks like: # ID DATE REVISION # 9 2026-06-05 10:12:00 +0000 UTC abc1234 ← bad deploy (current) # 8 2026-06-04 14:46:06 +0000 UTC def5678 ← known good # 7 2026-06-01 20:53:19 +0000 UTC 9a1b2c3 # Or check via kubectl (no argocd CLI needed): kubectl get application clearledger -n argocd \ -o jsonpath='{range .status.history[*]}{.id}{"\t"}{.deployedAt}{"\t"}{.revision}{"\n"}{end}' </code></pre> Use the ID (the number on the left), not the SHA. Step 3: Roll back to the good ID <pre><code class="language-bash">argocd app rollback clearledger 8 --grpc-web </code></pre> Step 4: Confirm the cluster is stable <pre><code class="language-bash">kubectl get pods -n clearledger # All pods should be Running argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status" # Sync Status: OutOfSync ← expected — cluster is at rev 8, Git is still at the bad HEAD # Health Status: Healthy ← this is what matters right now </code></pre> <code>OutOfSync</code> is correct and expected at this point. The cluster is running the old good revision. Git still has the bad commit. You'll fix that next. Step 5: Fix Git (don't leave it broken) <pre><code class="language-bash">cd ~/clearledger-infra git pull git revert <bad-commit-sha> --no-edit git push </code></pre> Step 6: Re-enable auto-sync <pre><code class="language-bash">argocd app set clearledger \ --sync-policy automated \ --self-heal \ --auto-prune \ --grpc-web # Trigger an immediate sync so you do not wait for the next auto-check argocd app sync clearledger --grpc-web # Confirm everything is clean argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status" # Expected: Sync Status: Synced, Health Status: Healthy </code></pre> Never leave auto-sync disabled longer than the incident. It's your drift-detection and tamper-evidence mechanism: without it, unauthorized <code>kubectl</code> changes go undetected. Re-enable it the moment you push the Git fix. <h4 id="heading-practise-the-rollback-now-before-you-need-it-under-pressure">Practise the rollback now (before you need it under pressure)</h4> Don't wait for a real incident to run this for the first time. The steps below simulate a bad image tag deploy and walk you through Method 1 (the preferred path). Step 1: Push a bad image tag to <code>clearledger-infra</code> <pre><code class="language-bash">cd ~/clearledger-infra git pull # Edit manifests/notification-service/deployment.yaml # Change the image tag to a tag that does not exist, e.g.: # image: docker.io/$DOCKER_USERNAME/clearledger-notification-service:broken-tag # Commit and push it git add manifests/notification-service/deployment.yaml git commit -m "test: simulate bad deploy with nonexistent image tag" git push </code></pre> Step 2: Watch ArgoCD sync the bad state <pre><code class="language-bash"># Give ArgoCD ~3 minutes to pick it up, or trigger immediately: argocd app sync clearledger --grpc-web # Watch the notification-service pod fail kubectl get pods -n clearledger -w # You will see: notification-service pod stuck in ImagePullBackOff or ErrImagePull </code></pre> Step 3: Roll back using Method 1 <pre><code class="language-bash">cd ~/clearledger-infra # Revert the bad commit git revert HEAD --no-edit git push # ArgoCD will auto-sync — or trigger it: argocd app sync clearledger --grpc-web # Watch pods recover kubectl get pods -n clearledger -w # notification-service should return to Running </code></pre> Step 4: Verify <pre><code class="language-bash">argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status" # Expected: Sync Status: Synced, Health Status: Healthy kubectl get pods -n clearledger # All pods Running, no ImagePullBackOff </code></pre> You have now practised a rollback end-to-end. The <code>git revert</code> commit is permanently in the infra repo's history: a real audit record of a simulated recovery. <h4 id="heading-quick-reference">Quick reference</h4> Use Method 1 (git revert) when: <ul> <li>A bad image tag or manifest was pushed to <code>clearledger-infra</code> and you have a few minutes </li> <li>Any config change in the infra repo caused pods to break </li> <li>This is almost always the right answer. It's fast, safe, and leaves a clean audit trail. </li> </ul> Use Method 2 (emergency ArgoCD rollback) when: <ul> <li>The cluster is broken right now, users are affected, and you need it stable in under 30 seconds </li> <li>You're not yet sure which commit caused the problem and need time to investigate: roll back to stabilise, then use <code>git log</code> to find the culprit, then fix forward with Method 1 </li> </ul> Neither method applies when a pod is crashing but nothing was pushed to the infra repo recently. This isn't a rollback problem. Check <code>kubectl logs</code>, Vault connectivity, and network policies instead. <code>revisionHistoryLimit: 10</code> in <code>stages/stage-2-gitops/argocd/clearledger-app.yaml</code> means ArgoCD always has 10 previous deployments available for emergency rollback. Increase it if your release cadence is high. <h3 id="heading-what-you-learned-in-stage-2">What You Learned in Stage 2</h3> <ul> <li>What GitOps means: Git is the single source of truth, and a tool enforces it </li> <li>What ArgoCD does: watches Git, compares it to the cluster, corrects drift automatically </li> <li>How the full flow works now: push code, CI builds image, CI updates infra repo, and ArgoCD syncs cluster. </li> <li>No one runs <code>kubectl</code> to deploy anymore. The pipeline updates Git, ArgoCD does the rest. </li> <li>How to roll back safely: <code>git revert</code> in the infra repo is the correct answer, while ArgoCD emergency rollback is the break-glass option. You must disable auto-sync first or selfHeal will silently undo it. </li> </ul> What you can now put on your CV / say in an interview: <blockquote> Implemented GitOps with ArgoCD so cluster state is driven from Git, with drift detection, auto-sync, and a Git-based rollback of a bad deploy. </blockquote> <code>make snapshot STAGE=2 && make snapshots</code>. Confirm <code>clearledger.stage2</code>. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>. <h2 id="heading-stage-3-security-gates">Stage 3 — Security Gates</h2> Every push runs security checks. Some failures stop the pipeline right away. Others you learn from now and enforce in the cluster later (Stage 4). Goal: understand six scanners: what each one looks at, what it catches, and how to read a failure. You'll break each gate on purpose (§3.4) so a failed CI job isn't a surprise. Ready for Stage 3? <ul> <li><code>make check-2</code> passes </li> <li><code>ENABLE_ARGOCD_SYNC=true</code> on GitHub (you set this in Stage 2) </li> <li><code>ENABLE_DAST</code> still unset (turn on later in this stage if you want) </li> <li>Argo CD at <code>http://argocd.local</code> shows Synced </li> <li>Optional: skim <a href="#heading-stage-1-security-posture-what-blocks-vs-what-waits">Stage 1 security posture</a>. Stage 1 already ran many of these tools </li> </ul> Done when: <code>make check-3</code> passes and you triggered each gate once (§3.4). Then <code>make snapshot STAGE=3</code> and <code>make snapshots</code>. <h3 id="heading-what-you-need-to-know-first">What You Need to Know First</h3> One tool isn't enough. Each scanner guards a different layer: <ul> <li>Gitleaks: secrets in code or Git history (API keys, tokens) </li> <li>Semgrep (SAST): bugs in your Python/JS source (injection, unsafe patterns) </li> <li>Trivy (SCA + images): finds known security vulnerabilities (called CVEs) in your Python/Node.js packages and Docker images. A CVE (Common Vulnerabilities and Exposures) is a publicly tracked software security flaw with a unique identifier. </li> <li>Checkov (IaC): misconfigurations in Dockerfiles, Kubernetes manifests, and Stage 8 Terraform. </li> <li>Cosign: proves images were built and signed by your pipeline </li> </ul> What blocks CI today: secrets, bad code (SAST), vulnerable images, Dockerfile issues on production images. What waits for later: Some Kubernetes issues are only reported in Stage 1. They show you what still needs hardening. In Stage 4, Kyverno turns the important rules into real cluster enforcement, so unsafe workloads are blocked before they run. When you <code>git commit</code>, hooks on your laptop can scan first (pre-commit). When you <code>git push</code>, GitHub Actions scans again on the runner. Same idea twice: catch mistakes before they waste a 10-minute pipeline. Pre-commit is optional to install, as CI always runs on push either way. <h3 id="heading-enable-dast-optional-after-stage-2">Enable DAST (Optional After Stage 2)</h3> DAST (Dynamic Application Security Testing) scans the running app at <code>http://clearledger.local</code>. It was off in Stages 1–2 on purpose: Stage 1 never deployed to the cluster, and Stage 2 was about getting GitOps healthy first. If <code>make check-2</code> passes and <code>curl http://clearledger.local/auth/health</code> returns <code>200</code>, you can turn DAST on: Go to GitHub and into your <code>clearledger</code> repo. Then go to Settings, Secrets and variables, Actions, Variables, and New repository variable: <table> <thead> <tr> <th>Name</th> <th>Value</th> </tr> </thead> <tbody><tr> <td><code>ENABLE_DAST</code></td> <td><code>true</code></td> </tr> </tbody></table> Push a small commit (or re-run the last workflow on <code>main</code>). The DAST (OWASP ZAP + fintech API tests) job should run instead of skipped. A failed ZAP scan is a real finding to investigate. Skipped before this step only means the toggle was off. <h3 id="heading-31-install-pre-commit-hooks">3.1: Install Pre-commit Hooks</h3> <pre><code class="language-bash"># macOS (Homebrew — avoids PEP 668 "externally-managed-environment" from pip3): brew install pre-commit # Linux/WSL2: # sudo apt install -y pre-commit # or: python3 -m pip install --user pre-commit pre-commit install pre-commit run --all-files </code></pre> If a hook fails, read the error first. Gitleaks and Ruff should pass before you commit. Some YAML or Terraform hook issues may come from later-stage files. If that happens, continue with the stage instructions and use <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md"><code>troubleshooting.md</code></a> for Gitleaks or CI scanner failures. Test it catches secrets locally before CI does: <pre><code class="language-bash">echo 'AWS_SECRET = "'$(printf '%s%s' 'AKIA' 'IOSFODNN7EXAMPLE')'"' >> app/auth-service/main.py git add app/auth-service/main.py && git commit -m "test" # Gitleaks fires and blocks the commit — see "What you should see" below git restore --staged app/auth-service/main.py git checkout app/auth-service/main.py </code></pre> The commit was blocked before it even reached Git. If the pre-commit hook wasn't installed, that fake AWS key would be in your Git history permanently (even if you delete the line later, Git remembers). If you already did Cosign in Stage 1, that counts. Stage 3 doesn't require regenerating keys. Confirm that <code>infra/cosign.pub</code> exists and GitHub has <code>COSIGN_PRIVATE_KEY</code> + <code>COSIGN_PASSWORD</code>. Stage 4 turns signing into enforcement at the cluster gate. ✋ Hands-on checkpoint: pre-commit actually blocks a secret Installed-but-not-wired is the classic silent failure. Prove the hooks fire: <pre><code class="language-bash">echo 'AWS_SECRET='"$(printf '%s%s' 'AKIA' 'IOSFODNN7EXAMPLE')" > leak-test.env git add leak-test.env pre-commit run --all-files; echo "exit=$?" git reset leak-test.env >/dev/null; rm -f leak-test.env </code></pre> Expected: the secret-scanning hook fails the run (<code>exit=1</code>) and flags <code>leak-test.env</code>. If <code>exit=0</code>, your hooks are installed but not catching anything: re-run <code>pre-commit install</code> and confirm <code>.git/hooks/pre-commit</code> exists. If you skip this, commits sail through unscanned and you'll believe Stage 3 is protecting you when it's not. <h3 id="heading-32-generate-cosign-keys">3.2: Generate Cosign Keys</h3> If you created Cosign keys in Stage 1 (§1.4), skip generation: go straight to inserting <code>cosign.pub</code> into the Kyverno policy and adding the GitHub secrets below. Cosign signs your Docker images with a cryptographic key. When you deploy to the cluster, Kyverno (Stage 4) can verify the signature and reject any image that wasn't signed by your pipeline. This prevents someone from pushing a malicious image to your Docker Hub and having the cluster run it. <pre><code class="language-bash"># macOS: brew install cosign # Linux/WSL2: curl -O -L https://github.com/sigstore/cosign/releases/download/v2.2.4/cosign-linux-amd64 && chmod +x cosign-linux-amd64 && sudo mv cosign-linux-amd64 /usr/local/bin/cosign cosign generate-key-pair # enter a password when prompted </code></pre> This creates two files: <code>cosign.key</code> (private, used by the pipeline to sign) and <code>cosign.pub</code> (public, used by Kyverno to verify). Insert your public key into the Kyverno policy (replace the placeholder block in <code>infra/policies/require-signed-images.yaml</code> with the contents of <code>cosign.pub</code>). Add secrets to GitHub (github.com/YOUR_USERNAME/clearledger → Settings → Secrets and variables → Actions): <table> <thead> <tr> <th>Secret</th> <th>Value</th> </tr> </thead> <tbody><tr> <td><code>COSIGN_PRIVATE_KEY</code></td> <td>Contents of <code>cosign.key</code></td> </tr> <tr> <td><code>COSIGN_PASSWORD</code></td> <td>The password you entered when generating keys</td> </tr> </tbody></table> ✋ Hands-on checkpoint: Cosign keys are ready Stage 4 uses <a href="http://cosign.pub"><code>cosign.pub</code></a> to verify signed images. Before you continue, confirm the key files exist and the private key isn't tracked by Git: <pre><code class="language-bash">test -f cosign.key && echo "private key present" test -f cosign.pub && echo "public key present" grep -q "BEGIN PUBLIC KEY" cosign.pub && echo "public key valid" git check-ignore cosign.key && echo "private key correctly ignored" </code></pre> Expected: all four lines should print. If <code>git check-ignore cosign.key</code> prints nothing, add <code>cosign.key</code> to <code>.gitignore</code> before committing anything. The private key must stay out of Git. Don't skip this check. Stage 4 needs the public key for the Kyverno image-signing policy, and the private key must remain local. <h3 id="heading-33-activate-the-full-security-pipeline">3.3: Activate the Full Security Pipeline</h3> The security gates are already in <code>.github/workflows/ci.yaml</code>. Push any change to trigger the full pipeline: <pre><code class="language-bash">git add . && git commit -m "ci: full DevSecOps pipeline" && git push origin main </code></pre> <h3 id="heading-34-break-each-gate-on-purpose">3.4: Break Each Gate on Purpose</h3> For each gate, you'll want to break something on purpose, read how the tool reports it, revert, and confirm green again. Try the local command first, then push once if you want a screenshot on GitHub Actions. <pre><code class="language-bash"># 1. Break it 2. Run locally or push 3. Read the failure # 4. git checkout -- path/to/file 5. pre-commit run --all-files (optional) 6. git push </code></pre> Start with Gate 1 end-to-end before the others. <h4 id="heading-gate-1-gitleaks-secrets">Gate 1: Gitleaks (secrets)</h4> Inject: hardcoded AWS key in any Python file. The goal is to prove the secret scanner works. This command adds a fake AWS-looking key to <code>app/auth-service/main.py</code>: <pre><code class="language-bash">echo 'AWS_KEY = "'$(printf '%s%s' 'AKIA' 'IOSFODNN7EXAMPLE')'"' >> app/auth-service/main.py git add app/auth-service/main.py && git commit -m "test: trigger gitleaks" # pre-commit blocks this commit locally — that is the test. # For a CI screenshot only: git commit --no-verify -m "test: trigger gitleaks" && git push </code></pre> Done looks like this (terminal: pre-commit): <pre><code class="language-text">🔑 Secrets scan (Gitleaks)...............................................Failed - hook id: gitleaks - exit code: 1 Finding: AWS_KEY = "REDACTED" RuleID: aws-access-token File: app/auth-service/main.py Line: 316 </code></pre> Expected: the commit should fail. Gitleaks should report one secret finding in <code>app/auth-service/</code><a href="http://main.py"><code>main.py</code></a>. That failure is good. It means the local pre-commit hook caught the secret before it reached Git. Revert: <pre><code class="language-bash">git restore --staged app/auth-service/main.py 2>/dev/null git checkout app/auth-service/main.py pre-commit run gitleaks --all-files # → Passed </code></pre> <h4 id="heading-gate-2-semgrep-sast">Gate 2: Semgrep (SAST)</h4> Local dry-run (no repo change): <pre><code class="language-bash">python3 -m venv /tmp/sec-gates-venv && /tmp/sec-gates-venv/bin/pip install semgrep cat > /tmp/semgrep-bad.py << 'EOF' import subprocess from fastapi import Request def bad(request: Request): subprocess.run(request.query_params.get("cmd"), shell=True) EOF /tmp/sec-gates-venv/bin/semgrep \ --config=p/python --config=p/security-audit --config=p/owasp-top-ten --error \ /tmp/semgrep-bad.py </code></pre> Break CI: add a temporary file Semgrep will scan, commit, and push: <pre><code class="language-bash">cat > app/auth-service/gate_test_semgrep.py << 'EOF' import subprocess from fastapi import Request def bad(request: Request): subprocess.run(request.query_params.get("cmd"), shell=True) EOF git add app/auth-service/gate_test_semgrep.py && git commit -m "test: trigger semgrep" && git push </code></pre> Expected result: Semgrep reports <code>subprocess-shell-true</code> as <code>Blocking</code>. The <code>SAST (Semgrep)</code> job turns red, and the image build jobs don't run. Revert: <pre><code class="language-bash">rm -f app/auth-service/gate_test_semgrep.py git add -A && git commit -m "revert: semgrep gate test" && git push </code></pre> <h4 id="heading-gate-3-checkov-iac-dockerfile">Gate 3: Checkov (IaC / Dockerfile)</h4> Checkov scans Dockerfiles and Kubernetes manifests for unsafe configuration. First, run a local demo. This removes the <code>HEALTHCHECK</code> from a copied Dockerfile and shows how Checkov reports it: <pre><code class="language-bash">python3 -m venv /tmp/sec-gates-venv && /tmp/sec-gates-venv/bin/pip install checkov sed '/^HEALTHCHECK/,+1d' app/auth-service/Dockerfile > /tmp/Dockerfile-nohc mkdir -p /tmp/checkov-demo/app/auth-service cp /tmp/Dockerfile-nohc /tmp/checkov-demo/app/auth-service/Dockerfile /tmp/sec-gates-venv/bin/checkov --directory /tmp/checkov-demo --framework dockerfile </code></pre> Now trigger a Checkov finding in CI by exposing SSH port <code>22</code> in the auth-service Dockerfile: <pre><code class="language-bash">echo 'EXPOSE 22' >> app/auth-service/Dockerfile git add app/auth-service/Dockerfile && git commit -m "test: trigger checkov" && git push </code></pre> Expected result: the Checkov log or artifact should show <code>CKV_DOCKER_1</code>, which means an SSH port was exposed. The <code>IaC Scan (Checkov)</code> job may or may not turn red, depending on the severity Checkov assigns. That's okay for this exercise. The goal is to find and understand the Checkov result. If you need a screenshot of a failed GitHub Actions job, use Gate 1, Gate 2, or Gate 4. Those are designed to turn the workflow red. Checkov is mainly for reading the finding, so it may stay green. Revert: <pre><code class="language-bash">git checkout app/auth-service/Dockerfile git commit -am "revert: checkov gate test" && git push </code></pre> <h4 id="heading-gate-4-trivy-image-cves">Gate 4: Trivy (image CVEs)</h4> Local dry-run: scan an old base image (no build): <pre><code class="language-bash">trivy image --exit-code 1 --severity CRITICAL,HIGH --ignore-unfixed python:3.8-slim </code></pre> Break CI: pin an old base in the Dockerfile, push, wait for <code>Scan images</code>: <pre><code class="language-bash">sed -i.bak 's/FROM python:3.13-slim/FROM python:3.8-slim/' app/auth-service/Dockerfile git add app/auth-service/Dockerfile && git commit -m "test: trigger trivy" && git push </code></pre> Pass: <code>Scan images</code> → Trivy scan all images exits 1 with a CVE table (<code>HIGH</code> / <code>CRITICAL</code>). <code>Publish images</code> and <code>Update Manifests</code> are skipped. Revert: <pre><code class="language-bash">git checkout app/auth-service/Dockerfile git commit -am "revert: trivy gate test" && git push </code></pre> <h3 id="heading-35-when-a-scan-fails-on-a-cve-you-didnt-inject">3.5: When a Scan Fails on a CVE You Didn't Inject</h3> §3.4 is deliberate. This section is for the other case where you push normal code, but the image scan fails because a new vulnerability was found. That's normal. CVE databases update all the time. Don't weaken the scan. Fix the vulnerable package or image. First, find the real CVE. In GitHub Actions, open Scan images then go to Trivy scan all images and look for the table with: <ul> <li>Package </li> <li>CVE </li> <li>Installed version </li> <li>Fixed version You can also download the artifact: </li> </ul> Ignore this red herring at the bottom of the log: <pre><code class="language-text">Version 0.71.2 of Trivy is now available Error: Process completed with exit code 1. </code></pre> The version notice doesn't fail the job. A fixable HIGH/CRITICAL CVE does. Don't add <code>--skip-version-check</code> to “fix” it. Instead, fix it with: <ul> <li>pip package: bump to the Fixed Version in <code>requirements.txt</code> (example: <code>python-multipart==0.0.30</code> for CVE-2026-53539). Apply the same bump to sibling services if they share that pin. </li> <li>OS package: newer base image or a targeted <code>apt</code>/<code>apk</code> upgrade in the Dockerfile. </li> <li>No stable fix yet documented exception only: add the CVE to <code>.trivyignore</code> and <code>.grype.yaml</code> with a comment (see <code>CVE-2026-7210</code>). </li> </ul> Don't remove <code>--exit-code 1</code>, lower the severity rule, or disable scanning. For help, see <a href="troubleshooting.md#trivy-version-x-is-now-available-notice-not-a-scan-failure">Trivy version notice</a> and <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">Trivy blocks Python service images</a>. <h3 id="heading-finish-stage-3">Finish Stage 3</h3> For screenshots, use one clear failed gate: <ul> <li>Gitleaks: <code>Secrets Scan</code> </li> <li>Semgrep: <code>SAST</code> </li> <li>Trivy: <code>Scan images</code> </li> <li>Checkov: look for <code>CKV_*</code> in the log or artifact. The job may stay green </li> </ul> After each test in §3.4, undo the test change, push the revert, and confirm the workflow is green again. One red GitHub Actions screenshot is enough for your portfolio. Run the stage check: <pre><code class="language-bash">make check-3 # must end: All checks passed. Ready for the next stage. </code></pre> Expected: <code>All checks passed. Ready for the next stage.</code> You should also have triggered at least one gate in §3.4. A local Gitleaks failure counts. <code>ENABLE_DAST=true</code> is optional. You only need it if you want to run ZAP later. Not required yet: Checkov blocking Kubernetes manifests or Cosign blocking deployments. Stage 4 turns those into cluster enforcement with Kyverno. Next, save your progress: <pre><code class="language-bash">make snapshot STAGE=3 && make snapshots </code></pre> <h2 id="heading-stage-4-admission-control-kyverno">Stage 4 — Admission Control (Kyverno)</h2> Even if CI passes, the cluster can still refuse. CI scans your code and images before they reach GitOps, but it can't watch everything that happens inside the cluster. Someone with <code>kubectl</code> access could apply a manifest directly. A Helm chart you install might create pods that violate your security standards. Those paths never hit the pipeline, which is why Stage 4 adds admission control: a checkpoint built into Kubernetes itself. Every time something tries to create or update a resource, the request passes through admission webhooks before it takes effect. If a webhook rejects the request, the resource is never created. Kyverno is a Kubernetes-native policy engine that uses those webhooks. You write policies as YAML files (not application code), and Kyverno enforces them on every matching resource in the cluster, for example, rejecting any pod that runs as root or requiring CPU and memory limits on every container. The difference from CI is timing: CI scans before code ships, while Kyverno enforces at the cluster gate. Together they give you two layers of defense. Your goal in this stage is to install Kyverno, apply the policies in <code>infra/policies/</code>, and prove in §4.4 that non-compliant pods are denied before the container runtime ever sees them. Before you start, make sure the foundation from earlier stages is still solid: <code>make check-3</code> should pass (pre-commit hooks and CI security gates are active), <code>infra/cosign.pub</code> should exist from Stage 3, and ArgoCD should still be syncing so the app responds at <code>http://clearledger.local</code>. If any of those are red, fix them first: Kyverno sits on top of a healthy cluster, not a broken one. You're done with Stage 4 when all three break-it scenarios in §4.4 are denied and <code>make check-4</code> passes. What changes from Stage 3 is enforcement, not scanning. In CI, Checkov reported Kubernetes misconfigurations but didn't block the pipeline. Kyverno now stops those same classes of problems at the cluster gate. Cosign has been signing your images since Stage 1. Kyverno now requires that signature before a ClearLedger image can deploy. This is where <a href="#heading-stage-1-security-posture-what-blocks-vs-what-waits">Stage 1 evidence becomes enforcement</a>. See that section if you want the full map of what blocked in Stage 1 versus what waited for Stage 4. Start with §4.1 to install Kyverno. If the install, policies, break-it scenarios, or <code>make check-4</code> fail, read <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md"><code>troubleshooting.md</code></a> and Stage 4: Admission Control (Kyverno) before changing Helm values or policy YAML. <h3 id="heading-what-kyverno-enforces">What Kyverno Enforces</h3> All policy files live in <code>infra/policies/</code>. Kyverno itself is installed via Helm using <code>stages/stage-4-admission-control/infra/kyverno/values.yaml</code>. <table> <thead> <tr> <th>Policy</th> <th>What it enforces</th> <th>Framework</th> </tr> </thead> <tbody><tr> <td><code>disallow-root-containers</code></td> <td><code>runAsNonRoot: true</code></td> <td>CIS K8s 5.2.6</td> </tr> <tr> <td><code>require-resource-limits</code></td> <td>CPU/memory requests and limits</td> <td>CIS K8s 5.2.4</td> </tr> <tr> <td><code>disallow-privilege-escalation</code></td> <td><code>allowPrivilegeEscalation: false</code></td> <td>CIS K8s 5.2.5</td> </tr> <tr> <td><code>drop-all-capabilities</code></td> <td><code>capabilities.drop: [ALL]</code></td> <td>CIS K8s 5.2.7</td> </tr> <tr> <td><code>require-signed-images</code></td> <td>Cosign signature on ClearLedger images</td> <td>SLSA Level 2</td> </tr> </tbody></table> <h3 id="heading-platform-stability-from-stage-4-onward">Platform Stability: From Stage 4 Onward</h3> From Stage 4 on, you're running more controllers on a single-node VM. Kyverno, storage provisioners, and later Prometheus and Loki. A pod can show <code>Running</code> while it's actually crash-looping in the background. When platform pods (Kyverno controllers, <code>hostpath-provisioner</code>, the Prometheus operator, and similar) accumulate high <code>RESTARTS</code>, the API server starts timing out, <code>kubectl</code> feels flaky, and you can waste days debugging the wrong component because the app pods look fine. After every stage from here on, give the cluster about ten minutes to settle, then run the stage health check: <pre><code class="language-bash">bash scripts/health-check.sh <stage> # for example, 4, 7, 7.5 # or the Makefile shortcut: make check-4 </code></pre> The script ends with a Platform stability section that flags pods with suspicious restart counts. You can also scan the worst offenders yourself. This lists the fifteen pods with the highest restart counts cluster-wide, which is useful when something feels slow but you're not sure which namespace is struggling: <pre><code class="language-bash">kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount' \ -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount' \ | tail -15 </code></pre> The gate: Kyverno controllers and other platform pods should show RESTARTS under 5 after the stage settles. If any platform pod is climbing past 10, stop and fix it with the documented Helm values or <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md</a>. Don't <code>kubectl patch</code> around it and move on. A stable platform layer is a prerequisite for every stage that follows. <h3 id="heading-41-install-kyverno">4.1: Install Kyverno</h3> <pre><code class="language-bash">helm repo add kyverno https://kyverno.github.io/kyverno/ helm repo update helm upgrade --install kyverno kyverno/kyverno \ --version 3.2.8 \ --namespace kyverno \ --create-namespace \ -f stages/stage-4-admission-control/infra/kyverno/values.yaml \ --wait --timeout=600s </code></pre> The values file does three important things for the lab: <ol> <li>Disables cleanup CronJobs: older Kyverno charts pull <code>bitnami/kubectl</code>, which was removed from Docker Hub and causes <code>ImagePullBackOff</code> on cleanup pods. </li> <li>Points Helm hooks at <code>bitnamilegacy/kubectl</code>, so future <code>helm uninstall</code> doesn't hang on a missing image. </li> <li>Extends liveness probe timeouts: the default <code>timeoutSeconds: 5, failureThreshold: 2</code> is too tight for a loaded single-node VM. Under CPU pressure, the health endpoint can take >5s to respond, which triggers a restart cascade that saturates the node and makes the API server intermittently unreachable. The values file sets <code>timeoutSeconds: 30, failureThreshold: 5</code> so Kyverno survives load spikes without crash-looping. </li> </ol> What you should see: <pre><code class="language-yaml">Release "kyverno" does not exist. Installing it now. NAME: kyverno NAMESPACE: kyverno STATUS: deployed ... Kyverno version: v1.12.6 </code></pre> Verify all four controllers are running (first pull can take several minutes on a slow connection): <pre><code class="language-markdown">kubectl get pods -n kyverno </code></pre> <pre><code class="language-plaintext">NAME READY STATUS RESTARTS AGE kyverno-admission-controller-bd685cd4b-f6kl6 1/1 Running 0 2m kyverno-background-controller-66fcfc6d87-59wgt 1/1 Running 0 2m kyverno-cleanup-controller-5c5bf8bc6b-7kspq 1/1 Running 0 2m kyverno-reports-controller-5cdd6f4c48-qf5wc 1/1 Running 0 2m </code></pre> If pods stay in <code>ContainerCreating</code> for a long time, the node is still pulling images from <code>ghcr.io/kyverno</code>. Wait. Don't start a second Helm install on top of a partial one. <h4 id="heading-stability-gate-kyverno-install-only-before-42">Stability gate: Kyverno install only (before §4.2):</h4> Before continuing, make sure the Kyverno pods are healthy: <pre><code class="language-bash">kubectl get pods -n kyverno </code></pre> Expected: the Kyverno controller pods show <code>1/1 Running</code>, with low restart counts such as <code>0</code>, <code>1</code>, or <code>2</code>, and the restart count isn't increasing. Don't run <code>make check-4</code> yet. That check also looks for the policies you apply later in §4.3, so it may fail at this point even if Kyverno installed correctly. <h3 id="heading-42-confirm-your-cosign-public-key-is-in-the-policy">4.2: Confirm your Cosign Public Key is in the Policy</h3> Stage 3 created <code>infra/cosign.pub</code>. Kyverno uses that same key to verify image signatures when a pod is created. The policy file ships with a placeholder. You must replace it with your key before applying policies in §4.3. <h4 id="heading-step-1-show-your-key-run-from-the-repo-root-on-the-vm">Step 1: Show your key (run from the repo root on the VM)</h4> <pre><code class="language-bash">cd ~/clearledger # or wherever you cloned the repo cat infra/cosign.pub </code></pre> You should see three lines: <code>-----BEGIN PUBLIC KEY-----</code>, a long base64 line, and <code>-----END PUBLIC KEY-----</code>. Copy that whole block (you'll paste it in the next step). <h4 id="heading-step-2-paste-the-key-into-the-policy">Step 2: Paste the key into the policy</h4> Open <code>infra/policies/require-signed-images.yaml</code> in your editor (<code>nano</code>, <code>vim</code>, or VS Code). Find this line: <pre><code class="language-yaml"> PASTE_YOUR_COSIGN_PUBLIC_KEY_HERE </code></pre> Delete only that placeholder line and paste the three lines from <code>cosign.pub</code> in its place. The result should look like this (your base64 line will differ): <pre><code class="language-yaml"> - keys: publicKeys: |- -----BEGIN PUBLIC KEY----- JFkwEwYHKoZIzj0CAQYIKoFIzj0DAQcDQgZEI... -----END PUBLIC KEY----- </code></pre> Save the file. Keep the pasted key indented under <code>publicKeys: |-</code>. The <code>BEGIN PUBLIC KEY</code> and <code>END PUBLIC KEY</code> lines should have spaces before them, just like the base64 line between them. <h4 id="heading-step-3-verify-three-quick-checks">Step 3: Verify (three quick checks)</h4> Run these one at a time from the repo root: <pre><code class="language-bash"># Check A — placeholder must be gone grep PASTE_YOUR_COSIGN_PUBLIC_KEY_HERE infra/policies/require-signed-images.yaml \ && echo "❌ FAIL: placeholder still in file — edit and save again" \ || echo "✓ OK: placeholder removed" </code></pre> <pre><code class="language-bash"># Check B — key block must be present exactly once grep -c "BEGIN PUBLIC KEY" infra/policies/require-signed-images.yaml </code></pre> Expected output for Check B: <code>1</code> (if you see <code>0</code>, the key was not pasted. If <code>2</code>, you pasted it twice). <pre><code class="language-bash"># Check C — policy key must match cosign.pub byte-for-byte diff infra/cosign.pub \ <(sed -n '/-----BEGIN PUBLIC KEY-----/,/-----END PUBLIC KEY-----/p' \ infra/policies/require-signed-images.yaml | sed 's/^[[:space:]]*//') </code></pre> Expected output for Check C: nothing. No diff lines means the keys match. If <code>diff</code> prints differences, open the policy file and fix the paste. If all three passed, continue to §4.3. If you skip this, Scenario 3 in §4.4 fails in a confusing way: unsigned images may slip through, or signed pods may be rejected because Kyverno is checking against the wrong key. <h3 id="heading-43-apply-the-five-core-policies">4.3: Apply the Five Core Policies</h3> Now you'll apply the five policies that map to CIS controls. Don't apply <code>verify-slsa-provenance.yaml</code> yet. It's an optional SLSA attestation policy (Audit mode) for a later enhancement. Stage 4 applies <code>infra/policies/require-signed-images.yaml</code>. This policy uses <code>failurePolicy: Fail</code>, so if Kyverno can't verify an image signature, the pod is blocked instead of allowed. The ECR policy with <code>failurePolicy: Ignore</code> is for Stage 8, not this step. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/67a638f5-65b8-41d2-be68-babe6c7b8c99.png" alt="screenshot image showing infra policy Yaml file failurePolicy as "Fail"" style="display:block;margin:0 auto" width="721" height="193" loading="lazy"> <pre><code class="language-bash">kubectl apply \ -f infra/policies/disallow-root.yaml \ -f infra/policies/disallow-privilege-escalation.yaml \ -f infra/policies/drop-all-capabilities.yaml \ -f infra/policies/require-resource-limits.yaml \ -f infra/policies/require-signed-images.yaml </code></pre> Wait a few seconds, then confirm all policies show <code>READY: True</code> and <code>VALIDATE ACTION: Enforce</code>: <pre><code class="language-bash">kubectl get clusterpolicy </code></pre> <pre><code class="language-plaintext">NAME ADMISSION BACKGROUND VALIDATE ACTION READY AGE disallow-privilege-escalation true true Enforce True 10s disallow-root-containers true true Enforce True 10s drop-all-capabilities true true Enforce True 10s require-resource-limits true true Enforce True 10s require-signed-images true false Enforce True 10s </code></pre> If <code>READY</code> stays empty, check Kyverno logs: <code>kubectl logs -n kyverno -l app.kubernetes.io/component=admission-controller --tail=50</code>. <h3 id="heading-44-breaking-it-on-purpose">4.4: Breaking it on Purpose</h3> Now you'll test the policies by trying to create bad pods. These pods are supposed to fail. That's the point. CI tools like Checkov warn you in a report. Kyverno goes further: it blocks unsafe pods before Kubernetes runs them. For each test, read the error message, as it should tell you which policy blocked the pod and what field was wrong. That error message is your proof that admission control is working. <table> <thead> <tr> <th>Scenario</th> <th>What you simulate</th> <th>Policy under test</th> <th>Success looks like</th> </tr> </thead> <tbody><tr> <td>1</td> <td>Attacker applies a bare pod (no hardening)</td> <td>Root, caps, privilege, limits</td> <td>Four policies fire, pod <code>NotFound</code></td> </tr> <tr> <td>2</td> <td>Developer fixes securityContext but forgets limits</td> <td>Resource limits only</td> <td>One policy fires, pod <code>NotFound</code></td> </tr> <tr> <td>3</td> <td>Attacker pushes unsigned image to Docker Hub</td> <td>Cosign signature</td> <td><code>require-signed-images</code> denies, pod <code>NotFound</code></td> </tr> </tbody></table> <h4 id="heading-scenario-1-root-container-no-securitycontext">Scenario 1: root container (no securityContext)</h4> What you're simulating: Someone with <code>kubectl</code> access bypasses CI and applies a minimal pod: no <code>securityContext</code>, no resource limits. This is exactly what Stage 1 Checkov flagged as evidence. Stage 4 now blocks it. What's wrong with this manifest: The container has only a name and image. It will run as root by default, keep all Linux capabilities, and has no CPU/memory bounds. <pre><code class="language-bash">cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: root-test namespace: clearledger spec: containers: - name: test image: nginx:alpine EOF </code></pre> What you should see: <pre><code class="language-yaml">Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request: resource Pod/clearledger/root-test was blocked due to the following policies disallow-privilege-escalation: check-allowPrivilegeEscalation: 'validation error: allowPrivilegeEscalation must be set to false. rule check-allowPrivilegeEscalation failed at path /spec/containers/0/securityContext/' disallow-root-containers: check-runAsNonRoot: |- validation error: Root containers are blocked in the clearledger namespace. Set securityContext.runAsNonRoot: true on the pod or container. . rule check-runAsNonRoot failed at path /spec/containers/0/securityContext/ drop-all-capabilities: check-capabilities: 'validation error: All containers must drop ALL capabilities. rule check-capabilities failed at path /spec/containers/0/securityContext/' require-resource-limits: check-resources: 'validation error: Resource requests and limits are required for all containers. rule check-resources failed at path /spec/containers/0/resources/limits/' </code></pre> How to read this output: The important line is: <pre><code class="language-text">resource Pod/clearledger/root-test was blocked due to the following policies </code></pre> That means Kyverno stopped the pod before it was created. Under that line, Kyverno lists every policy the pod failed. For example: <pre><code class="language-text">disallow-root-containers: check-runAsNonRoot: </code></pre> This means the pod failed the <code>disallow-root-containers</code> policy, specifically the <code>check-runAsNonRoot</code> rule. The fix is also shown in the message: <pre><code class="language-text">Set securityContext.runAsNonRoot: true </code></pre> The same pattern applies to the other policies: <ul> <li><code>disallow-privilege-escalation</code> means the pod didn't set <code>allowPrivilegeEscalation: false</code> </li> <li><code>drop-all-capabilities</code> means the pod didn't drop Linux capabilities with <code>capabilities.drop: [ALL]</code> </li> <li><code>require-resource-limits</code> means the pod didn't set CPU and memory requests/limits </li> </ul> The <code>path</code> part tells you where Kubernetes expected the missing setting. For example, <code>/spec/containers/0/securityContext/</code> means: look inside the pod spec, then the first container, then its <code>securityContext</code>. And <code>/spec/containers/0/resources/limits/</code> means: look inside the first container's resource limits. So this one bad pod failed four controls at once. That's the lesson: Kyverno doesn't just say "no." It tells you which policy failed and where to fix the YAML. Verify enforcement worked: <pre><code class="language-bash">kubectl get pod root-test -n clearledger # Error from server (NotFound): pods "root-test" not found </code></pre> If you see a pod in <code>Running</code> or <code>Pending</code>, policies aren't enforcing: re-check that <code>kubectl get clusterpolicy</code> shows all five <code>READY: True</code>. Take a screenshot. This is portfolio evidence for CIS Kubernetes Benchmark 5.2.6: enforced, not just configured. <h4 id="heading-scenario-2-missing-resource-limits">Scenario 2: missing resource limits</h4> What you're simulating: A developer who read the securityContext requirements and fixed root/caps/privilege. But skipped resource limits. This is common in real teams: “we hardened the container” but forgot CPU/memory bounds. What's wrong with this manifest: <code>securityContext</code> is correct, but there's no <code>resources.requests</code> or <code>resources.limits</code>. A container without limits can starve other workloads on the node. <pre><code class="language-bash">cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: nolimits-test namespace: clearledger spec: containers: - name: test image: nginx:alpine securityContext: runAsNonRoot: true runAsUser: 1000 allowPrivilegeEscalation: false capabilities: drop: [ALL] EOF </code></pre> What you should see: <pre><code class="language-plaintext">Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request: resource Pod/clearledger/nolimits-test was blocked due to the following policies require-resource-limits: check-resources: 'validation error: Resource requests and limits are required for all containers. rule check-resources failed at path /spec/containers/0/resources/limits/' </code></pre> Key observation: Only one policy fires this time: the securityContext fields satisfied the other four rules. Kyverno evaluates rules independently. Each container property is a separate gate. Verify: <pre><code class="language-bash">kubectl get pod nolimits-test -n clearledger # Error from server (NotFound): pods "nolimits-test" not found </code></pre> <h4 id="heading-scenario-3-unsigned-clearledger-image">Scenario 3: unsigned ClearLedger image</h4> What you're simulating: A supply-chain attack: someone pushes a malicious image to Docker Hub under your repo name (<code>clearledger-auth-service</code>) without going through your signed CI pipeline. Stage 3 made Cosign signing possible, while Stage 4 makes it mandatory at the cluster gate. Why this setup is needed: Kyverno checks image signatures against the image in Docker Hub, not against images on your laptop. The test image tag must exist in Docker Hub first. If you use a fake tag like <code>:unsigned</code> that was never pushed, Kubernetes may fail later with <code>ImagePullBackOff</code>. That only means the image can't be pulled; it doesn't prove Kyverno blocked an unsigned image. <h4 id="heading-step-1-push-a-deliberately-unsigned-test-image-one-time">Step 1: push a deliberately unsigned test image (one-time):</h4> <pre><code class="language-bash">export DOCKER_USERNAME=your-dockerhub-username docker pull nginx:alpine docker tag nginx:alpine ${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test docker push ${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test # Must fail — proves the image has no Cosign signature from your pipeline key: cosign verify --key infra/cosign.pub \ index.docker.io/${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test # Error: no signatures found </code></pre> <h4 id="heading-step-2-try-to-deploy-it-with-a-compliant-pod-spec">Step 2: try to deploy it with a compliant pod spec:</h4> The pod manifest is fully hardened (securityContext + limits) so only the signature policy can fail. Use <code>index.docker.io/</code> in the image URL: on Kyverno 1.12, <code>docker.io/...</code> may not trigger <code>verifyImages</code> matching. <pre><code class="language-bash">cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: unsigned-test namespace: clearledger spec: containers: - name: test image: index.docker.io/${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test securityContext: runAsNonRoot: true runAsUser: 1000 allowPrivilegeEscalation: false capabilities: drop: [ALL] resources: requests: memory: "64Mi" cpu: "50m" limits: memory: "128Mi" cpu: "200m" EOF </code></pre> What you should see: <pre><code class="language-plaintext">Error from server: error when creating "STDIN": admission webhook "mutate.kyverno.svc-fail" denied the request: resource Pod/clearledger/unsigned-test was blocked due to the following policies require-signed-images: verify-cosign-signature: 'failed to verify image index.docker.io/veeno-demo/clearledger-auth-service:unsigned-test: .attestors[0].entries[0].keys: no signatures found' </code></pre> How to read this output: <ul> <li>Note the webhook name is <code>mutate.kyverno.svc-fail</code>, not <code>validate</code>: image verification runs in Kyverno’s mutate pass (digest + signature check) before the pod is admitted. </li> <li><code>no signatures found</code> means Kyverno reached Docker Hub, found the image, and confirmed it was not signed with your <code>infra/cosign.pub</code> key. </li> <li>The pod never exists: the attacker can't get a shell even if the image is pullable. </li> </ul> Verify: <pre><code class="language-bash">kubectl get pod unsigned-test -n clearledger # Error from server (NotFound): pods "unsigned-test" not found </code></pre> What you should NOT see (these mean the test didn't prove signature enforcement): <table> <thead> <tr> <th>Symptom</th> <th>What went wrong</th> </tr> </thead> <tbody><tr> <td>Pod created, then <code>ImagePullBackOff</code></td> <td>Tag does not exist on Docker Hub, complete Step 1 first</td> </tr> <tr> <td>Pod created and <code>Running</code></td> <td>Image used <code>docker.io/...</code> instead of <code>index.docker.io/...</code></td> </tr> <tr> <td>No <code>require-signed-images</code> in the error</td> <td>Policy not applied, or <code>cosign.pub</code> not embedded in the policy YAML</td> </tr> </tbody></table> <h4 id="heading-contrast-signed-image-is-allowed">Contrast: signed image is allowed:</h4> The previous test used an unsigned image, so Kyverno blocked it. Your real ClearLedger images should be signed by the CI pipeline. If the pod also follows the security rules, Kyverno allows it to run. You can check the image currently used by <code>auth-service</code>: <pre><code class="language-bash"># Your deployed tag (signed in CI) should start if spec is compliant: kubectl get deployment auth-service -n clearledger \ -o jsonpath='{.spec.template.spec.containers[0].image}' # docker.io/veeno-demo/clearledger-auth-service:v0.1.0 </code></pre> Example output: <code>docker.io/veeno-demo/clearledger-auth-service:v0.1.0</code> Pods that were already running before the policies were applied will keep running. The important test is what happens when Kubernetes creates a new pod. New pods using signed ClearLedger images should pass Kyverno verification. Take a screenshot of the Scenario 3 denial. It proves the cluster blocks unsigned images, not just that CI signs images. <h3 id="heading-45-verify-clearledger-still-works">4.5: Verify ClearLedger Still Works</h3> Kyverno enforces on new pod creation. Existing deployments that already passed admission (or were synced before policies existed) keep running. Confirm your app pods are healthy: <pre><code class="language-bash">kubectl get pods -n clearledger </code></pre> <pre><code class="language-plaintext">NAME READY STATUS RESTARTS AGE auth-service-... 1/1 Running 0 ... frontend-... 1/1 Running 0 ... ledger-service-... 1/1 Running 0 ... notification-service-... 1/1 Running 0 ... postgres-0 1/1 Running 0 ... redis-... 1/1 Running 0 ... </code></pre> If ingress is configured: <pre><code class="language-bash">curl -s http://clearledger.local/auth/health | jq . # {"status": "ok", "service": "auth-service"} </code></pre> ArgoCD should still show Synced and Healthy: GitOps and admission control work together, not against each other. <h3 id="heading-46-policy-exceptions-when-a-legitimate-workload-needs-a-bypass">4.6: Policy Exceptions (When a Legitimate Workload Needs a Bypass)</h3> Kyverno blocks every pod that violates a policy. But what happens when a legitimate workload needs to bypass a specific rule? PostgreSQL is the example. The official Postgres Alpine image uses a specific internal user (UID 70) to manage its data directory. The <code>disallow-root-containers</code> policy requires every pod to set <code>runAsNonRoot: true</code>. Postgres does set that. But if Kyverno is configured to also check specific UID ranges, or if the pod's security context doesn't satisfy the rule for any reason, Kyverno blocks it. The database can't start, and the entire application fails. You can't weaken the policy cluster-wide to accommodate one database. That would let every pod bypass the rule. Instead, you create a PolicyException: a targeted exemption for exactly the pods that need it. Open <a href="../infra/policies/exceptions/postgres-root-exception.yaml"><code>infra/policies/exceptions/postgres-root-exception.yaml</code></a> and read the comments. Here's what each section does: The <code>spec.exceptions</code> block identifies which policy and rule to bypass: <pre><code class="language-yaml">exceptions: - policyName: disallow-root-containers ruleNames: - check-runAsNonRoot </code></pre> This says: "skip only the <code>check-runAsNonRoot</code> rule from the <code>disallow-root-containers</code> policy." Every other rule in that policy (and every other policy in the cluster) still enforces normally. The <code>spec.match</code> block limits which resources get the exception: <pre><code class="language-yaml">match: any: - resources: kinds: - Pod namespaces: - clearledger names: - postgres-* </code></pre> Only pods named <code>postgres-*</code> (matching <code>postgres-0</code>, <code>postgres-1</code>, and so on), only in the <code>clearledger</code> namespace, only for the <code>Pod</code> resource kind. Everything else in the cluster still follows the strict policy. The annotations are documentation for your team and auditors: <pre><code class="language-yaml">annotations: reason: "Postgres alpine image requires UID 70 for data directory ownership" approved-by: "platform-team" review-date: "2026-01-01" </code></pre> These have no technical effect: Kyverno ignores them. They exist so that six months from now, when someone asks "why does Postgres bypass this rule?", the answer is right there in the file. The rules for safe exceptions: <ol> <li>Scope narrowly: target the exact resource that needs it, nothing more </li> <li>Commit to Git: the exception is reviewed in a pull request, tracked in version history, and auditable </li> <li>Never weaken the policy itself: the rule stays strict for everything else </li> <li>Review periodically: exceptions should be temporary if possible, and re-evaluated on a schedule </li> </ol> Apply the exception only if Kyverno blocks your Postgres pods: <pre><code class="language-bash">kubectl apply -f infra/policies/exceptions/postgres-root-exception.yaml </code></pre> Verify Kyverno still blocks other non-compliant pods (same denial as Scenario 1): <pre><code class="language-bash">cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: another-root-test namespace: clearledger spec: containers: - name: test image: nginx:alpine EOF </code></pre> <h3 id="heading-47-cis-benchmark-evidence-kube-bench">4.7: CIS Benchmark Evidence <code>kube-bench</code>)</h3> You already installed Kyverno and proved it blocks unsafe pods. This step is different. <code>kube-bench</code> doesn't block pods and doesn't change the cluster. It only checks the Kubernetes node against the CIS benchmark and saves evidence. Think of the difference like this: <table> <thead> <tr> <th>Tool</th> <th>What it checks</th> <th>Question it answers</th> </tr> </thead> <tbody><tr> <td>Kyverno</td> <td>Pods and workloads</td> <td>"Is this pod allowed to run?"</td> </tr> <tr> <td>kube-bench</td> <td>Kubernetes node settings</td> <td>"Is this Kubernetes node hardened?"</td> </tr> </tbody></table> Both are useful, but only Kyverno blocks workloads in this lab. Run kube-bench: <pre><code class="language-bash">bash stages/stage-4-admission-control/scripts/run-kube-bench.sh </code></pre> The script runs kube-bench as a Kubernetes Job and saves the report here: <pre><code class="language-text">stages/stage-4-admission-control/scripts/kube-bench-report.json </code></pre> It also compares the result against this baseline: <pre><code class="language-text">stages/stage-4-admission-control/scripts/kube-bench-baseline.json </code></pre> On MicroK8s, you'll see many <code>FAIL</code> and <code>WARN</code> lines. That's expected. The lab isn't asking you to fix every CIS warning on a single-node local VM. What matters is the final result. Pass looks like this: <pre><code class="language-text">kube-bench: 1 FAIL control(s) present (documented in baseline — no regressions). kube-bench: no regressions vs baseline. </code></pre> That means the known MicroK8s issues are documented, and your cluster didn't get worse. If you see <code>REGRESSION</code> or <code>make check-4</code> fails on kube-bench, stop and investigate before Stage 5. Optional: confirm the report file exists: <pre><code class="language-bash">ls -la stages/stage-4-admission-control/scripts/kube-bench-report.json </code></pre> In production, you would either fix the CIS failures or document approved exceptions. In this lab, the baseline records the expected MicroK8s state. <h3 id="heading-48-health-check">4.8: Health Check</h3> <pre><code class="language-bash">make check-4 </code></pre> What you should see: <pre><code class="language-plaintext">▶ Stage 4 — Admission Control (Kyverno) ✓ Kyverno is running ✓ Policy disallow-root-containers — Enforce mode ✓ Policy require-resource-limits — Enforce mode ✓ Policy require-signed-images — Enforce mode ✓ Policy disallow-privilege-escalation — Enforce mode ✓ Policy drop-all-capabilities — Enforce mode ✓ Kyverno correctly rejects pods without securityContext ✓ kube-bench baseline exists (...) All checks passed. Ready for the next stage. </code></pre> If kube-bench reports regressions, run the script manually and update the baseline after reviewing. That diff is audit evidence. If Kyverno install, policies, break-it scenarios, or <code>make check-4</code> fail, see <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md. Stage 4</a>. <h3 id="heading-stage-4-complete-done-checklist-move-to-stage-5">Stage 4 Complete: Done Checklist (Move to Stage 5)</h3> You're done with Stage 4 when all of these are true: <table> <thead> <tr> <th>#</th> <th>Check</th> <th>How to verify</th> </tr> </thead> <tbody><tr> <td>1</td> <td>Kyverno running</td> <td><code>kubectl get pods -n kyverno</code> — four controllers <code>Running</code></td> </tr> <tr> <td>2</td> <td>Policies applied</td> <td><code>kubectl get clusterpolicy</code> — five policies, <code>READY: True</code>, <code>Enforce</code></td> </tr> <tr> <td>3</td> <td>Root pod blocked</td> <td>Scenario 1 denial in terminal (screenshot for portfolio)</td> </tr> <tr> <td>4</td> <td>Unsigned image blocked</td> <td>Scenario 3 denial — push <code>unsigned-test</code> tag first, use <code>index.docker.io/</code></td> </tr> <tr> <td>5</td> <td>App still healthy</td> <td><code>kubectl get pods -n clearledger</code> — all app pods <code>Running</code></td> </tr> <tr> <td>6</td> <td>Health check green</td> <td><code>make check-4</code> ends with <code>All checks passed. Ready for the next stage.</code></td> </tr> </tbody></table> Portfolio screenshots (optional): root-pod denial (§4.4 Scenario 1), unsigned-image denial (§4.4 Scenario 3), and <code>kubectl get clusterpolicy</code> showing five <code>Enforce</code> policies. Not yet: SLSA attestation (optional), Vault secrets (Stage 5), network policies (Stage 6). Passwords still live in Kubernetes Secrets. Stage 5 moves them into Vault. <h3 id="heading-what-you-learned-in-stage-4">What You Learned in Stage 4</h3> <ul> <li>The difference between CI scanning (before merge) and admission control (at the cluster gate) </li> <li>What Kyverno is: a policy engine that intercepts every Kubernetes API request </li> <li>That enforcement means the bad resource never exists, not "we detected it after the fact" </li> <li>How to read a Kyverno denial: policy name → rule name → JSON path that failed </li> <li>How to write and apply cluster-wide security policies as YAML </li> <li>How to scope a PolicyException without weakening the policy for everyone else </li> <li>That operational issues (Helm, image pulls, registry URL format) affect whether controls actually fire </li> <li>Why both CI and admission control are needed: CI catches problems in your code while Kyverno catches everything else that touches the cluster </li> <li>Evidence beats configuration: a policy file in Git means nothing: the break-it denials are proof CIS controls are enforced, not just documented. </li> </ul> What you can now put on your CV / say in an interview: <blockquote> Enforced admission control with Kyverno: blocking root containers, privilege escalation, unsigned images, and missing resource limits at deploy time: mapped to CIS Kubernetes benchmarks. </blockquote> <code>make snapshot STAGE=4 && make snapshots</code>. Confirm <code>clearledger.stage4</code>. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>. <h2 id="heading-stage-5-secrets-management-vault">Stage 5: Secrets Management (Vault)</h2> By the end of this stage, sensitive values no longer live in Git or in etcd-backed Kubernetes Secrets: Vault holds them centrally and injects them into pods only when they start. Your goal: remove <code>auth-service-secret</code> and <code>ledger-service-secret</code> from the cluster. Login and API calls must still work because Vault injects credentials at pod startup. That's the moment secrets management clicks. Before you start, confirm Stage 4 is solid: <code>make check-4</code> passes, all five Kyverno policies are enforcing, and the app responds at <code>http://clearledger.local</code>. Fix any crash-looping pods before installing Vault. <h3 id="heading-what-changes-in-this-stage">What Changes in This Stage</h3> Right now, database passwords and JWT keys sit in <code>secret.yaml</code> files on GitHub and in Kubernetes Secrets inside the cluster. In Stage 5 you move those values into HashiCorp Vault and teach the app to read them a different way. When an auth or ledger pod starts, the Vault agent injector adds a small sidecar container. That sidecar logs into Vault using the pod’s own service account, fetches the password and JWT, and writes them as files under <code>/vault/secrets/</code>. Your app already knows how to read those paths. It's the same data that used to arrive via <code>secretKeyRef</code>, just delivered at runtime instead of pulled from a Kubernetes Secret object. Once migration is complete, sensitive values live in Vault (the long-term store) and briefly on the pod filesystem while the container runs. They're not in Git anymore. You remove <code>secret.yaml</code> from <code>clearledger-infra</code> and ArgoCD syncs deployments that point at Vault instead. To load Vault the first time, you copy a template to a local <code>.env</code> file (§5.1). That file is gitignored. You run <code>seed-vault-secrets.sh</code> once to copy those values into Vault. Real secret values aren't written into committed scripts. The scripts read secrets from your local <code>.env</code> file or from your terminal, so passwords and tokens stay out of Git. <h3 id="heading-do-the-steps-in-this-order">Do the Steps in This Order</h3> Each step depends on the one before it. Skipping ahead is the most common way to get red auth/ledger pods that look like a broken app but really mean “Vault is not ready yet.” <ol> <li>§5.1: copy <code>stages/stage-5-secrets-management/.env.example</code> to <code>.env</code>, then fill it with your cluster passwords </li> <li>§5.2: install Vault and the agent injector with Helm </li> <li>§5.3. Run <code>setup.sh</code>, then <code>seed-vault-secrets.sh</code> (passwords now live in Vault) </li> <li>§5.4: push Vault-enabled deployments to <code>clearledger-infra</code>. Let ArgoCD sync. </li> <li>§5.5. Wait for 2/2 pods (app + Vault sidecar), then delete the old Kubernetes Secrets </li> <li>§5.5b: ArgoCD Synced / Healthy (after secret delete. OutOfSync before delete is normal) </li> <li>§5.6. Confirm login works and credentials appear under <code>/vault/secrets/</code> inside the pod </li> </ol> Start at §5.1. If anything fails, read <code>troubleshooting.md.</code> before changing manifests. <h3 id="heading-51-create-env-local-only-never-commit">5.1: Create <code>.env</code> (Local Only, Never Commit)</h3> This file holds two things: a dev Vault root token for Helm (§5.2), and the passwords you'll load into Vault in §5.3. It stays on your machine only. Never commit it. The <code>SEED_*</code> values must match what the app uses today so login still works after you delete Kubernetes Secrets later. Two different files. Don't mix them up: <table> <thead> <tr> <th>File</th> <th>What it is</th> </tr> </thead> <tbody><tr> <td><code>stages/stage-5-secrets-management/.env.example</code></td> <td>Blank template in the repo (empty fields). Copy this in step 1.</td> </tr> <tr> <td><code>stages/stage-5-secrets-management/.env</code></td> <td>Your real file (gitignored). You create it and fill it in steps 2–3.</td> </tr> </tbody></table> The sample block at the bottom of this section is only a picture of what a completed <code>.env</code> looks like: don't copy those placeholder passwords unless they happen to match your cluster. <h4 id="heading-step-1-copy-the-template-to-env">Step 1: copy the template to <code>.env</code></h4> <pre><code class="language-bash">cp stages/stage-5-secrets-management/.env.example \ stages/stage-5-secrets-management/.env </code></pre> That gives you a file with empty <code>VAULT_TOKEN=</code> and <code>SEED_*=</code> lines. Open it in your editor for steps 2–3. <h4 id="heading-step-2-read-the-current-passwords-from-the-cluster">Step 2: read the current passwords from the cluster</h4> Run these from the repo root. Each command prints one value: copy the output into <code>.env</code> in step 3. <pre><code class="language-bash"># → paste as SEED_AUTH_DATABASE_URL kubectl get secret auth-service-secret -n clearledger \ -o jsonpath='{.data.database_url}' | base64 -d; echo # → paste as SEED_AUTH_JWT_SECRET kubectl get secret auth-service-secret -n clearledger \ -o jsonpath='{.data.jwt_secret}' | base64 -d; echo # → paste as SEED_LEDGER_DATABASE_URL kubectl get secret ledger-service-secret -n clearledger \ -o jsonpath='{.data.database_url}' | base64 -d; echo </code></pre> <h4 id="heading-step-3-fill-in-env">Step 3: fill in <code>.env</code></h4> <table> <thead> <tr> <th>Variable</th> <th>What to put</th> </tr> </thead> <tbody><tr> <td><code>VAULT_TOKEN</code></td> <td>Any dev-only string you choose (for example, <code>my-dev-root-token</code>): same value in §5.2 Helm install</td> </tr> <tr> <td><code>SEED_AUTH_DATABASE_URL</code></td> <td>Output of first command above</td> </tr> <tr> <td><code>SEED_AUTH_JWT_SECRET</code></td> <td>Output of second command</td> </tr> <tr> <td><code>SEED_LEDGER_DATABASE_URL</code></td> <td>Output of third command</td> </tr> </tbody></table> Sample only: shape of a completed <code>.env</code> (use your kubectl output from step 2, not these example strings unless they match): <pre><code class="language-text">VAULT_TOKEN=my-dev-root-token SEED_AUTH_DATABASE_URL=postgresql://clearledger:changeme-stage0@postgres:5432/clearledger SEED_AUTH_JWT_SECRET=stage0-jwt-secret-change-in-production SEED_LEDGER_DATABASE_URL=postgresql://clearledger:changeme-stage0@postgres:5432/clearledger </code></pre> If <code>auth-service-secret</code> is already deleted (you skipped ahead: recover like this): <pre><code class="language-bash"># Database URL from Postgres bootstrap secret (lab default password is often changeme-stage0) PG_PASS=$(kubectl get secret postgres-secret -n clearledger \ -o jsonpath='{.data.password}' | base64 -d) echo "postgresql://clearledger:${PG_PASS}@postgres:5432/clearledger" # Use that line for both SEED_AUTH_DATABASE_URL and SEED_LEDGER_DATABASE_URL # JWT: same value you used at Stage 0, or read from Vault if you already seeded: kubectl exec -n vault vault-0 -- vault kv get -field=jwt_secret clearledger/auth-service 2>/dev/null \ || echo "(set SEED_AUTH_JWT_SECRET manually — must match tokens already issued)" </code></pre> Continue to §5.2 once <code>.env</code> has all four variables set. <h3 id="heading-52-install-vault-and-the-agent-injector">5.2: Install Vault and the Agent Injector</h3> <pre><code class="language-bash">set -a && source stages/stage-5-secrets-management/.env && set +a helm repo add hashicorp https://helm.releases.hashicorp.com && helm repo update # First install: helm install vault hashicorp/vault \ --namespace vault --create-namespace \ --set server.dev.enabled=true \ --set server.dev.devRootToken="${VAULT_TOKEN}" \ --set ui.enabled=true \ --set injector.enabled=true # If helm install fails with "cannot re-use a name", use upgrade instead: # helm upgrade --install vault hashicorp/vault \ # --namespace vault --create-namespace \ # --set server.dev.enabled=true \ # --set server.dev.devRootToken="${VAULT_TOKEN}" \ # --set ui.enabled=true \ # --set injector.enabled=true kubectl wait --for=condition=ready pod \ -l app.kubernetes.io/name=vault -n vault --timeout=120s kubectl wait --for=condition=ready pod \ -l app.kubernetes.io/name=vault-agent-injector -n vault --timeout=120s kubectl apply -f stages/stage-5-secrets-management/infra/vault-ingress.yaml </code></pre> Open <a href="http://vault.local"><code>http://vault.local</code></a> in your browser. Log in with the value you set as <code>VAULT_TOKEN</code> in <code>stages/stage-5-secrets-management/.env</code>. For example, if your <code>.env</code> has <code>VAULT_TOKEN=my-dev-root-token</code>, use <code>my-dev-root-token</code> as the Vault login token. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/8a2b0002-2ee3-4b04-89f0-e43da18fc9b9.png" alt="screenshot showing vault UI" style="display:block;margin:0 auto" width="1301" height="696" loading="lazy"> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/ec5fdb0b-7258-4e41-aeb5-adaa8754dbf2.png" alt="screenshot showing vault UI" style="display:block;margin:0 auto" width="1283" height="703" loading="lazy"> Verify: list Vault pods: <pre><code class="language-bash">kubectl get pods -n vault </code></pre> Expected: Vault pods: <pre><code class="language-text">NAME READY STATUS RESTARTS AGE vault-0 1/1 Running 0 1m vault-agent-injector-8d6b668b4-xxxxx 1/1 Running 0 1m </code></pre> If <code>helm install</code> fails with “cannot re-use a name”: Vault is already installed. Use the <code>helm upgrade --install</code> block above. <h3 id="heading-53-configure-vault-platform-seed-kv">5.3: Configure Vault (Platform + Seed KV)</h3> Run both scripts in order. Each reads <code>VAULT_TOKEN</code> from your <code>.env</code>. <pre><code class="language-bash">bash stages/stage-5-secrets-management/infra/vault/setup.sh bash stages/stage-5-secrets-management/infra/vault/seed-vault-secrets.sh </code></pre> <code>setup.sh</code>: prepares Vault for the cluster: Kubernetes auth, the KV secret store, policies, and roles so auth/ledger pods can fetch secrets later. It doesn't write your database passwords yet and nothing goes to Git. <code>seed-vault-secrets.sh</code>: takes the <code>SEED_*</code> lines from <code>.env</code> and stores them in Vault at <code>clearledger/data/auth-service</code> and <code>clearledger/data/ledger-service</code>. It doesn't echo those values to the terminal. Re-running either script is safe for the lab. Expected, <code>setup.sh</code> (tail): <pre><code class="language-text">==> Enabling Kubernetes auth method... ==> Configuring Kubernetes auth... ==> Enabling KV secrets engine... ==> Creating Vault policies... ==> Creating Kubernetes auth roles... ==> Applying RBAC + ServiceAccounts... ✓ Vault platform setup complete (no secrets written yet). Next: bash stages/stage-5-secrets-management/infra/vault/seed-vault-secrets.sh </code></pre> Expected, <code>seed-vault-secrets.sh</code>: <pre><code class="language-text">==> Logging into Vault... ==> Writing secrets to Vault KV (values are not printed)... ======== Secret Path ======== clearledger/data/auth-service ======= Metadata ======= Key Value --- ----- created_time 2026-06-01T15:31:53.538991153Z version 1 ✓ Secrets stored at clearledger/data/auth-service and clearledger/data/ledger-service </code></pre> Verify metadata only (no secret values printed): <pre><code class="language-bash">kubectl exec -n vault vault-0 -- vault kv metadata get clearledger/auth-service </code></pre> <pre><code class="language-text">Key Value --- ----- cas_required false created_time 2026-06-01T15:31:53.538991153Z current_version 1 delete_version_after 0s max_versions 0 oldest_version 0 updated_time 2026-06-01T15:31:53.538991153Z </code></pre> <h3 id="heading-54-gitops-update-clearledger-infra-fixes-argocd-outofsync">5.4: GitOps: Update <code>clearledger-infra</code> (Fixes ArgoCD OutOfSync)</h3> ArgoCD deploys from your <code>clearledger-infra</code> GitHub repo, not from the main <code>clearledger</code> app repo where you're working now. You edit manifests here first, then copy the same changes to <code>clearledger-infra</code> so ArgoCD can sync them. Work slowly and verify after each sub-step. <h4 id="heading-54a-update-manifests-in-the-app-repo-clearledger">5.4a. Update manifests in the app repo (<code>clearledger</code>)</h4> <pre><code class="language-bash">cp stages/stage-5-secrets-management/infra/manifests/auth-service/deployment.yaml \ infra/manifests/auth-service/deployment.yaml cp stages/stage-5-secrets-management/infra/manifests/ledger-service/deployment.yaml \ infra/manifests/ledger-service/deployment.yaml\ mkdir -p infra/manifests/vault cp infra/deferred-by-stage/stage-5-secrets-management/vault/rotation-cronjob.yaml \ infra/manifests/vault/rotation-cronjob.yaml rm -f infra/manifests/auth-service/secret.yaml infra/manifests/ledger-service/secret.yaml </code></pre> <h4 id="heading-54b-edit-inframanifestskustomizationyaml-by-hand">5.4b. Edit <code>infra/manifests/kustomization.yaml</code> by hand</h4> Open the file in your editor. In the <code>resources:</code> list: <ul> <li>Remove the app secret entries: delete these two lines, or comment them out with <code>#</code> (both work, as Kustomize ignores <code>#</code> lines): <pre><code class="language-yaml">- auth-service/secret.yaml - ledger-service/secret.yaml </code></pre> </li> <li>Add this line (with the other resources): <pre><code class="language-yaml">- vault/rotation-cronjob.yaml </code></pre> </li> </ul> Leave <code>postgres/postgres-secret.yaml</code>, that is Postgres bootstrap only, not app credentials. Save. Verify: <pre><code class="language-bash"># Active (uncommented) app secret lines must be gone — postgres-secret is OK grep -E '^[[:space:]]*-[[:space:]]+(auth-service|ledger-service)/secret\.yaml' \ infra/manifests/kustomization.yaml && echo "STOP: app secrets still active" || echo "OK" grep vault/rotation-cronjob.yaml infra/manifests/kustomization.yaml grep vault.hashicorp infra/manifests/auth-service/deployment.yaml | head -1 kustomize build infra/manifests >/dev/null && echo "OK: kustomize build" </code></pre> Expected: <code>OK</code>, rotation cronjob listed, first line shows <code>vault.hashicorp.com/agent-inject</code>, kustomize build succeeds. Commit in the app repo when ready: <code>git add infra/manifests && git commit -m "feat(stage-5): Vault deployments in canonical manifests"</code>. <h4 id="heading-54c-push-the-same-changes-to-clearledger-infra">5.4c. Push the same changes to <code>clearledger-infra</code></h4> <pre><code class="language-bash">git clone https://github.com/YOUR_USERNAME/clearledger-infra.git /tmp/clearledger-infra </code></pre> If clone fails with <code>destination path '/tmp/clearledger-infra' already exists</code> (you cloned in §1.3 or an earlier step), reuse that folder. Don't clone again: <pre><code class="language-bash">cd /tmp/clearledger-infra && git pull && cd - </code></pre> Or start fresh: <code>rm -rf /tmp/clearledger-infra</code> then run <code>git clone</code> again. Run the <code>cp</code> commands from the main <code>clearledger</code> app repo, not from <code>/tmp/clearledger-infra</code>. Your shell prompt should say <code>clearledger</code>, not <code>clearledger-infra</code>. The source path <code>infra/manifests/...</code> only exists in the app repo. <pre><code class="language-bash">cd ~/clearledger # main app repo — adjust path if yours differs cp infra/manifests/auth-service/deployment.yaml /tmp/clearledger-infra/manifests/auth-service/ cp infra/manifests/ledger-service/deployment.yaml /tmp/clearledger-infra/manifests/ledger-service/ mkdir -p /tmp/clearledger-infra/manifests/vault cp infra/manifests/vault/rotation-cronjob.yaml /tmp/clearledger-infra/manifests/vault/ cp infra/manifests/kustomization.yaml /tmp/clearledger-infra/manifests/kustomization.yaml rm -f /tmp/clearledger-infra/manifests/auth-service/secret.yaml rm -f /tmp/clearledger-infra/manifests/ledger-service/secret.yaml cd /tmp/clearledger-infra git add -A git status git commit -m "feat(stage-5): Vault injection; remove app secrets from GitOps" git push cd - </code></pre> ✋ Hands-on checkpoint. Stage 5 GitOps landed <pre><code class="language-bash">git clone --depth 1 https://github.com/YOUR_USERNAME/clearledger-infra.git /tmp/verify-s5 test ! -f /tmp/verify-s5/manifests/auth-service/secret.yaml && echo "OK: app secret removed from Git" grep vault.hashicorp /tmp/verify-s5/manifests/auth-service/deployment.yaml | head -1 grep vault/rotation-cronjob.yaml /tmp/verify-s5/manifests/kustomization.yaml rm -rf /tmp/verify-s5 </code></pre> Expected: <code>OK</code>, Vault annotation present, rotation job in kustomization. Expected, <code>git status</code> before commit (step 5.4c): <pre><code class="language-text">modified: manifests/auth-service/deployment.yaml modified: manifests/ledger-service/deployment.yaml modified: manifests/kustomization.yaml new file: manifests/vault/rotation-cronjob.yaml deleted: manifests/auth-service/secret.yaml deleted: manifests/ledger-service/secret.yaml </code></pre> After <code>git push</code>, ArgoCD will roll out Vault-enabled deployments automatically. Continue to §5.5. Don't expect Synced yet, as app secrets are still in the cluster until you delete them there. Common rollout failures: <table> <thead> <tr> <th>Symptom</th> <th>Fix</th> </tr> </thead> <tbody><tr> <td><code>Duplicate value: "vault-secrets"</code></td> <td>Do not declare a <code>vault-secrets</code> volume in <code>deployment.yaml</code>: the injector creates it</td> </tr> <tr> <td><code>Service appeared 2 times</code></td> <td>Keep <code>Service</code> only in <code>service.yaml</code>, not at the bottom of <code>deployment.yaml</code></td> </tr> <tr> <td>Kyverno <code>containers/0</code> <code>runAsNonRoot</code></td> <td>Add <code>runAsNonRoot: true</code> on the app container <code>securityContext</code>, not only on <code>spec.securityContext</code></td> </tr> <tr> <td>Pods stuck <code>1/1</code> (no sidecar)</td> <td>Confirm <code>injector.enabled=true</code> and deployment has <code>vault.hashicorp.com/agent-inject: "true"</code></td> </tr> <tr> <td><code>permission denied</code> in vault-agent-init</td> <td>Run <code>setup.sh</code> : K8s auth role not bound to service account</td> </tr> <tr> <td>ArgoCD Sync failed on <code>CronJob/vault-secret-rotation</code></td> <td>Kyverno blocked the job: <code>infra/manifests/vault/rotation-cronjob.yaml</code> must include <code>runAsNonRoot</code>, <code>allowPrivilegeEscalation: false</code>, <code>capabilities.drop: [ALL]</code>, and CPU/memory limits. Push fix to <code>clearledger-infra</code>.</td> </tr> </tbody></table> <h3 id="heading-55-wait-for-vault-injected-pods-then-delete-k8s-app-secrets">5.5: Wait for Vault-injected Pods, Then Delete K8s App Secrets</h3> Wait until auth/ledger show Vault sidecars (<code>READY 2/2</code> = app + vault-agent): <pre><code class="language-bash">kubectl get pods -n clearledger -l app=auth-service kubectl get pods -n clearledger -l app=ledger-service </code></pre> Expected: <pre><code class="language-text">NAME READY STATUS RESTARTS AGE auth-service-5756d9fcb9-bmdlr 2/2 Running 0 2m auth-service-5756d9fcb9-jtgss 2/2 Running 0 2m </code></pre> Inspect sidecar pulled secrets (init container logs): <pre><code class="language-bash">kubectl logs -n clearledger \ $(kubectl get pod -n clearledger -l app=auth-service -o name | head -1) \ -c vault-agent-init # ... Authentication successful, rendering templates ... </code></pre> Only after pods are 2/2, delete app Secrets: <pre><code class="language-bash">kubectl delete secret auth-service-secret ledger-service-secret -n clearledger </code></pre> Expected: secrets remaining: <pre><code class="language-bash">kubectl get secret -n clearledger </code></pre> <pre><code class="language-text">NAME TYPE DATA AGE postgres-secret Opaque 2 6d </code></pre> <code>postgres-secret</code> is Postgres bootstrap only, not app credentials. That stays until you harden Postgres separately. If delete says <code>NotFound</code>: secrets were already removed. Continue to §5.6. <h3 id="heading-55b-argocd-should-be-synced-after-secret-delete">5.5b: ArgoCD Should Be Synced After Secret Delete</h3> Run this after §5.5, not right after §5.4. Before you delete app Secrets, OutOfSync is normal. Git no longer lists <code>auth-service-secret</code> / <code>ledger-service-secret</code>, but they still exist in the cluster until you delete them in the step above. <pre><code class="language-bash">kubectl get application clearledger -n argocd \ -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}' </code></pre> Before secret delete: expect <code>sync=OutOfSync health=Healthy</code> or <code>Progressing</code> while Vault pods roll out. That's fine if auth/ledger are 2/2. After secret delete, hard-refresh and sync if still OutOfSync: <pre><code class="language-bash">kubectl annotate application clearledger -n argocd argocd.argoproj.io/refresh=hard --overwrite argocd app sync clearledger --grpc-web --prune </code></pre> If sync says another operation is already in progress, wait a minute: ArgoCD auto-sync is already running. Wait until: <pre><code class="language-bash">kubectl get application clearledger -n argocd \ -o jsonpath='{.status.sync.status} {.status.health.status}{"\n"}' # Synced Healthy </code></pre> Don't update app deployments with <code>kubectl apply</code> after ArgoCD is managing them. ArgoCD keeps the cluster matched to <code>clearledger-infra</code>. If you change a deployment by hand, ArgoCD may revert it. For Stage 5, update the manifests in Git and let ArgoCD sync the Vault-enabled deployments. <h3 id="heading-56-login-and-injected-files">5.6: Login and Injected Files</h3> <pre><code class="language-bash">kubectl exec -n clearledger \ $(kubectl get pod -n clearledger -l app=auth-service -o name | head -1) \ -c auth-service -- ls /vault/secrets/ </code></pre> <pre><code class="language-text">database_url jwt_secret </code></pre> <pre><code class="language-bash">curl -s -X POST http://clearledger.local/auth/login \ -H "Content-Type: application/json" \ -d '{"email":"test@clearledger.io","password":"SecurePass123"}' | jq . </code></pre> Expected: <pre><code class="language-json">{ "access_token": "<jwt-returned-by-auth-service>", "token_type": "bearer" } </code></pre> Take a screenshot: working login JSON + <code>kubectl get secret -n clearledger</code> showing no <code>auth-service-secret</code> / <code>ledger-service-secret</code>. <h3 id="heading-57-health-check">5.7: Health Check</h3> <pre><code class="language-bash">make check-5 </code></pre> What you should see: <blockquote> <code>make check-5</code> re-runs Stage 4 checks first, that is expected. Look for the Stage 5 block below to confirm Vault is working. </blockquote> <pre><code class="language-text">▶ Stage 4: Admission Control (Kyverno) ✓ Kyverno is running ✓ Policy disallow-root-containers — Enforce mode ... ✓ kube-bench matches baseline (no new FAIL regressions) ▶ Stage 5: Secrets Management (Vault) ✓ Vault pod is running ✓ Vault agent injector is running ✓ Vault is unsealed ✓ Vault Kubernetes auth method is enabled ✓ auth-service-secret removed — Vault is the secret source ✓ Vault injected /vault/secrets/database_url into auth-service All checks passed. Ready for the next stage. </code></pre> If Vault injection or ArgoCD sync fails, see <code>troubleshooting.md</code>. <h3 id="heading-stage-5-is-done-checklist-before-moving-to-stage-6">Stage 5 is Done: Checklist Before Moving to Stage 6</h3> <table> <thead> <tr> <th>#</th> <th>Check</th> <th>How to verify</th> </tr> </thead> <tbody><tr> <td>1</td> <td>Secrets in Vault only</td> <td><code>vault kv metadata get clearledger/auth-service</code> shows <code>current_version >= 1</code></td> </tr> <tr> <td>2</td> <td>No app secrets in infra Git</td> <td><code>secret.yaml</code> absent from <code>clearledger-infra/manifests/auth-service/</code> and <code>ledger-service/</code></td> </tr> <tr> <td>3</td> <td>ArgoCD synced</td> <td><code>Synced Healthy</code> on Application <code>clearledger</code></td> </tr> <tr> <td>4</td> <td>K8s app secrets deleted</td> <td><code>kubectl get secret -n clearledger</code> no auth/ledger app secrets</td> </tr> <tr> <td>5</td> <td>Injection works</td> <td>Auth pods <code>2/2</code>; <code>ls /vault/secrets/</code> shows <code>database_url</code>, <code>jwt_secret</code></td> </tr> <tr> <td>6</td> <td>App works</td> <td>Login curl returns <code>access_token</code></td> </tr> <tr> <td>7</td> <td>Health check</td> <td><code>make check-5</code> ends with <code>All checks passed. Ready for the next stage.</code></td> </tr> </tbody></table> Stage 5 moves app credentials out of Git and Kubernetes Secrets. It doesn't make Vault production-grade yet. This lab still uses Vault dev mode, not HA or auto-unseal. Also, a running pod can still read the files under <code>/vault/secrets/</code> because the app needs those credentials to work. That's normal. Stage 6 adds Falco so you can detect suspicious runtime access. <h3 id="heading-what-you-learned-in-stage-5">What You Learned in Stage 5</h3> <ul> <li>Kubernetes Secrets aren't enough for real secret management. </li> <li>Vault now stores the app credentials. </li> <li><code>.env</code> was only used locally to load the first secrets into Vault. It's never committed. </li> <li>Vault injects secrets into the pod when the app starts. </li> <li><code>clearledger-infra</code> must stop storing <code>secret.yaml</code>, because ArgoCD deploys from that repo. </li> <li>The order matters: install Vault, seed secrets, update GitOps, wait for healthy pods, then delete old Kubernetes Secrets. </li> </ul> What you can now say in an interview: <blockquote> I replaced Kubernetes Secrets with HashiCorp Vault agent injection, removed app credentials from Git and Kubernetes Secrets, and verified the app still worked after Vault injected the credentials at runtime. </blockquote> Save your progress: <pre><code class="language-bash">make snapshot STAGE=5 && make snapshots </code></pre> Confirm <code>clearledger.stage5</code> appears in the snapshot list. <h2 id="heading-stage-6-runtime-security-falco">Stage 6 — Runtime Security (Falco)</h2> Stages 1–5 secured what gets deployed and how secrets are stored. Stage 6 watches what happens inside running containers after they start. Your goal is to learn what runtime security catches and why it matters, then prove it by triggering a Falco alert and reading it the way an on-call engineer would. CI, Kyverno, and Vault all act before or at pod startup. Falco fills the gap they leave open. It watches what running software actually does inside the container. That's the layer incident response and forensics care about, not just another chart to install. Before you start Stage 6: <ul> <li><code>make check-5</code> passes </li> <li>Login and transactions still work at <code>http://clearledger.local</code> </li> <li>Platform pods have low restart counts </li> </ul> You're done with Stage 6 when: <ul> <li>You trigger at least one Falco alert </li> <li>You apply the network policies </li> <li><code>make check-6</code> passes </li> </ul> Then save your VM: <pre><code class="language-bash">make snapshot STAGE=6 make snapshots </code></pre> <h3 id="heading-do-the-steps-in-this-order">Do the Steps in This Order</h3> Each step depends on the one before it. Don't run <code>make check-6</code> until §6.4. It checks network policies you haven't applied yet. <ol> <li>§6.1: <code>bash stages/stage-6-runtime-security/scripts/install-falco.sh</code>. Confirm <code>falco-*</code> pods <code>2/2 Running</code> and custom rules loaded. </li> <li>§6.2: <code>make demo-6</code>: read <code>✓ Runtime detection confirmed</code> in the terminal </li> <li>§6.3 (optional) manual break-it scenarios (skip if <code>make demo-6</code> already worked) </li> <li>§6.4: <code>kubectl apply -f infra/deferred-by-stage/stage-6-runtime-security/netpol/network-policies.yaml</code>. Confirm <code>curl http://clearledger.local/</code> returns 200. </li> <li>§6.6: <code>make check-6</code> </li> </ol> Start at §6.1. If anything fails, see <code>troubleshooting.md</code>. Optional reading: <a href="#heading-how-stage-6-fits-the-full-stack-optional-reading">How Stage 6 fits the full stack</a>: why Falco and netpol exist and how they differ from Stages 3–5. <h3 id="heading-if-you-get-stuck-in-stage-6">If You Get Stuck in Stage 6</h3> Stage 6 has three jobs: <ol> <li>Install Falco </li> <li>Trigger one test alert </li> <li>Apply network policies </li> </ol> Don't worry about every row in the Falco UI. The UI may show noise. You pass the Falco part when you can find one alert from your demo, either in the terminal or in the UI. For the portfolio screenshot, open: <code>http://falco.local</code> Login: <ul> <li>Username: <code>admin</code> </li> <li>Password: <code>admin</code> </li> </ul> Take a screenshot only after your demo alert appears. Common stuck points <table> <thead> <tr> <th>You think…</th> <th>What is actually true</th> </tr> </thead> <tbody><tr> <td>“The UI shows 200+ Critical alerts, maybe I broke something”</td> <td>No. <code>postgres-0</code> reads <code>/etc/passwd</code> on a loop and Falco flags it. Ignore those rows.</td> </tr> <tr> <td>“I can't find my demo alert”</td> <td>Search the UI with Cmd+F → <code>Shell Spawned</code>, or use the terminal grep in step 4 above. If grep shows <code>auth-service</code> + <code>id && exit</code>, you passed.</td> </tr> <tr> <td>“<code>make check-6</code> failed on NetworkPolicy”</td> <td>You ran the check before §6.4. Apply netpol first, then re-run.</td> </tr> <tr> <td>“§6.3 vs §6.2 — which do I run?”</td> <td>Run <code>make demo-6</code> (§6.2) only. §6.3 is the same attacks as manual commands. Skip it if demo-6 already worked.</td> </tr> <tr> <td>“What is Shell Spawned?”</td> <td>Falco saw a <code>sh</code> process start inside <code>auth-service</code>. That's suspicious in production. In the lab, you caused it on purpose. See §6.2.</td> </tr> <tr> <td>“Scenario 4 hangs or exit 137”</td> <td>Old <code>wget</code> command + Terminating pod. Skip Scenario 4 or use the python3 command in §6.4. Checkpoint + <code>make check-6</code> is enough.</td> </tr> </tbody></table> <h3 id="heading-61-install-falco-and-falcosidekick-ui">6.1: Install Falco and Falcosidekick UI</h3> <pre><code class="language-bash">bash stages/stage-6-runtime-security/scripts/install-falco.sh </code></pre> This runs <code>helm upgrade --install</code> with <code>modern_ebpf</code>, enables Falcosidekick + Web UI, enables the k8s-metacollector (<code>collectors.kubernetes.enabled: true</code>) so custom rules can match <code>k8smeta.ns.name = clearledger</code>, loads rules from <code>infra/falco/clearledger-rules-content.yaml</code>, applies the rules ConfigMap and ingress. If Falco is already installed, the script is safe to re-run (upgrade). Verify Falco pods: <pre><code class="language-bash">kubectl get pods -n falco </code></pre> Expected output: <pre><code class="language-text">NAME READY STATUS RESTARTS AGE falco-w4fh6 2/2 Running 0 2m falco-falcosidekick-... 1/1 Running 0 2m falco-falcosidekick-ui-... 1/1 Running 0 2m falco-falcosidekick-ui-redis-0 1/1 Running 0 2m </code></pre> The Falco DaemonSet should show 2/2 Running. Sidekick, UI, and Redis pods should each show 1/1 Running. Pod name suffixes on your cluster will differ from the example. Open <code>http://falco.local</code>. You'll see the Falcosidekick UI. Log in with the chart defaults: <table> <thead> <tr> <th>Field</th> <th>Value</th> </tr> </thead> <tbody><tr> <td>Login</td> <td><code>admin</code></td> </tr> <tr> <td>Password</td> <td><code>admin</code></td> </tr> </tbody></table> To read the credentials from the cluster instead of trusting the lab defaults: <pre><code class="language-bash">kubectl get secret falco-falcosidekick-ui -n falco \ -o jsonpath='{.data.FALCOSIDEKICK_UI_USER}' | base64 -d && echo # admin:admin </code></pre> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/eeac67ee-182a-403d-806d-328e1fdcd8b7.png" alt="Screenshot fo Falco UI" style="display:block;margin:0 auto" width="1293" height="1318" loading="lazy"> <h4 id="heading-falcosidekick-ui-quick-orientation">Falcosidekick UI: quick orientation</h4> After login you land on the Events tab. The table can look busy before you run any demo, which is normal. <ul> <li>Rule: detection name (what fired) </li> <li>Priority: Critical / Warning / Notice (focus on Critical and Warning for this lab) </li> <li>Output: pod name, file, or command details </li> <li>Tags: look for <code>clearledger</code> on lab alerts </li> </ul> Background noise you can ignore: Notice rows from ArgoCD. Critical Sensitive File Read rows from <code>postgres-0</code> reading <code>/etc/passwd</code> (repeats every few seconds). Your demo alert is different. See §6.2. Verify custom rules loaded (do this before §6.2): <pre><code class="language-bash">kubectl get pods -n falco # Falco pod 2/2 Running kubectl get configmap clearledger-falco-rules -n falco kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --tail=200 \ | grep 'rules.d/clearledger_rules' </code></pre> Expected: <code>clearledger_rules.yaml | schema validation: ok</code> An empty grep with <code>--tail=30</code> alone isn't a failure. Use <code>--tail=200</code>. If you see <code>LOAD_ERR_COMPILE_CONDITION</code>, see <code>troubleshooting.md</code>. If rules didn't load, §6.2 and §6.3 will look like they passed when nothing fired. <h3 id="heading-62-guided-demo-make-demo-6">6.2: Guided Demo (<code>make demo-6</code>)</h3> Run this after §6.1 (Falco installed, rules verified, UI opens at <code>http://falco.local</code>). <pre><code class="language-bash">make demo-6 # or: bash stages/stage-6-runtime-security/scripts/demo-falco-alerts.sh </code></pre> <h4 id="heading-what-the-demo-script-does">What the demo script does</h4> The demo proves Falco can detect suspicious activity inside a running container. The script checks that Falco is running, opens <code>http://falco.local</code>, and waits while you log in with: <ul> <li>Username: <code>admin</code> </li> <li>Password: <code>admin</code> </li> </ul> Then it runs this test command inside the <code>auth-service</code> container: <pre><code class="language-bash">kubectl exec -n clearledger \ auth-service-<pod-suffix> \ -c auth-service -- /bin/sh -c 'id && exit' </code></pre> The script picks the real pod name for you. Non-interactive (CI or no Enter prompts): <code>SKIP_PROMPT=1 make demo-6</code>. This starts a shell inside the app container. That's suspicious in production because app containers should run the app, not open shells. Falco should detect it and create an alert called: <code>Shell Spawned in ClearLedger Container</code> When the script prints: <pre><code class="language-text">✓ Runtime detection confirmed </code></pre> refresh the Falco UI. Look for a Critical alert with: <ul> <li>Rule: <code>Shell Spawned in ClearLedger Container</code> </li> <li>Pod: <code>auth-service-...</code> </li> <li>Command: <code>sh -c id && exit</code> </li> </ul> Ignore alerts from <code>postgres-0</code>, especially <code>Sensitive File Read</code>. Those are background noise for this lab. If the UI is noisy, search the page for <code>Shell Spawned</code> or check from the terminal: <pre><code class="language-bash">kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --tail=500 \ | grep 'Shell Spawned' </code></pre> For your screenshot, capture the <code>Shell Spawned</code> alert for <code>auth-service</code>. <h3 id="heading-63-break-it-scenarios-manual-optional">6.3: Break-it Scenarios (Manual, Optional)</h3> These are the same detections as §6.2, but you run each command yourself. Skip this section if you already completed <code>make demo-6</code>. <table> <thead> <tr> <th>Rule name</th> <th>You trigger it by…</th> </tr> </thead> <tbody><tr> <td>Shell Spawned in ClearLedger Container</td> <td>Scenario 1 — <code>kubectl exec … /bin/sh</code></td> </tr> <tr> <td>Sensitive File Read in ClearLedger</td> <td>Scenario 2 — <code>cat /etc/passwd</code></td> </tr> <tr> <td>Package Manager / Outbound connection</td> <td>Scenario 3 — <code>wget</code> or <code>curl</code></td> </tr> </tbody></table> After each command, refresh <code>http://falco.local</code> or use the find methods in §6.2. <h4 id="heading-scenario-1-shell-in-a-running-pod-command-injection-simulation">Scenario 1 – Shell in a running pod (command injection simulation):</h4> <pre><code class="language-bash">kubectl exec -n clearledger \ $(kubectl get pod -n clearledger -l app=auth-service -o name | head -1) \ -c auth-service -- /bin/sh -c "id && exit" </code></pre> Expected in Falco UI / logs (within ~10 seconds): <pre><code class="language-text">CRITICAL: Shell spawned in ClearLedger container user=... container=auth-service pod=auth-service-... cmd=sh -c id && exit </code></pre> What this means: Stage 4 allowed the pod (it is compliant). Stage 6 detected behavior inside the pod: exactly what an attacker would do after command injection. If you see no alert: confirm the exec used <code>-c auth-service</code> (not the vault-agent sidecar), rules show <code>schema validation: ok</code>, and the pod image name contains <code>clearledger</code>. <h4 id="heading-scenario-2-read-a-sensitive-file-reconnaissance">Scenario 2 – Read a sensitive file (reconnaissance):</h4> <pre><code class="language-bash">kubectl exec -n clearledger \ $(kubectl get pod -n clearledger -l app=auth-service -o name | head -1) \ -c auth-service -- cat /etc/passwd </code></pre> Expected: <pre><code class="language-text">CRITICAL: Sensitive file read in ClearLedger file=/etc/passwd container=auth-service pod=auth-service-... </code></pre> <h4 id="heading-scenario-3-download-tool-at-runtime-optional">Scenario 3 – Download tool at runtime (optional):</h4> <pre><code class="language-bash">kubectl exec -n clearledger \ $(kubectl get pod -n clearledger -l app=auth-service -o name | head -1) \ -c auth-service -- sh -c "wget -q ifconfig.me -O - 2>/dev/null || true" </code></pre> May fire Package manager executed and/or Unexpected outbound connection (WARNING). Take screenshots of Scenarios 1 and 2: portfolio evidence for runtime detection. <h3 id="heading-64-apply-network-policies-zero-trust-segmentation">6.4: Apply Network Policies (Zero-trust Segmentation)</h3> Network policies are firewall rules between pods. Apply them after the Falco demo. <code>make check-6</code> checks for these policies, so run it only after this section. The <code>default-deny-all</code> policy blocks traffic by default. The <code>allow-*</code> policies open only the paths ClearLedger needs to work. Falco detects suspicious behavior. Network policies limit where a pod can connect. Apply: <pre><code class="language-bash">kubectl apply -f infra/deferred-by-stage/stage-6-runtime-security/netpol/network-policies.yaml kubectl get networkpolicy -n clearledger </code></pre> Expected: seven policies: <code>default-deny-all</code> plus six <code>allow-*</code> (<code>auth-service</code>, <code>ledger-service</code>, <code>notification-service</code>, <code>postgres</code>, <code>redis</code>, <code>frontend</code>). Verify the app still works: <pre><code class="language-bash">curl -s http://clearledger.local/auth/health | jq . # {"status":"ok","service":"auth-service"} curl -s http://clearledger.local/notifications/health | jq . # {"status":"ok",...} </code></pre> Checkpoint (required): proves netpol didn't break the real app: <pre><code class="language-bash">kubectl get networkpolicy -n clearledger curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/ kubectl get pods -n clearledger --field-selector=status.phase!=Running </code></pre> <table> <thead> <tr> <th>Result</th> <th>Meaning</th> </tr> </thead> <tbody><tr> <td>Seven policies listed</td> <td>Netpol applied</td> </tr> <tr> <td><code>200</code> from curl</td> <td>Users can still reach the app through ingress</td> </tr> <tr> <td>Third command prints nothing</td> <td>No crashed pods</td> </tr> </tbody></table> If auth or ledger start restarting after netpol, egress rules are too strict. See <code>troubleshooting.md</code>. <h4 id="heading-scenario-4-blocked-cross-service-traffic-optional">Scenario 4 – blocked cross-service traffic (optional)</h4> Skip if the checkpoint passed and you plan to run <code>make check-6</code>. This proves ledger can't call notification directly (no allow rule for that path). Failure to connect is success. Don't use the old <code>wget</code> one-liner: the ledger image has no <code>wget</code>/<code>curl</code>, and <code>head -1</code> can pick a Terminating pod (exec hangs or exit 137). <pre><code class="language-bash">LEDGER_POD=$(kubectl get pods -n clearledger -l app=ledger-service --no-headers \ | awk '$2=="2/2" && $3=="Running" {print $1; exit}') echo "Using pod: $LEDGER_POD" kubectl exec -n clearledger "$LEDGER_POD" -c ledger-service -- python3 -c " import urllib.request try: urllib.request.urlopen('http://notification-service/', timeout=5) print('UNEXPECTED: connection succeeded') except Exception as e: print('BLOCKED (expected):', e) " </code></pre> Expected: <pre><code class="language-text">BLOCKED (expected): <urlopen error timed out> </code></pre> or <code>Connection refused</code>, not <code>UNEXPECTED: connection succeeded</code>. <h3 id="heading-66-health-check">6.6: Health Check</h3> Run this after §6.4 (network policies). It confirms Falco, custom rules, and netpol are installed. It does not prove an alert fired (that is §6.2). <pre><code class="language-bash">make check-6 </code></pre> What you should see: <pre><code class="language-text">▶ Stage 6 — Runtime Security (Falco) ✓ Falco DaemonSet: 1/1 nodes ✓ ClearLedger custom Falco rules ConfigMap exists ✓ NetworkPolicy default-deny-all exists ✓ NetworkPolicy allow-auth-service exists ✓ NetworkPolicy allow-ledger-service exists ✓ NetworkPolicy allow-notification-service exists ✓ auth-service reachable after network policies ✓ notification-service reachable after network policies All checks passed. Ready for the next stage. </code></pre> <h3 id="heading-how-stage-6-fits-the-full-stack-optional-reading">How Stage 6 fits the full stack (optional reading)</h3> Each stage guards a different point in the lifecycle. Stages 1 through 5 work before or during pod startup. Stage 6 watches what happens inside a container that is already running. <ul> <li>In Stage 3, CI catches bad code and images on <code>git push</code>. </li> <li>Stage 4, Kyverno blocks bad pods at admission. </li> <li>Stage 5, Vault injects secrets at startup. </li> <li>Stage 6, Falco watches syscalls after the pod is running (shell spawns, sensitive file reads). </li> <li>Stage 6, Network policies filter pod-to-pod traffic. </li> </ul> They answer three different questions: Kyverno asks whether this pod may be created. Falco asks what the pod is doing right now. Network policies ask who the pod may talk to. Falco doesn't replace CI or Kyverno. If you skip Stages 3–5, Falco can still alert, but you already shipped vulnerable code and secrets in Git. <h3 id="heading-stage-6-is-complete-proceed-to-stage-65-or-7">Stage 6 is Complete. Proceed to Stage 6.5 or 7.</h3> <table> <thead> <tr> <th>#</th> <th>Check</th> <th>How to verify</th> </tr> </thead> <tbody><tr> <td>1</td> <td>Falco running</td> <td><code>kubectl get pods -n falco</code> — DaemonSet <code>2/2</code></td> </tr> <tr> <td>2</td> <td>Custom rules loaded</td> <td>`kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --tail=200</td> </tr> <tr> <td>3</td> <td>Shell alert fired and you read it</td> <td><code>make demo-6</code> → Critical row with <code>cmd=sh -c id && exit</code>, pod <code>auth-service-…</code> — §6.2</td> </tr> <tr> <td>4</td> <td>Network policies applied</td> <td><code>kubectl get networkpolicy -n clearledger</code> — §6.4</td> </tr> <tr> <td>5</td> <td>App still healthy</td> <td><code>curl</code> auth + notification health return 200</td> </tr> <tr> <td>6</td> <td>Health check</td> <td><code>make check-6</code> green — §6.6</td> </tr> </tbody></table> Portfolio screenshots (optional): shell-in-container alert · sensitive-file read alert in Falco UI. What comes next: Stage 6 gives you Falco alerts and basic network policies. You can refine the network policies later. Stage 6.5 is optional chaos testing with Litmus, and Stage 7 adds Grafana dashboards so you can see security events over time. <h3 id="heading-what-you-learned-in-stage-6">What You Learned in Stage 6</h3> <ul> <li>What runtime security catches that CI and admission control can't: threats inside running containers </li> <li>What Falco is: eBPF syscall monitoring with custom YAML rules </li> <li>What network policies are: Kubernetes firewall rules between pods </li> <li>How to trigger and interpret alerts: incident response skills </li> <li>The full stack: code scanning, admission control, secrets management, runtime detection which leads to (next) observability </li> </ul> What you can now put on your CV / say in an interview: <blockquote> Deployed Falco for runtime threat detection with custom rules, and can trigger and read an alert for a shell-in-container or sensitive-file read the way an on-call engineer would. </blockquote> <code>make snapshot STAGE=6 && make snapshots</code>. Confirm <code>clearledger.stage6</code>. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>. <h2 id="heading-stage-65-chaos-engineering-optional">Stage 6.5 — Chaos Engineering (Optional)</h2> Most learners skip this. If Stage 6 is done and <code>make check-6</code> passes, jump straight to <a href="#heading-stage-7-security-observability">Stage 7</a>. Nothing in Stages 7–8 requires Litmus. If you want chaos/resilience (~1 hour): LitmusChaos deletes one <code>auth-service</code> pod and proves <code>/auth/health</code> stays 200 while Kubernetes replaces it. <h3 id="heading-do-the-steps-in-this-order">Do the Steps in This Order</h3> <table> <thead> <tr> <th>Step</th> <th>Section</th> <th>What you do</th> </tr> </thead> <tbody><tr> <td>1</td> <td><a href="#heading-650-before-you-start-auth-pods-must-be-22">§6.5.0</a></td> <td><code>make fix-65-prereqs</code> — auth pods 2/2 Ready</td> </tr> <tr> <td>2</td> <td><a href="#heading-651-install-litmuschaos-operator-ui-cluster-connection">§6.5.1</a></td> <td><code>bash ...install-litmus.sh</code> — UI shows Active 1</td> </tr> <tr> <td>3</td> <td><a href="#heading-652-run-your-first-experiment-pod-delete">§6.5.2</a></td> <td>UI: Pod-delete experiment + <code>curl</code> stays 200</td> </tr> <tr> <td>4</td> <td><a href="#heading-657-health-check">§6.5.7</a></td> <td><code>make check-65</code>, snapshot</td> </tr> </tbody></table> Optional: <a href="#heading-653-same-experiment-from-the-terminal-make-demo-65-optional">§6.5.3</a>: same test via <code>make demo-65</code> (terminal path) instead of the UI wizard. <h3 id="heading-650-before-you-start-auth-pods-must-be-22">6.5.0: Before You Start (Auth Pods Must be 2/2)</h3> Chaos deletes pods. If replacements fail to start, you debug CrashLoopBackOff instead of learning resilience. <pre><code class="language-bash">export GITHUB_OWNER=YOUR_GITHUB_USERNAME # required — without this, fix-argocd breaks ArgoCD repoURL make fix-65-prereqs kubectl get pods -n clearledger -l app=auth-service </code></pre> Pass: two pods, both 2/2 Ready. Don't install Litmus until this is true. If something fails: <table> <thead> <tr> <th>Symptom</th> <th>Fix</th> </tr> </thead> <tbody><tr> <td>ArgoCD ComparisonError after <code>fix-65-prereqs</code></td> <td><code>kubectl apply -f stages/stage-2-gitops/argocd/clearledger-app.yaml</code></td> </tr> <tr> <td>Auth Init:0/1, Vault <code>permission denied</code></td> <td>Re-run Stage 5 <code>setup.sh</code> + <code>seed-vault-secrets.sh</code>, delete auth/ledger pods</td> </tr> <tr> <td>Auth 1/2 or postgres timeout</td> <td><code>make fix-65-prereqs</code> again (adds netpol + startup probes)</td> </tr> </tbody></table> <h3 id="heading-651-install-litmuschaos-operator-ui-cluster-connection">6.5.1: Install LitmusChaos (Operator, UI, Cluster Connection)</h3> <pre><code class="language-bash">bash stages/stage-6.5-chaos-engineering/scripts/install-litmus.sh kubectl get pods -n litmus open http://litmus.local # login: admin / litmus </code></pre> Pass before §6.5.2: Overview shows Infrastructures: Active 1 (not 0, not Pending). Verify pods: <pre><code class="language-bash">kubectl get pods -n litmus # litmus-core, chaos frontend/server, mongodb, subscriber — all Running </code></pre> <h4 id="heading-if-overview-shows-0-infrastructures-or-pending">If Overview shows 0 infrastructures or PENDING</h4> The UI is empty until a subscriber agent connects your cluster: <pre><code class="language-bash">export LITMUS_PASSWORD='litmus' # only if you changed the default bash stages/stage-6.5-chaos-engineering/scripts/connect-litmus-infra.sh </code></pre> Hard-refresh the browser. Start at <a href="http://litmus.local">http://litmus.local</a> only, not old <code>/account/.../settings</code> bookmarks. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/69212567-ae17-4f8f-b60d-7c3aac1592b8.png" alt="screenshot showing litmus ui" style="display:block;margin:0 auto" width="1255" height="627" loading="lazy"> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/66cfe3f9-99b5-4c1b-bfed-aa90fbcca2e8.png" alt="screenshot showing litmus ui" style="display:block;margin:0 auto" width="1267" height="951" loading="lazy"> <h4 id="heading-ui-navigation-click-order-for-652">UI navigation (click order for §6.5.2)</h4> <ol> <li>Overview: Confirm Active 1 </li> <li>ChaosHubs, Pod Delete, Launch Experiment </li> <li>Chaos Experiments: watch Running to Completed </li> </ol> Left nav: Overview, Environments, ChaosHub, Chaos Experiments. Skip Resilience Probes and deep Settings URLs for this lab. <h3 id="heading-652-run-your-first-experiment-pod-delete">6.5.2: Run Your First Experiment (Pod Delete)</h3> Goal: Kill one <code>auth-service</code> pod and prove <code>/auth/health</code> stays 200. Before you click Run in the UI, open two terminals: <pre><code class="language-bash"># Terminal A — watch pods kubectl get pods -n clearledger -l app=auth-service -w # Terminal B — watch health every 5 seconds while true; do date +%H:%M:%S curl -s -o /dev/null -w "health=%{http_code}\n" http://clearledger.local/auth/health sleep 5 done </code></pre> In the UI (<code>http://litmus.local</code>): Left nav → ChaosHubs → Pod Delete card → Launch Experiment. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/ca183e75-6680-4743-9ffe-03a144e09e13.png" alt="screenshot showing litmus ui" style="display:block;margin:0 auto" width="1267" height="951" loading="lazy"> Litmus UI note: ChaosCenter labels change between versions (for example, “Tune fault”, “Target selection”, “Chaos Experiment”). Match fields by concept, not exact button text. Accept wizard defaults unless the table below lists a value. In the Litmus UI, open: <code>ChaosHubs</code> → <code>Pod Delete</code> → <code>Launch Experiment</code> This opens the experiment wizard. Use these values when the wizard asks for them: <ul> <li>Infrastructure: <code>clearledger-cluster</code> and it must be <code>Active</code> </li> <li>Namespace: <code>clearledger</code> </li> <li>Target label: <code>app=auth-service</code> </li> <li>Target kind: <code>Deployment</code> </li> <li>Pods affected: <code>50%</code> </li> <li>Duration: <code>30</code> seconds </li> <li>Fault/experiment name: <code>pod-delete</code> </li> </ul> Finish the wizard with Save or Create, then click Run. Don't choose Schedule. What success looks like: <table> <thead> <tr> <th>Where</th> <th>Good sign</th> </tr> </thead> <tbody><tr> <td>Terminal A</td> <td>One pod Terminating, then back to 2/2 Ready</td> </tr> <tr> <td>Terminal B</td> <td><code>health=200</code> even while one pod is down</td> </tr> <tr> <td>Litmus UI</td> <td>Experiment Running → Completed</td> </tr> </tbody></table> Prefer terminal over UI? Skip the wizard and run <a href="#heading-653-same-experiment-from-the-terminal-make-demo-65-optional">§6.5.3</a> (<code>make demo-65</code>) instead. <h3 id="heading-653-same-experiment-from-the-terminal-make-demo-65-optional">6.5.3 — Same experiment from the terminal (<code>make demo-65</code>) — optional</h3> Use this if you want to run the pod-delete test without clicking through the Litmus UI. Make sure auth pods are healthy first: <pre><code class="language-bash">make fix-65-prereqs </code></pre> Then run the demo: <pre><code class="language-bash">make demo-65 </code></pre> The script applies the <code>auth-service-pod-delete</code> ChaosEngine in the <code>litmus</code> namespace. Litmus deletes one <code>auth-service</code> pod, Kubernetes replaces it, and the script checks that <code>/auth/health</code> keeps returning <code>200</code>. After it finishes, verify the result: <pre><code class="language-bash">kubectl get chaosresult -n litmus kubectl get pods -n clearledger -l app=auth-service </code></pre> You passed if the script ends with <code>PASS</code>, the <code>ChaosResult</code> is <code>Completed / Pass</code>, and two <code>auth-service</code> pods are running again. You can also see the run in the Litmus UI: Chaos Experiments → refresh → open the latest run. If new auth pods get stuck in <code>Init:0/1</code>, re-apply the Stage 6 network policies: <pre><code class="language-bash">kubectl apply -f infra/deferred-by-stage/stage-6-runtime-security/netpol/network-policies.yaml </code></pre> <h3 id="heading-653a-real-output-examples-verified-on-the-lab-cluster">6.5.3a: Real Output Examples (Verified on the Lab Cluster)</h3> These samples were captured from a working cluster after <code>make fix-65-prereqs</code>, <code>make connect-litmus</code>, and <code>make demo-65</code>. <h4 id="heading-make-check-65"><code>make check-65</code></h4> <pre><code class="language-text">▶ Stage 6.5 — Chaos Engineering (LitmusChaos) ✓ litmus namespace exists ✓ litmus-admin ServiceAccount exists in litmus ✓ pod-delete ChaosExperiment installed in litmus ✓ Litmus chaos operator is running ✓ Litmus ChaosCenter reachable at http://litmus.local ✓ Litmus subscriber running (UI connected to cluster) ✓ auth-service healthy (baseline before chaos) ✓ auth-service has 2/2 Ready replicas (stable for chaos) ✓ allow-postgres NetworkPolicy exists (Stage 6 fix) All checks passed. Ready for the next stage. </code></pre> <h4 id="heading-make-demo-65-captured-from-a-real-run-2026-06-01"><code>make demo-65</code> captured from a real run (2026-06-01)</h4> <pre><code class="language-text">Stage 6.5 — auth-service pod-delete Preflight: 2 auth-service pods Running Applying ChaosEngine auth-service-pod-delete (namespace litmus) Watching http://clearledger.local/auth/health 10s health=200 pods=2 20s health=200 pods=1 30s health=200 pods=1 40s health=200 pods=2 50s health=200 pods=2 60s health=200 pods=2 Result: ChaosResult: Completed / Pass Recovery: 2 auth-service pod(s) Running Health: 6/6 checks returned 200 PASS </code></pre> If health lines show <code>000</code>, run <code>bash scripts/setup-hosts.sh</code> on your Mac and re-run. The script also tries <code>multipass exec clearledger -- curl</code> when the VM is present. <h4 id="heading-terminal-b-health-loop-expected-output">Terminal B (health loop, expected output)</h4> <pre><code class="language-text">22:05:01 health=200 22:05:06 health=200 22:05:11 health=200 </code></pre> Pod count may show 1 while the replacement pod is starting, which is expected. <h4 id="heading-terminal-a-during-chaos-kubectl-get-pods-w">Terminal A during chaos (<code>kubectl get pods -w</code>)</h4> <pre><code class="language-text">NAME READY STATUS RESTARTS AGE auth-service-84cc988c4d-hdb45 2/2 Running 0 67m auth-service-84cc988c4d-b59sj 2/2 Terminating 0 15m ← killed auth-service-84cc988c4d-dxz9q 0/2 Pending 0 0s ← replacement auth-service-84cc988c4d-dxz9q 0/2 Init:0/1 0 2s auth-service-84cc988c4d-dxz9q 2/2 Running 0 90s </code></pre> <h4 id="heading-after-demo-verify">After demo: verify</h4> <pre><code class="language-bash">kubectl get chaosresult -n litmus # auth-service-pod-delete-pod-delete Completed Pass kubectl get pods -n clearledger -l app=auth-service # auth-service-84cc988c4d-xxxxx 2/2 Running # auth-service-84cc988c4d-yyyyy 2/2 Running kubectl get cm subscriber-config -n litmus -o jsonpath='{.data.IS_INFRA_CONFIRMED}' # true </code></pre> <h4 id="heading-subscriber-connected-infrastructure-active-in-ui">Subscriber connected (infrastructure Active in UI)</h4> <pre><code class="language-text">kubectl logs -n litmus -l app.kubernetes.io/name=subscriber --tail=3 level=info msg="AgentID: a63c2a2c-... has been confirmed" level=info msg="Server connection established, Listening...." </code></pre> <h3 id="heading-654-understand-the-yaml-files-read-before-running">6.5.4: Understand the YAML Files (Read Before Running)</h3> Each file is a <code>ChaosEngine</code>: a request to Litmus: “run experiment X against app Y for Z seconds.” <h4 id="heading-litmus-installyaml"><code>litmus-install.yaml</code></h4> This creates the <code>litmus</code> namespace only. Platform workloads live here, separate from <code>clearledger</code> app pods. <h4 id="heading-litmus-rbacyaml"><code>litmus-rbac.yaml</code></h4> <table> <thead> <tr> <th>Resource</th> <th>What it does</th> </tr> </thead> <tbody><tr> <td><code>ServiceAccount litmus-admin</code> (namespace <code>litmus</code>)</td> <td>Identity for Litmus runner pods</td> </tr> <tr> <td><code>ClusterRoleBinding → cluster-admin</code></td> <td>Allows deleting pods / injecting faults in <code>clearledger</code> (lab simplification. Production would use least-privilege)</td> </tr> </tbody></table> <h4 id="heading-auth-service-pod-deleteyaml-experiment-1-used-by-demo"><code>auth-service-pod-delete.yaml</code> (Experiment 1: used by demo)</h4> <pre><code class="language-yaml">metadata: namespace: litmus # engine lives here (Kyverno-safe) spec: appinfo: appns: clearledger # target app namespace applabel: app=auth-service appkind: deployment experiments: - name: pod-delete spec: components: env: - name: PODS_AFFECTED_PERC value: "50" # 50% of 2 replicas = 1 pod killed - name: TOTAL_CHAOS_DURATION value: "30" # chaos window in seconds </code></pre> What happens when applied: <ol> <li>Operator reads <code>ChaosEngine</code> and creates <code>auth-service-pod-delete-runner</code> pod in <code>litmus</code> </li> <li>Runner selects one <code>auth-service</code> pod in <code>clearledger</code> and sends SIGTERM / delete </li> <li>Kubernetes Deployment controller sees 1/2 replicas and schedules a replacement pod </li> <li>Service routes traffic to the surviving replica during recovery </li> <li><code>ChaosResult</code> CR records pass/fail from Litmus’s perspective </li> </ol> <h4 id="heading-ledger-service-network-latencyyaml-experiment-2-manual"><code>ledger-service-network-latency.yaml</code> (Experiment 2 — manual)</h4> Adds 2000 ms network latency to <code>ledger-service</code> pods for 60 seconds. Proves timeouts return 503 instead of hanging the UI. <h4 id="heading-notification-service-memory-hogyaml-experiment-3-manual"><code>notification-service-memory-hog.yaml</code> (Experiment 3 — manual)</h4> Fills 80% of pod memory limit for 60 seconds. Proves OOMKill + restart behavior. Never apply all three at once. Run one experiment, verify recovery, then the next. <h3 id="heading-655-after-the-demo-what-to-look-for-do-not-skip">6.5.5: After the Demo, What to Look For (Do Not Skip)</h3> 1. During chaos: availability <table> <thead> <tr> <th>Signal</th> <th>Good</th> <th>Bad</th> </tr> </thead> <tbody><tr> <td><code>curl http://clearledger.local/auth/health</code></td> <td>200 while one pod is down</td> <td>502/503/timeout</td> </tr> <tr> <td><code>kubectl get pods -l app=auth-service</code></td> <td>1 Running + 1 Init/Pending (replacement starting)</td> <td>0 Running</td> </tr> </tbody></table> 2. After chaos: recovery <table> <thead> <tr> <th>Signal</th> <th>Good</th> <th>Bad</th> </tr> </thead> <tbody><tr> <td>Pod count</td> <td>2/2 Ready (may take 1–2 min — Vault agent init)</td> <td>Stuck at 1 replica</td> </tr> <tr> <td>Events</td> <td><code>Killing</code> then <code>Scheduled</code> / <code>Started</code> on new pod</td> <td>Repeated CrashLoopBackOff</td> </tr> <tr> <td>ArgoCD</td> <td>Synced</td> <td>—</td> </tr> </tbody></table> 3. Litmus <code>ChaosResult</code> verdict <pre><code class="language-bash">kubectl get chaosresult -n litmus </code></pre> Your pass criteria: <ul> <li><code>/auth/health</code> returned 200 at least once during the chaos window </li> <li>A pod was Killed (see events) </li> <li>Deployment returned to 2 replicas </li> </ul> <h3 id="heading-656-manual-experiments-after-experiment-1-succeeds">6.5.6 Manual Experiments (After Experiment 1 Succeeds)</h3> Wait until both auth-service pods show 2/2 Ready, then run one experiment at a time: <pre><code class="language-bash"># Experiment 2 — 2s network latency on ledger-service (60s) kubectl delete chaosengine ledger-service-network-latency -n litmus --ignore-not-found kubectl apply -f stages/stage-6.5-chaos-engineering/infra/chaos/ledger-service-network-latency.yaml # Experiment 3 — memory pressure on notification-service (60s) kubectl delete chaosengine notification-service-memory-hog -n litmus --ignore-not-found kubectl apply -f stages/stage-6.5-chaos-engineering/infra/chaos/notification-service-memory-hog.yaml </code></pre> <table> <thead> <tr> <th>Experiment</th> <th>File</th> <th>What to verify</th> </tr> </thead> <tbody><tr> <td>Pod delete</td> <td><code>auth-service-pod-delete.yaml</code></td> <td>Health 200 during kill, 2 replicas after</td> </tr> <tr> <td>Network latency</td> <td><code>ledger-service-network-latency.yaml</code></td> <td>API returns 503/timeout, not infinite hang</td> </tr> <tr> <td>Memory hog</td> <td><code>notification-service-memory-hog.yaml</code></td> <td>Pod OOMKills and restarts, Redis subscription recovers</td> </tr> </tbody></table> Clean up an experiment: <pre><code class="language-bash">kubectl delete chaosengine auth-service-pod-delete -n litmus </code></pre> <h3 id="heading-657-health-check">6.5.7: Health Check</h3> <pre><code class="language-bash">make check-65 </code></pre> Expected: see full sample in <a href="#heading-653a-real-output-examples-verified-on-the-lab-cluster">§6.5.3a</a> (<code>make check-65</code> block). Minimum: <pre><code class="language-text">▶ Stage 6.5 Chaos Engineering (LitmusChaos) ✓ Litmus subscriber running (UI connected to cluster) ✓ auth-service has 2/2 Ready replicas (stable for chaos) ... All checks passed. Ready for the next stage. </code></pre> <h3 id="heading-stage-65-complete-checklist">Stage 6.5 Complete: Checklist</h3> <table> <thead> <tr> <th>#</th> <th>Check</th> <th>How to verify</th> </tr> </thead> <tbody><tr> <td>1</td> <td>Litmus operator running</td> <td><code>kubectl get pods -n litmus</code> — <code>litmus-*</code> Running</td> </tr> <tr> <td>2</td> <td>Experiments installed</td> <td><code>kubectl get chaosexperiment pod-delete -n litmus</code></td> </tr> <tr> <td>3</td> <td>Pod-delete demo run</td> <td><code>make demo-65</code> health 200 during chaos</td> </tr> <tr> <td>4</td> <td>Recovery observed</td> <td>2 auth-service replicas Ready. Killing/Scheduled events</td> </tr> <tr> <td>5</td> <td>Evidence saved</td> <td>Terminal output from <code>run-chaos.sh</code> (DORA artifact)</td> </tr> <tr> <td>6</td> <td>Health check</td> <td><code>make check-65</code> green</td> </tr> <tr> <td>7</td> <td>UI infrastructure connected</td> <td>Overview → Active: 1 (§6.5.2)</td> </tr> </tbody></table> <h3 id="heading-what-you-learned-in-stage-65">What You Learned in Stage 6.5</h3> <ul> <li>Detection ≠ resilience: Falco alerts don't prove HA </li> <li>Replicas + Services + probes: why <code>replicas: 2</code> isn't cosmetic </li> <li>ChaosEngine YAML: declarative failure injection as code </li> <li>Platform vs app namespaces: Kyverno blocks chaos runners in <code>clearledger</code>, engines run in <code>litmus</code> </li> <li>MTTR: time from pod kill to 2/2 Ready again (Stage 7 graphs this) </li> </ul> What you can now put on your CV / say in an interview: <blockquote> Ran chaos experiments with LitmusChaos (pod-delete, network latency, memory pressure) to prove the system recovers, and can distinguish detection from resilience. </blockquote> <code>make snapshot STAGE=65 && make snapshots</code>. Confirm <code>clearledger.stage65</code>. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>. <h2 id="heading-stage-7-security-observability">Stage 7 — Security Observability</h2> Security you can't measure, you can't prove. The goal here is to understand how metrics, logs, and dashboards fit together. Then prove it by running commands in the terminal, watching the same events appear in Grafana, and explaining what each panel means. This stage is not “install Grafana and move on.” Stage 7 isn't complete until your dashboards show real Kyverno violations and Falco alerts that you triggered in §7.4: plus portfolio screenshots (§7.6). <code>make check-7</code> only proves the stack is up; it does not prove you can detect security events. Before you start: <code>make check-6</code> should pass (Stage 6.5 is optional. Skip is fine). Check the VM is not overloaded: <code>multipass exec clearledger -- uptime</code>. If you ran Stage 6.5, do <a href="#heading-70-free-node-resources-scale-down-litmus">§7.0</a> first to scale Litmus down. Plan about half a day. This is the heaviest stage on a single-node VM. You'll be done when §7.6 is complete: dashboards show your Kyverno denial and Falco alert, not empty panels. Then <code>make check-7</code> (§7.7), <code>make snapshot STAGE=7</code>, and <code>make snapshots</code> (confirm <code>clearledger.stage7</code>). Already installed? If <code>kubectl get pods -n monitoring</code> shows Grafana 3/3 and Loki 1/1, skip §7.1. Start at §7.2 (verify the stack), then §7.4 (hands-on lab). <h3 id="heading-what-you-need-to-know-first">What You Need to Know First</h3> Up to now, each stage had its own window into the cluster. Stage 3 gave you CI scan results in GitHub Actions. Stage 4 showed Kyverno blocking a bad deploy in the terminal. Stage 6 gave you Falco alerts in its UI, and you could always run <code>kubectl logs</code> on a pod. Those views are useful, but they are scattered. Stage 7 brings them together in one place: Grafana. Instead of jumping between five different tools, you open a dashboard and see whether security events, policy violations, and app health are happening over time. <h4 id="heading-the-three-tools-youre-installing">The three tools you're installing</h4> Prometheus collects numbers from the cluster: things like “how many Kyverno denials in the last hour” or “how many HTTP requests per second.” It checks those numbers every 15–30 seconds and keeps a history you can graph. Loki collects log lines: the same kind of text you see from <code>kubectl logs</code>, but from many pods at once. Falco alerts, failed login attempts, and application errors all land here so you can search them later. Grafana is the web UI where charts and tables pull data from Prometheus and Loki. This is what you would show an auditor: not a one-off terminal screenshot, but proof that you can find and measure events after they happen. Prometheus doesn't magically know what to collect. ServiceMonitors and PodMonitors are small config objects that point it at the right targets. If Kyverno has no monitor, the Kyverno dashboard stays empty even when Kyverno is working fine. The same applies to application request rates. Those panels stay blank until §7.5, when metrics-enabled images are deployed through GitOps. Logs follow a similar path. Promtail reads container logs and sends them to Loki. If Loki isn't running, Grafana log panels show “No data” even though <code>kubectl logs</code> still works on individual pods. <h4 id="heading-how-this-connects-to-what-you-already-built">How this connects to what you already built</h4> When you blocked a bad <code>kubectl apply</code> in Stage 4, Kyverno recorded that denial. In Stage 7, that shows up on the Kyverno Policy Violations dashboard (via Prometheus). When you triggered a shell inside a pod in Stage 6, Falco wrote an alert. In Stage 7, that appears on the Security Event Timeline (via Loki). When ClearLedger handles HTTP traffic or a failed login, those events feed the Service Health dashboards (Loki and Prometheus together). Vault (Stage 5) and network policies (Stage 6) don't always have their own panel, but they still matter: fewer secrets in Git and blocked pod traffic show up indirectly in a healthier, quieter cluster. <h4 id="heading-what-youll-do-in-this-stage">What you'll do in this stage</h4> You'll run a command in the terminal (for example, a Kyverno violation or a Falco trigger) and then wait a short time while Prometheus or Loki ingests the event. Within about 15–90 seconds, the matching Grafana panel should update. That's the whole point of observability for security: the terminal proves the event happened once, while the dashboard proves you can detect and measure it later without being logged into the cluster at that exact moment. <h3 id="heading-70-free-node-resources-scale-down-litmus">7.0: Free Node Resources (Scale Down Litmus)</h3> Stage 6.5 is complete. You don't need the Litmus UI, MongoDB, or chaos operator running while Prometheus, Loki, and Grafana start. They compete for the same CPUs on a single-node lab VM (6 by default, see <code>scripts/setup-cluster.sh</code>). Scaling Litmus to zero frees ~500–800MB RAM and reduces CPU churn before the observability install. <pre><code class="language-bash">kubectl scale deployment,statefulset -n litmus --replicas=0 --all kubectl get pods -n litmus # Expected: no Running pods (Succeeded job pods from chaos experiments are OK) multipass exec clearledger -- uptime # Expected: load average (1m) ideally below ~8 before continuing </code></pre> You can scale Litmus back up later if you want to re-run chaos experiments (<code>bash stages/stage-6.5-chaos-engineering/scripts/install-litmus.sh</code>). For Stages 7–7.5, keep it scaled down. <h3 id="heading-71-install-the-observability-stack">7.1: Install the Observability Stack</h3> This is safe to run more than once. The script checks what's already installed. If Grafana, Prometheus, and Loki are healthy, it skips the heavy install and only updates dashboards and scrape configs. Running it again after a partial failure won't duplicate or break a working stack. Only add <code>FORCE=1</code> if something is genuinely stuck, for example you edited the Helm values files and need a full reinstall, or Loki keeps crashing in a restart loop: <pre><code class="language-bash">FORCE=1 bash stages/stage-7-observability/scripts/install-observability.sh </code></pre> On a first-time install, use the plain command in Step 1 below. Don't use <code>FORCE=1</code> unless the troubleshooting section tells you to. macOS, Linux, and WSL2: <code>FORCE=1 bash ...</code> works as written. Native Windows PowerShell doesn't use that syntax. Run the lab inside WSL2 Ubuntu (recommended), or set the variable first: <code>$env:FORCE=1; bash stages/stage-7-observability/scripts/install-observability.sh</code>. Step 1: install (wait until the script prints <code>✓ Stage 7 installed.</code>): <pre><code class="language-bash">bash stages/stage-7-observability/scripts/install-observability.sh </code></pre> <h4 id="heading-if-you-see-waiting-for-falco-during-the-stage-7-install-thats-expected">If you see “Waiting for Falco” during the Stage 7 install, that's expected.</h4> You already installed Falco in Stage 6. Stage 7 is not adding a second Falco. It is making sure the existing Falco setup can feed logs and metrics into the observability stack. The flow is: <ul> <li>Falco still runs in the <code>falco</code> namespace. </li> <li>Promtail sends Falco logs to Loki. </li> <li>Grafana reads those logs from Loki. </li> <li>The Security Event Timeline dashboard shows the Falco alerts. </li> </ul> Right after install, the Grafana panels may be empty. That's normal. You need to trigger a new alert in §7.4 before the dashboard has something fresh to show. Step 2. Check pods (run this after Step 1 finishes): <pre><code class="language-bash">kubectl get pods -n monitoring </code></pre> You want something like this (pod name suffixes vary): <pre><code class="language-text">NAME READY STATUS RESTARTS AGE kube-prometheus-stack-grafana-.... 3/3 Running 0 5m kube-prometheus-stack-prometheus-.... 2/2 Running 0 5m loki-0 1/1 Running 0 5m loki-promtail-.... 1/1 Running 0 5m </code></pre> Grafana must show 3/3 Ready (not 2/3). Loki must show 1/1. If pods are still <code>Pending</code> or <code>ContainerCreating</code>, wait a few minutes and run <code>kubectl get pods -n monitoring</code> again. Expected – Loki healthy: <pre><code class="language-bash">kubectl exec -n monitoring loki-0 -- wget -qO- http://127.0.0.1:3100/ready </code></pre> <pre><code class="language-text">ready </code></pre> Expected – Grafana can reach Loki (same path log panels use): <pre><code class="language-bash">kubectl exec -n monitoring deploy/kube-prometheus-stack-grafana -c grafana -- \ wget -qO- --timeout=5 http://loki:3100/ready </code></pre> <pre><code class="language-text">ready </code></pre> Expected – Grafana UI reachable: <pre><code class="language-bash">curl -sI http://grafana.local | head -n 1 </code></pre> <pre><code class="language-text">HTTP/1.1 302 Found </code></pre> Log into <a href="http://grafana.local">http://grafana.local</a>: <code>admin</code> / <code>admin123</code> Empty panels right after install are normal. You haven't generated events yet. Continue to §7.2–§7.4. If Helm fails: wait 30s, then <code>FORCE=1 bash stages/stage-7-observability/scripts/install-observability.sh</code>. See <code>troubleshooting.md. Stage 7</code>. ✋ Hands-on checkpoint: confirm Loki and dashboards are ready Before you open Grafana, confirm the logging stack and dashboards actually installed. On a single-node VM, Grafana can look fine while Loki is crash-looping or the ClearLedger dashboards never loaded. If you skip this check, you may spend the rest of Stage 7 debugging empty panels. Run: <pre><code class="language-bash">kubectl get pods -n monitoring kubectl get pods -n monitoring -l app.kubernetes.io/name=loki \ -o jsonpath='{.items[*].status.containerStatuses[*].restartCount}{"\n"}' kubectl get configmap -n monitoring -l clearledger_dashboard=1 --no-headers | wc -l </code></pre> Expected: <ul> <li>All monitoring pods are <code>Running</code> </li> <li>Grafana shows <code>3/3</code> Ready </li> <li>Loki shows <code>1/1</code> Ready </li> <li>Loki restart count is <code>0</code>, or low and not climbing </li> <li>The dashboard count is <code>6</code> </li> </ul> If Loki keeps restarting or the dashboard count is <code>0</code>, stop here and fix the install before continuing. Empty Grafana panels usually mean Loki or the dashboards are missing, not that the security events failed. <h3 id="heading-72-verify-prometheus-loki-and-grafana-before-opening-dashboards">7.2: Verify Prometheus, Loki, and Grafana (before opening dashboards)</h3> Run these three checks so you know which layer is broken if a panel is empty. <h4 id="heading-check-1-prometheus-has-kyverno-metrics">Check 1: Prometheus has Kyverno metrics</h4> <pre><code class="language-bash">kubectl exec -n monitoring deploy/kube-prometheus-stack-grafana -c grafana -- \ wget -qO- 'http://kube-prometheus-stack-prometheus.monitoring:9090/api/v1/query?query=kyverno_admission_requests_total' 2>/dev/null \ | head -c 400 </code></pre> (Prometheus runs as a StatefulSet pod, not a Deployment. This query goes through Grafana to the Prometheus Service.) Expected: JSON with <code>"status":"success"</code> and a <code>"metric"</code> block (values may be <code>0</code> until you trigger a violation in §7.4). If you see <code>"status":"success"</code> but <code>"result":[]</code>, Prometheus is up but Kyverno hasn't recorded admissions yet. That's fine before the lab. <h4 id="heading-check-2-loki-has-falco-logs">Check 2: Loki has Falco logs</h4> <pre><code class="language-bash">kubectl exec -n monitoring loki-0 -- wget -qO- \ 'http://127.0.0.1:3100/loki/api/v1/labels' 2>/dev/null | head -c 300 </code></pre> Expected: JSON listing labels such as <code>"namespace"</code> (and after Falco events, you'll see <code>"falco"</code> in label values). Quick log search (may return empty lines until §7.4 Exercise B): <pre><code class="language-bash">kubectl exec -n monitoring loki-0 -- wget -qO- \ 'http://127.0.0.1:3100/loki/api/v1/query?query=%7Bnamespace%3D%22falco%22%7D&limit=3' 2>/dev/null \ | head -c 500 </code></pre> Expected: <code>"status":"success"</code>. <code>"result":[]</code> means no Falco lines in Loki yet, not a broken Loki. <h4 id="heading-check-3-grafana-imported-clearledger-dashboards">Check 3: Grafana imported ClearLedger dashboards</h4> <pre><code class="language-bash">curl -s -u admin:admin123 'http://grafana.local/api/search?tag=clearledger' | jq -r '.[].title' </code></pre> Expected: six titles: <pre><code class="language-text">ClearLedger - Compliance Posture ClearLedger - DORA Metrics ClearLedger - Kubernetes Audit Log Analysis ClearLedger - Kyverno Policy Violations ClearLedger - Security Event Timeline ClearLedger - Service Health + Auth Security </code></pre> Or in the UI: go. toDashboards then filter tag <code>clearledger</code>. You should see exactly these six (no missing names). <h3 id="heading-73-your-first-10-minutes-in-grafana">7.3: Your First 10 Minutes in Grafana</h3> This section is only a tour. You're not proving anything yet. Rule for all of Stage 7: an empty panel usually means no events have happened in the selected time range, not that Grafana is broken. You create the real events in §7.4. <h4 id="heading-step-1-open-grafana">Step 1: Open Grafana</h4> Go to <code>http://grafana.local</code> and log in: <ul> <li>Username: <code>admin</code> </li> <li>Password: <code>admin123</code> </li> </ul> <h4 id="heading-step-2-set-the-time-range">Step 2: Set the Time Range</h4> In the top-right corner, choose Last 15 minutes. Keep this setting for all of Stage 7. Wider ranges like Last 24 hours can overload Loki on a single-node lab VM. <h4 id="heading-step-3-open-dashboards-one-at-a-time">Step 3: Open Dashboards One at a Time</h4> Open one dashboard, look around, then move to the next. Don't open all six at once. <ol> <li><a href="http://grafana.local/d/clearledger-kyverno-violations">Kyverno Policy Violations</a>: policy blocks from Stage 4. </li> <li><a href="http://grafana.local/d/clearledger-security-events">Security Event Timeline</a>: Falco alerts from Stage 6. You may see old <code>postgres</code> noise in the log table. </li> <li><a href="http://grafana.local/d/clearledger-service-health">Service Health + Auth</a>: app traffic and login attempts. </li> <li><a href="http://grafana.local/d/clearledger-compliance">Compliance Posture</a>: summary view for auditors. Skim it and come back after §7.4. </li> <li><a href="http://grafana.local/d/clearledger-audit-logs">Audit Log Analysis</a>: empty on MicroK8s by design (audit pipeline not enabled by default). </li> <li><a href="http://grafana.local/d/clearledger-dora-metrics">DORA Metrics</a>: deploy-frequency charts. Needs multiple CI runs to accumulate data. May show blank on first look. Optional. </li> </ol> Use the short dashboard links in this guide. Avoid old bookmarked URLs with long random slugs. You can also find them in Grafana: go to Dashboards then search tag <code>clearledger</code>. <h4 id="heading-step-4-how-to-read-what-you-see">Step 4: How to Read What You See</h4> Grafana panels pull data from two places: <ul> <li>Prometheus shows numbers over time, like Kyverno violation counts and request rates </li> <li>Loki shows log lines, like Falco alerts and auth-service messages </li> </ul> A big number panel asks: did this count go above zero? A line chart asks: was there a spike after I ran something? A logs panel shows the actual text, like rule names, <code>CRITICAL</code>, or <code>Failed login attempt</code>. If only log panels show <code>connection refused</code>, check Loki again in §7.1. If number panels work but log panels fail, the problem is likely Loki, not Grafana itself. <h4 id="heading-step-5-move-on">Step 5: Move On</h4> Open dashboards 1–3, then continue to §7.4. That's where you'll run commands in the terminal and watch the panels update with real security events. <h3 id="heading-74-hands-on-lab-terminal-dashboard-proof">7.4: Hands-on Lab: Terminal → Dashboard Proof</h3> This is the core learning section. For each exercise: run the command, wait, then confirm in Grafana. Timing: wait 30–90 seconds after each command for Prometheus scrape and Loki ingestion. <h3 id="heading-two-ways-to-do-this-lab">Two Ways to Do This Lab</h3> <h4 id="heading-option-1-follow-the-exercises-below-recommended-for-learning">Option 1: follow the exercises below (recommended for learning)</h4> Run each command yourself, then check Grafana. That's Exercise A, B, and C. <h4 id="heading-option-2-use-the-guided-script">Option 2: use the guided script</h4> The script runs the same steps and pauses so you can check Grafana between them: <pre><code class="language-bash">bash stages/stage-7-observability/scripts/generate-dashboard-data.sh </code></pre> Or: <pre><code class="language-bash">make demo-7 </code></pre> Both commands do the same thing. The script will say things like “Press Enter after you checked the Kyverno dashboard.” Switch to Grafana, look at the panel, then come back and press Enter. Want it to run without pauses? (faster, less hand-holding) <pre><code class="language-bash">SKIP_PROMPT=1 make demo-7 </code></pre> Use Option 1 if you want to understand each step. Use Option 2 if you want a walkthrough. Use <code>SKIP_PROMPT=1</code> if you just want the data generated quickly. <h4 id="heading-exercise-a-kyverno-block-prometheus-kyverno-dashboard">Exercise A: Kyverno block → Prometheus → Kyverno dashboard</h4> Terminal: apply a pod that violates Stage 4 policy (runs as root): <pre><code class="language-bash">cat <<'YAML' | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: stage7-kyverno-lab namespace: clearledger spec: containers: - name: test image: nginx:alpine YAML </code></pre> How to know it worked: You're testing whether Kyverno blocks a deliberately bad pod. Success means the pod never gets created. Pass. You should see: <ul> <li>The terminal prints <code>Error from server</code> and <code>denied the request</code> </li> <li>The exact policy names in the error don't matter. Your output might list one rule or several (<code>disallow-root-containers</code>, <code>require-resource-limits</code>, <code>drop-all-capabilities</code>, …). More lines just means more rules failed, that's still a pass. </li> <li>The pod name never shows up in the cluster: </li> </ul> <pre><code class="language-bash">kubectl get pods -n clearledger | grep stage7-kyverno-lab </code></pre> Expected: no output. If it fails: stop and fix Stage 4 first <ul> <li>The command ends quietly with <code>created</code> (no error) </li> <li><code>kubectl get pods -n clearledger</code> shows <code>stage7-kyverno-lab</code> </li> </ul> That means Kyverno let a root pod through. Run <code>make check-4</code> before continuing Stage 7. Example of a passing terminal (yours may list more policies): <pre><code class="language-text">Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc" denied the request: policy disallow-root-containers/validate-run-as-non-root fail: Running as root is not allowed </code></pre> Confirm Prometheus saw it (optional but useful if Grafana is empty): <pre><code class="language-bash">kubectl exec -n monitoring deploy/kube-prometheus-stack-grafana -c grafana -- \ wget -qO- 'http://kube-prometheus-stack-prometheus.monitoring:9090/api/v1/query?query=kyverno_admission_requests_total{request_allowed="false"}' 2>/dev/null \ | grep -o '"value":\[[^]]*\]' | head -3 </code></pre> Expected: a <code>"value"</code> entry with a recent Unix timestamp and a number greater than 0 (for example <code>"value":[..., "1"]</code>). If you see this, Kyverno and Prometheus are working even when Grafana panels say No data. Grafana: open <a href="http://grafana.local/d/clearledger-kyverno-violations?from=now-15m&to=now">Kyverno Policy Violations</a>. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/7f5cf039-f2de-41d6-b71e-f5af2fc3ccab.png" alt="screenshot of Kyverno Policy Violations" style="display:block;margin:0 auto" width="1325" height="1288" loading="lazy"> What you're proving: the terminal denial showed up in Grafana. You don't need every panel to light up. You need one clear sign that Kyverno blocks are being counted. Step 1: quick sanity check (top row, left to right) <ol> <li>Policy Violations (time range): big number. Pass: shows 1 or more. Fail: says No data. </li> <li>Violations (time range): same idea, second counter. Pass: 1 or more. </li> <li>Active Kyverno Rules — usually 18. If this number shows up, Grafana can talk to Prometheus. That is good even when the first two panels are still empty. </li> </ol> Step 2: if the top two numbers work, skim the charts <ul> <li>Violation Rate by Resource Kind (middle chart): look for a bump labeled Pod around the time you ran <code>kubectl apply</code>. </li> <li>Top Blocked Resource Types (bottom-left table): look for a Pod row. </li> <li>Violations by Namespace (trend) (bottom-right chart): look for a bump for clearledger. </li> </ul> Charts can lag. A big number > 0 in Step 1 is enough to move on. The charts are bonus proof for §7.6 screenshots. If the top two panels say "No data" but the terminal denial worked: This is common. Those panels count new denials during the time range, not the total ever recorded. One denial sometimes lands in Prometheus before Grafana's counter moves. Try this: <ol> <li>Run the same <code>kubectl apply</code> command again (denied again, that is expected). </li> <li>Wait 60 seconds. </li> <li>Click Refresh (circular arrow, top-right). </li> </ol> After a second denial you should see 2 in the top stat panels. Screenshot that for §7.6. Still empty? Use Explore as backup proof: <ol> <li>Grafana left menu → Explore </li> <li>Datasource: Prometheus </li> <li>Paste: <code>sum(kyverno_admission_requests_total{request_allowed="false"})</code> </li> <li>Click Run query </li> </ol> Pass: the result is 1 or 2. A screenshot of the terminal denial plus Explore showing a number > 0 counts as portfolio proof even if the dashboard stats stay slow. <h4 id="heading-exercise-b-falco-shell-loki-security-event-timeline">Exercise B: Falco shell → Loki → Security Event Timeline</h4> What you're doing (same idea as Exercise A): <ul> <li>Exercise A: you did something bad, Kyverno blocked it, and the Grafana Kyverno dashboard updated. </li> <li>Exercise B: you do something suspicious inside a running pod, Falco detects it, and Grafana Security Event Timeline updates. </li> </ul> You already did this in Stage 6 (<code>make demo-6</code>). Here you do it again and prove the alert shows up in Grafana, not only in <code>http://falco.local</code>. The story in one line: pretend you're an attacker who got shell access inside <code>auth-service</code>: Falco should scream, and the scream should appear on the timeline dashboard. Step 1: trigger the alert (terminal way) You're pretending an attacker got into <code>auth-service</code> and ran a quick command (<code>id</code>) to see who they're logged in as. That's suspicious. Falco is supposed to catch it. The block below is three commands in order. Copy-paste the whole block: <pre><code class="language-bash">AUTH_POD=$(kubectl get pod -n clearledger -l app=auth-service \ --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}') echo "Using pod: $AUTH_POD" kubectl exec -n clearledger "$AUTH_POD" -c auth-service -- /bin/sh -c 'id && exit' </code></pre> What each line does: <ol> <li>Line 1: finds the name of a running <code>auth-service</code> pod and saves it in <code>AUTH_POD</code>. </li> <li>Line 2: prints that name so you can see it worked (not empty). </li> <li>Line 3: runs <code>/bin/sh -c 'id && exit'</code> inside that pod. This is the fake “attack.” Falco watches for shells like this. </li> </ol> Pass: you only need these two lines in the output: <pre><code class="language-text">Using pod: auth-service-77b7d9cd99-xxxxx uid=1000 gid=1000 groups=1000 </code></pre> <ul> <li>First line: a real pod name (not blank). </li> <li>Second line: the <code>id</code> command ran inside the container. </li> </ul> That's Step 1 done. The pod is still running. You didn't break anything. Fail: stop and fix before Step 2: <ul> <li><code>error: Internal error</code> or <code>container not found</code> </li> <li><code>Using pod:</code> with nothing after it </li> </ul> Run <code>kubectl get pods -n clearledger -l app=auth-service</code> and retry when one pod shows Running. Step 2: Confirm Falco saw it (terminal, right away) The Falco log is one long JSON line. Don't try to read the whole thing. Run: <pre><code class="language-bash">kubectl logs -n falco -l app.kubernetes.io/name=falco --tail=50 | grep -i 'Shell spawned' </code></pre> Pass. You should see one short phrase somewhere in the line: <pre><code class="language-text">Shell spawned in ClearLedger container ... pod=auth-service-... cmd=sh -c id && exit </code></pre> Or the rule name: <pre><code class="language-text">"rule":"Shell Spawned in ClearLedger Container" </code></pre> That one grep hit means Exercise B worked in the terminal. Screenshot this line for your portfolio. Ignore: <ul> <li><code>Defaulted container "falco" out of: ...</code>: normal kubectl noise </li> <li>Lines about <code>postgres-0</code> and <code>/etc/passwd</code>: background noise from Stage 6, not your test </li> <li>The rest of the JSON (<code>output_fields</code>, <code>k8smeta</code>, and so on). You don't need to parse it </li> </ul> If grep prints nothing: run Step 1 again, wait 5 seconds, then re-run the grep. Step 3: Confirm Loki stored it (wait ~60 seconds first) The story so far: <ul> <li>Step 1: you triggered the alert inside <code>auth-service</code> </li> <li>Step 2: Falco wrote the alert to its own logs ✓ </li> </ul> Step 3 asks: did that log line make it into Loki which is Grafana's log database? Falco doesn't talk to Grafana directly. Promtail copies Falco's logs into Loki. That copy takes 60–90 seconds. Wait after Step 1, then run this check. What this command does: "Search Loki for Falco logs that contain <code>Shell spawned</code>, then show only lines that also mention <code>auth-service</code>." <pre><code class="language-bash">kubectl exec -n monitoring loki-0 -- wget -qO- \ 'http://127.0.0.1:3100/loki/api/v1/query?query=%7Bnamespace%3D%22falco%22%2Ccontainer%3D%22falco%22%7D%20%7C%3D%20%22Shell%20spawned%22&limit=3' 2>/dev/null \ | grep -i 'auth-service' </code></pre> Pass: you see a line with both <code>auth-service</code> and <code>Shell spawned</code>. That means Loki has your alert and Grafana can show it. Fail (misleading pass): you grep for <code>ClearLedger</code> alone and get a hit from <code>postgres-0</code> reading <code>/etc/passwd</code>. That's background noise from Stage 6, not your shell test. Always look for <code>auth-service</code>. Empty output? That's OK. If Step 2 passed, continue to Step 4. Promtail may still be catching up, or the JSON is too long for this quick grep. Grafana often shows the alert even when this command prints nothing. Step 4: open Grafana Open <a href="http://grafana.local/d/clearledger-security-events?from=now-1h&to=now">Security Event Timeline</a>. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/c8f78ce4-8857-485b-9e4a-77c83bb95bd9.png" alt="screenshot of security timeline dashboard" style="display:block;margin:0 auto" width="1114" height="1024" loading="lazy"> This is the right dashboard. The title at the top should say ClearLedger - Security Event Timeline. Before you look at panels: <ol> <li>Time range: Last 1 hour (top-right) </li> <li>Auto-refresh: Off </li> <li>Re-run Step 1 if your shell command was more than a few minutes ago </li> <li>Wait 90 seconds, then click Refresh </li> </ol> What you'll probably see (and this is normal): <ul> <li>CRITICAL Alerts (1h): a big number like 1.08 K. That is mostly <code>postgres-0</code> reading <code>/etc/passwd</code> on a loop (Stage 6 background noise). It does not mean you failed. </li> <li>Alerts by Rule Name (pie chart): dominated by Sensitive File Read in ClearLedger. Also normal. </li> <li>Recent CRITICAL / WARNING Events: lots of Postgres rows. Your shell alert is in there, but buried. </li> </ul> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/577e14fa-205d-462a-beb2-7a6a08514295.png" alt="anothre screenshot showing security even timeline grafana dashboard" style="display:block;margin:0 auto" width="1060" height="1009" loading="lazy"> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/e3a363af-676b-47a6-a61a-fe5ee8b7c130.png" alt="e3a363af-676b-47a6-a61a-fe5ee8b7c130" style="display:block;margin:0 auto" width="1119" height="977" loading="lazy"> The top timeline (Falco Alerts by Priority - Timeline) may say No data. That's a known quirk. Don't panic, just use the log panel and browser search instead. How to find your alert (on this dashboard): <ol> <li>Stay on ClearLedger - Security Event Timeline — not Explore, not Tempo. </li> <li>Click inside Recent CRITICAL / WARNING Events (the log list on the right). </li> <li>Press Cmd+F (Mac) or Ctrl+F (Windows/Linux). </li> <li>Search for <code>auth-service</code> or <code>Shell spawned</code>. </li> </ol> If the search finds a row mentioning your pod and Shell spawned, screenshot it. Wrong place (common mistake): Grafana Explore with datasource Tempo showing <code>ledger-service</code> traces. That's Stage 7.5 (OpenTelemetry), not Exercise B. Tempo shows request traces, not Falco security alerts. Pass for Exercise B (pick one): <ol> <li>Best: Step 2 terminal grep shows <code>Shell spawned</code> and the Security Event Timeline log search finds <code>auth-service</code> / <code>Shell spawned</code> — screenshot both. </li> <li>Also fine: Step 2 grep screenshot plus the Security Event Timeline dashboard with CRITICAL Alerts (1h) showing a number (proves that Falco → Loki → Grafana works, even if your shell row is buried in postgres noise). </li> <li>Fallback (only if the dashboard search fails): Step 2 grep plus Grafana Explore with datasource Loki (not Tempo): <ul> <li>Left menu, go to Explore </li> <li>Top-left datasource dropdown: choose Loki </li> <li>Query: <code>{namespace="falco", container="falco"} |= "Shell spawned"</code> </li> <li>Click Run query </li> <li>Look for a line with <code>auth-service</code> </li> </ul> </li> </ol> Screenshot for §7.6. <h4 id="heading-exercise-c-failed-login-loki-and-service-health">Exercise C: Failed login, Loki, and Service Health</h4> The story: someone is guessing passwords on your login API. You send ten bad login attempts from the terminal. <code>auth-service</code> writes <code>Failed login attempt</code> to its logs. Grafana Service Health + Auth Security should show the count go up. Same pattern as A and B: terminal action, then logs, then dashboard. Step 1: send bad login attempts (terminal) Copy-paste the whole block: <pre><code class="language-bash">for i in $(seq 1 10); do curl -s http://clearledger.local/auth/health >/dev/null curl -s -X POST http://clearledger.local/auth/login \ -H 'Content-Type: application/json' \ -d '{"email":"lab-attacker@evil.com","password":"wrong"}' >/dev/null done echo "done" </code></pre> Pass: the only output you need is: <pre><code class="language-text">done </code></pre> No output from the <code>curl</code> lines is normal. The loop hits <code>/auth/health</code> (keeps the app warm) and <code>/auth/login</code> with a wrong password ten times. Fail: <code>curl: (6) Could not resolve host</code>. Run <code>bash scripts/setup-hosts.sh</code> on your Mac. <code>curl: (7) Failed to connect</code>. Check <code>kubectl get pods -n clearledger -l app=auth-service</code>. Step 2: Confirm auth-service logged it <pre><code class="language-bash">kubectl logs -n clearledger -l app=auth-service --tail=30 | grep -i 'Failed login' | tail -3 </code></pre> Pass. You should see lines like: <pre><code class="language-text">Failed login attempt for email: lab-attacker@evil.com </code></pre> You may see several lines (one per failed attempt). One line is enough. Screenshot this for your portfolio. If grep prints nothing: wait 10 seconds and run again. If still empty, check the auth pod is Running: <code>kubectl get pods -n clearledger -l app=auth-service</code>. Step 3: open Grafana (wait ~60 seconds after Step 1) Open <a href="http://grafana.local/d/clearledger-service-health?from=now-1h&to=now">Service Health + Auth Security</a>. This is the right dashboard. The title should say ClearLedger - Service Health + Auth Security. <ol> <li>Time range: Last 1 hour </li> <li>Auto-refresh: Off </li> <li>Click Refresh once </li> </ol> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/a6df00f0-3f4a-4951-ab5f-3efb498beaa1.png" alt="screenshot showing Service Health + Auth Security grafana dashboard" style="display:block;margin:0 auto" width="1118" height="1028" loading="lazy"> What to check (only these matter for Exercise C): <ol> <li>Failed Login Attempts (1h): big number. Pass: > 0. This is your main proof. </li> <li>Failed Login Log Stream: log lines in the panel. Pass: lines with <code>Failed login attempt</code> or <code>lab-attacker@evil.com</code>. Use Cmd+F inside the panel if needed. </li> </ol> Panels you can ignore if empty: <ul> <li>Successful Logins: fine at 0 (you only sent bad passwords) </li> <li>Request Rate by Service: may be empty until §7.5 metrics images. Not required for Exercise C. </li> </ul> You pass Exercise C when you have two screenshots: Screenshot 1 (required): your Step 2 terminal output showing <code>Failed login attempt for lab-attacker@evil.com</code>. This proves the app logged the bad logins. Screenshot 2 (pick one of these): <ul> <li>Option A: the Failed Login Attempts (1h) panel showing a number greater than zero (for example 10). This proves Grafana counted the failures. </li> <li>Option B: the Failed Login Log Stream panel showing a line with <code>lab-attacker@evil.com</code>. Use this if the big number panel is still empty but the log stream has your email. </li> </ul> You need Screenshot 1 and either Option A or Option B. That's enough for §7.6. <h4 id="heading-exercise-d-compliance-dashboard-the-auditor-summary">Exercise D: Compliance dashboard (the auditor summary)</h4> What you're doing: open one dashboard that rolls up Exercises A, B, and C. This is the “show the auditor” view: admission control + runtime detection + application security in one screen. When: only after you finished A, B, and C. Step 1: open the dashboard <a href="http://grafana.local/d/clearledger-compliance?from=now-1h&to=now">Compliance Posture</a> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/060714bc-c7c3-4d53-ad0e-d6d642970550.png" alt="screenshot of grafana Compliance Posture dashboard" style="display:block;margin:0 auto" width="1115" height="1132" loading="lazy"> Set Last 1 hour, auto-refresh Off, click Refresh. Step 2: Check the top row stats <table> <thead> <tr> <th>Stat on dashboard</th> <th>Came from</th> <th>Pass</th> </tr> </thead> <tbody><tr> <td>Policy Violations</td> <td>Exercise A (Kyverno)</td> <td>> 0</td> </tr> <tr> <td>Runtime Threats</td> <td>Exercise B (Falco)</td> <td>> 0 (postgres noise counts — that is OK)</td> </tr> <tr> <td>Failed Auth Attempts</td> <td>Exercise C (bad logins)</td> <td>> 0</td> </tr> </tbody></table> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/ba6d0d41-bc88-4d8b-a93b-a1b78f99f4da.png" alt="screenshot of grafana Compliance Posture dashboard" style="display:block;margin:0 auto" width="1068" height="1058" loading="lazy"> All three don't need to be huge numbers. They just need to be above zero after your tests. If one stat is still 0: re-run that exercise (A, B, or C), wait 90 seconds, refresh. Policy Violations may need a second Kyverno denial like Exercise A. This is screenshot #3 for §7.6: the single frame that proves defense-in-depth. ✋ Hands-on checkpoint: are you actually done with Stage 7? Installing Grafana isn't the goal. Detection is: you triggered real events and can see them on dashboards. Optional terminal check (proves Grafana is wired up): <pre><code class="language-bash">curl -s -u admin:admin123 'http://grafana.local/api/search?tag=clearledger' | jq -r '.[].title' curl -s -u admin:admin123 'http://grafana.local/api/datasources' | jq -r '.[].name' </code></pre> First command, you should see six dashboard names: <ul> <li>ClearLedger - Kyverno Policy Violations </li> <li>ClearLedger - Security Event Timeline </li> <li>ClearLedger - Service Health + Auth Security </li> <li>ClearLedger - Compliance Posture </li> <li>ClearLedger - Kubernetes Audit Log Analysis </li> <li>ClearLedger - DORA Metrics </li> </ul> Second command, you should see at least: <ul> <li>Prometheus </li> <li>Loki </li> </ul> What does NOT mean you're done: <code>make check-7</code> only checks that monitoring pods are running. Green output there does not replace §7.4. What DOES mean you are done: You ran Exercises A, B, and C in §7.4 and saved the §7.6 screenshots: <ol> <li>Kyverno denial (terminal + dashboard) </li> <li>Falco shell alert (terminal + Security Event Timeline) </li> <li>Failed logins (terminal + Service Health) </li> <li>Compliance Posture summary (all three stats above zero) </li> </ol> If you have those four screenshots, Stage 7 is complete. <h3 id="heading-75-fill-in-the-request-rate-chart-optional">7.5: Fill in the Request Rate Chart (Optional)</h3> This is not required for Stage 7. Exercises A–C and §7.6 screenshots don't need this section. Skip it if you're happy moving on. Also, this is not the same as Stage 7.5 (OpenTelemetry/Tempo). This subsection is only about the Request Rate by Service chart on the Service Health dashboard. <h4 id="heading-what-this-section-is-for">What this section is for:</h4> On Service Health + Auth Security, the Failed Login panels work from logs (Loki). The Request Rate by Service chart needs something different: app pods must expose a <code>/metrics</code> endpoint so Prometheus can scrape request counts. The code is already in the repo (<code>app/*/prom_metrics.py</code>). Prometheus is already configured to scrape it (<code>clearledger-podmonitor.yaml</code>). The usual problem: your cluster is still running older images from before that code was in your build. <h4 id="heading-step-1-check-if-you-already-have-metrics-30-seconds">Step 1: check if you already have metrics (30 seconds)</h4> Run this first. If it passes, skip the rest of §7.5. <pre><code class="language-bash">kubectl exec -n monitoring deploy/kube-prometheus-stack-grafana -c grafana -- \ wget -qO- 'http://kube-prometheus-stack-prometheus.monitoring:9090/api/v1/query?query=http_requests_total' 2>/dev/null \ | grep -o '"__name__":"http_requests_total"' | head -1 </code></pre> Pass: prints <code>"__name__":"http_requests_total"</code>. Open Service Health, refresh, and the Request Rate by Service chart should already have lines. No output: continue to Step 2. <h4 id="heading-step-2-deploy-images-that-expose-metrics">Step 2: deploy images that expose <code>/metrics</code></h4> Pick one path. Path A: GitOps (if you have been using CI/CD since Stage 1–2) <ol> <li>Push a commit to <code>main</code> on your app repo. </li> <li>Wait for CI to build new images and update <code>clearledger-infra</code>. </li> <li>Wait for ArgoCD to show Synced and Healthy on the clearledger app. </li> <li>Go to Step 3. </li> </ol> Path B: lab shortcut (faster, local only) <pre><code class="language-bash">export DOCKER_USERNAME=your-dockerhub-user bash stages/stage-7-observability/scripts/build-metrics-images.sh </code></pre> This builds, pushes, and rolls out metrics-enabled images for all three services. Heads-up: ArgoCD self-heal may revert these image tags within a few minutes if <code>clearledger-infra</code> still points at older tags. That's fine for a quick lab demo. For a lasting fix, use Path A or update the infra repo (see §2 rollback notes). <h4 id="heading-step-3-verify-metrics-landed-60-seconds-after-rollout">Step 3: verify metrics landed (~60 seconds after rollout)</h4> <pre><code class="language-bash">kubectl exec -n clearledger deploy/auth-service -c auth-service -- \ wget -qO- http://127.0.0.1:8000/metrics 2>/dev/null | head -5 </code></pre> Pass: lines starting with <code># HELP</code> or <code>http_requests_total</code>. Then confirm Prometheus sees them: <pre><code class="language-bash">kubectl exec -n monitoring deploy/kube-prometheus-stack-grafana -c grafana -- \ wget -qO- 'http://kube-prometheus-stack-prometheus.monitoring:9090/api/v1/query?query=http_requests_total' 2>/dev/null \ | grep -o '"__name__":"http_requests_total"' | head -1 </code></pre> Pass: <code>"__name__":"http_requests_total"</code> Generate a little traffic (re-run the Exercise C curl loop or hit <code>http://clearledger.local/auth/health</code> a few times), wait 60 seconds, then open Service Health + Auth Security and refresh. Request Rate by Service should show lines for <code>auth-service</code>, <code>ledger-service</code>, or <code>notification-service</code>. <h4 id="heading-when-to-stop">When to stop:</h4> <ul> <li>Request Rate still empty but Failed Login panels work? You're done with Stage 7. Request Rate is a nice-to-have. </li> <li>Prometheus query passes but chart empty? Widen time range to Last 1 hour, generate traffic, wait 60s, refresh. </li> </ul> <h3 id="heading-76-wrap-up-stage-7-screenshots-done-check">7.6: Wrap up Stage 7 (Screenshots + Done Check)</h3> You're almost done. This section is just about saving proof, then moving on. <h4 id="heading-are-you-actually-finished">Are you actually finished?</h4> Opening Grafana and seeing six dashboards isn't enough. <code>make check-7</code> passing isn't enough either. That only proves pods are running. You're done when you ran §7.4, waited for the panels to update, and saved three screenshots from your cluster. If the panels are empty or only show old Postgres noise, go back to §7.4 first. Before each screenshot: set time range to Last 15 minutes (or Last 1 hour for Exercise B). Include the time picker and panel titles in the frame. Screenshot 1: Falco alert (Exercise B) Open <a href="http://grafana.local/d/clearledger-security-events">Security Event Timeline</a>. Capture Recent CRITICAL / WARNING Events with a row that mentions <code>Shell spawned</code> or <code>auth-service</code>. If postgres rows bury it, use Cmd+F inside the log panel, that still counts. Screenshot 2: Kyverno denial (Exercise A) Open <a href="http://grafana.local/d/clearledger-kyverno-violations">Kyverno Policy Violations</a>. Capture Policy Violations (time range) or Violations (time range) showing a number of 1 or more. Screenshot 3: Compliance summary (Exercise D) Open <a href="http://grafana.local/d/clearledger-compliance">Compliance Posture</a>. Capture the top row with all three stats above zero: Policy Violations, Runtime Threats, and Failed Auth Attempts. Screenshot 4 (optional): Failed logins (Exercise C) Open <a href="http://grafana.local/d/clearledger-service-health">Service Health + Auth Security</a>. Capture Failed Login Attempts (1h) above zero, or Failed Login Log Stream showing <code>lab-attacker@evil.com</code>. Save files somewhere sensible, like <code>docs/evidence/stage-7-screenshot-1-falco.png</code>. Name them so you know what each proves. Final check: run <code>make check-7</code> (§7.7), save your VM, and you can claim Stage 7. <h3 id="heading-77-verify">7.7: Verify</h3> <pre><code class="language-bash">make check-7 </code></pre> Expected: <pre><code class="language-text">▶ Stage 7 — Observability (Grafana + Prometheus + Loki) ✓ Prometheus is running ✓ Grafana reachable (http://grafana.local or in-cluster health OK) ✓ Loki pod is running (0 restarts) ✓ Loki reachable from Grafana (http://loki:3100/ready) ✓ ClearLedger alerting rules exist ✓ ClearLedger dashboards imported (6 found) </code></pre> Warnings about Loki restarts or missing dashboards: fix with §7.1 before claiming Stage 7 complete. Save your VM after §7.6 and <code>make check-7</code>. See the block at the end of Stage 7 below. <h3 id="heading-78-what-broke-lab-notes-interview-talking-points">7.8: What Broke (Lab Notes + Interview Talking Points)</h3> The stack in one sentence: Prometheus stores numbers (metrics), Loki stores log lines, and Grafana displays both visually. Nothing appears until something actually happens in the cluster. <h4 id="heading-what-tripped-you-up-in-the-lab">What tripped you up in the lab</h4> <ol> <li>Empty dashboards right after install: Normal. Grafana doesn't create events. You trigger them in §7.4 (Kyverno denial, Falco shell, failed logins). </li> <li>Loki slow or refresh stuck on “Cancel”: Falco logs are huge. Last 24 hours overloads a small cluster. Use Last 1 hour, one dashboard at a time, and wait ~10 seconds. </li> <li><code>make check-7</code> passed but panels still empty (lab checklist only, not an interview topic): The health check confirms Prometheus/Loki/Grafana pods are up. It does not mean events exist. You still need §7.4 + §7.6 before you snapshot and move on. </li> </ol> <h4 id="heading-if-someone-asks-about-this-in-an-interview">If someone asks about this in an interview</h4> Empty dashboards? Grafana only shows what already happened. No event in the time range means an empty panel. That's normal until you trigger something. Loki slow on a small cluster? Falco logs are huge. We kept time ranges short (15 minutes, not 24 hours) and opened one dashboard at a time. Same trade-off you would make in prod on limited hardware. How did you prove it worked? I ran the attacks myself: denied a bad pod, spawned a shell in a running container, and sent failed logins. Then I checked Grafana and screenshot the matching panels. Terminal action first, dashboard proof second. Short version you can say out loud: <blockquote> "I connected Kyverno and Falco into Grafana. To prove it, I triggered a policy block and a runtime alert, then showed both on security dashboards. On a single-node lab, Loki got slow with wide time ranges, so we kept queries tight." </blockquote> Pipeline problems from earlier stages (Trivy, Kyverno, image tags, and so on) are in <code>docs/troubleshooting.md</code> — not something you need to rehearse for Stage 7. <h3 id="heading-79-if-panels-look-wrong-after-a-repo-update">7.9: If Panels Look Wrong After a Repo Update</h3> Re-apply dashboards, then generate real events (§7.4, not fake data): <pre><code class="language-bash">bash stages/stage-7-observability/scripts/install-observability.sh # Then run Exercises A–C from §7.4 (Kyverno denial, Falco shell, failed logins) </code></pre> Open Grafana at Last 1 hour, wait ~30–60s after each exercise, and refresh once. The §7.4 exercises cover expected appearance for each dashboard. <h3 id="heading-what-you-learned-in-stage-7">What You Learned in Stage 7</h3> <ul> <li>Prometheus proves countable security events (Kyverno denials, HTTP rates) </li> <li>Loki proves forensic detail (Falco JSON, auth log lines) </li> <li>Grafana is the narrative layer, not a second install step after the lab </li> <li>You can trace: terminal action, backend signal, panel update </li> <li>ServiceMonitors / PodMonitors are what connect Stages 4–6 to charts </li> <li>Empty dashboards mean “no events yet” or “wrong time range”, not “broken security” </li> <li>Compliance posture is how you answer an auditor in one screen </li> <li>Network policies must explicitly allow the <code>monitoring</code> namespace to reach app pods on port 8000, otherwise PodMonitor scrapes silently fail with <code>context deadline exceeded</code> </li> <li>Kubernetes Audit Log dashboard is empty on MicroK8s by design: the API server audit pipeline (audit-policy → file → Promtail → Loki) isn't enabled by default </li> <li>Request Rate requires the full chain: app image with <code>/metrics</code>, PodMonitor, and network policy: any one missing means the panel stays empty </li> </ul> What you can now put on your CV / say in an interview: <blockquote> Built security observability with Prometheus, Loki, and Grafana (dashboards correlating Kyverno violations, Falco alerts, and DORA metrics) and can prove a security event end-to-end from terminal to dashboard. </blockquote> <h4 id="heading-stage-7-done-checklist">Stage 7 done checklist:</h4> <ul> <li><code>make check-7</code> → 6/6 ✓ (Stage 6.5 Litmus failure is expected: scaled down for memory) </li> <li><code>http://grafana.local/d/clearledger-kyverno-violations</code>. Violations stat > 0 </li> <li><code>http://grafana.local/d/clearledger-security-events</code>. CRITICAL Falco alert visible </li> <li><code>http://grafana.local/d/clearledger-compliance</code>. Policy Violations + Runtime Threats + Failed Auth Attempts all > 0 </li> <li><code>http://grafana.local/d/clearledger-service-health</code>. Failed Login Attempts > 0. Request Rate > 0 only if you did §7.5 </li> <li>Portfolio screenshots 1–3 saved </li> </ul> <code>make snapshot STAGE=7 && make snapshots</code>. Confirm <code>clearledger.stage7</code>. Don't skip this. Stage 7 is heavy, and disk pressure is common. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>. After a Mac reboot or sleep, auth/ledger pods may show Unknown or Init:0/1 even though the cluster is up (see <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md) Mac reboot</a>). <h2 id="heading-stage-75-opentelemetry-optional">Stage 7.5 — OpenTelemetry (Optional)</h2> You can skip this whole stage. Stage 7 (metrics + logs) is enough to finish the homelab and move to Stage 8. Only do Stage 7.5 if you want distributed traces for your portfolio or interviews, and your VM has spare RAM (about 1.5 Gi free). <h3 id="heading-what-you-are-adding">What You Are Adding</h3> Stage 7 answers: did something happen? (Kyverno blocked a pod, Falco saw a shell, or login failed.) Traces answer: what steps ran on this one request, and how long did each take? <ul> <li>Metrics: how many requests, how many errors </li> <li>Logs: what the app printed in its log file, such as errors, warnings, login failures) </li> <li>Traces: ledger-service called auth-service (12ms), then Postgres (8ms) </li> </ul> In this stage, you send one real transaction, then open that request in Grafana Explore (Tempo). You'll see each step listed with its timing: ledger-service, auth-service, Postgres. <h3 id="heading-before-you-start">Before You Start</h3> <ol> <li>Finish Stage 7: §7.4 exercises done, §7.6 screenshots saved, <code>SKIP_CHAOS_CHECK=1 make check-7</code> passes. </li> <li>Check VM memory: <code>multipass exec clearledger -- free -h</code> , want about 1.5 Gi free. </li> <li>If you ran Stage 6.5 Litmus, scale it down first (§7.0). </li> </ol> You're done when you see the full request trace in Grafana Explore (Tempo datasource) and <code>make check-75</code> passes. Then <code>make snapshot STAGE=75</code>. <h3 id="heading-ignore-this-warning-in-app-logs">Ignore This Warning in App Logs</h3> Since Stage 7 you may see: <pre><code class="language-plaintext">WARNING: Transient error StatusCode.UNAVAILABLE encountered while exporting traces </code></pre> That is harmless. The apps are already set up to send trace data, but the receiver isn't installed until §7.5.3. Your apps still work fine, the trace data just gets thrown away. Installing the collector in §7.5.3 makes the warning go away. <h3 id="heading-how-tracing-is-wired">How Tracing is Wired</h3> <ol> <li>Your apps send trace data when a request runs </li> <li>OTel Collector receives it (port 4317) and passes it along </li> <li>Grafana Tempo stores it </li> <li>Grafana Explore (Tempo selected) is where you look at one request step by step </li> </ol> Apps talk to the collector only, not to Tempo directly. That way you can change where traces are stored later without rebuilding the apps. <h3 id="heading-751-check-memory-and-load">7.5.1: Check Memory and Load</h3> Tempo needs ~300MB. Confirm headroom before installing: <pre><code class="language-bash">multipass exec clearledger -- free -h # want ~1.5Gi available multipass exec clearledger -- uptime # load should be reasonable for your CPU count SKIP_CHAOS_CHECK=1 bash scripts/health-check.sh 7 </code></pre> If Litmus is still running from Stage 6.5, scale it down first (§7.0): <pre><code class="language-bash">kubectl get pods -n litmus --field-selector=status.phase=Running # Expected: no resources found </code></pre> <h3 id="heading-752-install-grafana-tempo">7.5.2: Install Grafana Tempo</h3> Tempo is the trace storage backend. Install it into the <code>monitoring</code> namespace next to Prometheus and Loki: <pre><code class="language-bash">helm repo add grafana https://grafana.github.io/helm-charts helm repo update helm install tempo grafana/tempo \ --namespace monitoring \ --set tempo.storage.trace.backend=local \ --set tempo.storage.trace.local.path=/var/tempo \ --set persistence.enabled=true \ --set persistence.size=5Gi \ --wait </code></pre> Verify Tempo is running: <pre><code class="language-bash">kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo # Expected: tempo-0 1/1 Running </code></pre> <pre><code class="language-bash">kubectl exec -n monitoring tempo-0 -- wget -qO- http://localhost:3200/ready # Expected: ready </code></pre> <h3 id="heading-753-deploy-otel-collector-and-wire-grafana">7.5.3: Deploy OTel Collector and Wire Grafana</h3> This applies the OTel Collector (receives spans from app pods) and registers Tempo as a Grafana datasource automatically via the sidecar: <pre><code class="language-bash">kubectl apply -f stages/stage-7.5-opentelemetry/infra/otel/otel-collector.yaml kubectl apply -f stages/stage-7.5-opentelemetry/infra/otel/grafana-datasource-tempo.yaml </code></pre> Verify the collector is running: <pre><code class="language-bash">kubectl get pods -n monitoring -l app=otel-collector # Expected: otel-collector-xxxxx 1/1 Running </code></pre> Verify the collector started (not trace receipt yet): Apps push spans to the collector over OTLP: the collector doesn't scrape pods. At this step you're only confirming that it's listening. <pre><code class="language-bash">kubectl logs -n monitoring deploy/otel-collector --tail=15 # Expected: # Starting GRPC server ... endpoint: 0.0.0.0:4317 # Starting HTTP server ... endpoint: 0.0.0.0:4318 # Everything is ready. Begin running and processing data. # No crash loops or repeated errors. </code></pre> Proof that traces are actually flowing comes later: after you generate traffic in §7.5.6, check collector logs for span export lines from the <code>debug</code> exporter, then confirm the trace in Grafana Tempo (§7.5.7). <h3 id="heading-754-enable-prometheus-remote-write-receiver">7.5.4: Enable Prometheus Remote Write Receiver</h3> The OTel Collector also forwards OTel metrics to Prometheus via remote write. Prometheus needs to accept them: <pre><code class="language-bash">helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack \ --namespace monitoring \ -f stages/stage-7-observability/infra/helm/kube-prometheus-stack-values.yaml \ --wait </code></pre> This applies the <code>enableRemoteWriteReceiver: true</code> setting added to the Helm values in Stage 7.5. Wait for Prometheus to restart (about 60 seconds). <h3 id="heading-755-verify-app-pods-connect-to-the-collector">7.5.5: Verify App Pods Connect to the Collector</h3> The deployments in <code>clearledger-infra</code> already have <code>OTEL_EXPORTER_OTLP_ENDPOINT</code> set. Once the collector is running, the pods auto-connect. Confirm that the OTEL warnings are gone: <pre><code class="language-bash">kubectl logs -n clearledger deploy/ledger-service -c ledger-service --tail=20 2>/dev/null \ | grep -v "opentelemetry\|otlp\|Transient" | tail -10 # Expected: only INFO request logs, no WARNING: Transient error </code></pre> If warnings persist, the network policy may not have port 4317 egress. Apply the latest policies: <pre><code class="language-bash">kubectl apply -f infra/deferred-by-stage/stage-6-runtime-security/netpol/network-policies.yaml </code></pre> <h3 id="heading-756-generate-a-trace">7.5.6: Generate a Trace</h3> Now create a transaction and watch it flow through the system: <pre><code class="language-bash"># Step 1: register (skip if already registered) curl -s -X POST http://clearledger.local/auth/register \ -H "Content-Type: application/json" \ -d '{"email":"trace-demo@clearledger.io","password":"TracePass123"}' | python3 -m json.tool # Step 2: login and grab the token TOKEN=$(curl -s -X POST http://clearledger.local/auth/login \ -H "Content-Type: application/json" \ -d '{"email":"trace-demo@clearledger.io","password":"TracePass123"}' \ | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])") echo "Token acquired: ${TOKEN:0:20}..." # Step 3: create a transaction (this is the request you will trace) curl -s -X POST http://clearledger.local/ledger/transactions \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"amount": 5000, "direction": "credit"}' | python3 -m json.tool </code></pre> Verify the collector received spans: <pre><code class="language-bash">kubectl logs -n monitoring deploy/otel-collector --tail=30 \ | grep -iE "Traces|spans|ResourceSpans" || echo "No span lines yet — see §7.5.5 (OTEL env / netpol)" # Expected after a successful transaction: debug exporter lines mentioning exported traces/spans </code></pre> <h3 id="heading-757-view-the-trace-in-grafana">7.5.7: View the Trace in Grafana</h3> Open <a href="http://grafana.local">http://grafana.local</a> and go to the left sidebar Explore (compass icon). <h4 id="heading-step-1-select-tempo-and-open-search">Step 1: Select Tempo and open Search</h4> At the top of the query pane: <ol> <li>Datasource dropdown (orange T logo) → Tempo </li> <li>Query row labeled A (Tempo) → three tabs: Search | TraceQL | Service Graph </li> <li>Click Search. This shows dropdown filters. TraceQL is a text box only. If you land there with nothing typed you get <code>0 series returned</code>. </li> </ol> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/b005b7ed-ad35-4c2f-8ca4-7d685717754f.png" alt="screenshot of grafana showing tempo and ledger service" style="display:block;margin:0 auto" width="1158" height="408" loading="lazy"> <h4 id="heading-step-2-filter-by-service">Step 2: Filter by service</h4> In the Search tab: <ul> <li>Service Name → type or select <code>ledger-service</code> </li> <li>Leave Span Name, Status, Duration, and Tags empty for now </li> <li>Grafana shows the query it will run: <code>{resource.service.name="ledger-service"}</code> </li> </ul> Set the time range (top-right clock icon) to Last 15 minutes so your §7.5.6 transaction is included. <h4 id="heading-step-3-run-the-query">Step 3: Run the query</h4> Grafana Explore has no “Run query” button: results appear automatically after selecting a service. If the table stays empty, use the blue refresh button top-right of the pane. <h4 id="heading-step-4-open-the-trace-waterfall">Step 4: Open the trace waterfall</h4> Below the query editor, find Table - Traces. You should see at least one row like: <table> <thead> <tr> <th>Column</th> <th>Example</th> </tr> </thead> <tbody><tr> <td>Trace ID</td> <td><code>5730edf3…</code> (blue link)</td> </tr> <tr> <td>Start time</td> <td>when you ran the <code>curl</code></td> </tr> <tr> <td>Service</td> <td><code>ledger-service</code></td> </tr> <tr> <td>Name</td> <td><code>POST /transactions</code></td> </tr> <tr> <td>Duration</td> <td>~200ms (yours may differ)</td> </tr> </tbody></table> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/ac4aad25-4ca4-4ffc-9fb6-25c30f0214bd.png" alt="screenshot of grafana showing tempo and ledger service and query result" style="display:block;margin:0 auto" width="1179" height="1055" loading="lazy"> Click the Trace ID link. The right panel opens the trace detail view. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/d3c15dc0-e4b3-49e4-93e6-28b49feab6eb.png" alt="screenshot of grafana showing tempo and ledger service and query results" style="display:block;margin:0 auto" width="1226" height="1289" loading="lazy"> <h4 id="heading-what-the-trace-detail-view-shows">What the trace detail view shows</h4> Header: <code>ledger-service: POST /transactions</code> <ul> <li>Trace ID: unique ID for this request </li> <li>Duration: total end-to-end time </li> <li>Services: <code>2</code> (<code>ledger-service</code> and <code>auth-service</code> for a normal transaction) </li> </ul> Expand spans in the timeline: <pre><code class="language-plaintext">ledger-service POST /transactions (~total duration) ├── auth-service GET /verify ← JWT check over HTTP ├── ledger-service INSERT / sqlalchemy ← Postgres write └── (optional) redis PUBLISH ← only if amount ≥ notification threshold </code></pre> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/0b69babc-90c6-4cdc-a0bf-bad012854a36.jpg" alt="trace transaction flow" style="display:block;margin:0 auto" width="1536" height="957" loading="lazy"> Reading the trace detail screen: <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/3a72b4a4-2243-403d-9f03-ebc3b22e5337.png" alt="Tempo trace detail: ledger-service transaction with auth-service verify step." style="display:block;margin:0 auto" width="1226" height="1289" loading="lazy"> Each row is one step in the request (Grafana calls it a span). The colored bar on the right shows how long that step took. That's the span bar. A longer bar = more time spent on that step. Click a row or its bar to open the details panel on the right. You'll see two kinds of metadata: <ul> <li>Span attributes: what happened in this step. Examples: HTTP method (<code>POST</code>, <code>GET</code>), status code (<code>200</code>), or SQL text on a database step. In your trace you might see <code>asgi.event.type: http.request</code> on the FastAPI receive step. </li> <li>Resource attributes: where the step ran. Examples: <code>service.name: ledger-service</code>, <code>k8s.cluster.name: clearledger</code>, <code>deployment.environment: production</code>. </li> </ul> Quick mental model: span attributes = what the step did. Resource attributes = which service produced it. Connecting traces to logs: once you have a step selected, the Logs tab will take you straight to the matching Loki log lines for that pod at the same moment in time. Screenshot this trace detail view: portfolio proof for Stage 7.5. <h4 id="heading-traceql-alternative">TraceQL alternative</h4> If you prefer the text box, Switch to the TraceQL tab, and paste: <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/679d687d-1a2e-474b-8f7e-672f3ab2eb2b.png" alt="screenshot showing direction for where traceql button is" style="display:block;margin:0 auto" width="1157" height="263" loading="lazy"> <pre><code class="language-traceql">{ resource.service.name = "ledger-service" } </code></pre> <h4 id="heading-if-the-table-is-empty">If the table is empty</h4> If TraceQL says <code>0 series returned</code>: Use the Search tab instead, or paste the TraceQL query from above into the TraceQL tab. If search tab has no rows: Widen the time range to Last 15 minutes, re-run the transaction curl from §7.5.6, wait a few seconds, and refresh. If grafana can't connect to Tempo: The datasource URL needs port 3200. Re-apply the datasource and restart Grafana: <pre><code class="language-bash">kubectl apply -f stages/stage-7.5-opentelemetry/infra/otel/grafana-datasource-tempo.yaml kubectl rollout restart deployment/kube-prometheus-stack-grafana -n monitoring </code></pre> If collector logs show no trace data: Work through §7.5.5: usually the OTEL environment variables or network policy blocking port 4317. <h3 id="heading-757b-understand-when-a-trace-happens">7.5.7b: Understand When a Trace Happens</h3> You ran one curl command in §7.5.6. Grafana shows every place that single request traveled. Think of it like tracking a package: <ol> <li>You sent <code>POST /transactions</code> to ledger-service </li> <li>ledger-service asked auth-service: "is this user logged in?" </li> <li>ledger-service saved the row to the database </li> <li>redis only runs if the amount is big (10,000 or more) </li> </ol> Each of those is a row you see in the Tempo detail screen. You're not looking at four separate requests. It's one request with multiple stops. Why do I see both ledger-service and auth-service? Because ledger had to call auth before it could save the transaction. Grafana groups those stops into one trip so you can see the full path, not just the first hop. Why did my demo have no Redis row? You used <code>"amount": 5000</code>. The app only talks to Redis when the amount is 10,000 or higher. So seeing ledger + auth + database but no Redis is correct. Want to see Redis? Run §7.5.6 again with <code>"amount": 15000</code> and search Tempo again. Optional: connect it to the code Open <code>app/ledger-service/main.py</code>, find <code>create_transaction</code>, and read top to bottom. The Tempo rows follow that function in order: check the user, save to the database, and maybe notify Redis. Optional: same request, three tools At the time you ran the curl: <ul> <li>Tempo (this stage): which services ran and how long each took </li> <li>Loki (Stage 7): what the apps wrote in their log files </li> <li>Prometheus (Stage 7): how many requests happened around that time </li> </ul> Same moment, three different views. You already used Loki and Prometheus in Stage 7. <h3 id="heading-758-verify">7.5.8: Verify</h3> <pre><code class="language-bash">make check-75 </code></pre> Expected output: <pre><code class="language-text">▶ Stage 7.5 — OpenTelemetry (Distributed Tracing) ✓ OTel Collector is running (1 replica(s)) ✓ Grafana Tempo datasource ConfigMap exists ✓ Tempo is running ✓ auth-service has OTEL_EXPORTER_OTLP_ENDPOINT set </code></pre> If you see a warning instead: <pre><code class="language-text">⚠ OTel env vars not found on auth-service, redeploy with updated manifests </code></pre> <code>check-75</code> looks for <code>OTEL_EXPORTER_OTLP_ENDPOINT</code> in the deployment manifest. Older Stage 5 manifests may not list it even though tracing works: the Python apps default to <code>http://otel-collector.monitoring.svc.cluster.local:4317</code> when the env var is missing. You can proceed if collector logs show spans and Tempo shows your trace. To clear the warning, apply only the app deployments (not the whole kustomize tree: Kyverno may block redis/postgres patches): <pre><code class="language-bash">kubectl apply -f infra/manifests/auth-service/deployment.yaml kubectl apply -f infra/manifests/ledger-service/deployment.yaml kubectl rollout restart deployment/auth-service deployment/ledger-service -n clearledger make check-75 </code></pre> Save your VM after <code>make check-75</code>. See the block at the end of Stage 7.5 below. <h3 id="heading-what-you-learned">What You Learned</h3> Stage 7 gave you metrics (how busy?) and logs (what was printed?). Stage 7.5 adds traces (for one slow request, which step took the time?). In the lab you proved it with one <code>POST /transactions</code> curl. In production the idea is the same: a user hits an API, the request crosses multiple services, and you need to see that full path in one place. <h4 id="heading-if-someone-asks-in-an-interview">If someone asks in an interview:</h4> Why traces at all? Metrics might tell you p99 latency doubled. Logs might show an error on one pod. Traces tell you which downstream call in the chain caused the delay, auth, database, cache, or third-party API, without guessing. How did you implement it? We instrumented the services with OpenTelemetry, sent telemetry to a collector, and stored traces in Grafana Tempo. Apps talk to the collector, not directly to the backend, so we can change storage later without redeploying every service. What would you do in an incident? Find a slow or failing trace ID (from logs, metrics, or an alert), open it in Tempo, walk the call chain service by service, see where time stacked up, then jump to logs for that service at the same timestamp. That's faster than tailing logs on five pods and hoping they line up. Short version you can say out loud: <blockquote> "We use the three pillars together: Prometheus for rates and errors, Loki for log detail, and Tempo for request-level debugging across microservices. When latency spikes, I start from a trace, identify the slow hop, often a database or downstream API, and correlate back to logs and metrics for that service." </blockquote> <code>make check-75 && make snapshot STAGE=75 && make snapshots</code>. Confirm <code>clearledger.stage75</code>. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>. <h2 id="heading-stage-8-aws-migration">Stage 8 — AWS Migration</h2> Your goal here is to run the same ClearLedger app on AWS instead of your laptop VM. You're not rewriting the application. Stages 0–7 built containers on Kubernetes with GitOps, Kyverno, secrets, and observability. Stage 8 changes where it runs. You keep the same images, the same ArgoCD workflow, and the same security policies. Only the cloud services underneath change (MicroK8s → EKS, Vault → Secrets Manager, and so on). <ul> <li>Homelab: MicroK8s, Postgres in a pod, dev Vault, Docker Hub, <code>clearledger.local</code> </li> <li>AWS: EKS, RDS, Secrets Manager, ECR, ALB hostname </li> </ul> Am I ready for Stage 8? <ul> <li>Homelab complete through Stage 7 (Stage 7.5 optional) </li> <li>make check-7 passes (and make check-75 if you did traces) </li> <li>AWS account with billing alerts enabled. make aws-up creates billable resources </li> <li>Skim §8.2 so you know what make aws-up does (even if you use the quick path) </li> </ul> Done when the app is reachable on the AWS ALB, ArgoCD syncing, and you run <code>make aws-down</code> when finished to stop charges. <h3 id="heading-what-make-aws-up-gives-you">What <code>make aws-up</code> Gives You</h3> This is a demo stack: production-shaped, but not production-ready. It has HTTP only (no TLS cert). Stage 7 observability is installed automatically. CI still runs Gitleaks, Semgrep, Checkov, Trivy, and Cosign. For real production, you would add HTTPS (see <a href="https://github.com/Osomudeya/clearledger/blob/main/stages/stage-8-aws-migration/manifests/ingress-aws-https.example.yaml"><code>ingress-aws-https.example.yaml</code></a>), staging before promote, and alert routing. Those are documented but not applied by the spinup script. GitOps rule: after bootstrap, don't <code>kubectl apply</code> app Deployments by hand. ArgoCD owns the cluster (Stage 2). Push manifest changes to Git and let ArgoCD sync. <h3 id="heading-secrets-on-aws">Secrets on AWS</h3> On the homelab, Vault wrote secret files into the pod. On AWS, secrets live in AWS Secrets Manager (created by Terraform). Your app still needs them as environment variables like <code>DATABASE_URL</code>. ESO (default in this lab): the simple mental model: <ol> <li>Terraform stores the real password in AWS Secrets Manager (for example <code>clearledger/auth-service</code>) </li> <li>External Secrets Operator (ESO) watches that AWS secret </li> <li>ESO copies it into a normal Kubernetes Secret inside the cluster (for example <code>auth-service-secret</code>) </li> <li>Your deployment reads <code>DATABASE_URL</code> from that Kubernetes Secret, same as Stage 0, but the values come from AWS instead of a YAML file in Git </li> </ol> You never put passwords in Git. ESO keeps the Kubernetes Secret in sync with Secrets Manager. CSI (optional, §8.5 exercise): same AWS secrets but different delivery: mounted as files at <code>/mnt/secrets/*</code> instead of env vars. This is closer to how Vault worked on the homelab. IRSA: how ESO is allowed to read Secrets Manager without storing AWS access keys in the cluster. AWS trusts a Kubernetes service account instead. IRSA lets AWS trust a Kubernetes ServiceAccount, no <code>AWS_ACCESS_KEY_ID</code> in Git or in the cluster. Details here: <a href="https://github.com/Osomudeya/clearledger/tree/main/stages/stage-8-aws-migration/docs"><code>stages/stage-8-aws-migration/docs/secrets-patterns.md</code></a>. <h3 id="heading-81-two-ways-through-stage-8">8.1: Two Ways Through Stage 8</h3> Quick path (~45–60 min): edit <code>terraform/secrets.tf</code> (replace <code>CHANGE_ME_BEFORE_APPLY</code>), then: <pre><code class="language-bash">make aws-up # runs stages/stage-8-aws-migration/scripts/aws-spinup.sh make aws-down # destroys billable resources when you are done </code></pre> Read §8.2 afterward so you know what ran. Manual path (§8.3): run Terraform, ECR push, ArgoCD, Kyverno, ESO, and deploy yourself. Use this when learning, interviewing, or debugging a failed spinup. Don't skip §8.2–§8.5 if you only ran <code>make aws-up</code>. Otherwise you won't know what Terraform, ESO, or ArgoCD each did. Before your first Stage 8 push, read <a href="#heading-ci-routing-stages-17-vs-stage-8">§8: CI routing and <code>CLEARLEDGER_CI_TARGET</code></a> and set <code>CLEARLEDGER_CI_TARGET=aws</code> only after Terraform succeeds, not while you are still on Stages 1–7. <h3 id="heading-82-what-make-aws-up-runs">8.2: What <code>make aws-up</code> Runs</h3> The spinup script runs 15 steps in order: Setup (1–6): Check tools and AWS login; <code>terraform apply</code> (VPC, EKS, RDS, ECR, Secrets Manager, GuardDuty, CloudTrail, IAM), confirm security services, build and push images to ECR, patch <code>manifests/kustomization.yaml</code> with your registry and git SHA, and configure <code>kubectl</code> for EKS. Platform (7–12): install ArgoCD; Kyverno + cluster policies, Falco, External Secrets Operator + IRSA service accounts, CSI secrets driver, and Stage 7 observability stack. Deploy (13–15): ArgoCD app <code>clearledger-aws</code> syncs <code>stages/stage-8-aws-migration/manifests/</code>, wait for ALB hostname, and print URL and tear-down reminder. After the script finishes, open the printed <code>http://<alb-dns>/</code> in your browser (ClearLedger login UI), or follow <a href="#heading-when-to-open-what-checkpoint-map">§8.3. When to open what</a> for Argo CD and Grafana port-forwards. Default app deploy uses ESO for secrets. CSI is also installed so you can try file mounts in §8.5 without extra setup. Terraform layout: there's no <code>terraform.tf</code> file. The <code>terraform {}</code> block (version, providers, optional S3 backend) is at the top of <code>main.tf</code>. Resources are split by topic: <code>vpc.tf</code>, <code>eks.tf</code>, <code>rds.tf</code>, <code>ecr.tf</code>, <code>alb.tf</code>, <code>iam.tf</code>, <code>secrets.tf</code>, <code>security.tf</code>. Run all commands from <code>stages/stage-8-aws-migration/terraform/</code>. <h3 id="heading-83-manual-walkthrough">8.3: Manual Walkthrough</h3> Go to Before you start in this section and run the manual steps from Step A yourself at least once instead of <code>make aws-up</code>. Paths are from the repo root. Commands install things, while UIs prove they work. Homelab Stages 2 and 7 already taught you to open Argo CD and Grafana in a browser. Stage 8 is the same idea. But on AWS there's no <code>clearledger.local</code> or <code>grafana.local</code> in <code>/etc/hosts</code>. You use port-forward for control-plane UIs and the public ALB hostname for the app. <h4 id="heading-when-to-open-what-checkpoint-map">When to open what (checkpoint map)</h4> <code>make aws-up</code> runs fifteen steps. You don't need every UI open at once, just know when to look and what success looks like as the script moves along. First, Terraform builds the AWS foundation. When step 2 finishes, open the AWS Console and confirm the cluster, registry, and database exist before any pods run: EKS <code>clearledger</code> is Active, ECR has four repos including <code>frontend</code> (empty is fine for now), and RDS <code>clearledger-postgres</code> is Available. See <a href="#heading-aws-console-after-step-2">AWS Console (after step 2)</a> for the walkthrough. Next come container images. After step 4, or after CI — AWS (ECR + OIDC) goes green in GitHub Actions, check ECR: each repo should list your git SHA tag. That's what ArgoCD will pull when the app deploys. Around step 7 the script installs Argo CD. Port-forward to the UI and confirm the login page loads. You won't see the app yet. You're only checking that GitOps is reachable. Details: <a href="#heading-step-13-watch-argocd-sync-ui-cli">Argo CD UI</a>. Step 12 adds observability. Port-forward to Grafana, log in, and confirm the six ClearLedger dashboards are listed. Panels can stay empty until you generate events. This is the same as Stage 7 on the homelab. Step 13 applies the <code>clearledger-aws</code> app. Go back to Argo CD → Applications → <code>clearledger-aws</code>. You want Synced, Healthy, and running pods for auth, ledger, and notification. Step 14 exposes the app on a public URL. Open <code>http://<alb-dns>/</code> in your browser: you should see the same ClearLedger login UI as homelab <code>clearledger.local</code>, served from the ALB with no <code>/etc/hosts</code> entry. Use <code>/auth/health</code> and the other health URLs when you want a quick API check from the terminal. See <a href="#heading-step-15-open-the-app-in-your-browser">ALB — first time the app is public</a>. If you want extra confirmation, the optional check is EC2 → Load Balancers → <code>clearledger</code>: status Active, with healthy targets for frontend and the API services. On AWS the app is four services behind one ALB: the frontend at <code>/</code> (login, dashboard, transactions) and the three APIs at <code>/auth</code>, <code>/ledger</code>, and <code>/notifications</code>. Your portfolio screenshot for Stage 8 is the ALB URL showing the UI, like <code>http://clearledger-xxxxxxxxxx.eu-west-1.elb.amazonaws.com</code> with the ClearLedger login or dashboard visible. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/411a7ae0-8e7e-4a19-9267-e207f78ece93.png" alt="screenshot of clearledger ui with ALB Url" style="display:block;margin:0 auto" width="1348" height="364" loading="lazy"> For Argo CD and Grafana, keep a dedicated terminal running <code>kubectl port-forward</code> while the browser tab is open. <code>Ctrl+C</code> closes the tunnel. <h4 id="heading-before-you-start">Before you start</h4> Step A: set real passwords in <code>secrets.tf</code> Open <code>stages/stage-8-aws-migration/terraform/secrets.tf</code> and search for the literal text <code>CHANGE_ME_BEFORE_APPLY</code>. It appears four times in the file (Postgres password, JWT secret, and two database URLs). Replace every occurrence: <ul> <li>Postgres password: pick a strong password (same value in all three places that reference it) </li> <li>JWT secret: run <code>openssl rand -base64 64</code> and paste the output </li> </ul> <code>make aws-up</code> will refuse to run if any <code>CHANGE_ME_BEFORE_APPLY</code> text is still in that file. Step B: terminal checks <pre><code class="language-bash">aws sts get-caller-identity terraform --version # REQUIRED before first terraform apply; GitHub Actions OIDC (ci-aws.yaml) reads this at apply time: cp stages/stage-8-aws-migration/terraform/terraform.tfvars.example \ stages/stage-8-aws-migration/terraform/terraform.tfvars # Edit terraform.tfvars: github_owner = "YOUR_GITHUB_USERNAME" # your GitHub user or org, not a placeholder terraform -chdir=stages/stage-8-aws-migration/terraform validate # Fails with "Set github_owner in terraform.tfvars" until you replace YOUR_GITHUB_USERNAME </code></pre> Don't run <code>terraform apply</code> until <code>github_owner</code> is set. If you apply with the placeholder, AWS creates IAM role <code>clearledger-github-actions-ecr</code> with trust <code>repo:YOUR_GITHUB_USERNAME/...</code>. CI then fails at Publish images → ECR with <code>Not authorized to perform sts:AssumeRoleWithWebIdentity</code>. Fix: edit <code>terraform.tfvars</code> → <code>terraform apply</code> again → verify with <code>aws iam get-role</code> below then Re-run failed jobs on the failed Actions run (not the full pipeline). <h4 id="heading-steps-12-terraform">Steps 1–2: Terraform</h4> <pre><code class="language-bash">cd stages/stage-8-aws-migration/terraform terraform init -upgrade terraform apply # Save outputs: terraform output -raw ecr_registry_url terraform output -raw github_actions_ecr_role_arn terraform output -raw eso_role_arn terraform output -raw auth_service_irsa_role_arn terraform output -raw kubeconfig_command cd ../../.. </code></pre> <h4 id="heading-aws-console-after-step-2">AWS Console after step 2.</h4> Confirm Terraform created resources before you touch the cluster: <ol> <li>EKS → Clusters → <code>clearledger</code> → Status: Active, 3 nodes </li> <li>ECR → Repositories → <code>clearledger/auth-service</code>, <code>ledger-service</code>, <code>notification-service</code>, <code>frontend</code> (0 images until step 4 or CI) </li> <li>RDS → Databases → <code>clearledger-postgres</code> → Available </li> </ol> Verify GitHub can push to ECR (only if you plan to use AWS CI later) GitHub Actions needs permission to push images to your AWS account. Terraform creates an IAM role for that, but only if you set your real GitHub username in <code>terraform.tfvars</code> before <code>terraform apply</code>. Check it worked: <pre><code class="language-bash">aws iam get-role --role-name clearledger-github-actions-ecr \ --query 'Role.AssumeRolePolicyDocument.Statement[0].Condition.StringEquals."token.actions.githubusercontent.com:sub"' \ --output text </code></pre> Good: <code>repo:your-real-username/clearledger:environment:production</code> Bad: <code>repo:YOUR_GITHUB_USERNAME/clearledger:...</code> you forgot to edit <code>terraform.tfvars</code>. Fix the file, run <code>terraform apply</code> again, then in GitHub go to Actions and then the failed CI, AWS (ECR + OIDC) run → click Re-run failed jobs. That retries only the push step. You don't need to rebuild and rescan everything. Skip this whole block if you are only using <code>make aws-up</code> for now and not enabling AWS CI yet. When do ECR repos appear? During <code>terraform apply</code> (step 2), not when you <code>docker push</code>. Terraform creates empty image repositories: <code>clearledger/auth-service</code>, <code>ledger-service</code>, <code>notification-service</code>, and <code>frontend</code>, so seeing 0 images right after apply is normal. Images land later in step 4 (manual <code>docker push</code>) or when GitHub Actions CI succeeds. Set your AWS CLI region to <code>eu-west-1</code> Everything in this lab lives in eu-west-1 (Ireland). If your CLI defaults to <code>us-east-1</code>, commands will say resources are missing even though they exist: <pre><code class="language-bash">aws configure set region eu-west-1 aws configure get region # expect: eu-west-1 </code></pre> <h4 id="heading-steps-34-security-services-ecr-images">Steps 3–4: Security services + ECR images</h4> <pre><code class="language-bash">AWS_REGION=eu-west-1 # or rely on aws configure set region above # Step 3: verify security services (must pass --region eu-west-1) aws guardduty list-detectors --region "${AWS_REGION}" # Expect: DetectorIds: ["<id>"] — empty [] means wrong region, not "not created" aws cloudtrail get-trail-status --name clearledger-trail --region "${AWS_REGION}" # Expect: IsLogging: true # Error "Unknown trail ... us-east-1" → you forgot --region eu-west-1 # Step 4: build and push images to the ECR repos Terraform already created ECR_REGISTRY=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw ecr_registry_url) AUTH_ECR=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw auth_service_ecr_url) LEDGER_ECR=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw ledger_service_ecr_url) NOTIFY_ECR=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw notification_service_ecr_url) TAG=$(git rev-parse --short HEAD) aws ecr get-login-password --region "${AWS_REGION}" \ | docker login --username AWS --password-stdin "${ECR_REGISTRY}" docker build -t "${AUTH_ECR}:${TAG}" app/auth-service && docker push "${AUTH_ECR}:${TAG}" docker build -t "${LEDGER_ECR}:${TAG}" app/ledger-service && docker push "${LEDGER_ECR}:${TAG}" docker build -t "${NOTIFY_ECR}:${TAG}" app/notification-service && docker push "${NOTIFY_ECR}:${TAG}" # Confirm images landed (optional) aws ecr describe-images --repository-name clearledger/auth-service --region "${AWS_REGION}" \ --query 'imageDetails[*].imageTags' --output table </code></pre> ECR console (after step 4 or green CI): open each repository and go. tothe Images tab. You should see tags matching your git commit SHA. If repos are empty, ArgoCD will show <code>ImagePullBackOff</code> later. GitHub Actions (if using CI instead of manual push): repo → Actions → workflow CI. AWS (ECR + OIDC). If all jobs are green, publish the images. ECR succeeded. This is the supply-chain proof before deploy. <h4 id="heading-step-5-gitops-source-of-truth">Step 5: GitOps source of truth</h4> Patch placeholders in <code>kustomization.yaml</code> (same <code>sed</code> as <code>aws-spinup.sh</code> step 5): <pre><code class="language-bash">AWS_REGION=eu-west-1 ECR_REGISTRY=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw ecr_registry_url) TAG=$(git rev-parse --short HEAD) KUST=stages/stage-8-aws-migration/manifests/kustomization.yaml sed -i.bak \ -e "s|REPLACE_ECR_REGISTRY|${ECR_REGISTRY}|g" \ -e "s|REPLACE_IMAGE_TAG|${TAG}|g" \ "${KUST}" rm -f "${KUST}.bak" # Region in ESO + CSI manifests (only if not eu-west-1) if [[ "${AWS_REGION}" != "eu-west-1" ]]; then sed -i.bak "s|region: eu-west-1|region: ${AWS_REGION}|g" \ stages/stage-8-aws-migration/manifests/external-secrets.yaml \ stages/stage-8-aws-migration/manifests/csi/auth-service-spc.yaml \ stages/stage-8-aws-migration/manifests/csi/ledger-service-spc.yaml rm -f stages/stage-8-aws-migration/manifests/external-secrets.yaml.bak \ stages/stage-8-aws-migration/manifests/csi/*.bak 2>/dev/null || true fi # Verify before commit grep -E 'newName:|newTag:' "${KUST}" # Expect: YOUR_AWS_ACCOUNT.dkr.ecr.eu-west-1.amazonaws.com/clearledger/... and your git SHA git add stages/stage-8-aws-migration/manifests/kustomization.yaml git commit -m "stage8: ECR images ${TAG}" git push </code></pre> Also fix the ArgoCD Application repo URL once (replace with your GitHub username): <pre><code class="language-bash"># Example: YOUR_GITHUB_USERNAME/clearledger — check: git remote get-url origin sed -i.bak 's|YOUR_GITHUB_USERNAME|YOUR_ACTUAL_GITHUB_USER|g' \ stages/stage-8-aws-migration/argocd/clearledger-aws-app.yaml rm -f stages/stage-8-aws-migration/argocd/clearledger-aws-app.yaml.bak </code></pre> <h4 id="heading-step-6-cluster-access-terraform-outputs">Step 6: Cluster access + Terraform outputs</h4> Run from the repo root. Set the CLI region first (EKS and IAM outputs are regional), then kubeconfig, then export IRSA role ARNs: steps 9–10 need them. <pre><code class="language-bash">aws configure set region eu-west-1 eval "$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw kubeconfig_command)" kubectl get nodes export AWS_REGION=eu-west-1 export ESO_ROLE_ARN=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw eso_role_arn) export FALCO_ROLE_ARN=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw falco_role_arn) export REPLACE_AUTH_IRSA_ROLE_ARN=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw auth_service_irsa_role_arn) export REPLACE_LEDGER_IRSA_ROLE_ARN=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw ledger_service_irsa_role_arn) export REPLACE_NOTIFICATION_IRSA_ROLE_ARN=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw notification_service_irsa_role_arn) # Sanity check (all should print ARNs, not empty) echo "ESO: ${ESO_ROLE_ARN}" echo "Falco: ${FALCO_ROLE_ARN}" echo "Auth IRSA: ${REPLACE_AUTH_IRSA_ROLE_ARN}" </code></pre> <h4 id="heading-steps-712-platform-stack-on-the-cluster">Steps 7–12: Platform stack on the cluster</h4> You finished steps 1–6 (AWS exists, images in ECR, <code>kubectl</code> works). Now for steps 7-12 you'll install the platform stack, the same components as <code>aws-spinup.sh</code>, but you run the commands from the sections below, not the script. For each step, run the Install code block, then run the Verify block right under it. Don't move to the next step until you see Running pods (or a ClusterPolicy list). “Command finished with no output” isn't enough. <table> <thead> <tr> <th>Step</th> <th>Namespace</th> <th>What you are installing</th> <th>Rough pod count</th> </tr> </thead> <tbody><tr> <td>7</td> <td><code>argocd</code></td> <td>GitOps controller</td> <td>~7 pods</td> </tr> <tr> <td>8</td> <td><code>kyverno</code></td> <td>Admission policies</td> <td>~4 pods + ClusterPolicies</td> </tr> <tr> <td>9</td> <td><code>falco</code></td> <td>Runtime detection</td> <td>1 DaemonSet pod per node (3 on this cluster)</td> </tr> <tr> <td>10</td> <td><code>external-secrets</code> + <code>clearledger</code></td> <td>ESO + IRSA ServiceAccounts</td> <td>~3 ESO pods + 3 ServiceAccounts</td> </tr> <tr> <td>11</td> <td><code>kube-system</code> + <code>clearledger</code></td> <td>CSI driver + AWS provider</td> <td>3 driver + 3 provider (one per node)</td> </tr> <tr> <td>12</td> <td><code>monitoring</code></td> <td>Prometheus, Grafana, Loki</td> <td>~10+ pods</td> </tr> </tbody></table> Steps 13–15 (deploy app, wait for ALB, verify UI) come after step 12 below. <h4 id="heading-step-7-argocd">Step 7: ArgoCD</h4> <pre><code class="language-bash">kubectl create namespace argocd --dry-run=client -o yaml | kubectl apply -f - kubectl apply -n argocd --server-side --force-conflicts \ -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml kubectl rollout status deployment/argocd-server -n argocd --timeout=180s </code></pre> Verify what got created: <pre><code class="language-bash">kubectl get pods -n argocd kubectl get svc -n argocd kubectl get deploy -n argocd </code></pre> Expected: <code>argocd-server</code>, <code>argocd-repo-server</code>, <code>argocd-application-controller</code>, and so on: most pods Running 1/1 or 2/2. <code>argocd-server</code> Service exposes port 443. UI (optional now, required after step 13): new terminal, leave running. Use any free local port (<code>8081</code> if <code>8080</code> is in use): <pre><code class="language-bash">kubectl port-forward svc/argocd-server -n argocd 8080:443 # Or if 8080 is taken: # kubectl port-forward svc/argocd-server -n argocd 8081:443 # https://localhost:8080 (or 8081) user: admin kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath='{.data.password}' | base64 -d; echo </code></pre> Applications list is empty until step 13. That's normal. <h4 id="heading-step-8-kyverno-policies">Step 8: Kyverno + policies</h4> <code>cosign.pub</code> / <code>infra/cosign.pub</code> are gitignored (private key must never commit; public key is learner-specific). The repo ships example keys in <code>require-signed-images.yaml</code> / <code>require-signed-images-ecr.yaml</code>. If you regenerated keys in Stage 3, sync your local public key into policies before apply: <pre><code class="language-bash"># infra/cosign.pub exists locally but is gitignored — safe to copy into committed policy YAMLs bash scripts/embed-cosign-pub-in-policies.sh diff infra/cosign.pub <(grep -A3 'BEGIN PUBLIC KEY' infra/policies/require-signed-images-ecr.yaml | grep -v publicKeys) </code></pre> <pre><code class="language-bash">helm repo add kyverno https://kyverno.github.io/kyverno/ --force-update helm upgrade --install kyverno kyverno/kyverno \ --namespace kyverno --create-namespace \ -f stages/stage-4-admission-control/infra/kyverno/values.yaml \ --set admissionController.replicas=1 \ --wait --timeout=180s kubectl apply -f infra/policies/ </code></pre> Verify: <pre><code class="language-bash">kubectl get pods -n kyverno kubectl get clusterpolicy kubectl get clusterpolicy require-signed-images-ecr -o jsonpath='{.spec.rules[0].verifyImages[0].attestors[0].entries[0].keys.publicKeys}' | head -3 </code></pre> Expected: admission-controller, background-controller, cleanup-controller, reports-controller pods Running. <code>kubectl get clusterpolicy</code> lists 6+ policies including <code>require-signed-images-ecr</code>, <code>disallow-root-containers</code>, and so on. The <code>publicKeys</code> output must show <code>-----BEGIN PUBLIC KEY-----</code>, not <code>PASTE_YOUR_COSIGN_PUBLIC_KEY_HERE</code> (Kyverno treats a placeholder as a file path and blocks all deploys). <code>require-signed-images-ecr</code> defaults to Audit until CI Cosign-signs ECR images (<code>COSIGN_PRIVATE_KEY</code> + <code>COSIGN_PASSWORD</code> in GitHub). Unsigned images still deploy. Signed-image enforcement is optional later. If <code>verify-slsa-provenance</code> fails to apply (Audit + <code>mutateDigest</code>), set <code>mutateDigest: false</code> in that file, or skip it. It's optional for Stage 8. <h4 id="heading-step-9-falco">Step 9: Falco</h4> <pre><code class="language-bash">helm repo add falcosecurity https://falcosecurity.github.io/charts --force-update helm upgrade --install falco falcosecurity/falco \ --namespace falco --create-namespace \ -f stages/stage-6-runtime-security/infra/falco/helm-values.yaml \ --set driver.kind=modern_ebpf \ --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${FALCO_ROLE_ARN}" \ --wait --timeout=300s </code></pre> Verify: <pre><code class="language-bash">kubectl get pods -n falco -o wide kubectl get daemonset -n falco kubectl get sa falco -n falco -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}'; echo </code></pre> Expected: Falco DaemonSet with DESIRED = number of nodes (3). Each pod Running. ServiceAccount annotation shows your <code>FALCO_ROLE_ARN</code>. <h4 id="heading-step-10-external-secrets-operator-irsa-serviceaccounts">Step 10: External Secrets Operator + IRSA ServiceAccounts</h4> <pre><code class="language-bash">helm repo add external-secrets https://charts.external-secrets.io --force-update helm upgrade --install external-secrets external-secrets/external-secrets \ --namespace external-secrets --create-namespace \ --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${ESO_ROLE_ARN}" \ --wait --timeout=180s kubectl apply -f stages/stage-8-aws-migration/manifests/resources/namespace.yaml envsubst < stages/stage-8-aws-migration/manifests/clearledger-serviceaccounts.yaml | kubectl apply -f - </code></pre> Verify: <pre><code class="language-bash">kubectl get pods -n external-secrets kubectl get sa -n external-secrets external-secrets -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}'; echo kubectl get sa -n clearledger </code></pre> Expected: <code>external-secrets</code> deployment Running (often 3 containers / 1 pod). Three ServiceAccounts in <code>clearledger</code>: <code>auth-service</code>, <code>ledger-service</code>, <code>notification-service</code>: each with an <code>eks.amazonaws.com/role-arn</code> annotation. No app pods yet (ArgoCD deploys those in step 13). Step 11: CSI driver + SecretProviderClasses <pre><code class="language-bash">bash stages/stage-8-aws-migration/scripts/install-csi-secrets.sh </code></pre> Verify: <pre><code class="language-bash">kubectl get pods -n kube-system | grep -E 'secrets-store|provider-aws' kubectl get secretproviderclass -n clearledger helm list -n kube-system | grep -E 'csi-secrets|secrets-provider' </code></pre> Expected: CSI driver pods 3/3 Running (one per node). AWS provider pods 1/1 Running per node. Two <code>SecretProviderClass</code> objects in <code>clearledger</code>. Helm shows <code>csi-secrets-store</code> and/or <code>secrets-provider-aws</code> deployed. If Helm reports <code>meta.helm.sh/release-name</code> conflicts, re-run the script. It installs the AWS provider without duplicating the driver chart. <h4 id="heading-step-12-observability">Step 12: Observability</h4> <pre><code class="language-bash">bash stages/stage-7-observability/scripts/install-observability.sh </code></pre> Verify: <pre><code class="language-bash">kubectl get pods -n monitoring kubectl get svc -n monitoring | grep -E 'grafana|prometheus|loki' kubectl get configmap -n monitoring -l grafana_dashboard=1 --no-headers | wc -l </code></pre> Expected: Grafana 3/3 Running, Prometheus and Loki pods Running. ConfigMap count for dashboards is 6 (ClearLedger dashboards). Script prints <code>http://grafana.local</code>: on EKS use port-forward instead: <pre><code class="language-bash"># New terminal — keep running kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80 # http://localhost:3000 admin / admin123 # http://localhost:3000/dashboards?tag=clearledger </code></pre> Panels may show No data until you trigger events (§7.4 exercises work on this cluster too). Platform stack summary: quick sanity check before step 13: <pre><code class="language-bash">for ns in argocd kyverno falco external-secrets monitoring clearledger; do echo "=== ${ns} ===" kubectl get pods -n "${ns}" --no-headers 2>/dev/null | awk '{print $3}' | sort | uniq -c || echo "(no pods yet)" done kubectl get clusterpolicy --no-headers | wc -l | xargs echo "ClusterPolicies:" kubectl get secretproviderclass -n clearledger --no-headers | wc -l | xargs echo "SecretProviderClasses:" </code></pre> Expected: every namespace shows only <code>Running</code> (or <code>Completed</code> for jobs). <code>clearledger</code> may be empty until ArgoCD syncs. ClusterPolicies ≥ 6. SecretProviderClasses = 2. EKS API timeout on namespace create? You may see <code>Unexpected error when reading response body</code> / <code>context deadline exceeded</code> and still get <code>namespace/argocd created</code>. That's a transient client timeout talking to the EKS API (first request, slow network, or control plane catching up), not a failed create. Confirm with <code>kubectl get namespace argocd</code> and continue. If commands keep timing out, retry once or run <code>kubectl cluster-info</code> to verify connectivity. <h4 id="heading-steps-1314-deploy-via-argocd-see-the-alb">Steps 13–14: Deploy via ArgoCD + see the ALB</h4> The app YAMLs under <code>stages/stage-8-aws-migration/manifests/</code> aren't applied by hand. Step 13 tells Argo CD to sync Git. Argo CD then creates Deployments, Services, Ingress, and the rest. Repo access first If your GitHub repo is private, add a PAT in Argo CD → Settings → Repositories. If you made the repo public, refresh the app, <code>ComparisonError: authentication required</code> should clear. If sync still fails, check the usual causes: <ul> <li><code>external-secrets.io/v1beta1</code> not found, your cluster has a newer ESO API. Push <code>external-secrets.yaml</code> with <code>apiVersion: external-secrets.io/v1</code>. </li> <li>Kyverno complains about <code>PASTE_YOUR_COSIGN_PUBLIC_KEY_HERE</code> , run <code>bash scripts/embed-cosign-pub-in-policies.sh</code>, then <code>kubectl apply -f infra/policies/</code>. </li> <li><code>SecretSyncedError</code> on auth, <code>database_url</code> or <code>jwt_secret</code> not found — the AWS secret <code>clearledger/auth-service</code> must contain both keys (Terraform writes them in <code>secrets.tf</code>). Re-run <code>terraform apply</code> after fixing <code>CHANGE_ME_BEFORE_APPLY</code> values, or check the secret in the AWS console. </li> <li>Pods stuck <code>Pending</code> or “too many pods” the lab nodes are small. Scale the node group in Terraform or lower replica counts in the manifests. </li> </ul> Register the app: <pre><code class="language-bash">kubectl apply -f stages/stage-8-aws-migration/argocd/clearledger-aws-app.yaml </code></pre> Watch Argo CD until <code>clearledger-aws</code> is Synced and Healthy. That's when app pods appear in <code>clearledger</code>. <h4 id="heading-step-13-watch-argocd-sync-ui-cli">Step 13: Watch ArgoCD sync (UI + CLI)</h4> Open the Argo CD browser tab you kept open (port-forward from step 7). <pre><code class="language-plaintext">https://localhost:8080 ← or 8081 if 8080 was busy </code></pre> Click <code>clearledger-aws</code>. Wait for Healthy + Synced (2–5 minutes on first deploy). You can watch the same info from the terminal without touching the browser: <pre><code class="language-bash">kubectl get application clearledger-aws -n argocd -w # Ctrl-C when HEALTH STATUS shows Healthy </code></pre> While that's settling, watch pods start up in a second terminal: <pre><code class="language-bash">kubectl get pods -n clearledger -w # All pods should reach 1/1 Running within 2 minutes # Ctrl-C when everything is Running </code></pre> <h4 id="heading-step-14-get-your-public-app-url-alb">Step 14: Get your public app URL (ALB)</h4> AWS takes 2–5 minutes after ArgoCD syncs to provision the load balancer. Run this and wait until the ADDRESS column fills in: <pre><code class="language-bash">kubectl get ingress clearledger-ingress -n clearledger -w # ADDRESS is empty at first, then shows something like: # clearledger-xxxxxxxxxx.eu-west-1.elb.amazonaws.com # Ctrl-C once the hostname appears </code></pre> Export the URL for the steps below: <pre><code class="language-bash">export ALB_DNS=$(kubectl get ingress clearledger-ingress -n clearledger \ -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') echo "Your app is live at: http://${ALB_DNS}" </code></pre> Still empty after 10 minutes? See <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md</a> for ALB/ingress recovery steps. <h4 id="heading-step-15-open-the-app-in-your-browser">Step 15: Open the app in your browser</h4> Paste the ALB root URL into your browser. No DNS entry, port-forward, or VPN: <pre><code class="language-plaintext">http://clearledger-xxxxxxxxxx.eu-west-1.elb.amazonaws.com/ </code></pre> You should see the ClearLedger login screen (same SPA as homelab <code>clearledger.local</code>). Register or log in, submit a transaction, and confirm the dashboard loads. That's your Stage 8 portfolio screenshot. Quick API health checks (terminal or browser): <pre><code class="language-bash">curl -fsS "http://${ALB_DNS}/auth/health" && echo curl -fsS "http://${ALB_DNS}/ledger/health" && echo curl -fsS "http://${ALB_DNS}/notifications/health" && echo </code></pre> Each should return JSON like <code>{"status":"ok","service":"auth-service"}</code>. <h4 id="heading-step-16-verify-in-the-aws-console-optional-but-recommended">Step 16: Verify in the AWS Console (optional but recommended)</h4> This is what the deployed stack looks like from AWS side: <table> <thead> <tr> <th>Console location</th> <th>What to look for</th> </tr> </thead> <tbody><tr> <td>EC2 → Load Balancers</td> <td>A load balancer named <code>clearledger-…</code> with state Active</td> </tr> <tr> <td>EC2 → Target Groups</td> <td>Two or three target groups, all targets showing healthy</td> </tr> <tr> <td>ECR → Repositories</td> <td><code>clearledger/auth-service</code>, <code>clearledger/ledger-service</code>, <code>clearledger/notification-service</code>, <code>clearledger/frontend</code> — each with a recently pushed image tag</td> </tr> <tr> <td>EKS → Clusters → clearledger → Workloads</td> <td>Your pods shown as Running in the <code>clearledger</code> namespace</td> </tr> <tr> <td>Secrets Manager</td> <td><code>clearledger/auth-service</code>, <code>clearledger/ledger-service</code>, <code>clearledger/postgres</code> — all present</td> </tr> </tbody></table> 502/503 from the ALB? The load balancer is up but the pods aren't healthy yet, or the secrets haven't synced. Check: <code>kubectl get pods -n clearledger</code> (all <code>1/1 Running</code>?) and <code>kubectl get externalsecret -n clearledger</code> (both <code>SecretSynced True</code>?). ✋ Hands-on checkpoint: app is publicly reachable <pre><code class="language-bash"># All three must print {"status":"ok",...} curl -fsS "http://${ALB_DNS}/auth/health" && echo curl -fsS "http://${ALB_DNS}/ledger/health" && echo curl -fsS "http://${ALB_DNS}/notifications/health" && echo # All pods Running kubectl get pods -n clearledger # Nothing printed here = all pods Running (non-Running pods would show) kubectl get pods -n clearledger --field-selector=status.phase!=Running </code></pre> <code>ImagePullBackOff</code> in the pod list means ECR images aren't there yet. Check GitHub Actions and re-run the workflow. A <code>502</code> from the health URL means the pod isn't ready yet. Wait 30 seconds and retry. <h3 id="heading-84-verify-eso-default-secret-path">8.4: Verify ESO (Default Secret Path)</h3> After Argo CD syncs, confirm External Secrets Operator copied values from AWS Secrets Manager into normal Kubernetes Secrets: <pre><code class="language-bash">kubectl get externalsecret,secret -n clearledger kubectl describe externalsecret auth-service-secret -n clearledger | grep -A6 "Conditions:" kubectl get pods -n clearledger -l app=auth-service kubectl exec -n clearledger deploy/auth-service -c auth-service -- env | grep DATABASE_URL </code></pre> <h4 id="heading-command-1-externalsecrets-secrets">Command 1: ExternalSecrets + Secrets</h4> You should see two ExternalSecrets and two matching Secrets (auth has 2 keys, ledger has 1): <pre><code class="language-plaintext">NAME STORE REFRESH INTERVAL STATUS READY externalsecret.external-secrets.io/auth-service-secret aws-secrets-manager 1h SecretSynced True externalsecret.external-secrets.io/ledger-service-secret aws-secrets-manager 1h SecretSynced True NAME TYPE DATA AGE secret/auth-service-secret Opaque 2 3m secret/ledger-service-secret Opaque 1 3m </code></pre> <code>STATUS</code> must be SecretSynced and READY must be True. If you see <code>SecretSyncedError</code>, stop here and fix IRSA before §8.5. <h4 id="heading-command-2-describe-auth-externalsecret">Command 2: describe auth ExternalSecret</h4> Look for <code>Reason: SecretSynced</code> and <code>Status: True</code>: <pre><code class="language-plaintext"> Conditions: Last Transition Time: 2026-07-10T22:15:00Z Message: Secret was synced Reason: SecretSynced Status: True Type: Ready </code></pre> <h4 id="heading-command-3-auth-pods-running">Command 3: auth pods running</h4> <pre><code class="language-plaintext">NAME READY STATUS RESTARTS AGE auth-service-xxxxxxxxxx-xxxxx 1/1 Running 0 2m auth-service-xxxxxxxxxx-xxxxx 1/1 Running 0 2m </code></pre> Both replicas 1/1 Running. If pods are <code>CrashLoopBackOff</code> or <code>CreateContainerConfigError</code>, the K8s Secret may be missing or empty. <h4 id="heading-command-4-databaseurl-is-an-env-var-eso-path-not-a-file-path">Command 4: DATABASE_URL is an env var (ESO path), not a file path</h4> <pre><code class="language-plaintext">DATABASE_URL=postgresql://clearledger:*****@clearledger-postgres.xxxxx.eu-west-1.rds.amazonaws.com:5432/clearledger </code></pre> Good: a <code>postgresql://...</code> connection string (password shown as <code>*****</code> or your real password). Bad for this section: <code>/mnt/secrets/database_url</code> that means CSI file mounts (§8.5), not the default ESO env-var path. You can also spot-check the secret exists without printing values: <pre><code class="language-bash">kubectl get secret auth-service-secret -n clearledger -o jsonpath='{.data}' | grep -o 'database_url\|jwt_secret' # Expect: database_url and jwt_secret (two keys) </code></pre> If <code>SecretSynced=False</code>, check ESO logs and IRSA: <pre><code class="language-bash">kubectl logs -n external-secrets deploy/external-secrets -c external-secrets | tail -30 kubectl get sa auth-service -n clearledger -o yaml | grep role-arn </code></pre> ✋ Hands-on checkpoint. External Secrets actually synced from AWS <pre><code class="language-bash">kubectl get externalsecret -n clearledger kubectl get secret -n clearledger </code></pre> Expected: <code>auth-service-secret</code> and <code>ledger-service-secret</code> each show <code>SecretSynced</code> / Ready <code>True</code>. The matching Kubernetes Secrets exist in <code>clearledger</code>. A <code>SecretSyncedError</code> means IRSA/IAM can't reach Secrets Manager: fix the role binding before §8.5. If you skip this, §8.5 (CSI driver) builds on working secret access, and a silent IAM failure here surfaces as an unrelated-looking pod error two sections later. <h3 id="heading-85-hands-on-csi-driver-file-mounts">8.5: Hands-on, CSI Driver (File Mounts)</h3> The default pods already use ESO: secrets arrive as environment variables from a Kubernetes Secret object. This exercise switches <code>auth-service</code> to the CSI path instead: secrets are mounted as plain files under <code>/mnt/secrets/</code>, and the app reads them from disk. It's the same code path the homelab uses with Vault (<code>DATABASE_URL_FILE</code> / <code>JWT_SECRET_FILE</code>). CSI was already installed at spinup step 11, so there's nothing extra to install. <h4 id="heading-step-1-confirm-csi-is-running">Step 1: Confirm CSI is running</h4> <pre><code class="language-bash">kubectl get pods -n kube-system -l app=secrets-store-csi-driver kubectl get secretproviderclass -n clearledger </code></pre> You should see one CSI driver pod per node, and two <code>SecretProviderClass</code> objects: one for auth-service and one for ledger-service. <h4 id="heading-step-2-swap-the-deployment-in-git">Step 2: swap the deployment in Git</h4> Open <code>stages/stage-8-aws-migration/manifests/kustomization.yaml</code> and change one line: <pre><code class="language-yaml"># Before - deployments/auth-service.yaml # After - deployments/auth-service-csi.yaml </code></pre> Commit and push, then sync: <pre><code class="language-bash">argocd app sync clearledger-aws kubectl rollout status deployment/auth-service -n clearledger </code></pre> ArgoCD will roll out a new auth-service pod with the CSI volume attached. <h4 id="heading-step-3-confirm-the-files-are-there">Step 3: Confirm the files are there</h4> <pre><code class="language-bash"># Find the new pod kubectl get pod -n clearledger -l secrets=csi # List the mounted secret files kubectl exec -n clearledger deploy/auth-service -- ls /mnt/secrets # Check the database URL was written correctly kubectl exec -n clearledger deploy/auth-service -- cat /mnt/secrets/database_url # Confirm the service is still healthy curl -s "http://$(kubectl get ingress clearledger-ingress -n clearledger \ -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')/auth/health" </code></pre> You should see <code>database_url</code> and <code>jwt_secret</code> listed as files, and the health check should return <code>{"status":"ok"}</code>. ESO vs CSI: what actually changed? Both paths read the same passwords from AWS Secrets Manager. Only the delivery method changes. ESO (default, what you verified in §8.4) Think of ESO as a copy clerk that runs in the cluster: <ol> <li>ESO has its own AWS permission (IAM role). </li> <li>It reads <code>clearledger/auth-service</code> from Secrets Manager. </li> <li>It copies the values into a normal Kubernetes Secret named <code>auth-service-secret</code>. </li> <li>The auth pod reads <code>DATABASE_URL</code> and <code>JWT_SECRET</code> as environment variables. </li> </ol> The password lives briefly inside the cluster as a Kubernetes Secret object. CSI (this exercise, file mounts) Think of CSI as the pod picking up secrets itself when it starts: <ol> <li>The auth-service pod has its own AWS permission (IRSA on its ServiceAccount). </li> <li>When the pod starts, the CSI driver asks Secrets Manager for the values. </li> <li>They appear as files under <code>/mnt/secrets/</code> (<code>database_url</code>, <code>jwt_secret</code>). </li> <li>The pod is told <code>DATABASE_URL_FILE=/mnt/secrets/database_url</code> , it reads from disk, not from a copied K8s Secret. </li> </ol> No Kubernetes Secret copy is created for those values on this path. Why does the same app code work for both? <code>app/auth-service/main.py</code> uses a small helper <code>_read_secret()</code>: <ul> <li>If <code>DATABASE_URL_FILE</code> points to a file that exists → read the file (CSI or homelab Vault). </li> <li>Otherwise → read <code>DATABASE_URL</code> directly (ESO / Stage 0–4). </li> </ul> Same image, same code: you only change which deployment YAML Argo CD syncs. To switch back to ESO: in <code>kustomization.yaml</code>, change <code>auth-service-csi.yaml</code> back to <code>auth-service.yaml</code>, commit, push, and <code>argocd app sync clearledger-aws</code>. Terraform provisions all the AWS resources (VPC, EKS, RDS, ECR, Secrets Manager, and IAM roles) from <code>.tf</code> files in <code>stages/stage-8-aws-migration/terraform/</code>. <h3 id="heading-two-oidc-ideas-in-stage-8">Two OIDC Ideas in Stage 8</h3> Stage 8 uses OIDC in two different places. They sound similar, but they solve different problems. GitHub Actions OIDC lets the CI pipeline push images to ECR without storing long-lived AWS keys in GitHub. When a job runs, GitHub mints a short-lived token that proves the job's identity. AWS trusts that token and hands back temporary credentials: enough to push images and nothing else. IRSA does the same thing, but for pods running inside EKS. Instead of a GitHub token, the pod presents its Kubernetes ServiceAccount token. AWS trusts the EKS cluster's OIDC provider, verifies the token, and returns temporary credentials scoped to exactly what that pod needs. It helps to see what each one says: <pre><code class="language-text">GitHub Actions OIDC: Pipeline says → "I am a job in the production environment of YOUR_USERNAME/clearledger" AWS replies → "Here are credentials to push to ECR, valid for one hour" IRSA: Pod says → "I am the auth-service ServiceAccount in the clearledger namespace" AWS replies → "Here are credentials to read only the auth-service secret, valid for one hour" </code></pre> <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/ac83e9bf-dbfd-42b3-bba8-0bbf327b03c5.png" alt="Image flow diagram showing difference between OIDC and IRSA" style="display:block;margin:0 auto" width="1536" height="1024" loading="lazy"> The key is what is not stored anywhere: <pre><code class="language-text">No AWS_ACCESS_KEY_ID in GitHub Secrets No AWS_SECRET_ACCESS_KEY in GitHub Secrets No AWS keys inside Kubernetes Secrets </code></pre> Terraform creates the role <code>clearledger-github-actions-ecr</code> and wires up the trust policies for both. The pipeline in <code>.github/workflows/ci-aws.yaml</code> assumes that role, pushes images to ECR, and updates <code>kustomization.yaml</code>. ArgoCD picks up the change and deploys the new images. <h3 id="heading-ci-routing-stages-17-vs-stage-8">CI Routing: Stages 1–7 vs Stage 8</h3> The repo ships two workflow files. You don't need both running at the same time. <code>ci.yaml</code> is the homelab pipeline from Stages 1–7. It runs on your self-hosted Multipass VM, pushes images to Docker Hub, and updates your <code>clearledger-infra</code> GitOps repo. This is the default: nothing to configure. <code>ci-aws.yaml</code> is the AWS pipeline for Stage 8. It runs on GitHub-hosted <code>ubuntu-latest</code> runners, pushes images to ECR, and updates <code>kustomization.yaml</code> directly in this repo. It only activates when you set the repo variable <code>CLEARLEDGER_CI_TARGET=aws</code>. If you're on Stages 1–7, do nothing. The <code>CLEARLEDGER_CI_TARGET</code> variable is unset by default, so every push runs <code>ci.yaml</code> on your self-hosted runner as normal. The AWS workflow file exists in the repo but its jobs are skipped. Do not set <code>CLEARLEDGER_CI_TARGET=aws</code> until your EKS cluster is running. If you set it early, <code>ci.yaml</code> stops running on push (no more Docker Hub builds), and <code>ci-aws.yaml</code> will fail immediately because there's no ECR, no OIDC role, and no AWS infrastructure yet. If you accidentally set it, delete the variable: GitHub → repo Settings → Secrets and variables → Actions → Variables → delete <code>CLEARLEDGER_CI_TARGET</code>. Enabling AWS CI (do this after <code>terraform apply</code> completes) You need three repository variables and one secret in a <code>production</code> environment. First, set the variables: replace <code>YOUR_USERNAME</code> with your GitHub username: <pre><code class="language-bash">gh variable set CLEARLEDGER_CI_TARGET --body aws --repo YOUR_USERNAME/clearledger gh variable set AWS_ACCOUNT_ID --body "$(aws sts get-caller-identity --query Account --output text)" --repo YOUR_USERNAME/clearledger gh variable set AWS_REGION --body eu-west-1 --repo YOUR_USERNAME/clearledger </code></pre> Then create the <code>production</code> environment and add the OIDC role ARN as a secret: <pre><code class="language-bash"># Create the environment first, gh secret set returns 404 if it does not exist gh api --method PUT "repos/YOUR_USERNAME/clearledger/environments/production" gh secret set AWS_ACTIONS_ROLE_ARN \ --env production \ --body "$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw github_actions_ecr_role_arn)" \ --repo YOUR_USERNAME/clearledger </code></pre> Note: GitHub blocks secret names that start with <code>GITHUB_</code>. Use <code>AWS_ACTIONS_ROLE_ARN</code>, not <code>GITHUB_ACTIONS_ROLE_ARN</code>. Also make sure <code>github_owner</code> is set correctly in <code>terraform.tfvars</code> (see <code>terraform.tfvars.example</code>) before running <code>terraform apply</code>. This wires up the OIDC trust policy so AWS will accept tokens from your specific GitHub account. Once <code>CLEARLEDGER_CI_TARGET=aws</code> is set, every push to <code>main</code> runs the AWS pipeline: Gitleaks → Semgrep → Checkov → build → Trivy scan → ECR push → kustomization update. The homelab <code>ci.yaml</code> is skipped. If CI fails at the ECR push step: The most common failure is <code>Not authorized to perform sts:AssumeRoleWithWebIdentity</code>. This means the IAM role trust policy still has a placeholder <code>YOUR_GITHUB_USERNAME</code> in the <code>:sub</code> condition. Fix it by setting <code>github_owner</code> in <code>terraform.tfvars</code> and running <code>terraform apply</code> again, then re-run only the failed job (not the whole pipeline: the earlier scan steps already passed). <pre><code class="language-text">GitHub → Actions → failed run → Re-run failed jobs </code></pre> If you see <code>404</code> when running <code>gh secret set</code>, the <code>production</code> environment doesn't exist yet. Run the <code>gh api --method PUT</code> command above first. Re-run after fixing OIDC: failed jobs only, not the full pipeline. Earlier gates (Gitleaks, build, scan) already passed, and their artifacts are still in the workflow run. Use Re-run all jobs only if you changed app code or want a clean scan from scratch. <h3 id="heading-production-hardening-checklist">Production Hardening Checklist</h3> The lab architecture is production-style, but a real production setup needs extra guardrails. Add these before you describe it as production-ready. <h4 id="heading-1-protect-the-main-branches">1. Protect the main branches</h4> Protect both GitHub repos: <pre><code class="language-text">github.com/YOUR_GITHUB_USERNAME/clearledger github.com/YOUR_GITHUB_USERNAME/clearledger-infra </code></pre> Go to each repo: <pre><code class="language-text">Settings → Rules → Rulesets → New ruleset → Branch targeting: main </code></pre> Enable: <pre><code class="language-text">Require a pull request before merging Require approvals Require status checks to pass Require branches to be up to date before merging Block force pushes Block branch deletion </code></pre> Why this matters: nobody should push straight to the code repo or the GitOps repo in production. A bad direct push to <code>clearledger-infra</code> is a direct deployment request. <h4 id="heading-2-use-github-environments-with-approvals">2. Use GitHub Environments with approvals</h4> Create a protected environment: <pre><code class="language-text">clearledger repo → Settings → Environments → New environment → Name: production → Required reviewers: add yourself or the team → Deployment branches: main only </code></pre> The AWS workflow uses: <pre><code class="language-yaml">environment: production </code></pre> That means GitHub pauses the AWS deployment until an approved reviewer allows it. This creates a real promotion gate instead of "every push deploys to prod." <h4 id="heading-3-prefer-fine-grained-tokens-or-a-github-app">3. Prefer fine-grained tokens or a GitHub App</h4> For the basic lab, <code>INFRA_REPO_TOKEN</code> can be a classic PAT. For production, tighten it. Better option: <pre><code class="language-text">Fine-grained personal access token → Repository access: only YOUR_GITHUB_USERNAME/clearledger-infra → Permissions: Contents: Read and write Metadata: Read </code></pre> Best option for teams: use a GitHub App installed only on <code>clearledger-infra</code>, with permission to write contents. That gives better audit logs and easier rotation than a personal token. Store <code>INFRA_REPO_TOKEN</code> as a production environment secret, not a general repository secret: <pre><code class="language-text">clearledger → Settings → Environments → production → Environment secrets → INFRA_REPO_TOKEN </code></pre> <h4 id="heading-4-lock-aws-oidc-to-the-production-environment">4. Lock AWS OIDC to the production environment</h4> This isn't a shell command. It's a trust rule Terraform writes into AWS when you run <code>terraform apply</code>. In <code>iam.tf</code>, the IAM role <code>clearledger-github-actions-ecr</code> only accepts GitHub tokens whose subject claim matches: <pre><code class="language-text">repo:YOUR_GITHUB_USERNAME/clearledger:environment:production </code></pre> Only GitHub Actions jobs running in the <code>production</code> environment of your <code>clearledger</code> repo can assume the ECR push role. A random branch, fork, or workflow without that environment can't get AWS credentials. What you do: <ol> <li>Set <code>github_owner</code> in <code>terraform.tfvars</code>, then <code>terraform apply</code> (Stage 8 step 2). </li> <li>On GitHub: Settings → Environments → production. Create it if missing, and add protection rules if you want. </li> <li>Add environment secret <code>AWS_ACTIONS_ROLE_ARN</code> = <code>terraform output -raw github_actions_ecr_role_arn</code>. </li> <li><code>ci-aws.yaml</code> already sets <code>environment: production</code> on the ECR jobs, that is what makes GitHub mint a matching token. </li> </ol> Verify the rule exists (optional): <pre><code class="language-bash">aws iam get-role --role-name clearledger-github-actions-ecr \ --query 'Role.AssumeRolePolicyDocument.Statement[0].Condition.StringEquals."token.actions.githubusercontent.com:sub"' \ --output text </code></pre> Expect: <code>repo:your-username/clearledger:environment:production</code> On the manual Stage 8 path you can skip GitHub CI entirely, this lock only matters when you enable CI, AWS (ECR + OIDC). <h4 id="heading-5-staging-before-production-promote-dont-rebuild">5. Staging before production (promote, don't rebuild)</h4> Lab flow: push to <code>main</code> → <code>ci-aws.yaml</code> builds and scans → CI updates <code>stages/stage-8-aws-migration/manifests/kustomization.yaml</code> with the new image tag → Argo CD syncs <code>clearledger-aws</code>. Homelab Stages 1–7 still use <code>clearledger-infra</code> and Docker Hub. Stage 8 AWS uses the in-repo kustomize path. Real production adds a staging step in the middle: build the image once, deploy that same tag or digest to staging, run smoke tests or get manual approval, then promote to production, without building again. Why? Because, if you rebuild for prod, you might ship different code than what passed staging. The safe pattern is one artifact, tested once, promoted twice. <pre><code class="language-text">Build once (one image SHA) → deploy to staging → test / approve → deploy the same SHA to production </code></pre> <h4 id="heading-6-use-private-networking-where-possible">6. Use private networking where possible</h4> For production AWS: <pre><code class="language-text">- EKS nodes in private subnets - RDS in private subnets - Private EKS API endpoint, or restricted public endpoint - Security groups scoped to required ports only - ALB public only if the app is public - No SSH-based deployment path </code></pre> The pipeline should talk to AWS APIs through IAM/OIDC and deploy through GitOps. It shouldn't SSH into EC2 instances. <h4 id="heading-7-store-terraform-state-remotely">7. Store Terraform state remotely</h4> Local Terraform state is fine for a lab. Production should use encrypted remote state: <pre><code class="language-text">- S3 bucket for terraform.tfstate - DynamoDB table for state locking - SSE encryption enabled - Bucket versioning enabled - Public access blocked </code></pre> The Terraform backend block is already included in <code>stages/stage-8-aws-migration/terraform/main.tf</code> as a commented template. Uncomment it after you create the S3 bucket and DynamoDB lock table. <h4 id="heading-production-ready-summary">Production-ready summary:</h4> <pre><code class="language-text">- CI builds and proves the artifact. - GitHub Environments approve production. - OIDC gives short-lived AWS credentials. - ECR stores immutable images. - kustomization.yaml (Stage 8 path) records desired state. - ArgoCD clearledger-aws deploys from Git. - No SSH. No static AWS keys. No direct kubectl from CI. </code></pre> Open the URL. ClearLedger is running on AWS. Same architecture, same security layers, just new infrastructure. <img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/3e6a283d-add1-467b-83da-97c1915f0b92.png" alt="screenshot of clearledger UI running on EKS with ALB URL" style="display:block;margin:0 auto" width="1473" height="1269" loading="lazy"> Destroy when done. This stops all charges: <pre><code class="language-bash">make aws-down </code></pre> See <code>stages/stage-8-aws-migration/README.md</code> for the full walkthrough and cost reference. <h3 id="heading-what-you-learned-in-stage-8">What You Learned in Stage 8</h3> <ul> <li>That containerized applications are portable: the same code runs on your laptop and on AWS </li> <li>What Terraform does: declares infrastructure as code so environments are reproducible </li> <li>What changes in a cloud migration (managed services, IAM, networking) and what does not (application code, CI logic, security policies) </li> <li>Three AWS secret delivery paths: ESO (default), CSI file mounts (§8.5), vs Vault on homelab </li> <li>AWS-specific security services: GuardDuty (threat detection), CloudTrail (API audit), GitHub Actions OIDC (pipeline AWS auth without long-lived keys), and IRSA (pod-level IAM without long-lived credentials) </li> </ul> What you can now put on your CV / say in an interview: <blockquote> Migrated the same architecture to AWS (EKS, ECR, RDS, ALB, with secrets via External Secrets Operator and IRSA) provisioned by Terraform, without rewriting the application. </blockquote> When you're done on AWS, tear down to stop charges: <pre><code class="language-bash">make aws-down </code></pre> Your homelab VM is separate. If you plan to return to it, you should already have a snapshot from Stage 7 (<code>make snapshots</code> to confirm). See <a href="#heading-how-to-save-your-progress">Saving your progress</a>. <h2 id="heading-troubleshooting-see-troubleshootingmd">Troubleshooting (See <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md</a>)</h2> Pod stuck in Pending: <pre><code class="language-bash">kubectl describe pod POD_NAME -n clearledger # Insufficient memory/cpu → reduce resource requests # Image pull error → check Docker Hub repo name and credentials </code></pre> Kyverno blocking a deployment: <pre><code class="language-bash">kubectl get events -n clearledger --sort-by='.lastTimestamp' | tail -10 kubectl get policyreport -n clearledger -o yaml </code></pre> Vault agent not injecting secrets: <pre><code class="language-bash">kubectl logs POD_NAME -n clearledger -c vault-agent-init kubectl exec -n vault vault-0 -- vault read auth/kubernetes/role/auth-service </code></pre> Falco not firing alerts: <pre><code class="language-bash">kubectl logs -n falco daemonset/falco | grep -i error | tail -20 </code></pre> ArgoCD shows OutOfSync: <pre><code class="language-bash">argocd app sync clearledger --force argocd app get clearledger kubectl get events -n clearledger --sort-by='.lastTimestamp' </code></pre> clearledger.local not resolving: <pre><code class="language-bash">multipass info clearledger | grep IPv4 grep clearledger /etc/hosts # If the IP changed, update /etc/hosts </code></pre> VM disk full or pods Evicted (disk pressure): <pre><code class="language-bash">make doctor # PASS / WARN / FAIL + PVC and Prometheus TSDB sizes make reclaim # safe reclaim — unused images + journald only (not PVCs) </code></pre> If still FAIL after reclaim, tear down and recreate: <code>make teardown && make setup</code>. Full guidance: <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md: disk health</a> and <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">VM disk full</a>. <h2 id="heading-compliance-reference">Compliance Reference</h2> Every control maps to at least one framework. Full mapping: <a href="compliance-mapping.md"><code>docs/compliance-mapping.md</code></a>. <table> <thead> <tr> <th>Control</th> <th>Tool</th> <th>Stage</th> <th>PCI-DSS</th> <th>SOC2</th> <th>CIS K8s</th> </tr> </thead> <tbody><tr> <td>Secrets detection</td> <td>Gitleaks</td> <td>3</td> <td>6.2</td> <td>CC8.1</td> <td>—</td> </tr> <tr> <td>SAST</td> <td>Semgrep</td> <td>3</td> <td>6.3.2</td> <td>CC7.1</td> <td>—</td> </tr> <tr> <td>Dependency scan</td> <td>Trivy SCA</td> <td>3</td> <td>6.3.3</td> <td>CC7.1</td> <td>—</td> </tr> <tr> <td>IaC scan</td> <td>Checkov</td> <td>3</td> <td>6.3.1</td> <td>CC6.1</td> <td>—</td> </tr> <tr> <td>Image signing</td> <td>Cosign</td> <td>3</td> <td>6.3</td> <td>CC6.1</td> <td>—</td> </tr> <tr> <td>SBOM generation</td> <td>Syft</td> <td>3</td> <td>6.3.3</td> <td>CC6.1</td> <td>—</td> </tr> <tr> <td>Non-root containers</td> <td>Kyverno</td> <td>4</td> <td>6.5</td> <td>CC6.3</td> <td>5.2.6</td> </tr> <tr> <td>Resource limits</td> <td>Kyverno</td> <td>4</td> <td>—</td> <td>A1.1</td> <td>5.2.4</td> </tr> <tr> <td>No privilege escalation</td> <td>Kyverno</td> <td>4</td> <td>6.5</td> <td>CC6.3</td> <td>5.2.5</td> </tr> <tr> <td>Secrets management</td> <td>Vault</td> <td>5</td> <td>3.5</td> <td>CC6.1</td> <td>—</td> </tr> <tr> <td>Runtime detection</td> <td>Falco</td> <td>6</td> <td>10.7</td> <td>CC7.2</td> <td>—</td> </tr> <tr> <td>Network segmentation</td> <td>NetworkPolicy</td> <td>6</td> <td>1.3</td> <td>CC6.6</td> <td>5.3.2</td> </tr> <tr> <td>Security observability</td> <td>Grafana</td> <td>7</td> <td>10.6</td> <td>CC7.2</td> <td>—</td> </tr> <tr> <td>DORA metrics</td> <td>ArgoCD + Grafana</td> <td>7</td> <td>—</td> <td>—</td> <td>—</td> </tr> <tr> <td>Account threat detection</td> <td>GuardDuty</td> <td>8</td> <td>10.6</td> <td>CC7.2</td> <td>—</td> </tr> <tr> <td>API audit trail</td> <td>CloudTrail</td> <td>8</td> <td>10.2</td> <td>CC7.3</td> <td>—</td> </tr> </tbody></table> EU DORA (Digital Operational Resilience Act): applies to EU financial entities since January 2025. ClearLedger maps to all five DORA pillars. Full mapping in <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/compliance-mapping.md"><code>docs/compliance-mapping.md</code></a>. <h2 id="heading-interview-preparation">Interview Preparation</h2> Full weak/strong answers: <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/interview-prep.md"><code>docs/interview-prep.md</code></a> Practice these as you finish each stage: Stage 0: How does traffic reach your services in Kubernetes? What breaks first when deployment is manual? Stage 1: How do you prove what image is deployed for a given commit? What stops a developer bypassing CI? Stage 2: What does GitOps mean mechanically? How do you prove drift is corrected automatically? Stage 3: Difference between SAST, IaC scanning, and image scanning? Where do you draw the line for fail-on severity? Stage 4: What is admission control and why is it different from CI? How would you safely introduce a policy exception? Stage 5: Why are Kubernetes Secrets not "secret management"? How do you rotate secrets with minimal downtime risk? Stage 6: What does runtime detection catch that CI and admission can't? What is your first response to a shell-spawn alert? Stage 7: What's the difference between a dashboard and an alert? How do you produce audit evidence, not just claims? Stage 8: What actually changes when you move to EKS? What shouldn't change? How does IRSA reduce risk? <h2 id="heading-aws-cost-reference">AWS Cost Reference</h2> Default Stage 8 sizes (eu-west-1, approximate): <table> <thead> <tr> <th>Resource</th> <th>Monthly (8h/day)</th> <th>Monthly (24/7)</th> </tr> </thead> <tbody><tr> <td>EKS control plane</td> <td>~$24</td> <td>~$73</td> </tr> <tr> <td>3× t3.medium nodes</td> <td>~$30</td> <td>~$92</td> </tr> <tr> <td>NAT Gateway</td> <td>~$11</td> <td>~$33</td> </tr> <tr> <td>RDS db.t3.micro</td> <td>~$4</td> <td>~$13</td> </tr> <tr> <td>ALB</td> <td>~$2</td> <td>~$6</td> </tr> <tr> <td>GuardDuty + CloudTrail</td> <td>~$2</td> <td>~$5</td> </tr> <tr> <td>Total estimate</td> <td>~$73</td> <td>~$222</td> </tr> </tbody></table> Always destroy when not in use: <pre><code class="language-bash">make aws-down </code></pre> <h2 id="heading-conclusion">Conclusion</h2> You've now built a fintech application and layered eight security and reliability controls on top of it: all from a laptop. You started with raw Kubernetes and manual deploys in Stage 0. You added a CI pipeline that builds, scans, and signs images automatically in Stage 1. You connected Git to the cluster with ArgoCD in Stage 2. You gated every push with SAST, IaC, and image scanning in Stage 3. You blocked bad workloads at the cluster boundary with Kyverno in Stage 4. You moved credentials out of Git and Kubernetes secrets into Vault in Stage 5. You added runtime threat detection with Falco and network segmentation in Stage 6. You built observability dashboards that produce audit evidence in Stage 7. And you migrated the whole thing to AWS in Stage 8. None of these stages is a toy exercise. Each one represents a real problem that real teams hit in production. You felt the pain, then built the solution. That's the difference between reading about DevSecOps and being able to do it. Take your screenshots, update your CV with the specific tools and outcomes, and use the interview prep section when you need to talk through the decisions you made. You built every one of them. If you found this guide helpful, share it with someone breaking into DevOps or DevSecOps and <a href="https://www.linkedin.com/in/osomudeya-zudonu-17290b124">connect on LinkedIn</a>. I also post DevOps walkthroughs and interview tips for getting hired; follow or <a href="https://osomudeya.kit.com/23db7ca59f">subscribe there</a> if you want more.

How to apply

1 Read the full details above and confirm you meet the eligibility criteria.
2 Prepare your documents — an updated CV, and any cover letter, proposal or certificates required.
3 Click Apply on official site to complete your application on Osomudeya Zudonu’s official page.
4 Submit as early as possible — many close once filled.

Apply on official site

Sourced from freecodecamp. Always verify details on the official website. Opportunihub never charges you to apply.

Frequently asked questions

How do I apply for How to Build a Production-Ready DevSecOps Platform from Homelab to AWS [Full Book]?

Review the full details and eligibility on this page, prepare your documents, then use the “Apply on official site” button to complete your application on Osomudeya Zudonu’s official page.

Is this opportunity remote or location-based?

This opportunity is remote-friendly and open to applicants who can work from anywhere.

Is How to Build a Production-Ready DevSecOps Platform from Homelab to AWS [Full Book] free to apply for?

Opportunihub lists this Course for free. Legitimate Courses do not ask for payment to apply — never pay a fee to submit an application.