How to Encrypt Kubernetes Traffic with cert-manager, Let's Encrypt, and Internal TLS

<p>Most engineers assume their Kubernetes cluster encrypts all of its traffic. It doesn't. The commands you run with <code>kubectl</code> are encrypted — your client and the API server speak TLS. The API server talking to etcd is usually encrypted too, depending on how the cluster was provisioned.</p> <p>But traffic between your pods? Plaintext by default. Ingress traffic from the internet to your services? Only encrypted if you explicitly configure TLS. And certificates for internal services? You have to provision those yourself.</p> <p>This is not a Kubernetes oversight. It's a deliberate design choice — Kubernetes provides the primitives and leaves the implementation to you. The problem is that certificate management is notoriously painful. Certificates expire. Provisioning them manually doesn't scale. Forgetting to rotate them causes outages.</p> <p>cert-manager solves this. It runs as a controller inside your cluster, watches for <code>Certificate</code> resources, requests certificates from configured issuers, stores them in Kubernetes Secrets, and rotates them automatically before they expire. You declare what you want, cert-manager makes it happen and keeps it that way.</p> <p>In this article you'll work through how cert-manager's core model works, automate public Ingress TLS using Let's Encrypt, set up an internal Certificate Authority for service-to-service encryption, and understand how certificate rotation works so outages caused by expired certificates become a thing of the past.</p> <h2 id="heading-prerequisites">Prerequisites</h2> <ul> <li><p>A kind cluster with the nginx Ingress controller installed</p> </li> <li><p>Helm 3 installed</p> </li> <li><p>A domain name with DNS you control — needed for the Let's Encrypt demo</p> </li> <li><p>Basic understanding of TLS: you know what a certificate, a private key, and a CA are</p> </li> </ul> <p>All demo files are in the <a href="https://github.com/Caesarsage/DevOps-Cloud-Projects/tree/main/intermediate/k8/security/cert-manager">DevOps-Cloud-Projects GitHub repository</a>.</p> <h2 id="heading-table-of-contents">Table of Contents</h2> <ul> <li><p><a href="#heading-what-is-and-isnt-encrypted-in-kubernetes">What Is and Isn't Encrypted in Kubernetes</a></p> </li> <li><p><a href="#heading-how-cert-manager-works">How cert-manager Works</a></p> <ul> <li><p><a href="#heading-the-four-core-resources">The Four Core Resources</a></p> </li> <li><p><a href="#heading-issuers-and-clusterissuers">Issuers and ClusterIssuers</a></p> </li> <li><p><a href="#heading-the-certificate-lifecycle">The Certificate Lifecycle</a></p> </li> <li><p><a href="#heading-acme-challenges-http-01-vs-dns-01">ACME Challenges: HTTP-01 vs DNS-01</a></p> </li> </ul> </li> <li><p><a href="#heading-demo-1--install-cert-manager-and-issue-a-lets-encrypt-certificate">Demo 1 — Install cert-manager and Issue a Let's Encrypt Certificate</a></p> </li> <li><p><a href="#heading-how-to-get-a-wildcard-certificate-with-dns-01">How to Get a Wildcard Certificate with DNS-01</a></p> </li> <li><p><a href="#heading-demo-2--set-up-an-internal-ca-for-service-to-service-tls">Demo 2 — Set Up an Internal CA for Service-to-Service TLS</a></p> </li> <li><p><a href="#heading-how-certificate-rotation-works">How Certificate Rotation Works</a></p> </li> <li><p><a href="#heading-cleanup">Cleanup</a></p> </li> <li><p><a href="#heading-conclusion">Conclusion</a></p> </li> </ul> <h2 id="heading-what-is-and-isnt-encrypted-in-kubernetes">What Is and Isn't Encrypted in Kubernetes?</h2> <p>Before installing anything, it's worth being precise about what the cluster already protects and what it leaves open.</p> <table> <thead> <tr> <th>Traffic path</th> <th>Encrypted by default?</th> <th>Notes</th> </tr> </thead> <tbody><tr> <td><code>kubectl</code> → API server</td> <td>Yes</td> <td>TLS with the cluster CA</td> </tr> <tr> <td>API server → etcd</td> <td>Usually</td> <td>Depends on cluster provisioner — verify with your setup</td> </tr> <tr> <td>API server → kubelet</td> <td>Yes</td> <td>TLS, but kubelet cert verification depends on configuration</td> </tr> <tr> <td>Pod → Pod (same cluster)</td> <td><strong>No</strong></td> <td>Plaintext unless you add a service mesh or mTLS</td> </tr> <tr> <td>Internet → Ingress</td> <td><strong>No</strong></td> <td>Opt-in — requires TLS configuration on the Ingress resource</td> </tr> <tr> <td>Pod → Kubernetes API</td> <td>Yes</td> <td>Via the service account token and cluster CA</td> </tr> </tbody></table> <p>The two gaps that matter most in practice are pod-to-pod traffic and Ingress TLS. This article covers both Ingress TLS with Let's Encrypt and internal service-to-service encryption using a private CA.</p> <h2 id="heading-how-cert-manager-works">How cert-manager Works</h2> <p>cert-manager is a Kubernetes operator. It extends the Kubernetes API with custom resources that represent certificate requests and their configuration. When you create a <code>Certificate</code> resource, cert-manager's controller picks it up, requests a certificate from the configured issuer, and stores the resulting certificate and private key in a Kubernetes Secret. When the certificate approaches its expiry, cert-manager renews it automatically.</p> <p>This model means your application doesn't know or care about certificate management. It reads a Secret. cert-manager keeps that Secret fresh.</p> <h3 id="heading-the-four-core-resources">The Four Core Resources</h3> <p>cert-manager introduces four custom resources that you'll use regularly:</p> <table> <thead> <tr> <th>Resource</th> <th>What it represents</th> </tr> </thead> <tbody><tr> <td><code>Issuer</code></td> <td>A certificate authority or ACME account — namespace-scoped</td> </tr> <tr> <td><code>ClusterIssuer</code></td> <td>Same as Issuer, but available cluster-wide</td> </tr> <tr> <td><code>Certificate</code></td> <td>A request for a certificate — describes what you want</td> </tr> <tr> <td><code>CertificateRequest</code></td> <td>An individual signing request — created automatically by cert-manager, rarely touched directly</td> </tr> </tbody></table> <p>In practice you'll mostly deal with <code>ClusterIssuer</code> and <code>Certificate</code>. The <code>ClusterIssuer</code> defines where certificates come from. The <code>Certificate</code> defines what certificate you want and where to store it.</p> <h3 id="heading-issuers-and-clusterissuers">Issuers and ClusterIssuers</h3> <p>An <code>Issuer</code> can only issue certificates within its own namespace. A <code>ClusterIssuer</code> can issue certificates in any namespace. For shared infrastructure like Let's Encrypt, you almost always want a <code>ClusterIssuer</code>. For application-specific internal CAs, an <code>Issuer</code> scoped to that application's namespace is the safer choice.</p> <p>cert-manager supports several issuer types. The three you'll encounter most often are:</p> <p><strong>ACME</strong> — for public certificates from Let's Encrypt or any ACME-compatible CA. Ownership of the domain is proven via an HTTP-01 or DNS-01 challenge.</p> <p><strong>CA</strong> — for internal certificates signed by a CA whose private key is stored in a Kubernetes Secret. Used for service-to-service TLS within the cluster.</p> <p><strong>Self-signed</strong> — generates self-signed certificates. Rarely useful on its own, but essential as the bootstrap step when creating an internal CA.</p> <h3 id="heading-the-certificate-lifecycle">The Certificate Lifecycle</h3> <p>When you create a <code>Certificate</code> resource, cert-manager follows this sequence:</p> <ol> <li><p>Creates a <code>CertificateRequest</code> with a CSR (Certificate Signing Request)</p> </li> <li><p>Passes the CSR to the configured issuer</p> </li> <li><p>For ACME issuers: creates a <code>Challenge</code> resource and fulfils it (more on this below)</p> </li> <li><p>Receives the signed certificate from the issuer</p> </li> <li><p>Stores the certificate and private key in the Kubernetes Secret named in <code>spec.secretName</code></p> </li> <li><p>Monitors the certificate's expiry — by default, renews when 2/3 of the validity period has elapsed</p> </li> </ol> <p>Your application mounts the Secret. cert-manager updates it silently. Most applications that watch for file changes will pick up the new certificate without a restart.</p> <h3 id="heading-acme-challenges-http-01-vs-dns-01">ACME Challenges: HTTP-01 vs DNS-01</h3> <p>Let's Encrypt needs proof that you control the domain before it issues a certificate. ACME defines two challenge types for this.</p> <p><strong>HTTP-01</strong> works by having cert-manager create a temporary HTTP endpoint at <code>http://&lt;your-domain&gt;/.well-known/acme-challenge/&lt;token&gt;</code>. Let's Encrypt sends a request to that URL. If the response matches the expected token, the challenge passes. This requires your cluster to be reachable from the internet on port 80.</p> <p><strong>DNS-01</strong> works by having cert-manager create a temporary DNS TXT record at <code>_acme-challenge.&lt;your-domain&gt;</code>. Let's Encrypt checks for that record. This doesn't require inbound HTTP access, which makes it the right choice for private clusters, and it's the only way to get wildcard certificates (<code>*.example.com</code>).</p> <p>The trade-off: HTTP-01 is simpler to set up but only works for single domains and requires internet-accessible infrastructure. DNS-01 requires API access to your DNS provider but works for internal clusters and wildcards.</p> <h2 id="heading-demo-1-install-cert-manager-and-issue-a-certificate-using-pebble-and-lets-encrypt">Demo 1 — Install cert-manager and Issue a Certificate Using Pebble and Let's Encrypt</h2> <p>Pebble is Let's Encrypt's local ACME test server. It runs inside your cluster, issues certificates using the same ACME protocol as Let's Encrypt, and requires no public domain or internet access. Using Pebble lets you test the full cert-manager flow — challenge, issuance, renewal — on a plain kind cluster.</p> <p>Once you understand the flow locally, switching to real Let's Encrypt is a one-line change: replace the ClusterIssuer server URL and point a DNS record at a publicly reachable cluster. The rest of the configuration is identical.</p> <p>You'll install cert-manager, create a <code>ClusterIssuer</code> for Let's Encrypt, deploy a sample application with an Ingress, and watch a real certificate be issued and stored automatically.</p> <h3 id="heading-step-1-install-cert-manager">Step 1: Install cert-manager</h3> <p>cert-manager is now distributed via OCI Helm charts from <code>quay.io/jetstack</code>. The <code>--set crds.enabled=true</code> flag installs the Custom Resource Definitions as part of the chart:</p> <pre><code class="language-bash">helm upgrade cert-manager oci://quay.io/jetstack/charts/cert-manager \ --install \ --create-namespace \ --namespace cert-manager \ --set crds.enabled=true \ --version v1.17.0 \ --wait </code></pre> <p>You also need the nginx Ingress controller — cert-manager routes HTTP-01 challenges through it. The <code>controller.service.type=ClusterIP</code> override is for kind specifically: the default <code>LoadBalancer</code> Service never gets an <code>EXTERNAL-IP</code> on kind (there's no cloud LB), which makes <code>--wait</code> hang forever. On a real cluster, drop the override and keep <code>LoadBalancer</code>.</p> <pre><code class="language-bash">helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update helm install ingress-nginx ingress-nginx/ingress-nginx \ --namespace ingress-nginx \ --create-namespace \ --set controller.service.type=ClusterIP \ --wait </code></pre> <p>Confirm all four components are running:</p> <pre><code class="language-bash">kubectl get pods -n cert-manager kubectl get pods -n ingress-nginx </code></pre> <pre><code class="language-plaintext">NAME READY STATUS RESTARTS AGE cert-manager-76f84784c8-r4fx4 1/1 Running 0 6m45s cert-manager-cainjector-66fbf49587-gv25n 1/1 Running 0 6m45s cert-manager-webhook-577fddf86-l5wj4 1/1 Running 0 6m45s NAME READY STATUS RESTARTS AGE ingress-nginx-controller-6c7cd85885-h7zgx 1/1 Running 0 3m34s </code></pre> <blockquote> <p>kind-specific gotcha — remove the nginx admission webhook now.** On kind, the nginx admission webhook serves with a self-signed certificate that the Kubernetes API server cannot verify. The first time you try to create <em>any</em> Ingress resource you'll see <code>failed calling webhook "validate.nginx.ingress.kubernetes.io": ... x509: certificate signed by unknown authority</code>. Delete the webhook up front so the rest of the demo doesn't trip over it:</p> </blockquote> <pre><code class="language-bash">kubectl delete validatingwebhookconfiguration ingress-nginx-admission </code></pre> <h3 id="heading-step-2-install-pebble">Step 2: Install Pebble</h3> <p>Pebble is the local ACME test server, distributed by the JupyterHub project. It ships with a companion CoreDNS deployment (<code>pebble-coredns</code>) that Pebble uses to resolve names during ACME validation.</p> <pre><code class="language-bash">helm install pebble pebble \ --repo https://jupyterhub.github.io/helm-chart/ \ --namespace pebble \ --create-namespace \ --wait </code></pre> <p>Confirm both pods are running:</p> <pre><code class="language-bash">kubectl get pods -n pebble </code></pre> <pre><code class="language-plaintext">NAME READY STATUS RESTARTS AGE pebble-8d8d49d64-lz8ck 1/1 Running 0 36s pebble-coredns-7fb5c7cbf4-4jw9h 1/1 Running 0 36s </code></pre> <h3 id="heading-step-3-wire-up-dns-for-the-fake-hostname">Step 3: Wire up DNS for the fake hostname</h3> <p>We're going to issue a cert for <code>echo.pebble.local</code>. That hostname is fake — it doesn't exist in any real DNS — so we have to teach <strong>two</strong> independent resolvers about it before issuance will work:</p> <table> <thead> <tr> <th>Resolver</th> <th>Used by</th> <th>What we need it to do</th> </tr> </thead> <tbody><tr> <td><code>pebble-coredns</code> (in the <code>pebble</code> namespace)</td> <td>Pebble itself, when it makes the HTTP-01 validation request</td> <td>Resolve <code>echo.pebble.local</code> → ingress-nginx ClusterIP</td> </tr> <tr> <td>Cluster CoreDNS (<code>kube-system</code>)</td> <td>cert-manager's HTTP-01 <strong>self-check</strong> before reporting the challenge ready</td> <td>Forward <code>pebble.local</code> lookups to <code>pebble-coredns</code></td> </tr> </tbody></table> <p>If you skip either layer, the Order will go to <code>invalid</code> state with a DNS lookup failure.</p> <p>First grab the two IPs you'll need:</p> <pre><code class="language-bash">NGINX_IP=$(kubectl get svc -n ingress-nginx ingress-nginx-controller \ -o jsonpath='{.spec.clusterIP}') PEBBLE_DNS_IP=$(kubectl get svc pebble-coredns -n pebble \ -o jsonpath='{.spec.clusterIP}') echo "NGINX_IP=\(NGINX_IP PEBBLE_DNS_IP=\)PEBBLE_DNS_IP" </code></pre> <p><strong>Patch</strong> <code>pebble-coredns</code> to answer for <code>*.pebble.local</code> with the ingress controller's IP. The CoreDNS <code>template</code> plugin parses unreliably when the whole block is collapsed onto one line, so apply a real multi-line ConfigMap:</p> <pre><code class="language-bash">cat &lt;&lt;EOF | kubectl apply -f - apiVersion: v1 kind: ConfigMap metadata: name: pebble-coredns namespace: pebble data: Corefile: | .:8053 { errors health ready template ANY ANY pebble.local { answer "{{ .Name }} 60 IN A ${NGINX_IP}" } forward . /etc/resolv.conf cache 2 reload } EOF kubectl rollout restart deploy/pebble-coredns -n pebble kubectl rollout status deploy/pebble-coredns -n pebble </code></pre> <p>Verify it answers correctly:</p> <pre><code class="language-bash">kubectl run dnstest --rm -it --restart=Never --image=busybox -- \ nslookup echo.pebble.local ${PEBBLE_DNS_IP} </code></pre> <p>You should see <code>Address: &lt;NGINX_IP&gt;</code> in the response. If you get <code>SERVFAIL</code>, check <code>kubectl logs -n pebble deploy/pebble-coredns</code> — a parser error like <code>not a TTL: "}"</code> means the template block collapsed onto one line again.</p> <p><strong>Patch the cluster CoreDNS</strong> so cert-manager's self-check can resolve the same name. Add a stub zone that forwards <code>pebble.local</code> to <code>pebble-coredns</code>:</p> <pre><code class="language-bash">cat &lt;&lt;EOF | kubectl apply -f - apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } forward . /etc/resolv.conf { max_concurrent 1000 } cache 30 loop reload loadbalance } pebble.local:53 { forward . ${PEBBLE_DNS_IP} } EOF kubectl rollout restart deploy/coredns -n kube-system kubectl rollout status deploy/coredns -n kube-system </code></pre> <p>Verify the cluster resolver now answers for <code>echo.pebble.local</code> (without specifying a server — it'll use the default kube-dns):</p> <pre><code class="language-bash">kubectl run dnstest --rm -it --restart=Never --image=busybox -- \ nslookup echo.pebble.local </code></pre> <p>Both <code>Server: 10.96.0.10</code> and <code>Address: &lt;NGINX_IP&gt;</code> should appear.</p> <h3 id="heading-step-4-fetch-the-pebble-ca-and-create-the-clusterissuer">Step 4: Fetch the Pebble CA and create the ClusterIssuer</h3> <p>Pebble signs its certificates with a self-signed root that lives in the <code>pebble</code> ConfigMap under <code>root-cert.pem</code>. cert-manager needs to trust this CA to talk to Pebble's ACME directory, so we pass it as a base64-encoded <code>caBundle</code> in the ClusterIssuer:</p> <pre><code class="language-bash">kubectl get configmap pebble -n pebble \ -o jsonpath='{.data.root-cert\.pem}' &gt; pebble-ca.crt head -1 pebble-ca.crt # should print -----BEGIN CERTIFICATE----- CA_BUNDLE=$(base64 -i pebble-ca.crt | tr -d '\n') echo "CA_BUNDLE length: ${#CA_BUNDLE}" # ~1600 chars, one continuous line </code></pre> <p>Create the ClusterIssuer using the heredoc — the <code>${CA_BUNDLE}</code> shell variable gets substituted into the YAML before kubectl reads it:</p> <pre><code class="language-bash">kubectl apply -f - &lt;&lt;EOF apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: pebble spec: acme: server: https://pebble.pebble.svc.cluster.local/dir email: test@example.com privateKeySecretRef: name: pebble-account-key caBundle: ${CA_BUNDLE} solvers: - http01: ingress: ingressClassName: nginx EOF </code></pre> <p>Check the issuer is ready:</p> <pre><code class="language-bash">kubectl get clusterissuer pebble </code></pre> <pre><code class="language-plaintext">NAME READY AGE pebble True 5s </code></pre> <p>If <code>READY</code> stays <code>False</code>, the two most common causes are a malformed caBundle (verify it's a single unbroken base64 line with no newlines) or Pebble being unreachable from the <code>cert-manager</code> namespace. To check reachability:</p> <pre><code class="language-bash">kubectl run test-curl --rm -it --restart=Never \ --image=curlimages/curl:latest \ --namespace cert-manager -- \ curl -k https://pebble.pebble.svc.cluster.local/dir </code></pre> <p>If that returns JSON, Pebble is reachable.</p> <h3 id="heading-step-5-deploy-a-sample-application">Step 5: Deploy a sample application</h3> <pre><code class="language-yaml"># echo-app.yaml apiVersion: apps/v1 kind: Deployment metadata: name: echo namespace: default spec: replicas: 1 selector: matchLabels: app: echo template: metadata: labels: app: echo spec: containers: - name: echo image: ealen/echo-server:latest ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: echo namespace: default spec: selector: app: echo ports: - port: 80 targetPort: 80 </code></pre> <pre><code class="language-bash">kubectl apply -f echo-app.yaml </code></pre> <p>Verify the resources came up:</p> <pre><code class="language-bash">kubectl get deploy,pod,svc -n default </code></pre> <pre><code class="language-plaintext">NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/echo 1/1 1 1 32s NAME READY STATUS RESTARTS AGE pod/echo-5665fbcfdd-mbgxj 1/1 Running 0 36s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/echo ClusterIP 10.96.103.114 &lt;none&gt; 80/TCP 40s service/kubernetes ClusterIP 10.96.0.1 &lt;none&gt; 443/TCP 32m </code></pre> <h3 id="heading-step-6-create-an-ingress-with-tls">Step 6: Create an Ingress with TLS</h3> <p>The <code>cert-manager.io/cluster-issuer: pebble</code> annotation tells cert-manager to automatically create a <code>Certificate</code> resource for this Ingress, using the issuer we just created. The hostname <code>echo.pebble.local</code> doesn't need to resolve externally — we taught both DNS resolvers about it in Step 3.</p> <pre><code class="language-yaml"># echo-ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: echo namespace: default annotations: cert-manager.io/cluster-issuer: pebble spec: ingressClassName: nginx tls: - hosts: - echo.pebble.local secretName: echo-tls # cert-manager will create this Secret rules: - host: echo.pebble.local http: paths: - path: / pathType: Prefix backend: service: name: echo port: number: 80 </code></pre> <pre><code class="language-bash">kubectl apply -f echo-ingress.yaml </code></pre> <h3 id="heading-step-7-watch-the-certificate-being-issued">Step 7: Watch the certificate being issued</h3> <pre><code class="language-bash"># Watch the Certificate resource (Ctrl-C once Ready=True) kubectl get certificate echo-tls -n default -w </code></pre> <pre><code class="language-plaintext">NAME READY SECRET AGE echo-tls False echo-tls 5s echo-tls True echo-tls 28s </code></pre> <p>When <code>READY</code> becomes <code>True</code>, the certificate has been issued and stored in the <code>echo-tls</code> Secret. The full chain — CertificateRequest → Order → Challenge → solver pod → Secret — happens in well under a minute on a healthy cluster:</p> <pre><code class="language-bash">kubectl get certificate,certificaterequest,order,challenge -n default </code></pre> <pre><code class="language-plaintext">NAME READY SECRET AGE certificate.cert-manager.io/echo-tls True echo-tls 81s NAME APPROVED DENIED READY ISSUER AGE certificaterequest.cert-manager.io/echo-tls-1 True True pebble 81s NAME STATE AGE order.acme.cert-manager.io/echo-tls-1-1824732543 valid 81s </code></pre> <p>(Challenges are deleted automatically once an Order completes, so <code>kubectl get challenge -n default</code> typically shows nothing at this point — that's success, not failure.)</p> <p>If <code>READY</code> stays <code>False</code> for more than a minute, see the troubleshooting tips at the end of this section.</p> <p>Inspect the issued certificate to confirm Pebble signed it:</p> <pre><code class="language-bash">kubectl get secret echo-tls -n default -o jsonpath='{.data.tls\.crt}' | \ base64 -d | openssl x509 -noout -issuer -subject -dates </code></pre> <pre><code class="language-plaintext">issuer=CN=Pebble Intermediate CA 05478c subject= notBefore=May 17 19:09:22 2026 GMT notAfter=Aug 15 19:09:21 2026 GMT </code></pre> <p>Issuer is Pebble's intermediate CA — proof the full ACME flow worked end-to-end. The cert is valid for 90 days, and cert-manager will renew it automatically at day 60.</p> <p>Hit the ingress over HTTPS from inside the cluster to confirm everything is wired together:</p> <pre><code class="language-bash">kubectl run curltest --rm -it --restart=Never --image=curlimages/curl -- \ curl -sk https://echo.pebble.local/ </code></pre> <p>The echo server should return a JSON blob — note the <code>"x-forwarded-proto":"https"</code> field, which proves the request came through nginx over TLS.</p> <p><strong>Troubleshooting if the cert never goes Ready:</strong></p> <ul> <li><p><code>kubectl describe order -n default</code> — look for "DNS problem" or "Connection refused" in the events.</p> </li> <li><p><code>kubectl logs -n pebble deploy/pebble --tail=50</code> — Pebble logs the exact URL it tried to fetch during validation and any errors.</p> </li> <li><p>If the Order is stuck pending with no events: cert-manager hasn't reconciled yet. Wait 30s.</p> </li> <li><p>If the Order is <code>invalid</code>: one of the two DNS layers (Step 3) is misconfigured. Re-run both <code>nslookup</code> checks.</p> </li> <li><p>If the Ingress apply itself failed with an x509 webhook error: you skipped the <code>kubectl delete validatingwebhookconfiguration ingress-nginx-admission</code> step in Step 1.</p> </li> </ul> <h3 id="heading-step-8-switch-to-lets-encrypt-staging-real-public-domain">Step 8: Switch to Let's Encrypt staging (real public domain)</h3> <p>Pebble proved the flow works locally. Now move to a publicly-reachable domain pointed at a publicly-reachable cluster. The DNS gymnastics from Step 3 go away — the domain is real, so both resolvers find it without intervention.</p> <p>Use Let's Encrypt <strong>staging</strong> first. It speaks the same ACME protocol as production but with generous rate limits, so failed attempts during testing won't lock you out:</p> <pre><code class="language-yaml"># clusterissuer-staging.yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-staging spec: acme: server: https://acme-staging-v02.api.letsencrypt.org/directory email: your-email@example.com privateKeySecretRef: name: letsencrypt-staging-account-key solvers: - http01: ingress: ingressClassName: nginx </code></pre> <pre><code class="language-bash">kubectl apply -f clusterissuer-staging.yaml # Point the Ingress at staging and the real hostname, then force re-issuance kubectl annotate ingress echo \ cert-manager.io/cluster-issuer=letsencrypt-staging --overwrite -n default kubectl delete secret echo-tls -n default </code></pre> <p>The new cert's issuer will look something like <code>(STAGING) Let's Encrypt</code>.</p> <h3 id="heading-step-9-switch-to-lets-encrypt-production">Step 9: Switch to Let's Encrypt production</h3> <p>Once staging works, repeat with the production ClusterIssuer. The only difference is the <code>server</code> URL:</p> <pre><code class="language-yaml"># clusterissuer-prod.yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: your-email@example.com privateKeySecretRef: name: letsencrypt-prod-account-key solvers: - http01: ingress: ingressClassName: nginx </code></pre> <pre><code class="language-bash">kubectl apply -f clusterissuer-prod.yaml kubectl annotate ingress echo \ cert-manager.io/cluster-issuer=letsencrypt-prod --overwrite -n default kubectl delete secret echo-tls -n default </code></pre> <p>cert-manager detects the missing Secret and immediately requests a browser-trusted certificate from production Let's Encrypt.</p> <p>cert-manager detects the missing Secret and immediately triggers a new certificate request using the production issuer.</p> <h2 id="heading-how-to-get-a-wildcard-certificate-with-dns-01">How to Get a Wildcard Certificate with DNS-01</h2> <p>HTTP-01 challenges work well for single domains with public ingress. But there are two situations where you need DNS-01 instead: when your cluster is not publicly accessible (internal clusters, air-gapped environments, staging namespaces behind a VPN), and when you want a wildcard certificate that covers all subdomains of your domain.</p> <p>DNS-01 requires cert-manager to be able to create and delete TXT records in your DNS provider. cert-manager has built-in support for Route53, Cloud DNS, Cloudflare, Azure DNS, and many others.</p> <p>Here is a <code>ClusterIssuer</code> for DNS-01 using AWS Route53:</p> <pre><code class="language-yaml"># clusterissuer-dns01.yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-dns01 spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: your-email@example.com privateKeySecretRef: name: letsencrypt-dns01-account-key solvers: - dns01: route53: region: us-east-1 # Use IRSA (IAM Roles for Service Accounts) in production # rather than static credentials hostedZoneID: YOUR_HOSTED_ZONE_ID </code></pre> <p>A wildcard <code>Certificate</code> using that issuer:</p> <pre><code class="language-yaml"># wildcard-cert.yaml apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: wildcard-example-com namespace: default spec: secretName: wildcard-example-com-tls issuerRef: name: letsencrypt-dns01 kind: ClusterIssuer commonName: "*.example.com" dnsNames: - "*.example.com" - "example.com" # Also cover the apex domain duration: 2160h # 90 days renewBefore: 720h # Renew 30 days before expiry </code></pre> <p>The resulting Secret <code>wildcard-example-com-tls</code> can be referenced by any Ingress in the <code>default</code> namespace. All subdomains — <code>api.example.com</code>, <code>dashboard.example.com</code>, <code>staging.example.com</code> — are covered by a single certificate that rotates automatically.</p> <p>For Cloudflare instead of Route53, the solver section looks like this:</p> <pre><code class="language-yaml"> solvers: - dns01: cloudflare: email: your-email@example.com apiTokenSecretRef: name: cloudflare-api-token key: api-token </code></pre> <h2 id="heading-demo-2-set-up-an-internal-ca-for-service-to-service-tls">Demo 2 — Set Up an Internal CA for Service-to-Service TLS</h2> <p>Let's Encrypt certificates are great for public-facing services. But for internal services — a gRPC microservice calling another, a web application talking to its database — you don't need public trust. You need a CA that the cluster trusts, and you need it to issue certificates for service names that don't exist as public DNS records.</p> <p>cert-manager's CA issuer handles this. You create a root CA, tell cert-manager about it, and then issue certificates for internal services using that CA. Every service that trusts the root CA trusts every certificate it issues.</p> <h3 id="heading-step-1-create-a-self-signed-clusterissuer">Step 1: Create a self-signed ClusterIssuer</h3> <p>A self-signed issuer generates certificates that are signed by the certificate itself — it is its own CA. You use this as a bootstrap step to create the root CA certificate:</p> <pre><code class="language-yaml"># selfsigned-issuer.yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned spec: selfSigned: {} </code></pre> <pre><code class="language-bash">kubectl apply -f selfsigned-issuer.yaml </code></pre> <h3 id="heading-step-2-create-the-root-ca-certificate">Step 2: Create the root CA certificate</h3> <p>Use the self-signed issuer to create a CA certificate. The <code>isCA: true</code> field tells cert-manager this certificate can sign other certificates:</p> <pre><code class="language-yaml"># internal-ca.yaml apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: internal-ca namespace: cert-manager # Store in cert-manager namespace spec: isCA: true commonName: internal-ca secretName: internal-ca-secret duration: 87600h # 10 years — this is a root CA renewBefore: 720h privateKey: algorithm: ECDSA size: 256 issuerRef: name: selfsigned kind: ClusterIssuer </code></pre> <pre><code class="language-bash">kubectl apply -f internal-ca.yaml kubectl get certificate internal-ca -n cert-manager </code></pre> <pre><code class="language-plaintext">NAME READY SECRET AGE internal-ca True internal-ca-secret 8s </code></pre> <h3 id="heading-step-3-create-a-ca-clusterissuer-backed-by-the-root-ca">Step 3: Create a CA ClusterIssuer backed by the root CA</h3> <p>Now create a <code>ClusterIssuer</code> that uses the root CA Secret you just created. This is the issuer that will sign certificates for your internal services:</p> <pre><code class="language-yaml"># internal-ca-issuer.yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: internal-ca spec: ca: secretName: internal-ca-secret # References the Secret in cert-manager namespace </code></pre> <pre><code class="language-bash">kubectl apply -f internal-ca-issuer.yaml kubectl get clusterissuer internal-ca </code></pre> <pre><code class="language-plaintext">NAME READY AGE internal-ca True 5s </code></pre> <h3 id="heading-step-4-issue-a-certificate-for-an-internal-service">Step 4: Issue a certificate for an internal service</h3> <p>Now issue a certificate for an internal gRPC service. The <code>dnsNames</code> use Kubernetes internal DNS names — <code>&lt;service&gt;.&lt;namespace&gt;.svc.cluster.local</code>:</p> <pre><code class="language-yaml"># payments-cert.yaml apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: payments-tls namespace: production spec: secretName: payments-tls-secret issuerRef: name: internal-ca kind: ClusterIssuer commonName: payments.production.svc.cluster.local dnsNames: - payments.production.svc.cluster.local - payments.production.svc - payments duration: 2160h # 90 days renewBefore: 360h # Renew 15 days before expiry </code></pre> <pre><code class="language-bash">kubectl create namespace production kubectl apply -f payments-cert.yaml kubectl get certificate payments-tls -n production </code></pre> <pre><code class="language-plaintext">NAME READY SECRET AGE payments-tls True payments-tls-secret 6s </code></pre> <p>The Secret <code>payments-tls-secret</code> now contains <code>tls.crt</code>, <code>tls.key</code>, and <code>ca.crt</code>. Mount this into your application pod:</p> <pre><code class="language-yaml"># In your Deployment spec volumes: - name: tls secret: secretName: payments-tls-secret containers: - name: payments volumeMounts: - name: tls mountPath: /etc/tls readOnly: true </code></pre> <p>Your application reads <code>/etc/tls/tls.crt</code> and <code>/etc/tls/tls.key</code> to configure TLS. Other services that need to trust it read <code>/etc/tls/ca.crt</code>.</p> <h3 id="heading-step-5-distribute-the-ca-bundle-with-trust-manager">Step 5: Distribute the CA bundle with trust-manager</h3> <p>The problem with a custom CA is that every service needs to know about it. cert-manager's companion tool, trust-manager, handles this by distributing the CA bundle as a <code>ConfigMap</code> to every namespace:</p> <pre><code class="language-bash">helm upgrade trust-manager oci://quay.io/jetstack/charts/trust-manager \ --install \ --namespace cert-manager \ --wait </code></pre> <p>Create a <code>Bundle</code> resource that takes the CA certificate from the <code>internal-ca-secret</code> and distributes it cluster-wide:</p> <pre><code class="language-yaml"># ca-bundle.yaml apiVersion: trust.cert-manager.io/v1alpha1 kind: Bundle metadata: name: internal-ca-bundle spec: sources: - secret: name: internal-ca-secret key: ca.crt target: configMap: key: ca-bundle.crt namespaceSelector: matchLabels: # Distribute to all namespaces with this label kubernetes.io/metadata.name: production </code></pre> <pre><code class="language-bash">kubectl apply -f ca-bundle.yaml </code></pre> <p>After a few seconds, every matching namespace has a ConfigMap named <code>internal-ca-bundle</code> containing the CA certificate. Applications mount this ConfigMap to trust internally-issued certificates without any per-service configuration.</p> <h3 id="heading-step-6-verify-the-certificate-chain">Step 6: Verify the certificate chain</h3> <pre><code class="language-bash"># Extract the CA cert and service cert kubectl get secret payments-tls-secret -n production \ -o jsonpath='{.data.ca\.crt}' | base64 -d &gt; ca.crt kubectl get secret payments-tls-secret -n production \ -o jsonpath='{.data.tls\.crt}' | base64 -d &gt; payments.crt # Verify the cert was signed by the CA openssl verify -CAfile ca.crt payments.crt </code></pre> <pre><code class="language-plaintext">payments.crt: OK </code></pre> <h2 id="heading-how-certificate-rotation-works">How Certificate Rotation Works</h2> <p>Certificate rotation is the part of certificate management that breaks production clusters most often. cert-manager handles it automatically, but understanding the mechanism helps you tune it and debug it when things go wrong.</p> <p>cert-manager watches every <code>Certificate</code> resource it manages and checks the expiry of the underlying certificate in the Secret. When the remaining validity drops below the <code>renewBefore</code> threshold, cert-manager triggers a renewal. The default <code>renewBefore</code> is 1/3 of the certificate's total validity period — so a 90-day certificate starts renewing at day 60.</p> <p>The renewal creates a new <code>CertificateRequest</code>, goes through the full issuance flow, and updates the Secret in place. The new certificate replaces the old one atomically. Applications that use file mounts and watch for changes (most modern web servers and gRPC frameworks do) will pick up the new certificate without restarting.</p> <pre><code class="language-bash"># See the current rotation status kubectl describe certificate echo-tls -n default </code></pre> <p>Look for these fields in the output:</p> <pre><code class="language-plaintext">Status: Not After: 2024-06-18T10:00:00Z Not Before: 2024-03-20T10:00:00Z Renewal Time: 2024-05-18T10:00:00Z # When cert-manager will start renewing Conditions: Type: Ready Status: True Message: Certificate is up to date and has not expired </code></pre> <p>If a renewal fails — for example, because the HTTP-01 challenge can't be completed — cert-manager retries with exponential backoff. The existing certificate continues to serve until it actually expires, giving you a window to debug the issue.</p> <p>To see renewal events in real time:</p> <pre><code class="language-bash">kubectl get events -n default --field-selector reason=Issued kubectl get events -n default --field-selector reason=Failed </code></pre> <p><strong>Setting</strong> <code>renewBefore</code> <strong>correctly:</strong> For public-facing services, 30 days before a 90-day certificate is a sensible buffer. For internal short-lived certificates (24-hour validity), set <code>renewBefore</code> to 8 hours so rotation happens well before expiry even if the first attempt fails. Never set <code>renewBefore</code> to more than half the certificate's validity — cert-manager will immediately try to renew a certificate it just issued.</p> <h2 id="heading-cleanup">Cleanup</h2> <pre><code class="language-bash"># Remove demo resources kubectl delete ingress echo -n default kubectl delete service echo -n default kubectl delete deployment echo -n default kubectl delete secret echo-tls -n default kubectl delete certificate payments-tls -n production kubectl delete namespace production # Uninstall cert-manager and trust-manager helm uninstall trust-manager -n cert-manager helm uninstall cert-manager -n cert-manager kubectl delete namespace cert-manager # Remove ClusterIssuers kubectl delete clusterissuer letsencrypt-staging letsencrypt-prod \ internal-ca selfsigned 2&gt;/dev/null </code></pre> <h2 id="heading-conclusion">Conclusion</h2> <p>Kubernetes leaves TLS configuration entirely to you. In this article you worked through both the public and internal sides of that responsibility.</p> <p>On the public side, you installed cert-manager using the current OCI Helm chart, created a <code>ClusterIssuer</code> backed by Let's Encrypt, and watched cert-manager go through the full ACME HTTP-01 challenge flow — from creating a temporary solver pod to storing a valid certificate in a Kubernetes Secret. You saw how switching from staging to production is a one-line annotation change, and how cert-manager renews certificates automatically before they expire.</p> <p>On the internal side, you bootstrapped a private CA using cert-manager's self-signed issuer, created a <code>ClusterIssuer</code> backed by that CA, and issued certificates for internal service names that only exist inside the cluster. You used trust-manager to distribute the CA bundle cluster-wide so services can trust each other's certificates without per-service configuration. And you saw how to verify the certificate chain with <code>openssl</code> so you can confirm it's working before deploying to production.</p> <p>Understanding certificate rotation is what separates teams that manage TLS confidently from teams that get woken up at 3am by an expired certificate. cert-manager automates the renewal, but the <code>renewBefore</code> field is your safety margin — set it correctly and know how to read the renewal status.</p> <p>All YAML manifests and Helm values from this article are available in the <a href="https://github.com/Caesarsage/DevOps-Cloud-Projects/tree/main/intermediate/k8/security/cert-manager">DevOps-Cloud-Projects GitHub repository</a>.</p>