The EKS Cost Optimization Handbook: Reduce Your AWS Bill by 60% Using Karpenter and Rightsizing

<p>This handbook is a complete guide to the 7-step playbook that took one EKS bill from \(85,000/month to \)34,000/month — without touching a single line of product code.</p> <p>I've audited EKS clusters at more than 10 companies. The same waste patterns appear every time: over-provisioned nodes, cross-AZ data transfer, idle EBS volumes, and so on. And the most expensive mistake of all: buying compute commitments before rightsizing.</p> <p>This handbook is the fix. I've used this 7-step playbook to reduce EKS costs by 50–60% at every company where I've implemented it. There are no product code changes, and no downtime. Just infrastructure optimization executed in the right order.</p> <p>By the end of this guide, you'll know how to right-size pod resource requests, implement Karpenter for intelligent bin-packing and Spot diversification, migrate compatible workloads to Graviton for 20% cheaper compute, and eliminate NAT Gateway charges entirely with VPC endpoints.</p> <p>All Terraform modules, NodePool templates, and automation scripts referenced in this guide are available in the companion repository at <a href="https://github.com/aayostem/eks-cost-optimization">github.com/aayostem/eks-cost-optimization</a>. The repo includes ready-to-deploy configurations for every step so you can move from reading to implementing in the same afternoon.</p> <h2 id="heading-table-of-contents">Table of Contents</h2> <ul> <li><p><a href="#heading-what-youll-learn">What You'll Learn</a></p> </li> <li><p><a href="#heading-prerequisites">Prerequisites</a></p> </li> <li><p><a href="#heading-part-1-the-baseline-where-your-eks-money-is-going">Part 1: The Baseline — Where Your EKS Money Is Going</a></p> </li> <li><p><a href="#heading-part-2-right-sizing-pod-resource-requests">Part 2: Right-Sizing Pod Resource Requests</a></p> </li> <li><p><a href="#heading-part-3-karpenter-for-bin-packing-and-spot-diversification">Part 3: Karpenter for Bin-Packing and Spot Diversification</a></p> </li> <li><p><a href="#heading-part-4-graviton-migration">Part 4: Graviton Migration</a></p> </li> <li><p><a href="#heading-part-5-vpc-endpoints-for-data-transfer">Part 5: VPC Endpoints for Data Transfer</a></p> </li> <li><p><a href="#heading-part-6-ebs-volume-optimisation">Part 6: EBS Volume Optimisation</a></p> </li> <li><p><a href="#heading-part-7-load-balancer-consolidation">Part 7: Load Balancer Consolidation</a></p> </li> <li><p><a href="#heading-the-complete-7-step-sequence">The Complete 7-Step Sequence</a></p> </li> <li><p><a href="#heading-best-practices-for-eks-cost-optimisation">Best Practices Summary</a></p> </li> <li><p><a href="#heading-resources">Resources</a></p> </li> </ul> <h2 id="heading-what-youll-learn">What You'll Learn</h2> <ul> <li><p>How to right-size pod resource requests using VPA recommendations</p> </li> <li><p>The complete Karpenter setup with Spot diversification and automatic consolidation</p> </li> <li><p>Graviton3 migration for all non-GPU workloads</p> </li> <li><p>VPC endpoints to eliminate NAT Gateway data transfer charges</p> </li> <li><p>EBS gp2 to gp3 migration — 20% cheaper with zero performance loss</p> </li> <li><p>Load balancer consolidation with shared Ingress</p> </li> <li><p>The 7-step sequence that maximises ROI — and why the order isn't optional</p> </li> </ul> <p>Let's dive in.</p> <h2 id="heading-prerequisites">Prerequisites</h2> <p>Before following along, you should have:</p> <p><strong>Knowledge:</strong></p> <ul> <li><p>Working familiarity with Kubernetes — you can deploy an application and inspect pods</p> </li> <li><p>Basic AWS knowledge — you understand EC2 instance types, VPCs, and EBS volumes</p> </li> <li><p>Comfort reading Terraform HCL and Kubernetes YAML</p> </li> </ul> <p><strong>Tools and access:</strong></p> <ul> <li><p>An existing EKS cluster running Kubernetes 1.27 or later</p> </li> <li><p><code>kubectl</code> configured and pointing at your cluster</p> </li> <li><p>AWS CLI v2 installed and authenticated with appropriate permissions</p> </li> <li><p>Helm 3 installed (for Karpenter and Kubecost)</p> </li> <li><p><a href="https://github.com/kubernetes-sigs/metrics-server">Metrics Server</a> installed in your cluster</p> </li> </ul> <p><strong>Companion repository:</strong> Clone the repo before starting. It contains all YAML, Terraform, and shell scripts referenced in this guide:</p> <pre><code class="language-bash">git clone https://github.com/aayostem/eks-cost-optimization cd eks-cost-optimization </code></pre> <p><strong>Estimated savings:</strong> For a cluster running at \(85,000/month with typical over-provisioning, expect \)40,000–55,000/month in savings after completing all 7 steps. Smaller clusters under $10,000/month typically see 40–50% reduction.</p> <h2 id="heading-part-1-the-baseline-where-your-eks-money-is-going">Part 1: The Baseline — Where Your EKS Money Is Going</h2> <h3 id="heading-11-the-typical-eks-cost-breakdown">1.1 The Typical EKS Cost Breakdown</h3> <p>Before touching anything, you need to know exactly where the money is going. Optimising the wrong category first is how teams waste weeks of engineering time and see no meaningful reduction.</p> <p>Here's what a typical $85,000/month EKS cluster looks like when you break it down:</p> <table> <thead> <tr> <th>Category</th> <th>Monthly Cost</th> <th>Percentage</th> <th>Waste Potential</th> </tr> </thead> <tbody><tr> <td>Compute (EC2 nodes)</td> <td>$52,000</td> <td>61%</td> <td>High — over-provisioning, wrong instance types</td> </tr> <tr> <td>Data Transfer</td> <td>$15,300</td> <td>18%</td> <td>Very High — cross-AZ and NAT Gateway charges</td> </tr> <tr> <td>Storage (EBS volumes)</td> <td>$10,200</td> <td>12%</td> <td>Medium — unattached volumes and gp2 vs gp3</td> </tr> <tr> <td>Load Balancers</td> <td>$4,250</td> <td>5%</td> <td>Low to Medium — single-service ALBs</td> </tr> <tr> <td>EKS Control Plane</td> <td>$72</td> <td>&lt;1%</td> <td>None — this is a fixed cost</td> </tr> <tr> <td>Other</td> <td>$3,178</td> <td>4%</td> <td>Low</td> </tr> </tbody></table> <p>Compute and Data Transfer together represent 79% of the bill and account for 90% of the correctable waste. Those are the targets.</p> <p>Run this command to see your own breakdown before starting anything:</p> <pre><code class="language-bash"># Pull last month's cost breakdown by service # Save this output — it becomes your before number aws ce get-cost-and-usage \ --time-period Start=\((date -d 'last month' +%Y-%m-01),End=\)(date +%Y-%m-01) \ --granularity MONTHLY \ --group-by Type=DIMENSION,Key=SERVICE \ --metrics UnblendedCost \ --query 'ResultsByTime[0].Groups[*].{Service:Keys[0],Cost:Metrics.UnblendedCost.Amount}' \ --output table | sort -k3 -rn </code></pre> <p>Screenshot the output and save it. You'll compare against it after each step to verify actual savings before moving to the next one.</p> <h3 id="heading-12-the-most-expensive-mistake-wrong-optimisation-order">1.2 The Most Expensive Mistake: Wrong Optimisation Order</h3> <p>Here's what most teams do when they get a large AWS bill:</p> <ol> <li><p>Buy Savings Plans immediately, locking in waste at a 30% discount</p> </li> <li><p>Then implement Karpenter, discovering they've over-committed the wrong instance family</p> </li> <li><p>Then migrate to Graviton, discovering their Savings Plan doesn't cover ARM instances</p> </li> </ol> <p>The result: a 12–36 month commitment paying for waste they could have eliminated in three weeks.</p> <p>The correct sequence is:</p> <pre><code class="language-plaintext">Step 1: Right-size pod requests ← Always first Step 2: Implement Karpenter ← Dynamic provisioning on rightsized requests Step 3: Enable Spot for non-prod ← Karpenter handles fallback automatically Step 4: Migrate to Graviton ← Karpenter makes this seamless Step 5: Add VPC endpoints ← Eliminate data transfer charges Step 6: Optimise EBS volumes ← Quick win, run alongside other steps Step 7: Consolidate load balancers ← Final structural cleanup </code></pre> <p>Then, and only then, buy Savings Plans — against the optimised baseline you've just established.</p> <p>The one rule: optimise first, then commit. Every step before the Savings Plan purchase reduces what you're locking in for 1–3 years.</p> <h2 id="heading-part-2-right-sizing-pod-resource-requests">Part 2: Right-Sizing Pod Resource Requests</h2> <h3 id="heading-21-why-over-provisioned-requests-are-so-expensive">2.1 Why Over-Provisioned Requests Are So Expensive</h3> <p>Kubernetes schedules pods based on resource <em>requests</em> — not actual usage. A pod that requests 2 vCPUs and 4GB of memory requires a node with that capacity available, regardless of whether the pod is actually using it.</p> <p>Here's the incorrect approach with the requests set to worst-case peak estimates:</p> <pre><code class="language-yaml"># Bad: Resource requests set during initial deployment, never revisited # This pod actually uses 250m CPU and 512Mi memory on average resources: requests: cpu: "2" # 8x more than actual usage memory: "4Gi" # 8x more than actual usage limits: cpu: "4" memory: "8Gi" </code></pre> <p>When every pod is over-requested by 8x, your cluster needs 8x more nodes than your workloads actually require. That's where the 61% compute line in your bill comes from.</p> <p>First, verify actual usage before changing anything:</p> <pre><code class="language-bash"># Install Metrics Server if not already running kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml # Check actual CPU and memory usage per pod # Compare these numbers against your current resource requests kubectl top pods --all-namespaces --sort-by=cpu </code></pre> <p>Expected output showing the typical gap:</p> <pre><code class="language-plaintext">NAMESPACE NAME CPU(cores) MEMORY(bytes) production payment-api-xxx 25m 128Mi production user-api-xxx 15m 96Mi production notification-svc-xxx 5m 64Mi staging worker-xxx 10m 256Mi </code></pre> <p>If your pods are requesting 2 CPU cores each but using 25m–15m cores in practice, you have a 50–80x over-request ratio. Every node in your cluster is mostly empty space you're paying for.</p> <h3 id="heading-22-using-the-vertical-pod-autoscaler-for-recommendations">2.2 Using the Vertical Pod Autoscaler for Recommendations</h3> <p>The Vertical Pod Autoscaler (VPA) is a Kubernetes component that analyses historical CPU and memory usage for each deployment and recommends optimal resource requests. You use it in recommendation-only mode first — it tells you what to set without changing anything automatically, so you can review and apply the changes yourself with full control.</p> <p>Here's the correct implementation:</p> <pre><code class="language-yaml"># Good: VPA in recommendation-only mode # Watches your pod's actual usage for 24+ hours, then recommends right-sized requests # updateMode: "Off" means it only recommends — it never restarts your pods apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: payment-api-vpa namespace: production spec: targetRef: apiVersion: apps/v1 kind: Deployment name: payment-api updatePolicy: updateMode: "Off" # Recommendation only — you apply manually after review resourcePolicy: containerPolicies: - containerName: "*" minAllowed: cpu: "100m" # VPA will never recommend below this floor memory: "256Mi" maxAllowed: cpu: "2" # VPA will never recommend above this ceiling memory: "4Gi" </code></pre> <p>Install VPA and retrieve recommendations:</p> <pre><code class="language-bash"># Install VPA components kubectl apply -f https://github.com/kubernetes/autoscaler/releases/download/vertical-pod-autoscaler-1.0.0/vpa-v1.0.0.yaml # Apply the VPA manifest for each deployment you want to right-size kubectl apply -f vpa/payment-api-vpa.yaml # Wait 24 hours for VPA to collect usage data, then check recommendations kubectl describe vpa payment-api-vpa -n production </code></pre> <p>What a VPA recommendation looks like:</p> <pre><code class="language-plaintext">Recommendation: Container Recommendations: Container Name: payment-api Lower Bound: cpu: 50m memory: 128Mi Target: cpu: 250m ← Set your requests to this value memory: 512Mi ← Set your requests to this value Upper Bound: cpu: 500m memory: 1Gi </code></pre> <p>Apply the recommendation to your deployment:</p> <pre><code class="language-yaml"># Good: Right-sized requests based on VPA Target recommendation resources: requests: cpu: "250m" # Down from 2000m — an 8x reduction memory: "512Mi" # Down from 4096Mi — an 8x reduction limits: cpu: "500m" # 2x the request — headroom for genuine spikes memory: "1Gi" # 2x the request </code></pre> <p>All VPA manifests for common deployment types are in <code>vpa/</code> in the <a href="https://github.com/aayostem/eks-cost-optimization/tree/main/vpa">companion repo</a>.</p> <h3 id="heading-23-the-roi-of-right-sizing">2.3 The ROI of Right-Sizing</h3> <table> <thead> <tr> <th>Metric</th> <th>Before</th> <th>After</th> <th>Improvement</th> </tr> </thead> <tbody><tr> <td>Average CPU utilisation</td> <td>18%</td> <td>65%</td> <td>+47 percentage points</td> </tr> <tr> <td>Node count required</td> <td>42</td> <td>28</td> <td>-33%</td> </tr> <tr> <td>Monthly compute cost</td> <td>$52,000</td> <td>$36,400</td> <td>-$15,600/month</td> </tr> </tbody></table> <p>Verify the improvement after applying recommendations:</p> <pre><code class="language-bash"># Check cluster-level utilisation after right-sizing # Target: 60–75% CPU and memory utilisation across nodes kubectl top nodes </code></pre> <h2 id="heading-part-3-karpenter-for-bin-packing-and-spot-diversification">Part 3: Karpenter for Bin-Packing and Spot Diversification</h2> <p>Karpenter is an open-source Kubernetes node provisioner built by AWS and donated to the CNCF.</p> <p>Where the default Kubernetes Cluster Autoscaler scales pre-configured node groups up and down, Karpenter watches the actual resource requests of pending pods and provisions exactly the right EC2 instance type to satisfy them — selecting dynamically from thousands of available instance families rather than the two or three you pre-configured. It also continuously monitors running nodes for underutilisation and consolidates workloads onto fewer nodes, terminating the empty ones automatically.</p> <p>The result is a cluster that is always sized to what your workloads actually need right now, not what you anticipated at setup time.</p> <h3 id="heading-31-the-ceiling-with-cluster-autoscaler">3.1 The Ceiling with Cluster Autoscaler</h3> <p>Cluster Autoscaler works with pre-defined node groups. You configure which instance types are available and it scales those groups up and down.</p> <p>The limitation is that it can only provision instances from the types you pre-configured. It can't dynamically select the right instance type based on what the workload actually needs right now.</p> <p>Here's the incorrect approach using static node groups:</p> <pre><code class="language-bash"># Bad: Two static node groups, each over-provisioning against worst-case scenarios # CPU-optimised group runs even when workloads are memory-bound # Memory-optimised group runs even when workloads are CPU-bound eksctl create nodegroup \ --cluster my-cluster \ --name cpu-optimized \ --instance-types c5.2xlarge \ --nodes-min 5 --nodes-max 20 eksctl create nodegroup \ --cluster my-cluster \ --name memory-optimized \ --instance-types r5.2xlarge \ --nodes-min 3 --nodes-max 10 </code></pre> <p>You're provisioning for the worst case in each family simultaneously. At any given moment, one group is underutilised while the other is scaling. Neither is right.</p> <h3 id="heading-32-how-karpenter-solves-this">3.2 How Karpenter Solves This</h3> <p>Karpenter watches the actual resource requests of pending pods and provisions exactly the right instance type to fit them. It selects from thousands of available instance types, not just the two you pre-configured. It also consolidates running workloads onto fewer nodes when utilisation drops, automatically terminating underutilised nodes.</p> <p>Here's the correct implementation:</p> <pre><code class="language-yaml"># Good: Karpenter NodePool # Karpenter selects the optimal instance type based on pending pod requirements # Tries Spot first, falls back to On-Demand automatically when Spot isn't available apiVersion: karpenter.sh/v1beta1 kind: NodePool metadata: name: default spec: template: spec: requirements: # Allow both x86 and ARM (Graviton) — Karpenter picks the cheaper option - key: kubernetes.io/arch operator: In values: ["amd64", "arm64"] # Try Spot first, fall back to On-Demand if unavailable - key: karpenter.sh/capacity-type operator: In values: ["spot", "on-demand"] # Exclude families with poor price-to-performance ratio - key: karpenter.k8s.aws/instance-family operator: NotIn values: ["t2", "t3a"] limits: cpu: "1000" memory: "4000Gi" disruption: # Remove underutilised nodes and reschedule their pods automatically consolidationPolicy: WhenUnderutilized # Recycle nodes after 30 days to ensure fresh, patched AMIs expireAfter: 720h </code></pre> <p>What each setting does:</p> <ul> <li><p><code>consolidationPolicy: WhenUnderutilized</code>: Karpenter continuously monitors node utilisation and removes underused nodes, moving their pods elsewhere. Your node count decreases automatically as load drops without any manual intervention.</p> </li> <li><p><code>expireAfter: 720h</code>: Nodes older than 30 days are gracefully replaced, ensuring your infrastructure always runs the latest EKS-optimised AMI with current security patches.</p> </li> <li><p><code>values: ["spot", "on-demand"]</code>: Karpenter attempts Spot capacity first. If Spot is unavailable for the requested instance type, it falls back to On-Demand with no alerts and no manual action required.</p> </li> </ul> <p>Migrating from Cluster Autoscaler safely:</p> <pre><code class="language-bash"># Step 1: Install Karpenter alongside Cluster Autoscaler — do not remove CAS yet helm repo add karpenter https://charts.karpenter.sh helm install karpenter karpenter/karpenter \ --namespace karpenter \ --create-namespace \ --set settings.clusterName=your-cluster-name # Step 2: Apply NodePool and NodeClass configuration kubectl apply -f karpenter/nodepool.yaml kubectl apply -f karpenter/nodeclass.yaml # Step 3: Taint existing legacy nodes so new pods schedule on Karpenter nodes # This migrates workloads gradually — zero downtime kubectl taint nodes -l eks.amazonaws.com/nodegroup=cpu-optimized \ group=legacy:NoSchedule # Step 4: Watch pods reschedule to Karpenter-managed nodes over the next hour kubectl get pods -o wide --all-namespaces | grep -v legacy # Step 5: After 30 days of stable operation, remove the old node groups eksctl delete nodegroup --cluster my-cluster --name cpu-optimized eksctl delete nodegroup --cluster my-cluster --name memory-optimized </code></pre> <p>Ready-to-deploy NodePool and NodeClass templates are in <code>karpenter/</code> in the <a href="https://github.com/aayostem/eks-cost-optimization/tree/main/karpenter">companion repo</a>.</p> <h3 id="heading-33-spot-instances-for-non-production-workloads">3.3 Spot Instances for Non-Production Workloads</h3> <p>Staging and development workloads don't need the reliability guarantees of On-Demand instances. Moving them to Spot saves 60–90% on those node costs. Karpenter handles Spot interruptions by rescheduling pods automatically. For stateless workloads, interruptions are invisible to users.</p> <pre><code class="language-yaml"># Good: Spot-only NodePool for staging environments apiVersion: karpenter.sh/v1beta1 kind: NodePool metadata: name: staging-spot spec: template: metadata: labels: billing/environment: staging spec: taints: - key: environment value: staging effect: NoSchedule # Only pods that tolerate this taint schedule here requirements: - key: karpenter.sh/capacity-type operator: In values: ["spot"] # Spot only for non-production disruption: consolidationPolicy: WhenUnderutilized </code></pre> <h3 id="heading-34-the-roi-of-karpenter-and-spot">3.4 The ROI of Karpenter and Spot</h3> <table> <thead> <tr> <th>Metric</th> <th>Before (Cluster Autoscaler)</th> <th>After (Karpenter + Spot)</th> <th>Improvement</th> </tr> </thead> <tbody><tr> <td>Average node count</td> <td>28</td> <td>18</td> <td>-36%</td> </tr> <tr> <td>Average CPU utilisation</td> <td>65%</td> <td>82%</td> <td>+17 percentage points</td> </tr> <tr> <td>Staging environment cost</td> <td>$8,000/month</td> <td>$2,400/month</td> <td>-70%</td> </tr> <tr> <td>Scale-up time for new pods</td> <td>3–5 minutes</td> <td>30–60 seconds</td> <td>-80%</td> </tr> </tbody></table> <h2 id="heading-part-4-graviton-migration">Part 4: Graviton Migration</h2> <p>AWS Graviton is Amazon's own ARM-based processor family, available across EC2 instance types with names ending in <code>g</code> — <code>m7g</code>, <code>c7g</code>, <code>r7g</code>, and so on.</p> <p>Graviton instances are priced approximately 20% lower than equivalent Intel or AMD x86 instances. For most server-side workloads — Node.js, Python, Go, Java — they also deliver 20–40% better performance per dollar because the processor architecture is optimised specifically for these workload types.</p> <p>You don't change your application code to use Graviton. You change the architecture flag in your container image build and the node selector in your Kubernetes deployment.</p> <h3 id="heading-41-why-graviton-reduces-cost-without-reducing-performance">4.1 Why Graviton Reduces Cost Without Reducing Performance</h3> <p>The first question to answer before migrating is whether your container images support ARM64. Most official images from Docker Hub ship as multi-architecture images. Your own application images need to be built for both architectures explicitly.</p> <p>Check whether your images support ARM64:</p> <pre><code class="language-bash"># Check if an image has an ARM64 manifest docker manifest inspect your-registry/your-app:latest | jq '.manifests[].platform' </code></pre> <p>Expected output for a multi-arch image:</p> <pre><code class="language-json">{"architecture": "amd64", "os": "linux"}, {"architecture": "arm64", "os": "linux", "variant": "v8"} </code></pre> <p>If <code>arm64</code> appears, the image is ready. If not, you need to build and push a multi-arch image first.</p> <p>Build and push a multi-architecture image:</p> <pre><code class="language-bash"># Build for both x86 and ARM in a single command using Docker Buildx docker buildx create --use --name multi-arch-builder docker buildx build \ --platform linux/amd64,linux/arm64 \ --tag your-registry/your-app:latest \ --push \ . </code></pre> <h3 id="heading-42-migrating-workloads-to-graviton">4.2 Migrating Workloads to Graviton</h3> <p>With Karpenter already installed, Graviton migration is a single label change on your deployment. Karpenter provisions the appropriate ARM64 node automatically.</p> <p>Here's the correct implementation:</p> <pre><code class="language-yaml"># Good: nodeSelector directs the pod to Graviton nodes # Karpenter provisions an arm64 node if one isn't already available apiVersion: apps/v1 kind: Deployment metadata: name: payment-api spec: template: spec: nodeSelector: kubernetes.io/arch: arm64 # Schedule exclusively on Graviton nodes containers: - name: api image: your-registry/payment-api:latest # Must be multi-arch </code></pre> <p>Migrate gradually, starting with stateless services:</p> <pre><code class="language-bash"># Step 1: Migrate one stateless service and monitor for 48 hours kubectl patch deployment payment-api \ -p '{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/arch":"arm64"}}}}}' # Step 2: Watch for errors in the first 30 minutes kubectl logs -l app=payment-api --tail=100 -f # Step 3: Verify the pod is running on a Graviton node # The NODE column should show a Graviton instance type (m7g, c7g, r7g) kubectl get pods -l app=payment-api -o wide # Step 4: After 48 hours of stable operation, migrate the next service </code></pre> <p>There are some situations where you shouldn't migrate to Graviton: GPU workloads, applications with native x86 binary dependencies, or any workload where you haven't yet built multi-arch images.</p> <h3 id="heading-43-the-roi-of-graviton">4.3 The ROI of Graviton</h3> <table> <thead> <tr> <th>Workload Type</th> <th>x86 Monthly Cost</th> <th>Graviton Monthly Cost</th> <th>Saving</th> </tr> </thead> <tbody><tr> <td>Web services (Node.js, Python)</td> <td>$18,000</td> <td>$14,400</td> <td>$3,600/month</td> </tr> <tr> <td>Data processing</td> <td>$12,000</td> <td>$9,600</td> <td>$2,400/month</td> </tr> <tr> <td>API services (Go, Java)</td> <td>$8,000</td> <td>$6,400</td> <td>$1,600/month</td> </tr> <tr> <td><strong>Total</strong></td> <td><strong>$38,000</strong></td> <td><strong>$30,400</strong></td> <td><strong>$7,600/month</strong></td> </tr> </tbody></table> <h2 id="heading-part-5-vpc-endpoints-for-data-transfer">Part 5: VPC Endpoints for Data Transfer</h2> <h3 id="heading-51-the-nat-gateway-tax">5.1 The NAT Gateway Tax</h3> <p>Every byte that travels from your EKS pods to an AWS service — S3, DynamoDB, ECR, SQS — goes through a NAT Gateway if you haven't configured VPC endpoints. NAT Gateway charges $0.045 per GB of data processed.</p> <p>A busy EKS cluster pulling container images from ECR, writing to S3, and polling SQS queues can process hundreds of terabytes per month through NAT Gateway — generating thousands of dollars in charges for traffic that never actually left the AWS network.</p> <p>Measure your current NAT Gateway cost before adding endpoints:</p> <pre><code class="language-bash"># Get last month's NAT Gateway data processing charges aws ce get-cost-and-usage \ --time-period Start=\((date -d 'last month' +%Y-%m-01),End=\)(date +%Y-%m-01) \ --granularity DAILY \ --filter '{ "Dimensions": { "Key": "USAGE_TYPE", "Values": ["NATGateway-Bytes"] } }' \ --metrics UnblendedCost \ --query 'ResultsByTime[*].{Date:TimePeriod.Start,Cost:Total.UnblendedCost.Amount}' \ --output table </code></pre> <h3 id="heading-52-vpc-endpoints-the-fix-that-takes-30-minutes">5.2 VPC Endpoints — The Fix That Takes 30 Minutes</h3> <p>A VPC endpoint creates a private connection between your VPC and an AWS service, routing traffic through the AWS backbone without touching the NAT Gateway. The data transfer becomes free. Each endpoint costs approximately \(0.01/hour — roughly \)7.20/month — far less than the NAT Gateway processing charges it replaces.</p> <p>Here's the complete implementation for the four most common EKS traffic destinations:</p> <pre><code class="language-bash"># Get your VPC ID and primary route table ID first VPC_ID=$(aws eks describe-cluster --name your-cluster \ --query 'cluster.resourcesVpcConfig.vpcId' --output text) ROUTE_TABLE_ID=$(aws ec2 describe-route-tables \ --filters Name=vpc-id,Values=$VPC_ID Name=association.main,Values=true \ --query 'RouteTables[0].RouteTableId' --output text) echo "VPC: \(VPC_ID | Route Table: \)ROUTE_TABLE_ID" # S3 gateway endpoint — free to create, eliminates all S3 traffic through NAT aws ec2 create-vpc-endpoint \ --vpc-id $VPC_ID \ --service-name com.amazonaws.us-east-1.s3 \ --route-table-ids $ROUTE_TABLE_ID # DynamoDB gateway endpoint — also free, same mechanism as S3 aws ec2 create-vpc-endpoint \ --vpc-id $VPC_ID \ --service-name com.amazonaws.us-east-1.dynamodb \ --route-table-ids $ROUTE_TABLE_ID # ECR API interface endpoint — eliminates NAT charges on image pulls aws ec2 create-vpc-endpoint \ --vpc-id $VPC_ID \ --vpc-endpoint-type Interface \ --service-name com.amazonaws.us-east-1.ecr.api \ --subnet-ids $(aws ec2 describe-subnets \ --filters Name=vpc-id,Values=$VPC_ID Name=tag:Tier,Values=private \ --query 'Subnets[*].SubnetId' --output text) # ECR Docker endpoint — required alongside ECR API for complete image pull coverage aws ec2 create-vpc-endpoint \ --vpc-id $VPC_ID \ --vpc-endpoint-type Interface \ --service-name com.amazonaws.us-east-1.ecr.dkr \ --subnet-ids $(aws ec2 describe-subnets \ --filters Name=vpc-id,Values=$VPC_ID Name=tag:Tier,Values=private \ --query 'Subnets[*].SubnetId' --output text) </code></pre> <p>The Terraform module that creates all four endpoints in a single <code>apply</code> is in <code>terraform/vpc-endpoints/</code> in the <a href="https://github.com/aayostem/eks-cost-optimization/tree/main/terraform/vpc-endpoints">companion repo</a>.</p> <p>Verify that the endpoints are routing traffic correctly:</p> <pre><code class="language-bash">aws ec2 describe-vpc-endpoints \ --filters Name=vpc-id,Values=$VPC_ID \ --query 'VpcEndpoints[*].{Service:ServiceName,State:State,Type:VpcEndpointType}' \ --output table # Expected: all endpoints showing State=available </code></pre> <h3 id="heading-53-the-roi-of-vpc-endpoints">5.3 The ROI of VPC Endpoints</h3> <table> <thead> <tr> <th>Service</th> <th>Before (Through NAT)</th> <th>After (VPC Endpoint)</th> <th>Monthly Saving</th> </tr> </thead> <tbody><tr> <td>S3 data transfer</td> <td>$4,500</td> <td>$0</td> <td>$4,500</td> </tr> <tr> <td>ECR image pulls</td> <td>$800</td> <td>$0</td> <td>$800</td> </tr> <tr> <td>DynamoDB queries</td> <td>$1,200</td> <td>$0</td> <td>$1,200</td> </tr> <tr> <td>Endpoint cost</td> <td>—</td> <td>$29 (4 endpoints)</td> <td>-$29</td> </tr> <tr> <td><strong>Net saving</strong></td> <td></td> <td></td> <td><strong>$6,471/month</strong></td> </tr> </tbody></table> <h2 id="heading-part-6-ebs-volume-optimisation">Part 6: EBS Volume Optimisation</h2> <h3 id="heading-61-the-gp2-to-gp3-migration">6.1 The gp2 to gp3 Migration</h3> <p>EBS gp2 volumes price their IOPS based on storage size — 3 IOPS per GB, with a 100 IOPS minimum. EBS gp3 volumes provide 3,000 IOPS baseline regardless of size, and cost 20% less per GB. The migration runs online with no downtime.</p> <p>Find and migrate all gp2 volumes:</p> <pre><code class="language-bash"># Step 1: List all gp2 volumes and their sizes aws ec2 describe-volumes \ --filters Name=volume-type,Values=gp2 \ --query 'Volumes[*].{ID:VolumeId,Size:Size,State:State}' \ --output table # Step 2: Migrate each gp2 volume to gp3 — no instance stop required # The modify operation runs online while the volume stays attached and in use aws ec2 describe-volumes \ --filters Name=volume-type,Values=gp2 \ --query 'Volumes[*].VolumeId' \ --output text | tr '\t' '\n' | while read vol; do echo "Migrating $vol from gp2 to gp3..." aws ec2 modify-volume \ --volume-id $vol \ --volume-type gp3 done # Step 3: Verify all volumes are now gp3 aws ec2 describe-volumes \ --filters Name=volume-type,Values=gp2 \ --query 'Volumes[*].VolumeId' \ --output text # Expected: empty output — zero gp2 volumes remaining </code></pre> <h3 id="heading-62-finding-and-removing-orphaned-volumes-and-snapshots">6.2 Finding and Removing Orphaned Volumes and Snapshots</h3> <p>When Kubernetes PersistentVolumeClaims are deleted, the underlying EBS volumes sometimes aren't cleaned up. They keep running — and billing — indefinitely.</p> <pre><code class="language-bash"># Find unattached EBS volumes — status=available means not attached to any instance aws ec2 describe-volumes \ --filters Name=status,Values=available \ --query 'Volumes[*].{ID:VolumeId,Size:Size,Created:CreateTime}' \ --output table # Find EBS snapshots older than 90 days aws ec2 describe-snapshots \ --owner-ids self \ --query "Snapshots[?StartTime&lt;='$(date -d '90 days ago' --iso-8601=seconds)'].[SnapshotId,StartTime,VolumeSize]" \ --output table </code></pre> <p>Before deleting any snapshot, cross-reference with your RDS automated backup schedule to confirm it's not the only backup for a production database.</p> <h3 id="heading-63-the-roi-of-ebs-optimisation">6.3 The ROI of EBS Optimisation</h3> <table> <thead> <tr> <th>Resource</th> <th>Before</th> <th>After</th> <th>Monthly Saving</th> </tr> </thead> <tbody><tr> <td>gp2 → gp3 migration (1TB total)</td> <td>$102</td> <td>$72</td> <td>$30</td> </tr> <tr> <td>Unattached volumes removed (50 × 100GB)</td> <td>$500</td> <td>$0</td> <td>$500</td> </tr> <tr> <td>Old snapshots cleaned (500GB)</td> <td>$25</td> <td>$0</td> <td>$25</td> </tr> <tr> <td><strong>Total</strong></td> <td><strong>$627</strong></td> <td><strong>$72</strong></td> <td><strong>$555/month</strong></td> </tr> </tbody></table> <h2 id="heading-part-7-load-balancer-consolidation">Part 7: Load Balancer Consolidation</h2> <h3 id="heading-71-the-problem-one-load-balancer-per-service">7.1 The Problem — One Load Balancer Per Service</h3> <p>Many teams create a separate <code>LoadBalancer</code> Service for every microservice. On AWS, each Application Load Balancer costs approximately \(16.20/month base charge plus \)0.008/LCU-hour for traffic processed. At 20 microservices, that's $324/month before a single request is processed.</p> <p>Here's the incorrect approach:</p> <pre><code class="language-yaml"># Bad: This creates a dedicated AWS ALB every time it's applied # 20 microservices = 20 ALBs = $324+/month before any traffic charges apiVersion: v1 kind: Service metadata: name: payment-api spec: type: LoadBalancer # Creates a dedicated ALB ports: - port: 80 targetPort: 8080 </code></pre> <h3 id="heading-72-the-fix-shared-ingress-controller">7.2 The Fix — Shared Ingress Controller</h3> <p>An Ingress controller is a Kubernetes component that runs as a pod inside your cluster and programs a single external load balancer to route traffic to multiple services based on hostname and URL path. Instead of one AWS Application Load Balancer per microservice, you get one ALB total — with path-based routing directing each request to the right backend service. The result is the same routing behaviour at a fraction of the cost.</p> <p>Here's the correct implementation:</p> <pre><code class="language-yaml"># Good: One Ingress resource routes all external traffic # The AWS Load Balancer Controller creates one ALB for all services listed here apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: shared-ingress namespace: production annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]' alb.ingress.kubernetes.io/ssl-redirect: "443" spec: rules: - host: api.company.com http: paths: - path: /payments pathType: Prefix backend: service: name: payment-service port: number: 8080 - path: /users pathType: Prefix backend: service: name: user-service port: number: 8080 - host: dashboard.company.com http: paths: - path: / pathType: Prefix backend: service: name: dashboard-service port: number: 3000 tls: - hosts: - api.company.com - dashboard.company.com secretName: tls-wildcard-cert </code></pre> <p>Verify the Ingress is provisioned and the ALB DNS name is assigned:</p> <pre><code class="language-bash"># Watch until the ADDRESS column shows the ALB DNS name (typically 2–3 minutes) kubectl get ingress shared-ingress -n production -w </code></pre> <p>The cost difference:</p> <table> <thead> <tr> <th>Approach</th> <th>Load balancers</th> <th>Monthly cost</th> </tr> </thead> <tbody><tr> <td>LoadBalancer Service per microservice (20 services)</td> <td>20 ALBs</td> <td>~$400/month</td> </tr> <tr> <td>Single Ingress controller</td> <td>1 ALB</td> <td>~$27/month</td> </tr> <tr> <td><strong>Monthly saving</strong></td> <td></td> <td><strong>~$373/month</strong></td> </tr> </tbody></table> <p>The shared Ingress manifest is in <code>k8s/ingress/</code> in the <a href="https://github.com/aayostem/eks-cost-optimization/tree/main/k8s/ingress">companion repo</a>.</p> <h2 id="heading-the-complete-7-step-sequence">The Complete 7-Step Sequence</h2> <table> <thead> <tr> <th>Step</th> <th>Action</th> <th>Time to Implement</th> <th>Expected Monthly Saving</th> </tr> </thead> <tbody><tr> <td>1</td> <td>Right-size pod resource requests (VPA)</td> <td>1 week</td> <td>$15,600</td> </tr> <tr> <td>2</td> <td>Install Karpenter with consolidation</td> <td>1 week</td> <td>$8,400</td> </tr> <tr> <td>3</td> <td>Move staging and dev to Spot</td> <td>1 week</td> <td>$11,200</td> </tr> <tr> <td>4</td> <td>Migrate compatible workloads to Graviton</td> <td>2 weeks</td> <td>$7,600</td> </tr> <tr> <td>5</td> <td>Add VPC endpoints for S3, ECR, DynamoDB</td> <td>1 day</td> <td>$6,471</td> </tr> <tr> <td>6</td> <td>Migrate gp2 to gp3 and delete orphaned volumes</td> <td>1 day</td> <td>$555</td> </tr> <tr> <td>7</td> <td>Consolidate load balancers with shared Ingress</td> <td>1 day</td> <td>$373</td> </tr> <tr> <td><strong>Total</strong></td> <td></td> <td><strong>3–4 weeks</strong></td> <td><strong>$49,799/month</strong></td> </tr> </tbody></table> <p>Annual saving at this rate: <strong>$597,588</strong>. Engineering time required: one engineer, one sprint per step.</p> <h2 id="heading-best-practices-for-eks-cost-optimisation">Best Practices for EKS Cost Optimisation</h2> <p>✅ <strong>Do:</strong> Right-size pod resource requests before any other optimisation. Every subsequent step depends on accurate requests.</p> <p>✅ <strong>Do:</strong> Implement Karpenter with <code>consolidationPolicy: WhenUnderutilized</code>. Let it continuously optimise your node count automatically.</p> <p>✅ <strong>Do:</strong> Move staging and development workloads to Spot. 60–90% savings for workloads that tolerate interruption.</p> <p>✅ <strong>Do:</strong> Migrate compatible workloads to Graviton. Most web services and APIs run without code changes.</p> <p>✅ <strong>Do:</strong> Add VPC endpoints for S3, DynamoDB, and ECR before reviewing data transfer costs.</p> <p>✅ <strong>Do:</strong> Migrate gp2 volumes to gp3. It's online, zero downtime, and immediately 20% cheaper.</p> <p>✅ <strong>Do:</strong> Use a single shared Ingress controller for all external traffic instead of per-service load balancers.</p> <p>❌ <strong>Don't:</strong> Buy Savings Plans before completing steps 1–6. You'll lock in waste for 1–3 years.</p> <p>❌ <strong>Don't:</strong> Use static node groups with Cluster Autoscaler when your workload mix changes. Karpenter handles this dynamically.</p> <p>❌ <strong>Don't:</strong> Run staging and development environments on On-Demand instances. Spot interruptions are manageable, but the cost difference is not.</p> <h2 id="heading-resources">Resources</h2> <ul> <li><p><a href="https://karpenter.sh/docs/"><strong>Karpenter Documentation</strong></a> — Official NodePool configuration reference and installation guide</p> </li> <li><p><a href="https://github.com/aws/aws-graviton-getting-started"><strong>AWS Graviton Getting Started Guide</strong></a> — Language-specific compatibility notes and migration guidance from AWS</p> </li> <li><p><a href="https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler"><strong>Vertical Pod Autoscaler GitHub</strong></a> — VPA installation and configuration documentation</p> </li> <li><p><a href="https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints.html"><strong>AWS VPC Endpoints Documentation</strong></a> — Complete list of available VPC endpoints and configuration options</p> </li> <li><p><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/requesting-ebs-volume-modifications.html"><strong>EBS Volume Modification Documentation</strong></a> — AWS guide for online volume type migration with zero downtime</p> </li> <li><p><a href="https://kubernetes-sigs.github.io/aws-load-balancer-controller/"><strong>AWS Load Balancer Controller</strong></a> — Official documentation for the Ingress controller that provisions AWS ALBs</p> </li> <li><p><a href="https://docs.aws.amazon.com/cost-management/latest/APIReference/API_GetCostAndUsage.html"><strong>AWS Cost Explorer API Reference</strong></a> — Full reference for the cost breakdown commands used throughout this guide</p> </li> <li><p><a href="https://aws.github.io/aws-eks-best-practices/cost_optimization/cfm_framework/"><strong>EKS Best Practices Guide — Cost Optimisation</strong></a> — AWS's official EKS cost optimisation framework</p> </li> <li><p><a href="https://github.com/aayostem/eks-cost-optimization"><strong>Companion Repository</strong></a> — All Terraform modules, NodePool templates, VPA manifests, and automation scripts from this guide</p> </li> </ul>