We Cut an AWS Bill Without Rewriting the Product

September 23, 202610 mins
Cutting an AWS bill — Unsplash
When an AWS bill gets too high, the first proposal is usually a migration. Move off Kubernetes. Go serverless. Switch clouds. Rewrite the expensive service in a faster language.

Xperiencify's AWS bill went from $10,000 to $2,000 a month in 14 days, and nothing was migrated. The product stayed the same, and so did the stack and the deploy pipeline. We removed what nobody was using and made the database do less work.

Think of a cloud bill as a history of the company. Every launch, experiment, load test and "temporary" environment leaves something behind, and most of it keeps billing long after anyone remembers why it's there. Before you pay for a migration, find out how much of your bill is history.

The situation

Xperiencify is a gamified course platform for coaches and course creators. Its traffic is spiky: launch days and live webinars bring a wave of users all at once. We'd already moved it to AWS EKS with ArgoCD, with autoscaling and health checks. It grew from 500 to 20,000 users without outages and holds 99.99% uptime.

Reliability was solved. The bill was $10,000 a month.

Why a rewrite is the wrong first move

A migration is a project measured in months. You keep paying the old bill the whole time. Your engineers stop shipping product, and you take on delivery risk for the privilege. We'd hold off on it until the inventory is done.

It doesn't fix the cause, either. If nobody knows what the current spend is for, the new platform inherits the same habits: the forgotten environment, the oversized database, the NAT gateway nobody remembers creating, the logs kept forever. The cheapest dollars to save are the ones you pay for and don't use. You find those with an inventory, not an architecture diagram.

Step 1: Match every billed line to something the product uses

Two levers moved Xperiencify's number: an audit of unused resources, and database optimization. The inventory comes first.

Open Cost Explorer. Group spend by service, then by resource. For each item, ask what runs on it, who owns it and what breaks if it disappears. If nobody has a good answer, it's a candidate. In most AWS accounts, unused spend hides in the same few places:

  • Environments nobody opens. Staging copies, demo stacks, load-test clusters that were meant to live for a week.
  • Storage that outlived its data. Unattached volumes, old snapshots, buckets without lifecycle rules, log groups with no retention limit.
  • Networking left behind. Idle load balancers, NAT gateways and elastic IPs from workloads that moved or were retired.
  • Capacity sized for one big day. Node groups and instances provisioned for a launch that happened once.

One rule keeps this safe: nothing gets deleted on day one. Tag it, stop it or scale it to zero, then wait. If nobody notices after a full business cycle, take a final snapshot where data is involved, and only then delete.

Where is your unused spend?

Paste your biggest AWS cost lines into ChatGPT and get a first pass at where your bill is paying for things nobody uses.

|

Step 2: Make the database do less work before you make it smaller

The database is usually one of the biggest lines on the bill, and the scariest one to touch. Downsize it blindly and a cost project turns into an outage on the next launch day. So the order matters:

  1. Find where the load comes from. Rank queries by total time, not by the slowest single query. A 40-millisecond query that runs on every page view costs more than a 3-second report that runs once a day.
  2. Fix the causes. Missing indexes. Loops that issue one query per row. Queries that run on every request when the result could be cached. Background jobs that re-read whole tables. Data kept past its useful life. Much of this lives in application code, not in database settings.
  3. Resize last, to measured load. Leave headroom for your spikiest hour, not your average one.

Step 3: Change one thing at a time, from Git

Every infrastructure change went through Git and ArgoCD, so each one could be reviewed before it shipped and rolled back afterwards. Make one change. Watch error rates, latency and queue depth for a day. Then make the next. On a platform with launch-day spikes like Xperiencify's, schedule changes well away from launches and webinars.

The result

  • $10,000 to $2,000 a month. That's $8,000 a month, or $96,000 a year.
  • 14 days from the start of the work.
  • No rewrite and no migration. Same product, same stack, same deploy pipeline.

Before every big launch we were stressed. Now it just works and we spend less on AWS.

Murray Gray, CEO Xperiencify

Is your AWS bill growing faster than your product?

We run the same audit on your account: unused resources, the database, a safe change plan. Flat price, one senior engineer, working in your AWS account. Send your email and we'll set up a 30-minute call within 24 hours.

When this won't work

Cleanup has a ceiling. If the bill grew because usage grew, and spend tracks users and revenue, there may be little waste to remove. Then you need different levers: architecture changes on the hottest paths, savings plans, or moving specific workloads to cheaper services.

There's a quick way to tell which case you're in. Compare how your AWS spend changed over the last six months with how active users or revenue changed. If spend grew faster, or stayed flat while usage fell, you're paying for history. Start with the inventory.

The eight-question check

Three or more "no" answers usually mean the bill can fall without touching the architecture.

  1. Can you say what every line over $200 a month is for?
  2. Has every environment in the account been opened by someone in the last 60 days?
  3. Do volumes, snapshots and log groups have retention rules?
  4. Do you know the gap between requested and used CPU and memory across your cluster?
  5. Can you name your top 10 queries by total database time?
  6. Has the database been sized from measured load since your last big launch?
  7. Did AWS spend grow slower than users or revenue over the last six months?
  8. Can every infrastructure change be reviewed and rolled back from Git?

Related reading

The service: Cloud Engineering. If costs are only one of the fires: costs exploding.

Enjoyed this article? Share it with others

Related Posts