There was a database cluster running around the clock, sized for the worst moment it might ever see, holding years of multi-tenant data that was mostly read by reports and the occasional analytical query. It was expensive, it was always awake, and it mixed every tenant’s data in one place that a lot of things could reach. None of that was a bug. It was a database being asked to be a data platform, which is a job it was never built for. Pulling those two jobs apart is some of the most satisfying infrastructure work I have done lately, so here is how it went.
Table of contents
Open Table of contents
The short version
- An OLTP database and an analytical platform are different jobs. One serves fast transactional writes, the other answers big reads over history. Making one do both is where the pain and the bill come from.
- The move: replicate the data out into object storage as Parquet, organize it as Iceberg tables, and query it on demand with a serverless engine. No cluster running all night.
- Security was the spine, not an afterthought. Multi-tenant data got two isolation layers: a separate account so it never shared a blast radius with the client-facing network, and row-level security so one tenant cannot read another’s rows.
- Structure the data in layers, raw to refined, so a bad transform never corrupts the source of truth.
- The cost dropping by more than half was real and welcome, but it was a side effect. The point was the right tool for each job.
The mistake: one database wearing two hats
A transactional database is optimized to write and read small rows fast, one customer action at a time. An analytical query is the opposite: read enormous slices of history, scan millions of rows, aggregate. When you run the second kind of workload against a database built for the first, three things happen, all bad.
It gets slow, because a query scanning a huge fraction of the table competes with the transactional traffic that pays the bills. It gets expensive, because the only way to make it cope is to oversize the hardware for a peak that is mostly idle. And it stays risky, because all that history sits in the one place your live systems reach into, so every heavy report is a little bit of load and a little bit of exposure against production.
The database was not failing because it was bad. It was failing because we kept asking it to be two different products at the same time. The reframe that unlocked everything: stop asking the database to be the analytics layer. Let it do the transactional job it is good at, and move the reading-history job somewhere designed for it.
The architecture: replicate, store cheap, query on demand
The shape is now a fairly standard lakehouse, and standard is a compliment here. The whole path reads as one line:
OLTP database → CDC → object storage → Parquet → Iceberg → serverless query engine
It has three moving ideas.
Replicate the data out. A change-data-capture pipeline streams the database’s changes into object storage continuously, so the lake stays close to current without anyone running exports by hand. Crucially it runs alongside the existing database, reading a replica, so the migration never competed with production traffic. You can stand the whole thing up next to the running system and cut over only when you trust it.
Store it cheap and columnar. The data lands as Parquet files in object storage, then gets organized into table format that behaves like a real database table: schema evolution, ACID commits, snapshots you can time travel to. Object storage costs a fraction of a running database’s disk, and Parquet’s columnar layout means an analytical query reads only the columns it needs instead of dragging every row across the wire.
Query on demand, pay per query. A serverless query engine reads those tables directly. Nothing runs between queries. You are billed for the data a query scans, not for a cluster sitting awake at 4am waiting for a report that runs at noon. That single change, from paying for uptime to paying for work, is most of the cost story.
Layer the data so a bad transform cannot poison the source
The one design choice I would not skip is layering the data as it flows from raw to refined. The pattern goes by a few names, but the shape is always the same:
| Layer | What it holds | Why it exists |
|---|---|---|
| Raw | The data exactly as it landed, untouched | The immutable source of truth. If everything downstream burns, you rebuild from here |
| Cleaned | Deduplicated, typed, conformed | The trustworthy working copy most consumers read |
| Curated | Aggregated, business-level tables | Fast, purpose-built answers for reports and dashboards |
The reason this matters is failure isolation. Transforms have bugs. A dedup goes wrong, a join fans out, someone ships a bad column. If your only copy is the one you keep rewriting in place, a bad transform is a data-loss incident. With a raw layer that nothing writes over, the worst case is “reprocess the next layer,” not “restore from a backup and hope.” The raw layer is cheap insurance you will eventually be very glad you bought.
Security was the spine, because the data is multi-tenant
Here is the part that made this a security project as much as a cost one. The data belonged to many tenants, and “all tenants in one lake” is exactly the kind of quietly reachable arrangement that turns one mistake into everyone’s mistake. So isolation was designed in at two levels, not bolted on after.
Account-level isolation. The analytical data went into its own dedicated account, separate from the one where client-facing connectivity lived. That boundary means the lake full of every tenant’s history does not share a blast radius with the network surface that outside parties connect into. Two different risk profiles, two different accounts. The same instinct as segmenting a flat network: do not let the sensitive thing and the exposed thing sit in the same room.
Row-level isolation. Inside the lake, data was partitioned by tenant and fronted by row-level security, so an identity querying the lake sees only the rows for the tenants it is entitled to. The partitioning also makes queries cheaper, because a tenant-scoped query prunes straight to that tenant’s files instead of scanning the whole dataset. Isolation and performance pointing the same direction is the nicest kind of design, and rare enough that you notice it.
The principle underneath both:
Multi-tenant data should never rely on application code remembering to filter.
The isolation has to live in the platform, in the account boundary and in the access layer, where forgetting is not an option.
The cost was a side effect, and that is the point
The bill dropped by more than half, from a couple thousand a month to a few hundred. That number gets people’s attention, and it should, but I want to be honest about where it came from. It did not come from a clever discount. It came from stopping paying for a database to be awake and oversized around the clock so it could survive occasional heavy reads, and instead paying only for the queries actually run, against storage that costs a fraction of live database disk.
Cost followed correctness. Once each workload sat on the tool built for it, the transactional database right-sized to its real transactional load, and the analytics moved to something that charges for work instead of uptime, the savings were not a goal we chased. They were what falling into the right shape looks like on an invoice.
Final takeaway
If a database is running all night, oversized, holding history that a handful of queries read by day, it is not really a database anymore. It is a data platform in a costume, doing a job it was never designed for and charging you for the privilege. The fix is not a bigger instance. It is to separate the two jobs: let the transactional database be small and fast, and move the reading-history work onto storage that is cheap, a format that is columnar, and an engine that only runs when you ask it something.
Do that with the data layered so a bad transform cannot reach the source, and with tenant isolation living in the platform instead of in someone’s memory, and you get three wins that usually trade off against each other: it is faster, it is safer, and it is cheaper. Not because you optimized for all three, but because you finally gave each job the tool it was asking for.