Data Does Not Page You. It Just Goes Quietly Wrong.

When a service breaks, it has the decency to tell you. Error rates climb, a dashboard goes red, a probe fails, someone gets paged, and the whole machinery of on-call wakes up to deal with it. It is loud, and loud is a feature.

Data does not do you that courtesy. When data breaks, the pipeline usually keeps reporting success, the dashboards stay green, and nobody gets paged. You find out weeks later, when a human notices that a number looks wrong and asks an innocent question that ruins your afternoon.

This is the story of one of those. A replication pipeline that was healthy by every metric it exposed, quietly writing NULL into a column that absolutely could not be NULL.

Table of contents

The short version

• A replication pipeline reported healthy while corrupting data on the target.
• A tenant_id column was NULL on a chunk of rows on the target, correct on the source.
• Root cause: a past window where MySQL binlog_row_image was MINIMAL, which the change-data-capture stream needs to be FULL.
• Nothing errored, because every layer faithfully did its job with less information than it needed.
• The fix was FULL going forward, a selective reload to repair the damage, and data validation so the next one is caught by a machine, not a person.

The symptom arrived as a question, not an alert

There was no incident. There was a message. Someone looking at an analytics report noticed a bucket of rows that belonged to no tenant at all, which in a multi-tenant system is roughly as alarming as finding money in your account that has no owner. Nice, until you think about it for two seconds.

That is the first thing worth sitting with. The failure had been happening for a while, and the first detector in the entire stack was a person’s eyebrow. No monitor caught it, because every monitor we had was pointed at the pipeline, and the pipeline was fine.

NULLs where a tenant should be

The shape of the problem was easy to confirm and deeply unpleasant. On the target, a meaningful slice of rows had tenant_id IS NULL. On the source, those exact rows had a perfectly good tenant.

-- on the target: rows that should be impossible
SELECT COUNT(*) FROM orders WHERE tenant_id IS NULL;
-- a number that should be zero, and was very much not zero

In a multi-tenant database, tenant_id is not just another column. It is the thing that keeps one customer’s data from bleeding into another’s. Every tenant-scoped query filters on it. A NULL there does not throw an error. It just silently drops out of every WHERE tenant_id = ? filter, which means the data is both present and invisible, wrong in the quietest possible way.

The pipeline was healthy the entire time

Here is the part that makes this a data problem and not a plumbing problem. The replication was, by its own account, perfect. Change data capture was running, latency was low, and the error count sat at a proud zero.

CDC status:        running
Replication lag:   low
Errors:            0
Tables:            all "in sync"

Everything I would normally watch said green. And every one of those signals was telling the truth about the pipeline, while telling me nothing at all about the data flowing through it. Pipeline health and data health are two different questions, and I was only asking one of them.

Root cause: one binlog setting nobody was watching

The trail led to MySQL’s binary log, and specifically to binlog_row_image.

For row-based replication, MySQL writes a “before” and “after” image of each changed row to the binlog. The binlog_row_image setting decides how much of the row it writes:

• FULL writes every column, changed or not.
• MINIMAL writes only what it needs: the primary key to find the row, plus the columns that actually changed.

MINIMAL is smaller and faster, and it is completely fine for MySQL-to-MySQL replication, because the replica already has the rest of the row. The trouble is that change-data-capture tools like AWS DMS read that same binlog and expect the full picture. The documented requirement is FULL. Give a CDC stream MINIMAL and you are handing it an update event with holes in it.

At some point in the past, through a parameter change nobody connected to this, the source spent a window running MINIMAL. During that window, every UPDATE that did not touch tenant_id produced a change event that did not mention tenant_id. The key was there. The changed column was there. The tenant was simply not in the message.

SHOW VARIABLES LIKE 'binlog_row_image';
-- FULL is what CDC needs. MINIMAL is what quietly breaks it.

Why absolutely nothing errored

This is the lesson hiding inside the incident, and it is bigger than MySQL.

Every layer did its job correctly. MySQL logged exactly what MINIMAL says to log. The binlog was valid. DMS read a valid event and applied exactly what it received. The target accepted a valid write. There was no corrupt packet, no failed job, no exception to catch anywhere in the chain.

The corruption lived in the gap between what was logged and what was needed. And gaps do not raise exceptions. They just quietly produce the wrong answer while every component around them reports success. That is the entire failure mode of data infrastructure in one sentence.

The fix, in three parts

You cannot patch this with a restart, because the damage is already written into rows.

First, stop the bleeding. Set binlog_row_image = FULL on the source so the CDC stream stops receiving events with holes. Every change from this point forward carries the whole row.

Second, repair the damage. The correct tenant_id values only ever existed on the source, so there is nothing to compute or guess. The affected tables have to be reloaded from the source, replacing the corrupted rows with the truth. A targeted full reload of just those tables is the honest fix. Painful, but honest.

Third, make it impossible to miss next time. Turn on DMS data validation, which continuously compares source and target row by row and reports mismatches. If this exact drift happens again, it shows up as an alert, not as someone’s eyebrow three weeks later.

task setting:
  EnableValidation: true
  # DMS compares source and target and flags rows that disagree

Pipeline health is not data health

The whole incident collapses into one distinction that I now treat as a rule.

Watching only the left column is how you end up confidently green while the data rots. You need something asking the right column too, and it has to be automated, because the manual version of that check is a customer.

What I watch for data now

Pipeline metrics stay, but they are the floor, not the ceiling. The signals that actually catch this class of problem look at the data itself:

source_vs_target_row_count_diff
null_rate on critical columns (tenant_id, foreign keys, money)
validation_mismatch_count
target freshness against an SLO
distribution drift on key columns

The single most valuable one is the null rate on columns that are never supposed to be null. A tenant_id null rate creeping above zero is not a metric, it is a smoke alarm, and it would have caught this in hours instead of weeks.

Final takeaway

Data infrastructure gets treated as a lesser cousin of “real” production, right up until the wrong number reaches a customer, a report, or a regulator. Then everyone remembers that data is production too.

The reason it needs SRE is precisely because it fails so politely. A down service screams. Bad data whispers, and it keeps whispering while every dashboard around it smiles and says everything is fine. So build the thing that checks the data against itself, alert on the values and not just the job, and keep a healthy distrust of the color green.

Because data does not page you. It just goes quietly wrong, and quietly wrong is the most expensive kind of wrong there is.