What Is Change Data Capture?

How CDC Powers Modern AI Data Pipelines

Every time a customer updates a shipping address, a support agent closes a ticket, or a payment clears, that change happens in a source database somewhere. The question is how long it takes the rest of your data stack to find out. For a lot of teams, the honest answer is hours, or until the next batch job runs.

Change data capture, or CDC, is a technique for tracking exactly what changed in a source database the moment it changes, and delivering just that delta to whatever system needs it next. No table reloads. No waiting for the nightly job. Just inserts, updates, and deletes, streamed out as they happen.

That's the one-line version. The more useful part is how CDC actually works under the hood, and why it's become the default expectation for anything feeding an AI system rather than a nice-to-have for real-time dashboards.

How CDC Actually Works

CDC isn't one technique. It's three, and picking the wrong one is a common way teams get burned.

Log-based CDC reads the database's own transaction log: the write-ahead log (WAL) in PostgreSQL, the binlog in MySQL, or the redo log in Oracle. Every database already writes to this log to support crash recovery, so log-based CDC piggybacks on something that's happening anyway. It doesn't query production tables directly, and it adds close to zero overhead. This is why it's the standard for anything customer-facing or transaction-heavy, and why tools built to standardize CDC, like the open-source Debezium project, read logs directly rather than reaching for the alternatives below.

Trigger-based CDC fires a database trigger on every insert, update, and delete, writing the change to a shadow table. It's simpler to stand up than log parsing, but every trigger adds write latency to the source table, and a schema change means rebuilding triggers by hand. Teams running Oracle GoldenGate or similar tools have used this approach for decades. It's less common elsewhere for exactly that performance reason.

Query-based CDC polls tables on a schedule, comparing timestamps or version columns to spot what changed. It's the easiest to set up and the weakest option. It misses anything that changes and reverts between polls, and it typically can't detect deletes at all, since the row it would check is simply gone.

MethodHow it detects changesSource impactCatches deletesLog-basedReads the transaction logMinimalYesTrigger-basedDatabase triggers on writeAdds latency per writeYesQuery-basedPolls and compares timestampsRepeated query loadNo

At production scale, log-based CDC wins almost every time. The other two show up mostly where log access isn't possible.

CDC and Data Replication Aren't the Same Thing, but They're Joined at the Hip

Replication is the goal: keep a current copy of your data somewhere else, usually a cloud data warehouse. CDC is a way to get there, and specifically the way that avoids the two costs baked into full-table reloads: the load you put on the source system every time you re-extract everything, and the lag between when something happens and when your warehouse finds out.

Before CDC was practical at scale, most replication ran on batch reloads: pull the whole table, overwrite the copy, repeat on a schedule. That works fine until the table gets big or the business needs the copy current within minutes instead of a day. CDC replaces "reload everything" with "apply just what changed," so replication scales with how fast your data changes, not with how large the table has grown.

Why CDC Matters More Now: AI Pipelines Don't Forgive Stale Data

Batch-driven staleness used to be mostly a reporting problem. A dashboard that's six hours behind is annoying. An AI agent acting on data that's six hours behind can cause real damage, and it usually fails quietly rather than throwing an error.

That's the pattern industry analysts keep landing on. Gartner estimates that 88% of enterprise AI agent projects never reach production, and the infrastructure postmortems explaining why keep pointing at the same root cause: retrieval systems built for a demo, not a workload, where missing freshness guarantees and unmonitored schema drift only surface weeks after launch, once the system is quietly running on data nobody is still checking.

CDC doesn't fix bad retrieval logic or a poorly tuned model. What it removes is one whole category of failure: the vector index, feature store, or context layer an agent reads from being built on data that was true an hour ago and quietly isn't anymore. If an AI system touches operational data at all, CDC is what keeps "true an hour ago" from becoming the default state of the pipeline feeding it.

How This Actually Gets Built

Here's where the theory runs into practice. Writing a CDC pipeline by hand means standing up log parsing for each source database, handling schema drift without breaking the stream, and running an agent that can survive a network blip without losing changes. One frequently cited estimate puts the build time for a single new connector pipeline at four to six weeks, plus roughly a week a quarter just to keep it running once it's live.

Maia's CDC service handles the log-reading layer itself, using Debezium connectors under the hood for sources like Oracle, where it reads directly through Oracle's LogMiner rather than querying tables. A CDC agent runs inside the customer's own VPC, connects to the source database, and does an initial snapshot before moving into continuous streaming mode. Change events get written as partitioned batches to cloud storage in near real time, and from there, shared jobs pick them up and apply them into Snowflake, Databricks, Redshift, or BigQuery, running alongside existing batch pipelines rather than as a separate system to babysit.

That snapshot-then-stream pattern matters more than it sounds. It's the difference between a pipeline that only sees changes going forward and one that starts with an accurate full copy and stays current from there, without a separate backfill job.

Engage3 put this to the test on its own retail pricing data. After moving its PostgreSQL-to-Snowflake replication onto Matillion CDC, the company cut its time-to-complete by 95% compared with its previous approach. As CTO Anup Doshi has said of the broader initiative, the goal was always getting customers timely, trustworthy recommendations, not just technically-current data. The two turned out to depend on each other.

The Takeaway

Batch ETL isn't going away, and it doesn't need to. Scheduled transformations inside a warehouse are still the right tool for plenty of jobs. But treating CDC as an edge case for high-volume reporting is a bet that's aging badly. Once anything downstream, whether a dashboard, a feature store, or an AI agent, depends on data being current rather than eventually current, the reload-and-wait model becomes the thing you have to justify, not the default.