Change Data Capture (CDC): The Real-Time Guide with Debezium, Kafka & Iceberg

Last Updated: July 2026 | 15 min read

Quick Answer: Change Data Capture (CDC) streams every insert, update, and delete out of a database the instant it happens, by reading the database's own transaction log instead of polling tables. The standard modern stack is Debezium (the CDC engine) running inside Kafka Connect, publishing one change event per row into Kafka topics, with a sink that upserts those events into a data lake table like Apache Iceberg. Because delivery is at-least-once, you make the sink idempotent with a key-based MERGE so replays are safe. The result: an analytics copy that trails production by under a second, without touching the source with heavy queries.

Every data team eventually hits the same wall: the nightly batch job that copies your production database into the warehouse is too slow, too heavy, and blind to deletes. You add an updated_at column, then discover it misses rows that were inserted and deleted between runs. Change Data Capture is the way out — and after building CDC pipelines on Postgres, Kafka, and Iceberg in production, I'll show you exactly how it fits together, the config that actually matters, and the three failure modes that catch everyone the first time.

This guide covers what CDC is, log-based vs query-based, how Debezium works, a full Postgres → Kafka → Iceberg walkthrough, schema changes, exactly-once, and when not to use it.

What Is Change Data Capture?

Change Data Capture is a design pattern that captures row-level changes in a source database and delivers them as an ordered stream of events to other systems. Rather than asking "what does the table look like now?", CDC answers "what just changed, in what order?" — and answers it continuously.

A single change event describes one row mutation. It carries the operation type (c create, u update, d delete, r snapshot read), the row's state before and after the change, the primary key, and metadata like the source table, transaction id, and log position. Downstream consumers replay these events to reconstruct the table anywhere: a data lake, a search index, a cache, another database.

The key shift is from state to change. A batch export copies state; CDC ships the change itself. That difference is what makes deletes visible, ordering exact, and latency sub-second.

Why teams adopt CDC:

• Near-real-time analytics — the lake trails production by milliseconds to seconds, not hours.
• Zero query load on the source — reading the log costs the database almost nothing, unlike repeated SELECTs.
• Deletes are captured — a hard DELETE emits an event; polling a table can never see a row that's already gone.
• Exact ordering — events arrive in commit order, so you can rebuild history faithfully.
• Decoupling — one CDC stream fans out to many consumers (warehouse, cache, index) without extra load per consumer.

Log-Based vs Query-Based CDC: Why the Log Wins

The difference is where the changes come from: query-based CDC polls the table, log-based CDC reads the database's transaction log. That one distinction decides latency, load, and whether you can even see deletes.

Every transactional database already writes an ordered, durable log of every committed change so it can recover from a crash and feed replicas — PostgreSQL's write-ahead log (WAL), MySQL's binlog, MongoDB's oplog, SQL Server's transaction log. Log-based CDC connects as a replication client and reads that log. Query-based CDC ignores the log and instead runs SELECT ... WHERE updated_at > :last_run on a schedule.

The verdict: query-based CDC is fine as a quick hack when you control the schema and can tolerate minute-scale lag and missed deletes. For anything production-grade, log-based CDC with Debezium is the default — it's faster, lighter on the source, and the only option that never loses a delete. The rest of this guide is log-based.

How Debezium Works

Debezium is an open-source, log-based CDC engine that runs as connectors inside Kafka Connect and turns database transaction logs into Kafka topics. It's the most widely deployed CDC tool in the open-source world, with connectors for PostgreSQL, MySQL, MongoDB, SQL Server, Oracle, and more.

A Debezium connector goes through two phases:

1. Snapshot phase. On first start, the connector takes a consistent snapshot of the existing rows in the tables you configured and emits them as r (read) events. This gives consumers the full current state before any live change arrives.
2. Streaming phase. The connector then attaches to the transaction log at the exact position the snapshot ended and streams every subsequent commit as c, u, or d events — forever, in order.

Because it runs on Kafka Connect, Debezium inherits Connect's offset tracking, restarts, and horizontal scaling. The connector records how far it has read in the log (the WAL LSN in Postgres) as a Connect offset, so after a crash it resumes from exactly where it left off — no missed or (with idempotent consumers) meaningfully duplicated changes.

Here's the anatomy of a single Postgres update event, trimmed to the essentials:

{
  "op": "u",                          // c=create, u=update, d=delete, r=snapshot
  "ts_ms": 1752130000123,
  "source": {
    "db": "shop", "schema": "public", "table": "orders",
    "lsn": 24567891, "txId": 4412
  },
  "before": { "id": 42, "status": "PENDING", "total": 1999 },
  "after":  { "id": 42, "status": "PAID",    "total": 1999 }
}

The before/after envelope is what makes CDC powerful: you can see not just the new value but exactly what changed, which is essential for audit trails, slowly changing dimensions, and reversing bad writes.

Building a CDC Pipeline: Postgres → Debezium → Kafka → Iceberg

A production CDC pipeline has four moving parts: a source database with logical replication enabled, a Debezium source connector, Kafka topics, and a sink that upserts into your lake table. Let's build one end to end. This is the exact shape I use in production.

Step 1 — Enable logical replication on Postgres

Debezium's Postgres connector needs the WAL to contain full logical change information. Set the WAL level and restart:

-- postgresql.conf
wal_level = logical
max_replication_slots = 4
max_wal_senders = 4

Then create a role and grant replication, and set REPLICA IDENTITY FULL on tables where you want the complete before image on updates and deletes:

ALTER TABLE orders REPLICA IDENTITY FULL;

Gotcha #1 — the replication slot that ate the disk. A Debezium connector holds a replication slot on Postgres. If the connector goes down and stays down, Postgres retains WAL for that slot so no changes are lost — and the WAL grows until the disk fills and the database stops accepting writes. Always monitor pg_replication_slots for a growing retained_wal, and drop slots for connectors you've permanently removed.

Step 2 — Register the Debezium source connector

Debezium connectors are configured with a JSON payload posted to the Kafka Connect REST API:

{
  "name": "orders-cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "cdc_user",
    "database.password": "${env:CDC_PASSWORD}",
    "database.dbname": "shop",
    "topic.prefix": "shop",
    "table.include.list": "public.orders,public.customers",
    "plugin.name": "pgoutput",
    "snapshot.mode": "initial",
    "tombstones.on.delete": "true"
  }
}

curl -X POST http://connect:8083/connectors \
  -H 'Content-Type: application/json' -d @orders-cdc.json

This creates topics shop.public.orders and shop.public.customers — one topic per table. Each event is keyed by the table's primary key, which is what lets Kafka log compaction and downstream upserts work correctly.

Step 3 — Sink the change stream into Iceberg with a MERGE

The consumer reads each change topic and applies it to an Apache Iceberg table. The critical detail: you upsert by primary key, you never blind-insert. In PySpark with Iceberg:

from pyspark.sql import functions as F

# micro-batch of decoded Debezium events for the orders table
changes = (spark.read.format("kafka")
    .option("subscribe", "shop.public.orders")
    .load()
    .select(F.from_json("value", schema).alias("e"))
    .select("e.op", "e.after.*", "e.before.id"))

# keep only the latest change per key in this batch (ordering by log position)
latest = (changes
    .withColumn("rn", F.row_number().over(
        Window.partitionBy("id").orderBy(F.col("lsn").desc())))
    .where("rn = 1"))

latest.createOrReplaceTempView("orders_changes")

spark.sql("""
MERGE INTO lake.orders t
USING orders_changes s
ON t.id = s.id
WHEN MATCHED AND s.op = 'd' THEN DELETE
WHEN MATCHED             THEN UPDATE SET *
WHEN NOT MATCHED AND s.op != 'd' THEN INSERT *
""")

Three things make this correct: de-duplicating to the latest change per key within the batch, routing op = 'd' to a DELETE, and using MERGE so re-applying the same batch is a no-op. That last property is what turns Kafka's at-least-once delivery into an exactly-once result — the same idea covered in depth in Idempotent Data Pipelines.

Handling Schema Changes

When someone runs ALTER TABLE on the source, Debezium captures the schema change and adapts the event structure — but your downstream consumers won't unless you plan for it. This is the part that quietly breaks pipelines weeks after launch.

Debezium tracks the source schema and, when a column is added or dropped, the before/after payloads change shape. The clean way to manage this is a Schema Registry (Avro or Protobuf): Debezium registers each new schema version, the registry enforces compatibility rules, and consumers negotiate the version they can read. Configure backward-compatible evolution so adding a nullable column never breaks existing consumers.

Practical rules that keep schema changes from becoming incidents:

• Prefer additive changes. Adding a nullable column is safe; renaming or dropping one is not — treat those as breaking.
• Use a Schema Registry with BACKWARD compatibility so old consumers keep working when new fields appear.
• On the Iceberg side, rely on schema evolution — Iceberg adds columns by id, not position, so a new source column maps to a new table column without rewriting data.
• Alert on deserialization failures in the sink; a spike almost always means an unhandled schema change upstream.

Exactly-Once: Why CDC Is At-Least-Once and How to Fix It

Debezium and Kafka guarantee at-least-once delivery, not exactly-once — a consumer can see the same change event more than once after a connector restart, a Kafka rebalance, or a sink retry. Pretending otherwise is the fastest way to double-count revenue.

The reason is fundamental: after a crash, the connector resumes from its last committed log offset, and any events processed but not yet acknowledged get re-sent. You cannot eliminate this at the delivery layer. You defeat it at the processing layer by making the sink idempotent:

• Key every event by primary key (Debezium already does this).
• Upsert, don't insert — the MERGE in Step 3 applies the same change twice to the same result.
• Order within a key using the log position (lsn), so an out-of-order redelivery can't overwrite a newer value with an older one.
• Route deletes explicitly, and decide whether you want hard deletes or soft-delete tombstones in the lake.

Apply those four and at-least-once delivery becomes an exactly-once result. This is the single most important reliability property of a CDC pipeline, and it's covered end to end in our idempotent pipelines guide and its treatment of exactly-once processing.

Common CDC Mistakes to Avoid

• Ignoring the replication slot. An offline connector's slot pins WAL and can fill the disk (Gotcha #1). Monitor and clean up slots.
• Blind-inserting change events. Without a key-based upsert, at-least-once delivery double-counts. Always MERGE.
• Forgetting REPLICA IDENTITY FULL. Without it, Postgres update/delete events carry only the primary key in before, and any logic that needs the old values breaks.
• No schema evolution plan. The first ALTER TABLE in production silently poisons the sink. Use a Schema Registry from day one.
• One giant topic for everything. Keep the default one-topic-per-table; it lets you scale, compact, and reprocess tables independently.
• Snapshotting a huge table on the primary at peak. Use an incremental or read-replica snapshot so the initial load doesn't hurt production.

See how SolutionGigs can help → Standing up a reliable CDC pipeline — replication tuning, connector config, idempotent sinks, monitoring — is exactly the kind of data-engineering work our vetted experts do. Get matched on solutiongigs.in.

Frequently Asked Questions

What is Change Data Capture (CDC)?

Change Data Capture (CDC) is a technique that detects and streams every row-level change — insert, update, and delete — from a database as it happens. Instead of repeatedly querying a table, CDC reads the database's transaction log (the WAL in PostgreSQL, the binlog in MySQL) and emits an ordered event per change. Those events flow into a stream like Kafka so downstream systems stay in sync in near real time without loading the source.

What is the difference between log-based and query-based CDC?

Query-based CDC polls a table on a schedule using an updated_at column, so it adds query load, misses hard deletes, and can't see rows inserted and deleted between polls. Log-based CDC reads the transaction log directly, capturing every change including deletes, with almost no load on the source, exact commit order, and sub-second latency. Log-based CDC is the modern default, and Debezium is the most widely used engine for it.

What is Debezium and how does it work?

Debezium is an open-source, log-based CDC platform that runs as connectors inside Kafka Connect. A connector attaches to your database as a replication client, reads the transaction log, and publishes a change event to a Kafka topic for every changed row — one topic per table. It first takes a consistent snapshot of existing rows, then switches to streaming ongoing changes from the log, so consumers get the full history followed by every live change.

Is Change Data Capture real-time?

Log-based CDC is near real-time, typically delivering changes end to end in well under a second. Debezium reads the transaction log as the database flushes it, so latency depends on commit speed and connector tuning, not a polling interval. It isn't strictly synchronous — the change commits in the source first and is then captured — but for streaming, cache invalidation, and analytics it's fast enough to treat as real time.

Does CDC guarantee exactly-once delivery?

No. Debezium and Kafka deliver change events at least once, so a consumer can occasionally see the same change twice after a restart or rebalance. You reach exactly-once results by making the consumer idempotent: key each event by primary key and use an upsert (a MERGE into Iceberg or Delta) instead of a blind insert. Applying the same event twice then yields the same final row, so duplicates are harmless.

When should you not use CDC?

Skip CDC when a nightly batch load already meets your freshness needs, when you only need aggregates rather than row-level changes, or when you can't get replication access to the source. CDC adds operational weight — a Kafka Connect cluster, replication slots to monitor, schema-change handling — so it's worth it when you genuinely need low-latency, delete-aware sync from a transactional database, and overkill when a scheduled query would do.

Can CDC replace ETL?

CDC replaces the extract step of ETL, not the whole thing. It gives you a low-latency, complete stream of source changes, but you still transform and model that data downstream. In practice CDC feeds the raw/bronze layer of a lakehouse, and tools like dbt or Spark handle the transformation into clean, modeled tables. Think of CDC as a better extract, not a replacement for modeling.

Conclusion

Change Data Capture turns your database's own transaction log into a real-time stream of every insert, update, and delete — and once you have that stream, keeping a data lake, a search index, or a cache in sync with production becomes a solved problem instead of a nightly batch gamble. The winning stack in 2026 is Debezium reading the WAL, Kafka carrying one keyed topic per table, and an Iceberg sink that upserts with a MERGE so at-least-once delivery becomes an exactly-once result.

Get the fundamentals right and the rest follows: enable logical replication, watch your replication slots, upsert by key, and plan for schema evolution before the first ALTER TABLE. Do that, and you'll have an analytics copy that trails production by less than a second — the foundation every modern data platform is built on.

Building a streaming or CDC pipeline and want it done right the first time? Get matched with a vetted data engineer on solutiongigs.in today — it's free to post. For the wider picture, start with What Is Data Engineering? and The Modern Data Stack in 2026.

Mohammed Yaseen

Founder, SolutionGigs

Mohammed builds streaming and CDC pipelines on Kafka, Spark, and Iceberg, and writes about doing data engineering the reliable way. LinkedIn →