Exactly-Once Processing in Spark Structured Streaming

Last Updated: July 2026 | 15 min read

Quick Answer: Exactly-once processing in Spark Structured Streaming means every record affects your output exactly one time, even across failures and restarts. Spark delivers it only when all three pieces are in place: a replayable source (Kafka, which re-reads from a committed offset), checkpointing (so offsets and state survive a crash), and an idempotent or transactional sink (so a replayed batch produces no duplicates). Spark's engine guarantees exactly-once internal state, but the write is your job — a plain append to a database or API is only at-least-once and will duplicate rows after a failure. The practical tool is foreachBatch, which gives you a deterministic batchId you upsert or dedupe on.

Ask ten engineers whether Spark Structured Streaming is "exactly-once" and you'll get ten confident, contradictory answers. The truth is more precise — and more useful. Spark's checkpointing gives you exactly-once recovery of its own state, but whether your pipeline is exactly-once depends entirely on the source and the sink you bolt onto it. Get the sink wrong and you'll ship duplicate rows the first time an executor dies at 3 a.m., no matter how many checkpointLocation options you set. This guide explains what exactly-once actually guarantees, the two halves that must both hold, and the real PySpark patterns — foreachBatch + idempotent upserts — that make it true in production.

What exactly-once actually means (and what it doesn't)

Exactly-once is a statement about effects, not about physical processing. Spark may read, compute, and even write a record more than once after a failure — but if the final result reflects that record exactly one time, the pipeline is exactly-once. Nobody achieves it by literally touching each record once; they achieve it by processing at-least-once and making the write idempotent so replays are invisible.

The three delivery semantics, precisely:

The subtle point that trips people up: exactly-once is built on top of at-least-once, not instead of it. Spark's recovery deliberately re-processes the last uncommitted batch after a crash — that's how it avoids data loss. The duplicates that re-processing would create are then cancelled out by an idempotent sink. Remove the idempotency and you're left with plain at-least-once and visible duplicates.

The one-sentence mental model: Spark gives you exactly-once state; you must supply a replayable source and an idempotent sink. Miss either and the guarantee silently downgrades.

The two halves of exactly-once

Exactly-once is a contract between a replayable source and an idempotent sink, notarized by the checkpoint. Both halves are mandatory — the checkpoint alone is not enough. Here is what each half does.

Half 1 — A replayable source

A replayable source can re-deliver records from a known position after a failure. Apache Kafka, the canonical example, stores every record at a durable offset; Spark records exactly which offsets each batch consumed. If the job dies mid-batch, Spark restarts and re-reads from the last committed offset — no record is skipped (no loss) and none is permanently lost in flight.

Sources like a plain socket or a non-durable queue are not replayable: once a record is read and the job crashes, it's gone. With a non-replayable source, exactly-once — and even at-least-once — is impossible, because there's nothing to replay. This is why every serious streaming pipeline reads from Kafka, Kinesis, or a file source, never a raw socket.

Half 2 — An idempotent (or transactional) sink

An idempotent sink produces the same final state whether a batch is written once or five times. This is the half Spark cannot do for you, and the one that quietly breaks pipelines. Two ways to get it:

• Transactional sink — Delta Lake or Apache Iceberg commit each micro-batch atomically and record the batch's stream version, so a replayed batch is recognized and skipped. With these, exactly-once is essentially automatic.
• Idempotent write — for a non-transactional sink (JDBC database, key-value store, REST API), you make the operation idempotent: MERGE/upsert on a unique key, or record the last committed batchId and skip anything already applied.

If the sink is a plain append to a database table and nothing dedupes, you have at-least-once — full stop.

How Spark's checkpointing makes recovery exactly-once

The checkpoint directory is the durable ledger that lets Spark restart from precisely where it left off. It stores the offsets each batch consumed, a commit log of which batches finished, and the state store — and the write-ahead-log ordering across these is what makes recovery exactly-once.

Every streaming query needs a checkpoint location:

query = (result.writeStream
    .format("delta")
    .option("checkpointLocation", "s3://bucket/checkpoints/clicks_agg")
    .outputMode("append")
    .start())

Inside that directory Spark maintains:

The recovery sequence is what actually delivers the guarantee:

1. On start, Spark reads the offset log to find the last planned batch, and the commit log to find the last committed batch.
2. If a batch was planned but not committed (the crash window), Spark re-plans it with the exact same offsets — same input, same batchId.
3. It recomputes and re-writes that batch. A transactional/idempotent sink absorbs the replay; a plain sink duplicates.

The write-ahead-log ordering is the whole trick: offsets are logged before the batch runs, and the commit is logged after the sink succeeds. So a crash can only ever leave a batch in the "planned but not committed" state — which is safe to replay — never in a "committed but lost" state.

Two rules follow directly, and breaking either destroys exactly-once:

• Never delete or reuse a checkpoint directory. One query, one checkpoint. Pointing two queries at the same location, or wiping it to "reset", corrupts the ledger.
• Changing a stateful query (its keys, watermark, or aggregation) can make an old checkpoint incompatible. Plan a migration or a new checkpoint path — see the Spark streaming tuning guide.

foreachBatch: the workhorse for exactly-once sinks

foreachBatch is how you get exactly-once into any sink Spark doesn't natively support. It hands you each micro-batch as a static DataFrame plus a deterministic batchId, and because Spark replays the same batchId with the same data after a failure, you can make the write idempotent. This is the single most important pattern in the article.

def upsert_to_delta(micro_batch_df, batch_id):
    # micro_batch_df is a normal (static) DataFrame; batch_id is monotonically increasing
    (DeltaTable.forName(spark, "clicks_agg").alias("t")
        .merge(micro_batch_df.alias("s"), "t.window = s.window AND t.key = s.key")
        .whenMatchedUpdateAll()
        .whenNotMatchedInsertAll()
        .execute())

(result.writeStream
    .foreachBatch(upsert_to_delta)
    .option("checkpointLocation", "s3://bucket/checkpoints/clicks_agg")
    .outputMode("update")
    .start())

Because the MERGE keys on (window, key), running the same batch twice produces the same table state — the replay is a no-op. That's idempotency, and it's what upgrades an at-least-once flow to exactly-once effects.

Idempotent writes to a non-transactional sink

For a sink with no MERGE — a JDBC warehouse, a REST API, a cache — track the last committed batchId in the sink itself and skip anything already applied:

def write_idempotent(micro_batch_df, batch_id):
    # 1. Has this batch already been committed? (read a small control row)
    last = get_last_committed_batch_id(target="orders_summary")  # your query
    if batch_id <= last:
        return  # replay of an already-applied batch — skip, stay idempotent

    # 2. Do the real write and record the new batch_id in ONE transaction
    with target_transaction() as tx:
        micro_batch_df.write.jdbc(...)          # or POST to an API
        record_committed_batch_id(tx, batch_id) # commit marker + data together

The critical detail: the data write and the batchId marker must commit together. If they can't be atomic, prefer an upsert keyed on a business key so a partial replay still converges. When even that isn't possible (a fire-and-forget API), send a stable idempotency key — sha2(business_key) or batchId + row_key — and let the receiver dedupe. That gives exactly-once effects on top of an at-least-once HTTP call.

foreachBatch runs on the driver, per batch — not per row. The DataFrame inside is static, so you can MERGE, write to two sinks, or cache it. But it disables continuous-processing mode and runs once per micro-batch, which is exactly what makes the batchId a reliable idempotency key.

Kafka-to-Kafka exactly-once

When both the source and sink are Kafka, Spark's Kafka sink writes each batch inside a Kafka transaction tied to the checkpoint, giving end-to-end exactly-once as long as consumers read with isolation.level=read_committed. Since Spark 3.0 the Kafka sink is transactional.

(result.selectExpr("CAST(key AS STRING)", "to_json(struct(*)) AS value")
    .writeStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("topic", "enriched-events")
    .option("checkpointLocation", "s3://bucket/checkpoints/enriched")
    .start())

Two things you must get right for this to actually be exactly-once:

1. Downstream consumers must set isolation.level=read_committed. Otherwise they read aborted/uncommitted messages from a replayed batch — reintroducing duplicates on the read side.
2. Don't share transactional.id across queries. Spark derives it from the checkpoint; a shared checkpoint (already forbidden above) also breaks transactional fencing.

Common mistakes that silently break exactly-once

Most "we're getting duplicates but checkpointing is on!" incidents come from this short list:

1. Non-idempotent sink. The number-one cause. Checkpointing is on, but the sink is a plain append, so the replayed crash-batch duplicates. Fix: foreachBatch + MERGE, or a transactional sink (Delta/Iceberg).
2. Sharing or deleting the checkpoint. Two queries on one checkpoint, or wiping it to "start fresh", corrupts the offset/commit logs. One query, one checkpoint, forever.
3. Assuming the source is replayable when it isn't. A socket or a queue that acks-on-read can't replay; exactly-once is impossible there regardless of sink.
4. Non-deterministic transformations. foreachBatch idempotency assumes the same input produces the same output. A current_timestamp(), a random id, or a non-deterministic UDF makes the replayed batch different, so the upsert key drifts and duplicates slip through. Derive keys from event data, not wall-clock time — the same discipline behind idempotent data pipelines.
5. Committing the data and the batch marker separately. If the write succeeds but the marker write fails, the replay re-applies the batch. Commit them atomically, or upsert on a business key.
6. Kafka consumers on read_uncommitted. Your producer side is transactional, but downstream reads uncommitted messages and sees duplicates. Set isolation.level=read_committed.
7. Confusing exactly-once with ordering. Exactly-once says nothing about order. If you need ordered results, that's a separate concern (partitioning, sequence numbers) — don't assume the guarantee covers it.

See how SolutionGigs can help

Chasing phantom duplicates in a streaming pipeline, or unsure whether your sink is actually idempotent under failure? SolutionGigs connects you with vetted data engineers who've shipped production Spark + Kafka systems and debugged these exact recovery edge cases. See how SolutionGigs can help →

Frequently Asked Questions

Does Spark Structured Streaming guarantee exactly-once processing?

Spark guarantees exactly-once end-to-end only when three conditions hold: the source is replayable (like Kafka), the query uses checkpointing so offsets and state survive a restart, and the sink is transactional or idempotent. Spark itself provides exactly-once internal state, but the sink is your responsibility — a plain append to a database or API is only at-least-once and will duplicate rows after a failure.

What is the difference between at-least-once, at-most-once, and exactly-once?

At-most-once processes each record zero or one times — fast, but data can be lost. At-least-once processes each record one or more times — no loss, but failures create duplicates. Exactly-once means every record affects the result exactly one time. Crucially, exactly-once is built on at-least-once plus an idempotent write, not on literally processing each record once.

How does foreachBatch enable exactly-once writes in Spark?

foreachBatch gives you each micro-batch as a static DataFrame plus a deterministic batchId. Because Spark replays the same batchId with the same data after a failure, you can make the write idempotent — a MERGE/upsert on a business key, or skipping any batchId already recorded in the sink. That turns a non-transactional sink like JDBC or a REST API into an effectively exactly-once one.

What does the checkpoint directory store in Spark Structured Streaming?

It stores the source offsets planned for each batch (offset log), a commit log of which batches finished, the streaming state store for aggregations and joins, and query metadata. On restart Spark uses the offset and commit logs to find the last committed batch and re-reads any uncommitted offsets, which is what makes recovery exactly-once. Never delete or share a checkpoint between queries.

Why am I still getting duplicate rows with checkpointing enabled?

Checkpointing makes Spark's internal processing exactly-once, but duplicates come from a non-idempotent sink. If a batch is written and the driver dies before the commit is recorded, Spark replays that batch and writes the same rows again. Fix it with foreachBatch + a MERGE/upsert on a unique key, a transactional sink like Delta or Iceberg, or deduping by batchId.

Is exactly-once possible when writing to a non-transactional sink like a REST API?

Yes, effectively — by making the downstream operation idempotent instead of relying on transactions. Send a stable idempotency key (a hash of the business key, or batchId plus row key) so the receiver ignores a repeated request, or have the API upsert instead of insert. Combined with foreachBatch and checkpointing, this gives exactly-once effects even though the HTTP call itself is at-least-once.

Does exactly-once processing hurt streaming throughput?

There's a modest cost, not a cliff. Checkpointing writes offset/commit logs each batch, and MERGE/upsert sinks do more work than a blind append — but both are small relative to the shuffle and state costs of a real pipeline. Transactional sinks like Delta/Iceberg add commit overhead measured in the low hundreds of milliseconds per batch. For nearly all workloads, correctness is worth it; only extreme low-latency cases lean toward at-least-once with downstream dedup.

Conclusion

Exactly-once in Spark Structured Streaming isn't a checkbox — it's a contract you assemble from three parts that must all hold: a replayable source, checkpointing, and an idempotent or transactional sink. Spark gives you exactly-once recovery of its own state for free, but it can only carry the guarantee across the finish line if you supply the other two halves. The failure mode is quiet: everything looks fine until an executor dies, a batch replays, and a non-idempotent sink writes the same rows twice.

The playbook that keeps production streaming correct:

• Read from a replayable source (Kafka/Kinesis/files) — never a socket if you care about correctness.
• Always set checkpointLocation, one per query, and never delete or share it.
• Make the sink idempotent — foreachBatch + MERGE/upsert on a business key, a transactional sink (Delta/Iceberg), or a stable idempotency key for APIs.
• Keep transformations deterministic so replayed batches produce identical keys.
• For Kafka sinks, read downstream with isolation.level=read_committed.

Nail those and your pipeline survives restarts, rebalances, and spot-instance reclaims without a single duplicate. If you're architecting a real-time system and want a second set of eyes on where duplicates could sneak in — or whether you even need streaming over a simpler batch approach — get matched with a vetted data engineer on SolutionGigs. It's free to post your project.

Mohammed Yaseen

Founder, SolutionGigs

Mohammed runs Spark Structured Streaming pipelines on AWS EMR at Telemetrix, moving hundreds of millions of Kafka telemetry events a day into Iceberg on S3 with exactly-once guarantees. He's chased the duplicate rows, checkpoint-corruption, and non-idempotent-sink bugs this guide warns about — in production, not in theory. LinkedIn →