Apache Spark vs Apache Flink: Choosing a Stream Processing Engine

Last Updated: August 2026 | 14 min read

Quick Answer: Apache Spark is a batch engine that streams by running many small batches, so its default latency floor is the trigger interval. Apache Flink is a streaming engine whose operators run continuously and hand records to each other one at a time, so there is no batch boundary to wait for. That single difference generates every other difference you will read about. But the most-repeated version of this comparison is now out of date: Spark 4.1 shipped Real-Time Mode, which removes the micro-batch boundary and reaches millisecond-range p99 latency — though only for a restricted set of pipelines (no foreachBatch, no Delta sink, update output mode only, and stateless-only in open-source Spark). So in 2026 the decision is less about latency in the abstract and more about state, recovery granularity, sink support, and whether a second engine is worth its operating cost.

Almost every Spark vs Flink article you will find makes the same argument: Spark does micro-batching, Flink does true streaming, therefore pick Flink if you need low latency. That was a good summary for about eight years. It is now a version-dependent claim, and repeating it will lead you to add an engine to your platform you may not need.

This guide gives you what those articles don't: the one architectural idea both engines are built on, what actually changed with Spark 4.1 Real-Time Mode and Flink 2.0 disaggregated state — including the RTM limitation table the announcement coverage skips — how recovery and backpressure genuinely differ (this matters more day-to-day than latency), a latency ladder to test whether you need per-event processing at all, and the honest cost of running two engines, including why we chose not to adopt Flink on our own platform.

The one idea: is there a boundary between records?

The entire Spark vs Flink comparison reduces to one question: does the engine put a boundary around a group of records before processing them?

• Spark's default answer is yes. A Structured Streaming query wakes up on a trigger, reads the offsets that arrived since last time, plans a job, runs its stages, writes the output, commits the offsets, and goes back to sleep. Records are processed as a set. Nothing at all happens between triggers.
• Flink's answer is no. A Flink job deploys a graph of operators that stay running for the lifetime of the job. A record enters the source operator, gets handed to the next operator, then the next, and reaches the sink — without ever waiting for a peer.

Everything downstream follows from that:

Hold onto that table. It's the reason the two engines feel different to operate even when they compute the same result.

What is Apache Spark?

Apache Spark is a unified open-source engine for large-scale data processing that runs batch, SQL, streaming and machine-learning workloads on the same runtime. It came out of UC Berkeley's AMPLab in 2009, is now an Apache Software Foundation project, and its commercial centre of gravity is Databricks.

Spark's streaming API is Structured Streaming, and its defining idea is that a stream is an unbounded table. You write the same DataFrame code you'd write for a batch job, and Spark incrementalises it. That is a genuinely large productivity win: a team that knows PySpark DataFrames already knows most of Spark streaming.

Execution is micro-batch by default. trigger(processingTime="30 seconds") gives you a batch every 30 seconds; the default trigger runs batches back to back as fast as they complete.

What is Apache Flink?

Apache Flink is an open-source engine for stateful computations over unbounded and bounded data streams, built streaming-first with record-at-a-time processing, keyed state and event-time semantics as primitives rather than additions. It began as the Stratosphere research project in Berlin, became an Apache top-level project in 2014, and its main commercial backers are Confluent (via Immerok) and Alibaba.

Flink's model is a dataflow graph of long-running operators. You describe sources, transformations and sinks; Flink deploys them as parallel subtasks connected by data channels, and they stay up. State lives with the operator, keyed by the partitioning key, and is checkpointed to durable storage.

Flink gives you three API levels — SQL/Table API, DataStream API, and low-level process functions with explicit access to state and timers. That bottom layer is the part Spark has no direct equivalent for, and it's why event-driven applications gravitate to Flink.

What changed: Spark 4.1 Real-Time Mode

In 2026 the sentence "Spark can't do low latency because it micro-batches" stopped being unconditionally true. This is the most important recent development in this comparison, it is almost entirely missing from the articles currently ranking for it — and, as you'll see below, it comes with restrictions that most of the coverage also omits.

Apache Spark 4.1.0 shipped the first official support for Real-Time Mode (RTM) in Structured Streaming, from SPARK-52330. Two changes do the work:

1. Concurrent stage scheduling. Instead of running stage 1 to completion, then stage 2, RTM keeps long-lived stages running simultaneously so a record doesn't wait at a stage boundary.
2. In-memory streaming shuffle. Data moves between stages through memory rather than being materialised to disk, removing the write-then-read round trip between stages.

Checkpointing is also decoupled from processing: RTM checkpoints on a timer (default five minutes) rather than once per batch, so durability no longer sets your latency.

You enable it with a trigger change rather than a rewrite — the transformations above the writeStream call stay the same:

// Micro-batch (classic Structured Streaming)
events.writeStream
  .trigger(Trigger.ProcessingTime("30 seconds"))
  .foreachBatch(upsertToIceberg _)
  .start()

// Real-Time Mode — same DataFrame logic, different execution model
events.writeStream
  .trigger(RealTimeTrigger.apply("5 minutes"))   // arg = checkpoint interval, NOT batch size
  .option("checkpointLocation", "s3://bucket/ckpt/alerts")
  .format("kafka")
  .start()

Databricks reports p99 latencies from a few milliseconds up to roughly 300 ms depending on transformation complexity, with one customer case study at 15 ms end-to-end p99.

The RTM limitations nobody mentions

This is the part that decides whether RTM is actually available to you, and it's missing from essentially every article celebrating the feature. Two different things get called "Real-Time Mode," and they are not the same size.

• Open-source Apache Spark 4.1 ships RTM as Scala, stateless support (SPARK-53736). The release note is explicit that single-digit-millisecond latency is for stateless tasks. If your pipeline is a stateful aggregation in PySpark, OSS 4.1 RTM is not your answer yet.
• Databricks Runtime carries a wider version — Scala, Java and Python, plus transformWithState, dropDuplicates, tumbling and sliding windowed aggregations, stream-to-table joins, and stream-to-stream inner joins on newer runtimes.

Even on the wider version, the unsupported list is load-bearing:

That foreachBatch line matters more than it looks. foreachBatch is the standard Spark pattern for idempotent sink writes — a MERGE into Delta or Iceberg keyed on batchId, which is how most teams get exactly-once semantics. Combine "no foreachBatch" with "no Delta sink" and the shape of an RTM pipeline becomes clear: it is built for Kafka-in, Kafka-out or API-out event processing — not for writing a lakehouse table. Which is, not coincidentally, exactly the workload people reach for Flink to do.

So read the headline carefully. Spark 4.1 did not become Flink. It gained a genuinely useful low-latency mode for a specific pipeline shape, from the vendor whose engineers built it and published the benchmarks. Treat "Spark can now do millisecond latency" as plausible for your workload, worth benchmarking, and subject to the table above. If you already run Flink for a latency reason, RTM is a reason to re-measure — not, on this evidence, a reason to migrate.

What changed on the Flink side: disaggregated state

Flink 2.0, released in March 2025, moved primary state storage out of the worker and into object storage — the biggest change to Flink's architecture in a decade, and the answer to its oldest operational complaint.

Classic Flink kept keyed state in an embedded RocksDB instance on the worker's local disk. That is fast, but it means your state size is bounded by attached disk, checkpoints must copy local files to durable storage, and rescaling means shuffling state files around.

Flink 2.0 introduces ForSt, a state store that treats a remote distributed filesystem (S3, HDFS) as the primary home for state and uses local disk as a cache, paired with an asynchronous execution model to hide remote-storage latency. Per the disaggregated state documentation, the payoff is state size limited only by the object store, plus much lighter checkpoints and faster rescaling.

Flink has kept moving since: 2.2 added model-inference operations in the Table API and broader delta-join support, and 2.3 (June 2026) added changelog conversion operators, richer materialized tables and a rewritten native S3 filesystem.

The honest summary of both roadmaps: Spark spent 2025–26 attacking its latency weakness, and Flink spent it attacking its operability weakness. They are converging, which is precisely why the old one-line answer no longer decides anything.

Where the engines still genuinely differ

Latency is the headline and the least useful criterion. These four are what you'll actually feel.

1. State and timers — Flink's deepest advantage

Flink's state API is more expressive, and that difference doesn't disappear with a config flag. In a KeyedProcessFunction you get per-key value/list/map state and per-key event-time and processing-time timers, and you can express logic like "if I've seen a payment_started for this session and no payment_completed within 90 seconds, emit an abandonment event."

Spark answered with the transformWithState API (arbitrary stateful processing with state variables and timers), which covers a great deal of this ground. But Flink's version has more years of production behind it, and Flink 2.0's disaggregated state removes the "how big can state get" ceiling that used to force teams to redesign.

Rule of thumb: if your business logic is per-entity state machines with timeouts, Flink is the better fit. If it's windowed aggregations and joins over streams, both are fine and you should pick on other grounds. Our guide to streaming joins and windowing covers the semantics that are identical in both engines.

2. Failure recovery granularity

This is the difference that surprises teams the most, and it cuts in Spark's favour.

• Spark: a failed task is retried inside the same micro-batch. One bad executor loses a fraction of a batch's work. If the whole batch fails, Spark reruns it from the committed offsets. Blast radius: one batch.
• Flink: a failed subtask triggers a restart of its pipeline region, and every operator in that region rewinds to the last completed checkpoint and reprocesses from there. Blast radius: everything since the last checkpoint.

So Flink's recovery cost is tied to your checkpoint interval, and there's a real tension: frequent checkpoints mean cheap recovery but more overhead; infrequent checkpoints mean the opposite. Sizing that trade-off, and getting checkpoints to complete at all under backpressure, is where most Flink operational pain lives.

3. Backpressure handling

Flink propagates backpressure automatically through the whole pipeline; Spark limits how much data it admits per batch. Two different philosophies with two different failure signatures.

Flink uses credit-based flow control: a slow downstream operator stops granting credits, its upstream buffers fill, and the slowdown travels back to the source, which stops reading from Kafka. Nothing is dropped, nothing overflows — but a backpressured pipeline can struggle to complete checkpoints, because barriers get stuck behind buffered data. That's exactly what unaligned checkpoints exist to fix: barriers overtake buffered records, and the in-flight data becomes part of the checkpoint.

Spark's control is admission-based: you cap maxOffsetsPerTrigger (Kafka) so a batch never takes on more than it can finish. Simpler to reason about, and the symptom is easy to read — consumer lag grows and batch duration exceeds the trigger interval. Less automatic, but far fewer ways to be subtly wrong.

4. Batch is not a tie

Spark is the better batch engine, and this is not close. Flink genuinely supports bounded execution and it has improved a lot, but Spark has the mature cost-based optimizer, adaptive query execution, the deepest connector ecosystem, first-class Iceberg and Delta integration, and the notebook and SQL tooling analysts already use.

If your platform is 80% batch ETL and lakehouse maintenance with a couple of streaming pipelines — which describes most data platforms — Spark covers all of it with one engine and one skill set.

Spark vs Flink: full comparison

One-sentence verdict: default to Spark because it covers batch and streaming with one engine and one team's skills, and add Flink when you have a specific event-driven workload whose per-event latency or per-key state logic Spark genuinely can't serve — then measure that claim before you commit.

The latency ladder: do you actually need per-event processing?

Most teams that "need real time" need something narrower than that. Before choosing an engine, place your requirement on this ladder — it is much cheaper to move down a rung than to add an engine.

The honest test is one question: what decision gets made differently because the number arrived in 50 ms instead of 30 seconds? If a human looks at it, or an alert routes to a queue somebody reads within minutes, you are on rung 3 or 4 and Spark's micro-batch is not your problem.

Rung 5 is real, though, and it's where Flink earns its reputation: blocking a fraudulent card authorisation, pricing an ad impression, throttling an API abuser, triggering a safety interlock. If a machine acts on the result in a loop tighter than a second, you're on rung 5.

How to choose: a decision guide

Work through these in order and stop at the first clear answer.

1. Is more than half your workload batch or SQL analytics? → Spark. Don't add a streaming-first engine to a batch-first platform for a minority of pipelines.
2. Is your logic per-entity state machines with timeouts, or event-pattern detection? → Flink. Process functions, timers and CEP are the reason it exists.
3. Do you need rung-5 latency (a machine acting sub-second)? → Check the RTM constraint table first. Kafka-in/Kafka-out with update semantics → benchmark Spark RTM before adding an engine. Anything that must land in a Delta or Iceberg table at that latency → Flink, because RTM currently excludes those sinks. Measure your own pipeline either way; published numbers are for someone else's transformations.
4. Is your state very large — tens of terabytes of keyed state? → Flink 2.0+ with disaggregated state is the stronger story.
5. Are you already running Spark, and the ask is "make this pipeline faster"? → Tune before you migrate. Shorten the trigger, fix data skew, right-size executors, then try RTM. A migration is a quarter; a trigger change is an afternoon.
6. Is your team fewer than about five data engineers? → One engine. Whichever one you pick, the second engine's real cost is not licences, it's the second set of things you must be good at when something breaks at 2am.

Designing a streaming platform and want a second opinion before it carries production load? See how solutiongigs.in can help →

The two-engine tax (and why we didn't adopt Flink)

At SolutionGigs we run Telemetrix, an infrastructure-monitoring platform that ingests device and service telemetry through Kafka and processes it with Spark Structured Streaming on EMR, landing into Iceberg. When we designed the alerting path, we evaluated Flink seriously, and we chose not to adopt it. The reasoning is more useful than a benchmark.

The latency argument didn't survive contact with the requirement. Our alert path had to detect a threshold breach and notify. We had assumed "real time" meant per-event. When we actually traced the chain, the notification hop — queue, dedupe window, delivery — added its own seconds, and the humans receiving alerts were not going to act faster than that. We were on rung 4, not rung 5. A 10-second trigger satisfied the requirement with margin.

The cost we would have paid was concrete. Adding Flink meant a second cluster to size and patch, a second checkpointing model to understand, a second set of monitoring dashboards and alert rules, a second failure-recovery playbook, and — the part teams underestimate — a second thing every engineer had to be competent at during an incident. For one pipeline, on a small team, that is a bad trade.

Where we did feel a real limitation, it wasn't latency — it was state. One feature needed per-device session logic with timeouts ("device went quiet mid-deploy and never reported completion"). That's exactly Flink's sweet spot, and expressing it in Spark took more code than it should have. We implemented it with a stateful transform and a 15-minute watermark, and it works — but if we had three more features shaped like that one, Flink would have won on merit.

The generalisable lesson: the case for a second engine is almost never latency in isolation. It's a cluster of workloads whose shape doesn't fit your first engine. One awkward pipeline is a code smell. Five is an architecture decision.

Common mistakes to avoid

1. Choosing Flink for latency you can't use. The most common and most expensive error. Quantify the requirement end to end — including the hops after your engine — before it drives an architecture choice.

2. Quoting the micro-batch argument without checking versions — or over-correcting the other way. "Spark micro-batches, therefore it's slow" was true of Spark 3.x defaults, and on 4.1 with Real-Time Mode it's a configuration rather than a limit. But the opposite error is now just as common: assuming RTM applies to your pipeline. Check the output mode, the sink, and whether you rely on foreachBatch before you plan around it. Version-check every performance claim in this space, including the ones in this article.

3. Setting Flink's checkpoint interval by copying a blog post. Your checkpoint interval sets your recovery cost, and state size determines whether checkpoints complete at all. Start from "how much reprocessing can I tolerate on failure" and validate under load, with unaligned checkpoints enabled if you see backpressure.

4. Assuming exactly-once comes from the engine. It doesn't. It comes from a replayable source, durable checkpointed state, and a sink that can deduplicate or commit transactionally. Both engines give you two of the three; the sink is your job in both. See exactly-once processing for the full contract.

5. Running Flink batch jobs because "Flink does batch too." True and rarely wise. If Spark is already in your platform, batch on Flink buys you nothing and costs you the mature optimizer and connector ecosystem.

6. Ignoring the Kafka side of the problem. Partition count caps parallelism in both engines. Sixteen partitions means at most sixteen useful parallel readers — a hundred Flink slots or Spark cores will not help. Fix the topic before you blame the engine.

7. Benchmarking with a trivial pipeline. A stateless filter over Kafka tells you almost nothing. Latency and stability diverge between these engines under stateful load — large keyed state, skewed keys, late data, restarts. Benchmark the pipeline you're actually going to run.

Frequently Asked Questions

What is the main difference between Apache Spark and Apache Flink?

Spark was designed as a batch engine and streams by running many small batches, so its default latency floor is the trigger interval. Flink was designed streaming-first: its operators run continuously and pass records one at a time, with no batch boundary to wait for. Everything else — latency, state handling, recovery granularity, backpressure — follows from that choice. Spark 4.1's Real-Time Mode narrows the latency gap, but the designs still differ.

Is Apache Flink faster than Apache Spark?

On per-event latency, usually yes — Flink operates in the low-millisecond range because it never waits for a batch. On throughput for batch and analytical workloads, Spark is normally at least as fast, thanks to its optimizer and columnar execution. Spark 4.1's Real-Time Mode reports p99 latencies from a few milliseconds to around 300 ms, but only for pipelines that fit its constraints — no Delta sink, no foreachBatch, update mode only. So "Flink is faster" depends on both the axis you measure and your pipeline shape.

Should I learn Spark or Flink in 2026?

Learn Spark first. It spans batch, analytics, lakehouse ETL and streaming, so it appears in far more job descriptions and more of your daily work. Learn Flink second, especially if you work on event-driven products — fraud detection, real-time pricing, telemetry alerting, complex event processing. Flink skills are scarcer and priced accordingly, but the market is narrower. Event time, watermarks, keyed state and checkpointing transfer between both.

Does Apache Flink replace Apache Spark?

No. Flink runs batch and Spark runs streams, but few organisations swap one for the other. Spark stays the default for batch ETL, lakehouse maintenance, SQL analytics and ML data prep; Flink is chosen for long-running, low-latency, stateful event processing. Many mature platforms run both — the real question is whether the second engine earns its operational cost in upgrades, monitoring, tuning and on-call knowledge.

Do Spark and Flink both support exactly-once processing?

Yes, and both do it the same way in principle: a replayable source, durable checkpointed state, and a sink that is transactional or idempotent. Flink injects checkpoint barriers into the stream, snapshots operator state as they pass, and commits via two-phase commit. Spark commits offsets and state at batch boundaries and relies on an idempotent sink, typically foreachBatch with a MERGE. Neither gives you exactly-once if your sink can't deduplicate.

Which is better for large state, Spark or Flink?

Flink, historically and still. Its state API is richer — keyed state, per-key timers, low-level process functions — and Flink 2.0's disaggregated state keeps primary state in object storage like S3 with local disk as cache, so state size is bounded by the object store rather than your worker disks, and checkpoints get much lighter. Spark's transformWithState and RocksDB improvements cover most workloads, but very large state with per-key timers remains Flink's stronger suit.

Can I use SQL with both Spark and Flink?

Yes. Spark SQL is mature and is how most teams interact with Spark. Flink SQL is production-grade too, covering streaming joins, windowed aggregations, MATCH_RECOGNIZE pattern matching and materialized tables. The semantic difference matters most: a Flink SQL query is a long-running job continuously updating a changelog result, while a Spark SQL query is normally a bounded computation that finishes. Flink SQL means reasoning about watermarks and state retention, not just results.

Can Spark and Flink read from the same Kafka topic?

Yes — both are ordinary Kafka consumers, and running them in separate consumer groups lets each read the full topic independently without interfering. This is the sane way to run a migration or a bake-off: point both engines at production traffic, write to separate sinks, and compare results and latency on real data before switching anything. Just budget for the extra broker read load.

Conclusion

The Spark vs Flink question has been answered the same way for years — "Spark micro-batches, Flink truly streams, pick Flink for low latency" — and in 2026 that answer has aged badly. Spark 4.1's Real-Time Mode removes the batch boundary that the whole argument rested on, and Flink 2.0's disaggregated state removes the operational ceiling that was the main argument against Flink. Both engines spent the last two years fixing the exact thing critics used against them.

Aged badly is not the same as wrong, though. Real-Time Mode is narrow today — stateless-only in open-source Spark, and even on Databricks it rules out the Delta sink, foreachBatch, and append output mode. So the practical answer for a low-latency pipeline that must also land a lakehouse table is still Flink. Just make sure that's the pipeline you actually have.

So choose on the things that didn't converge. Spark is the right default because one engine and one skill set cover batch ETL, lakehouse maintenance, SQL analytics, ML prep and streaming that tolerates sub-second-to-seconds latency — which is the overwhelming majority of data platforms. Flink is the right choice for a specific shape of problem: per-entity state machines with timers, event-pattern detection, very large keyed state, and machine-in-the-loop decisions under a second.

Before you commit either way, do two cheap things. Put your requirement on the latency ladder honestly and see whether rung 5 is really where you live. Then price the two-engine tax — a second upgrade cycle, a second monitoring surface, a second incident playbook, and a second technology every engineer must know at 2am. On a small team, that tax decides more architectures than any benchmark.

If you're building a streaming platform, hiring for one, or want an experienced data engineer to review the design before it's carrying production traffic, post your project on solutiongigs.in and get matched with a vetted expert — it's free to post.

Mohammed Yaseen

Founder, SolutionGigs

Mohammed builds streaming data platforms on Kafka, Spark and Iceberg, and has spent enough time sizing checkpoint intervals to be suspicious of any latency claim that arrives without a benchmark. LinkedIn →