Spark Out of Memory Errors: Causes and Fixes (Complete 2026 Guide)

Last Updated: July 2026 | 16 min read

Quick Answer: A Spark out of memory error happens when one JVM — the driver or a single executor — is asked to hold more data than its heap allows, and it either throws java.lang.OutOfMemoryError: Java heap space or gets killed by YARN for exceeding memory limits. The fix depends entirely on which JVM died and why: reduce data per task by adding partitions and fixing skew (the most common real cause), raise spark.executor.memoryOverhead for "container killed by YARN", stop collect()-ing large data to the driver, and only then raise spark.executor.memory. Throwing more RAM at the job without diagnosing the cause usually just delays the same crash — or quietly doubles your cloud bill.

Almost every Spark engineer meets OutOfMemoryError early, reacts by bumping --executor-memory to a bigger number, and moves on — until the job fails again at 2× the cost. Having tuned Spark on AWS EMR across pipelines processing hundreds of millions of events per day, I've learned that an OOM is a symptom with at least five different root causes, each needing a different fix. This guide is the diagnostic playbook: how to tell driver OOM from executor OOM, how to read the exact error message, the Spark executor memory model in one diagram, and the fixes in the order you should actually try them. It pairs with our deep-dives on Spark data skew and the Spark small files problem, which are two of the biggest hidden triggers of OOM.

What is an out of memory error in Spark?

A Spark out of memory error is a failure of a single JVM process — not the cluster as a whole — when the data it must hold in memory exceeds its configured heap or container limit. Spark runs as one driver JVM that coordinates the job and many executor JVMs that do the work in parallel. Each is a separate Java process with its own fixed memory budget. When either one exceeds its budget, you get an OOM.

The most important thing to understand before touching a single config is where the memory goes inside an executor. Each region is controlled by a different setting, and each corresponds to a different kind of OOM — so identifying the region tells you which knob to turn.

An executor's container memory has two big parts: the JVM heap (spark.executor.memory) and the off-heap overhead (spark.executor.memoryOverhead). Inside the heap, Spark carves out a 300 MB reserved slice, a unified region shared by execution and storage (governed by spark.memory.fraction, default 0.6), and user memory for your own objects. A "Java heap space" error means the heap filled up; a "Container killed by YARN" error means the overhead overflowed. Same word — OOM — completely different fix.

Driver OOM vs executor OOM: identify this first

Before any config change, determine whether the driver or an executor ran out of memory — the fixes have almost nothing in common. This single distinction resolves more than half of misdiagnosed OOMs.

Executor OOM happens on a worker JVM while it processes a partition. Signs: failed tasks, stages that retry, "Lost executor", or "Container killed by YARN for exceeding memory limits" in the logs. The cause is data-side — a partition too big for the executor to hold.

Driver OOM happens on the single coordinator JVM. Signs: the error appears in the driver stderr, often right after an action, with no executor loss. The cause is almost always your code pulling data back to the driver:

# Driver-OOM landmines — each pulls the full dataset into ONE JVM
big_df.collect()          # materializes every row on the driver
big_df.toPandas()         # same, then converts to a pandas frame
list(big_df.toLocalIterator())  # safer, but still risky at scale
print(big_df.rdd.collect())     # the classic accidental OOM

If the stack trace is in the driver, no amount of --executor-memory will help. Fix the code (write to storage instead of collecting), or if the driver genuinely needs more room (large broadcast joins), raise spark.driver.memory and spark.driver.maxResultSize.

Fast rule: OOM right after collect(), toPandas(), show() on a huge frame, or a broadcast → driver. OOM with "Container killed by YARN" or "Lost executor" mid-stage → executor.

What causes Spark out of memory errors? (5 root causes)

Spark OOMs trace back to five recurring causes — and only one of them is genuinely "not enough RAM". Knowing which one you have is the whole game.

• Too much data per task. With too few partitions (or a huge input split), each task must hold more rows than the heap allows. spark.sql.shuffle.partitions defaults to 200 — fine for tens of GB, far too few for terabytes.
• Data skew. One hot key funnels a disproportionate share of rows into a single partition, so one task OOMs while the rest finish instantly. This is the most common hidden cause and the one most often misread as "need more memory".
• Overhead / off-heap overflow. Shuffle buffers, netty network buffers, and especially PySpark's Python + PyArrow processes live outside the JVM heap. When they exceed spark.executor.memoryOverhead, YARN kills the whole container even though the heap had room.
• Driver-side collection. collect(), toPandas(), oversized broadcasts, or unbounded accumulators pull data into the single driver JVM until it dies.
• Memory-hungry operations. Wide aggregations, explode() on large arrays, caching datasets you only read once, or many concurrent tasks per executor each competing for the same heap.

Rule of thumb: if the job OOMs on one task while others succeed, suspect skew or partitioning first. If every task struggles, suspect undersized executors or too many cores per executor.

How to read the Spark OOM error message

The exact wording of the error tells you the region that overflowed — read it before changing any config. Here are the messages you'll actually see and what each one means.

Note the trap: "Container killed by YARN" is not a heap error. Raising spark.executor.memory for it can even make things worse, because on a fixed-size container a bigger heap leaves less room for the overhead region that actually overflowed.

How to fix Spark out of memory errors (in the right order)

Work from cheapest and most-likely to most-expensive: partitions and skew first, overhead second, executor memory last. Following this order means you usually fix the OOM for free — and stop paying for RAM you don't need.

Fix 1 — Reduce data per task (partitions + skew)

The number-one executor OOM fix is making each partition smaller, not making the executor bigger. More, smaller partitions means each task holds less data at once. Aim for partitions in the ~128–200 MB range.

# Raise shuffle parallelism for large jobs (default is only 200)
spark.conf.set("spark.sql.shuffle.partitions", 800)

# Or explicitly repartition a skewed/oversized DataFrame before a wide op
df = df.repartition(800, "join_key")

Enable Adaptive Query Execution (AQE) so Spark right-sizes shuffle partitions and splits skewed ones automatically at runtime (on by default in Spark 3.2+):

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

If one task OOMs while the rest fly, it's skew — read our data skew guide for salting and broadcast-join fixes. This is the highest-leverage change you can make.

Fix 2 — Fix "Container killed by YARN": raise memory overhead

If the message says the container exceeded its limit, raise the off-heap overhead, not the heap. This is the correct fix for PySpark jobs, heavy shuffles, and native-library workloads.

# Explicit overhead (older style)
spark.conf.set("spark.executor.memoryOverhead", "2g")

# Or as a factor of executor memory (Spark 3.3+)
spark.conf.set("spark.executor.memoryOverheadFactor", "0.2")  # 20%

The default overhead is ~10% of executor memory (min 384 MB), which is frequently too small for PySpark because the Python interpreter and PyArrow buffers run outside the JVM. If you use Python UDFs or mapInPandas, budget 15–25% overhead.

Fix 3 — Stop driver OOMs at the source

The cure for driver OOM is architectural: don't move big data to the driver. Replace collection with distributed writes.

# ❌ Pulls the whole result into the driver JVM
result = big_df.collect()

# ✅ Write in parallel from the executors — no driver bottleneck
big_df.write.mode("overwrite").parquet("s3://bucket/output/")

# ✅ Only need a peek? Bound it.
sample = big_df.limit(1000).toPandas()

If a broadcast join is the culprit, confirm the broadcast table is truly small (under a few hundred MB) — Spark auto-broadcasts under spark.sql.autoBroadcastJoinThreshold. When the driver legitimately needs headroom, size it explicitly: spark.driver.memory=8g and a matching spark.driver.maxResultSize.

Fix 4 — Right-size executor memory and cores (last, not first)

Only after partitions, overhead, and driver code are handled should you add executor heap — and add cores carefully. A common mistake is one giant executor with many cores: all those concurrent tasks share the same heap, so more cores means more OOM risk, not less.

spark-submit \
  --executor-memory 8g \
  --executor-cores 4 \       # 4–5 cores per executor is a good sweet spot
  --conf spark.executor.memoryOverhead=2g \
  your_job.py

Prefer more medium executors (e.g. 8 GB / 4 cores) over fewer huge ones (64 GB / 32 cores). Medium executors garbage-collect faster and isolate failures. Also drop .cache()/.persist() on any dataset you read only once — cached blocks sit in the storage region and squeeze out execution memory.

Fix 5 — Tune memory fractions (advanced, rarely needed)

If you've done everything above and still spill or OOM, adjust how the heap is split — but this is a fine-tuning step, not a starting point. Raising spark.memory.fraction gives execution and storage more of the heap at the expense of user memory:

spark.conf.set("spark.memory.fraction", "0.7")        # default 0.6
spark.conf.set("spark.memory.storageFraction", "0.3")  # shrink cache, favor execution

Most teams never need to touch these. Reach for them only when profiling shows a specific execution-vs-storage imbalance — otherwise the defaults are well-chosen.

Which fix should you use? (decision guide)

Match the symptom to the fix — the Spark UI and the error text tell you which row you're on.

The 90% path: enable AQE, right-size shuffle partitions, raise overhead for PySpark, and stop collecting to the driver. That combination clears the vast majority of production OOMs without over-provisioning a single executor.

Common mistakes when fixing Spark OOM errors

• Raising spark.executor.memory for a "Container killed by YARN" error. That's an overhead overflow — a bigger heap in a fixed container leaves less room for overhead and can worsen it. Raise memoryOverhead instead.
• Fixing an executor OOM by giving the driver more memory (or vice versa). Always confirm which JVM died first. The two failures are unrelated.
• Throwing hardware at skew. A bigger executor just gives the one straggler task a roomier place to be slow; the other executors still idle. Skew is a distribution problem — fix the distribution.
• One massive executor with 32 cores. All tasks share that heap. Prefer several 4–5 core executors so concurrent tasks don't collide.
• Caching everything. .cache() on data you scan once wastes the storage region and starves execution memory, causing the OOM you're trying to prevent.
• Ignoring overhead in PySpark. Python and PyArrow live off-heap; default overhead is often too small for UDF-heavy jobs.

If you're firefighting a Spark pipeline that OOMs unpredictably, SolutionGigs can match you with a vetted data engineer who has tuned exactly these failures on EMR and Databricks — it's free to post a project.

Frequently Asked Questions

What causes out of memory errors in Spark?

Spark OOMs occur when one JVM — the driver or a single executor — must hold more data than its heap allows. The usual causes are too much data per task (few partitions or skew), the off-heap overhead region overflowing (shuffle buffers, Python/Arrow), the driver collecting a large result, or an oversized broadcast. It's a distribution and sizing problem far more often than a genuine total-RAM shortage.

What is the difference between driver OOM and executor OOM in Spark?

An executor OOM happens on a worker JVM while processing a partition — you see failed tasks, retried stages, or "Container killed by YARN". A driver OOM happens on the single coordinator JVM, almost always from collect(), toPandas(), or a huge broadcast. Executor OOM is fixed with partitions, overhead, and memory; driver OOM is fixed by not collecting large data and raising spark.driver.memory.

How do I fix java.lang.OutOfMemoryError: Java heap space in Spark?

First shrink the data per task: raise spark.sql.shuffle.partitions or repartition so partitions are smaller, and fix any data skew feeding one giant partition. Enable AQE so Spark splits skewed partitions automatically. Only if partitions are already small should you raise spark.executor.memory. Avoid caching read-once datasets and never collect() a large DataFrame. Adding memory without fixing partitioning just delays the same error.

What does 'Container killed by YARN for exceeding memory limits' mean?

It means the whole executor container used more physical memory than YARN granted it, so YARN killed it — this is not a JVM heap error. The overflow is usually the off-heap overhead region: shuffle buffers, netty buffers, or PySpark's Python/PyArrow processes. Fix it by raising spark.executor.memoryOverhead (or spark.executor.memoryOverheadFactor), not spark.executor.memory.

Does increasing executor memory always fix Spark OOM errors?

No. If the real cause is skew, too few partitions, a driver collect(), or overhead overflow, more executor memory just postpones the crash or wastes money. Bigger executors also mean fewer of them and heavier garbage collection. Diagnose which JVM died and why in the Spark UI, then turn the knob that matches the cause rather than blindly scaling RAM.

What is spark.executor.memoryOverhead and when should I increase it?

It's off-heap memory reserved on top of the JVM heap for shuffle buffers, native libraries, and Python/Arrow processes, defaulting to ~10% of executor memory (min 384 MB). Increase it when you see "Container killed by YARN for exceeding memory limits" or when running heavy PySpark UDFs, since Python runs outside the JVM heap and draws on overhead memory. Budget 15–25% for UDF-heavy PySpark jobs.

How do I avoid driver out of memory errors in PySpark?

Never pull a large dataset to the driver. Replace collect() and toPandas() on big DataFrames with df.write to storage, or use take(n)/limit(n) for a sample. Watch for accidental large broadcasts and unbounded accumulators. If the driver truly needs more memory (large broadcast joins), raise spark.driver.memory and spark.driver.maxResultSize deliberately.

Conclusion

A Spark out of memory error is a diagnosis problem before it's a configuration problem. The same OutOfMemoryError can mean a skewed partition, an overflowing off-heap region, or a collect() that dragged a billion rows into the driver — and each needs a different, often free, fix. The engineers who resolve OOMs quickly don't reach for --executor-memory first; they ask two questions: which JVM died (driver or executor), and what does the exact message say (heap, GC, or "container killed by YARN"). Those two answers point straight to the right row in the decision table above.

Climb the ladder in order: right-size partitions and fix skew, raise overhead for "container killed by YARN" and PySpark, stop collecting to the driver, and only then add executor heap — with a sensible 4–5 cores per executor. Most production OOMs die on the first two rungs, and the cluster gets cheaper, not bigger, in the process. Open the Spark UI on your failing job right now: check whether one task's input dwarfs the median, and read the failure message word for word. That five-minute diagnosis usually saves both the job and the bill.

Stuck on a Spark pipeline that keeps running out of memory? Get matched with a vetted data engineer on SolutionGigs — it's free to post your project.

Mohammed Yaseen

Founder, SolutionGigs

Mohammed tunes Spark pipelines on AWS EMR processing hundreds of millions of events per day, where diagnosing driver vs executor OOM and right-sizing memory is a routine performance and cost win. LinkedIn →