Spark Adaptive Query Execution (AQE): How It Works

Last Updated: August 2026 | 15 min read

Quick Answer: Adaptive Query Execution (AQE) is a Spark SQL feature that re-optimizes a query plan while the query runs, using the actual runtime statistics of data produced at each shuffle instead of the pre-execution estimates the optimizer had to guess with. At every shuffle boundary AQE looks at the real data size and distribution and adjusts the rest of the plan in three ways: it coalesces many tiny shuffle partitions into fewer right-sized ones, splits skewed partitions so one hot task can't stall a stage, and switches join strategies (e.g. sort-merge → broadcast) when a side turns out small. AQE has been enabled by default since Apache Spark 3.2.0 (spark.sql.adaptive.enabled=true), so most modern Spark jobs already benefit from it.

Spark's Catalyst optimizer plans your query before a single row moves — and it plans using estimates: table sizes, filter selectivity, join cardinalities it can only guess at. When those guesses are wrong (and on real data they often are), you get 200 shuffle partitions for 3 MB of data, a sort-merge join where a broadcast would have been 10× faster, or one skewed task that runs for 40 minutes while 199 others finished in seconds. Adaptive Query Execution is Spark's answer: instead of trusting the pre-flight plan, it re-plans mid-flight using numbers it can actually measure. This guide explains exactly how AQE works under the hood, walks through its three optimizations with real PySpark, gives you the configs that matter, and shows you where AQE still needs a human.

What is Adaptive Query Execution?

Adaptive Query Execution (AQE) is a runtime re-optimization framework in Spark SQL that adjusts the physical query plan based on statistics collected during execution. It closes the gap between what the optimizer estimated and what the data actually looks like.

A normal Spark query is planned once: Catalyst produces a logical plan, optimizes it, and picks a physical plan — all using static, pre-execution estimates. Those estimates come from table metadata and heuristics, and they are frequently off, especially after filters, joins, and aggregations distort row counts in ways the optimizer can't foresee.

AQE changes the model. It splits the physical plan at shuffle boundaries into query stages, runs one stage, and — before planning the next — reads the real statistics that stage produced (partition sizes, row counts, total bytes). Armed with facts instead of guesses, it re-optimizes the remaining plan. The result is a plan tuned to your actual data on this run, not a generic plan tuned to an estimate.

Key facts to anchor on:

• AQE lives in Spark SQL and applies to the DataFrame, Dataset, and SQL APIs (anything that goes through Catalyst). It does not optimize raw RDD code.
• It was introduced in Spark 3.0 (off by default), and made on by default in Spark 3.2.0 via spark.sql.adaptive.enabled=true.
• It only re-optimizes around shuffles — a query with no shuffle (a simple map/filter/write) gives AQE nothing to adapt.

Why AQE matters: the problem with static plans

Static query plans fail whenever the optimizer's estimate is wrong — and on real data, estimates are wrong constantly. AQE matters because it fixes the three most common and most expensive of those failures automatically.

Before AQE, every Spark engineer hit the same trio of pain:

1. The 200-partition problem. spark.sql.shuffle.partitions defaults to 200. Shuffle 5 MB of data and you get 200 near-empty partitions — 200 tasks, each doing almost nothing, all scheduling overhead and tiny output files. Shuffle 2 TB and 200 is far too few — each task is huge and risks spilling or an out-of-memory error. One static number can't be right for both.
2. Wrong join strategy. The optimizer plans a sort-merge join because it estimated both sides were large. At runtime a filter shrinks one side to 8 MB — a broadcast join would have skipped the shuffle entirely, but the plan was already fixed.
3. Silent skew. One key holds 80% of the rows. The plan splits work into equal partitions on paper, but one partition is enormous, so one task runs for an hour while the cluster sits idle. This is the classic Spark data skew tax.

AQE addresses all three at runtime, which is why it's one of the highest-leverage performance features Spark has shipped. At Telemetrix, turning AQE on (and leaving spark.sql.shuffle.partitions alone) removed an entire category of per-job partition tuning we used to do by hand.

How AQE works under the hood

AQE breaks the physical plan into query stages separated by shuffles, executes them one at a time, and re-optimizes the remaining plan after each stage using the real statistics that stage produced. The loop — run a stage, measure, re-plan — is the whole idea.

Here is the sequence AQE follows:

1. Plan normally. Catalyst produces an initial physical plan from static estimates — exactly as it always did.
2. Cut at shuffles. AQE divides that plan into query stages at each shuffle (exchange) boundary. Stages with no unfinished dependencies are ready to run.
3. Run a stage, collect stats. When a shuffle stage finishes, Spark has exact map-output statistics: how big each partition is, how many rows, total bytes.
4. Re-optimize the rest. Using those real numbers, AQE rewrites the remaining plan — coalescing partitions, splitting skew, and swapping join strategies — then continues.
5. Repeat until the whole query is done. The final executed plan can look quite different from the initial one.

You can see this in the Spark UI: the SQL tab shows an AdaptiveSparkPlan node, and once the query finishes it reports isFinalPlan=true. The nodes labelled AQEShuffleRead (marked coalesced or skewed) are AQE physically merging or splitting partitions. If the initial and final plans differ, AQE did work.

Mental model: Catalyst is a chess player planning ten moves ahead from the opening position. AQE is that same player looking at the actual board after every move and adjusting the rest of the plan. Same optimizer, but now it reacts to reality.

AQE optimization 1: dynamically coalescing shuffle partitions

AQE merges many small post-shuffle partitions into fewer, appropriately-sized ones based on the actual data, so you no longer hand-tune spark.sql.shuffle.partitions. This is the optimization you benefit from most often.

Set your shuffle partitions high (leave the default 200, or push it higher for big jobs) and let AQE coalesce down. After the shuffle, AQE reads the real partition sizes and combines adjacent small partitions until each is close to a target size — spark.sql.adaptive.advisoryPartitionSizeInBytes (default 64 MB).

from pyspark.sql import SparkSession

spark = (SparkSession.builder
         .config("spark.sql.adaptive.enabled", "true")                       # on by default in 3.2+
         .config("spark.sql.adaptive.coalescePartitions.enabled", "true")    # on by default
         .config("spark.sql.adaptive.advisoryPartitionSizeInBytes", "64m")   # target per partition
         .getOrCreate())

# Start with a high shuffle-partition count on purpose; AQE will coalesce it down.
spark.conf.set("spark.sql.shuffle.partitions", "400")

daily = (events
         .groupBy("event_date")
         .count())          # shuffle here → AQE coalesces the post-shuffle partitions
daily.write.parquet("s3://bucket/daily/")

Without AQE, that aggregation would emit 400 partitions no matter how few dates exist — 400 tiny files. With AQE, if the result is small it might coalesce to a handful of partitions, directly reducing the small files problem and cutting task overhead. The advisory size is a target, not a hard cap — AQE won't merge across certain boundaries, but it gets you close.

AQE optimization 2: skew join optimization

AQE detects a partition that is far larger than its peers during a sort-merge join and splits it into several smaller sub-partitions, so the skewed key no longer stalls the whole stage on one task. It's automatic skew handling, within limits.

A partition is treated as skewed when it is both bigger than spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes (default 256 MB) and larger than the median partition times spark.sql.adaptive.skewJoin.skewedPartitionFactor (default 5×). When AQE flags one, it splits that fat partition into multiple tasks (replicating the matching rows on the other side of the join as needed) so the work spreads across cores instead of piling onto one.

spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")                 # on by default
spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionFactor", "5")
spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes", "256m")

# If one customer_id dominates, AQE splits that skewed partition across tasks.
orders.join(customers, "customer_id").write.parquet("s3://bucket/joined/")

In the Spark UI you'll see AQEShuffleRead nodes marked skewed where this happened. The catch: this only covers sort-merge joins. It does not fix skew in a groupBy aggregation, skew from a single un-splittable hot key, or skew before the first shuffle — for those you still reach for salting or pre-aggregation. AQE is the first line of defense against skew, not the last. (See the full Spark data skew guide for the manual techniques.)

AQE optimization 3: dynamically switching join strategies

When runtime stats reveal that one side of a planned sort-merge join is actually small, AQE converts it to a broadcast (or shuffled-hash) join on the fly, skipping an expensive shuffle. It rescues joins the optimizer mis-planned from bad estimates.

Catalyst decides broadcast-vs-shuffle joins before execution using estimated sizes. Estimates are often too high — a table looks big, but a filter or a previous join shrinks it. AQE re-checks after the shuffle: if a side is now below spark.sql.adaptive.autoBroadcastJoinThreshold, it swaps the sort-merge join for a broadcast hash join, eliminating the shuffle for that join entirely. If both sides are moderate it may switch to a shuffled-hash join instead.

# Even if Catalyst planned a sort-merge join, AQE can promote it to broadcast
# once it sees the filtered dimension is only a few MB at runtime.
spark.conf.set("spark.sql.adaptive.autoBroadcastJoinThreshold", "30m")

fact.filter("region = 'APAC'") \
    .join(big_dimension.filter("active = true"), "dim_id") \
    .write.parquet("s3://bucket/out/")

A runtime-promoted broadcast isn't quite as good as planning a broadcast from the start (the shuffle write already happened), but it's far better than grinding through a needless sort-merge. For the full picture of how Spark chooses between join types, see the Spark join strategies guide.

Key AQE configuration settings

Most AQE settings default to sensible values — you rarely need to change them — but these are the knobs that matter when you do. Keep the master switch on and tune the advisory sizes only if profiling tells you to.

Don't disable AQE to "control" partitions. The most common anti-pattern is turning AQE off and hand-setting spark.sql.shuffle.partitions per job. That's exactly the manual work AQE removes — and it almost always does worse than letting AQE coalesce. Tune with AQE, not against it.

AQE vs manual tuning: where you still need repartition()

AQE tunes what happens around shuffles it already plans; it does not manage read parallelism, output file sizing, or pre-shuffle key layout — those are still your job. AQE is a complement to repartition()/coalesce(), not a replacement.

What AQE handles for you:

• Right-sizing post-shuffle partition counts (no more manual shuffle.partitions per job).
• Splitting skewed partitions inside sort-merge joins.
• Promoting joins to broadcast when a side turns out small.

What AQE will not do — still on you:

• Read parallelism. Reading a few large files gives few partitions; AQE won't grow them. Use repartition(n) for more parallelism on the read.
• Pre-write file sizing. Want one file per date, or a fixed output size? That's repartition("date") + partitionBy, or coalesce() before the write — see repartition vs coalesce.
• Aggregation skew and non-join skew. AQE's skew handling is join-only; a skewed groupBy still needs salting or pre-aggregation.
• Overall cluster and memory sizing. AQE won't right-size your executors; that's executor tuning.

The clean division: let AQE tune shuffle-partition counts and react to runtime skew; use repartition()/coalesce() explicitly for reads, writes, and layout you control.

Common AQE mistakes (and how to avoid them)

1. Disabling AQE and hand-tuning shuffle.partitions. You're re-doing by hand the exact work AQE does better with real stats. Leave AQE on.
2. Expecting AQE to fix aggregation skew. Its skew optimization is sort-merge-join-only. A skewed groupBy still needs salting or a two-stage aggregation.
3. Assuming AQE grows partitions. Coalescing only merges down. If a read gives you too few partitions, you must repartition() up — AQE won't.
4. Setting advisoryPartitionSizeInBytes too small. A tiny target (e.g. 4 MB) defeats the purpose and recreates the small-partition problem. 64–128 MB is the healthy range.
5. Running on Spark < 3.0. AQE simply doesn't exist before 3.0, and is off by default in 3.0–3.1. Confirm your runtime; on Spark 3.2+ it's already on.
6. Not checking the final plan. If a query is still slow, open the SQL tab and read the adaptive plan (isFinalPlan=true), not the initial one — they differ, and the final one is what actually ran.
7. Treating AQE as a substitute for good data layout. Partitioning, file formats, and file sizes still matter; AQE optimizes execution, not storage.

See how SolutionGigs can help

A Spark job that's slow even with AQE on usually has a skew or join problem AQE can't reach on its own — and finding it means reading adaptive query plans, not guessing. SolutionGigs connects you with vetted data engineers who tune production Spark on AWS EMR and Databricks every day. See how SolutionGigs can help →

Frequently Asked Questions

What is Adaptive Query Execution (AQE) in Spark?

Adaptive Query Execution (AQE) is a Spark SQL feature that re-optimizes the query plan while the query runs, using real runtime statistics instead of the pre-execution estimates the optimizer started with. After each shuffle it reads the actual data size and distribution and adjusts the rest of the plan — coalescing partitions, splitting skew, and switching join strategies. It's been enabled by default since Apache Spark 3.2.0.

Is Adaptive Query Execution enabled by default in Spark?

Yes. AQE is controlled by spark.sql.adaptive.enabled, which defaults to true from Apache Spark 3.2.0 onward (it existed but was off by default in 3.0 and 3.1). Databricks enables it by default too. Because it ships on, most modern Spark jobs already benefit, though each sub-optimization has its own toggle you can tune.

What are the three main features of Spark AQE?

AQE has three core optimizations: (1) dynamically coalescing shuffle partitions — merging many tiny post-shuffle partitions into fewer right-sized ones; (2) skew join optimization — splitting an oversized, hot partition so one task doesn't stall the stage; and (3) dynamically switching join strategies — converting a planned sort-merge join into a broadcast or shuffled-hash join once runtime stats show a side is small enough.

Does AQE replace spark.sql.shuffle.partitions tuning?

Mostly, for the post-shuffle count. With AQE coalescing on you can leave spark.sql.shuffle.partitions high and let AQE merge partitions down to a sensible number based on actual data, targeting advisoryPartitionSizeInBytes. You stop hand-tuning that number per job. AQE does not replace deliberate repartition()/coalesce() for read parallelism, pre-write file sizing, or partitioning by a key.

How do I know if AQE actually optimized my Spark query?

Check the SQL tab of the Spark UI or call df.explain() after the query runs. When AQE acts, the plan shows an AdaptiveSparkPlan node with isFinalPlan=true, plus AQEShuffleRead nodes marked coalesced or skewed. If a join switched, the final plan shows BroadcastHashJoin where the initial plan had SortMergeJoin. The difference between the initial and final plans is AQE at work.

Can AQE fix all data skew in Spark?

No. AQE's skew optimization handles skew inside sort-merge joins by splitting oversized partitions, covering many common cases automatically. But it doesn't fix skew in aggregations (groupBy), skew from a single un-splittable hot key, or skew before the first shuffle. Those still need salting, pre-aggregation, or an explicit repartition. Treat AQE as the first line of defense, not the whole strategy.

Does AQE add overhead or slow down small queries?

The overhead is negligible for almost all workloads. AQE re-plans only at shuffle boundaries using statistics Spark already collects, so it adds no extra passes over the data. For tiny queries with no shuffles there's nothing to adapt and effectively zero cost. The savings from coalescing partitions and handling skew almost always dwarf the small planning cost — which is why it ships on by default.

Conclusion

Adaptive Query Execution is Spark admitting a simple truth: the best time to optimize a query is while you can see the data, not before. By cutting the plan at shuffle boundaries and re-optimizing with real statistics, AQE fixes the three failures that used to eat hours of engineer time — wrong shuffle-partition counts, mis-planned joins, and silent skew — automatically, on by default, for every DataFrame and SQL query you run on Spark 3.2+.

The practical takeaways:

• Leave AQE on and stop hand-tuning spark.sql.shuffle.partitions — let coalescing size your partitions.
• Let AQE handle join skew and join-strategy switches, but remember its skew fix is join-only.
• Still use repartition()/coalesce() for read parallelism, output file sizing, and key layout — AQE doesn't touch those.
• Read the final adaptive plan (isFinalPlan=true) when debugging, not the initial one.

Get this right and a whole class of manual Spark tuning simply disappears. If your Spark pipelines are still slow with AQE on — skew, mystery shuffles, or joins that won't broadcast — get matched with a vetted data engineer on SolutionGigs. It's free to post your project.

Mohammed Yaseen

Founder, SolutionGigs

Mohammed runs Spark on AWS EMR at Telemetrix, turning hundreds of millions of events a day into Iceberg tables on S3. He's leaned on AQE to kill per-job shuffle-partition tuning and watched it split skewed joins that used to run for an hour — and still reaches for repartition() where AQE can't. LinkedIn →