Spark shuffle partitions

See how a wide transformation redistributes data, why partition size matters, and how Adaptive Query Execution can merge excess post-shuffle tasks.

Shape the shuffle

Both scenarios process the same data on the same executor cores. Only post-shuffle partition handling changes.

Approximate bytes written by the map stage and fetched by reduce tasks.

The configured starting count for joins and aggregations.

At most this many reduce tasks can run concurrently in the model.

Used with parallelismFirst=false so target-size behavior is easy to see.

Moves this share of the shuffled data into one hot partition. Coalescing alone does not remove key skew.

Illustrative model, not measured performance. Estimated task time = 120 ms scheduling overhead + 18 ms per MB + spill penalty above 256 MB. A stage finishes after all task waves, including the slowest partition.

One shuffle, two partition plans

Every colored bar is a reduce-side partition. Height represents bytes assigned to that task.

Fixed partition count

AQE off

Map tasks

Reduce tasks

Adaptive coalescing

AQE on

Map tasks

Reduce tasks

At or below advisory size Above advisory size Hot partition / straggler

What the stage is telling you

Use task size, task waves, and the long tail together. Partition count by itself is not a performance score.

A practical AQE starting point

A deliberately generous initial count gives AQE room to coalesce after Spark sees real shuffle statistics. Setting the initial count too low is different: ordinary coalescing cannot create the missing parallelism.

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.parallelismFirst", "false")
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "128MB")
spark.conf.set("spark.sql.shuffle.partitions", "800")