Two quiet numbers in the Spark UI explain most slow stages. What spill really means, why one number is always bigger, and three fixes that hold up.
Open a slow stage in the Spark UI and two quiet numbers tell you most of the story: Spill (Memory) and Spill (Disk).
Most engineers meet them for the first time when a job that should take minutes takes an hour. The numbers look alarming. The word spill sounds like something is leaking.
Nothing is leaking. Spill is Spark working exactly as designed. Once you understand the pair, a mysterious slowdown turns into a short checklist.
Picture yourself sorting a tall stack of papers at a small desk.
The desktop is the task's working memory. When the stack fits on the desk, sorting is fast. When it does not, you move some papers into a drawer so you can keep going. That is Spill (Memory): data Spark had to push out of the task's active workspace while sorting or grouping.
When the drawer fills up too, bundles go into the filing cabinet across the room. That is Spill (Disk): the same data, serialized and compressed, written to local disk.
Work continues in both cases. But every trip to the drawer is slower than the desktop, and every trip to the cabinet is slower than the drawer.
flowchart TD
A[Rows arrive for a sort or group by] --> B{Fits in task memory?}
B -->|Yes| C[Processed in memory. Fast.]
B -->|No| D[Spill: serialize and write to local disk]
D --> E[Read back and merged. Slower, but still correct.]The first time people read these metrics, the sizes look wrong. A stage might show 8 GB of Spill (Memory) and 800 MB of Spill (Disk).
That is expected. Spill (Memory) measures the data as it lives in memory, fully expanded. Spill (Disk) measures the same rows after Spark serializes and compresses them for the trip to storage. Live objects take far more room than their packed form.
Read the pair this way. Memory spill tells you how much pressure the task was under. Disk spill tells you how much data actually made the trip.
Spill is a symptom. The cause is almost always one of three things.
The first is skew. One key has far more rows than the others, so one task gets a stack taller than the desk while the other tasks wait. A group by on a low-cardinality key like a status flag is the classic case.
The second is a high-cardinality aggregation. Grouping by something like a user ID creates millions of small groups, and the structure that tracks them outgrows the workspace.
The third is a large sort-merge join. When both sides are big, Spark sorts both sides, and sorting is exactly the kind of work that fills a desk.
You can see spill yourself with a small skewed dataset. This runs entirely in Free Edition.
from pyspark.sql import functions as F
# One million rows share a single key. One hundred thousand rows get their own key.
hot_df = spark.range(0, 1000000).select(
F.lit("hot_key").alias("group_key"),
F.col("id").alias("value"),
)
cold_df = spark.range(0, 100000).select(
F.concat(F.lit("key_"), F.col("id")).alias("group_key"),
F.col("id").alias("value"),
)
events_df = hot_df.unionByName(cold_df)
result_df = events_df.groupBy("group_key").agg(
F.count("*").alias("row_count"),
F.sum("value").alias("total"),
)
result_df.collect()Run the cell, then open the Spark UI from the notebook's compute menu. Find the completed query and open the stage with the largest input. In the task summary, one task runs far longer than the rest, and it carries Spill (Memory) and Spill (Disk) values while the others show none.
That single heavy task is skew made visible. A stage finishes only when its slowest task finishes, so one hot key holds up everyone else.
If you want a fuller tour of reading stages and task metrics, we walked through one in Your Databricks job took 42 minutes instead of 8.
Start with the join, not the cluster. If one side of a join is small, broadcast it. A broadcast join sends the small table to every executor and skips the shuffle entirely, which means no sort and no spill.
from pyspark.sql import functions as F
joined_df = large_df.join(F.broadcast(small_df), "customer_id")If the problem is skew in a group by, salting is the honest fix. Give each row a random tag, aggregate per key and tag, then combine the partial results. It spreads the tall stack across many desks.
from pyspark.sql import functions as F
# Spread each hot key across ten buckets, then combine the partial counts.
salted_df = events_df.withColumn("salt", (F.rand() * 10).cast("int"))
partial_df = salted_df.groupBy("group_key", "salt").agg(
F.count("*").alias("partial_count")
)
final_df = partial_df.groupBy("group_key").agg(
F.sum("partial_count").alias("row_count")
)Third, let Adaptive Query Execution help before you tune anything by hand. AQE is enabled by default on recent Databricks runtimes. It can split skewed partitions and merge small shuffle partitions on its own.
SET spark.sql.adaptive.enabledIf that returns true, the platform is already doing part of this work for you. Manual fixes are for the cases it cannot see.
A small amount of spill on a heavy stage is normal. Spark chose to spill because it was the safe option, and the job still finished.
Worry when spill concentrates on one or two tasks, when the spilling stage sits on the critical path of a slow job, or when the same query slows month over month as data grows. Those are design signals, not hardware signals.
The fix is rarely more memory. It is almost always a better distribution of the work.
Spill (Memory) is data pushed out of a task's active workspace. Spill (Disk) is that same data serialized and compressed on local disk, which is why it is always the smaller number. Skew, high-cardinality aggregations, and large sort-merge joins cause most spill. Broadcast small tables, salt hot keys, and let Adaptive Query Execution help before reaching for manual tuning.