One hot key can make one task do all the work. Here is how to spot it and fix it.
It is late afternoon. Your Spark job has 200 tasks. 199 of them finish in about 15 seconds. The last one keeps running for 40 minutes.
So you add more workers. The job still takes 40 minutes.
This is data skew. And more hardware does not fix it.
When Spark groups or joins data, it sends every row with the same key to the same partition. That is how it guarantees all rows for one key meet in one place.
Most of the time this is fine. But real data is rarely even. One big customer, one busy country, or one NULL value can hold most of the rows.
If one key holds 70% of the data, one task must process 70% of the data. The other cores finish quickly and then sit idle, waiting.
flowchart LR
A[Input rows] --> B{Group by key}
B --> C[Task 1: small]
B --> D[Task 2: small]
B --> E[Task 3: HUGE hot key]
B --> F[Task 4: small]
E --> G[Job waits for Task 3]Adding workers adds more small tasks. It does not split the huge one.
Open the Spark UI for the slow stage and look at task durations. Compare the median with the max.
If the median is 2 seconds and the max is 30 minutes, you have skew. Also check "Shuffle Read Size". One task reading far more data than the others is the clearest sign.
Adaptive Query Execution is on by default in Databricks. It can detect a skewed join partition and split it into smaller pieces automatically.
SET spark.sql.adaptive.enabled = true;
SET spark.sql.adaptive.skewJoin.enabled = true;AQE works well for joins. It does not help much for a heavy GROUP BY on one hot key. For that, you can use salting.
Salting means adding a small random number to the hot key. Instead of one key IN, you get IN_0, IN_1, IN_2, IN_3.
Now Spark spreads those rows across four tasks instead of one. You aggregate in two steps: first by the salted key, then by the original key.
flowchart LR
A[Hot key rows] --> B[Add salt 0-3]
B --> C[Task: key_0]
B --> D[Task: key_1]
B --> E[Task: key_2]
B --> F[Task: key_3]
C & D & E & F --> G[Combine partial totals]This example builds a small skewed dataset and fixes it with salting. It runs on serverless compute in Free Edition.
from pyspark.sql import functions as F
# 1. Build skewed data: about 90% of orders come from one country
orders_df = (
spark.range(0, 2_000_000)
.withColumn(
"country",
F.when(F.rand() < 0.9, F.lit("IN"))
.otherwise(F.concat(F.lit("C"), (F.rand() * 50).cast("int")))
)
.withColumn("amount", (F.rand() * 100).cast("double"))
)
# 2. See the imbalance
display(orders_df.groupBy("country").count().orderBy(F.desc("count")).limit(5))
# 3. Add a salt between 0 and 3
SALT_BUCKETS = 4
salted_df = orders_df.withColumn(
"salt", (F.rand() * SALT_BUCKETS).cast("int")
)
# 4. Stage one: partial totals per country and salt
partial_totals_df = (
salted_df
.groupBy("country", "salt")
.agg(F.sum("amount").alias("partial_amount"))
)
# 5. Stage two: combine partials into the final total
country_totals_df = (
partial_totals_df
.groupBy("country")
.agg(F.sum("partial_amount").alias("total_amount"))
)
display(country_totals_df.orderBy(F.desc("total_amount")))On this small dataset you will not see a big time difference. The point is to see the pattern. Check that the final totals match a plain groupBy("country"). They should be identical.
Broadcast join. If one side of a join is small, broadcast it. No shuffle means no skew. This is often the simplest fix.
AQE skew join. For large joins with a few hot keys, let AQE split them first.
Salting. For heavy aggregations on hot keys, or when AQE is not enough.
Filter bad keys. Sometimes the hot key is NULL or "unknown". Handle those rows separately or drop them if they do not matter.
Start small, like 4 or 8. More buckets spread the work more, but add extra rows to the first stage.
A good rule is to salt only the hot keys if you know them. Every other key can keep salt 0.
Partitions and shuffles are covered step by step in the BricksNotes book. You can also read Spark spill explained and test yourself with the practice exam.