A new column, a renamed column, and a broken morning. How to handle schema change on purpose instead of by accident.
The message arrived at 6:40 in the morning. "Job failed. Bronze table half loaded. Can you look?"
Nothing had changed on our side. No new code, no new cluster, no new config. The pipeline had run every night for four months without a complaint.
What changed was the source. A team upstream shipped a release. They added a column called loyalty_tier, and they renamed cust_id to customer_id. Two small edits in their world. One broken morning in ours.
If you work with data long enough, you meet this problem. The schema changes, and the pipeline has an opinion about that.
This article is about having the right opinion.
A schema change is not one thing. It is at least three, and they behave very differently.
A new column appears. This is the friendliest case. Your old code still works because it never asked for that column. The only question is whether you want the new data or not.
A column is renamed. This looks small to the person who did it and large to everyone downstream. To Delta Lake, a rename is a column that vanished and a column that appeared. Anything that referenced the old name breaks.
A type changes. A string becomes an integer, or an integer becomes a decimal. This is the quiet one. Sometimes it fails loudly. Sometimes it succeeds and gives you wrong numbers, which is worse.
Naming the case before you fix it saves a lot of time. Most bad schema decisions come from treating all three as the same emergency.
When Delta Lake refuses to write data that does not match the table, that is not the tool being difficult. That is the tool protecting the table.
Without enforcement, a mismatched load quietly puts nulls where values used to be, or writes the same information under two different column names. You find out weeks later, from a report that never looked right.
So the goal is not to switch enforcement off. The goal is to decide, on purpose, which changes you accept automatically and which changes need a human.
A pipeline that accepts every schema change is not flexible. It is just unguarded.
If you are reading files from a landing zone, Auto Loader gives you the best place to handle change, because it tracks the schema for you and it can park anything unexpected instead of failing.
from pyspark.sql import functions as F
landing_path = "/Volumes/workspace/default/book_data/orders_raw"
schema_path = "/Volumes/workspace/default/book_data/_schema/orders"
orders_stream = (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", schema_path)
.option("cloudFiles.schemaEvolutionMode", "addNewColumns")
.option("rescuedDataColumn", "_rescued_data")
.load(landing_path)
)Three options are doing the real work here.
cloudFiles.schemaLocation is where Auto Loader remembers the schema it has seen. Keep this path stable and separate for every stream.
cloudFiles.schemaEvolutionMode decides what happens when a new column shows up. addNewColumns is the sensible default: the stream stops once, records the new column, and picks it up when it restarts. In a Lakeflow Declarative Pipeline the restart is automatic.
rescuedDataColumn is the safety net. Anything that does not fit the known schema lands in _rescued_data as JSON instead of being dropped. You lose nothing, and you get something you can look at.
That last point matters more than it sounds. Rescued data turns a silent loss into a visible question.
surprises = (
spark.read.table("bronze.orders")
.filter(F.col("_rescued_data").isNotNull())
.select("ingest_file", "_rescued_data")
.limit(20)
)
display(surprises)If you already know a field will be trouble, say so up front with a schema hint rather than letting inference guess.
.option("cloudFiles.schemaHints", "order_total DOUBLE, customer_id STRING")Hints are how you stop the classic problem where the first day of files has order_total as a long, and the second day has decimals.
Once the data is in a DataFrame, the question moves to the write.
(
orders_clean.write
.format("delta")
.mode("append")
.option("mergeSchema", "true")
.saveAsTable("bronze.orders")
)mergeSchema adds new columns to the table. It is a reasonable choice for bronze, where your job is to keep what arrived.
overwriteSchema is a different animal. It replaces the schema, which means it can drop columns and change types in one move.
(
orders_rebuilt.write
.format("delta")
.mode("overwrite")
.option("overwriteSchema", "true")
.saveAsTable("bronze.orders")
)Use this when you meant to rebuild the table. Do not use it inside a nightly job to make an error go away. It is the option that turns a failed load into a lost column.
A simple rule that has served me well: bronze may merge, silver and gold should be explicit. In silver you are making promises to other people, so choose your columns by name and let the job fail if a promise cannot be kept.
For the rename case, Delta has column mapping. With mapping on, the table stores a stable internal id for each column, so a rename becomes metadata only. No file rewrite, no long maintenance window.
ALTER TABLE silver.customers
SET TBLPROPERTIES ('delta.columnMapping.mode' = 'name');
ALTER TABLE silver.customers
RENAME COLUMN cust_id TO customer_id;Turn it on when you create the table, not on the morning you need it. Enabling column mapping raises the reader and writer version of the table, so check that everything reading it can keep up before you flip it in production.
Every technique above handles change after it arrives. The cheaper fix happens earlier.
Write down the columns you depend on. Not the whole schema, just the handful your silver tables and reports would break without. Five or six names and types is usually enough.
Then check them on the way in.
required = {"customer_id": "string", "order_total": "double", "order_ts": "timestamp"}
actual = {f.name: f.dataType.simpleString() for f in orders_stream.schema.fields}
missing = [name for name in required if name not in actual]
if missing:
raise ValueError(f"Contract broken. Missing columns: {missing}")Ten lines, and the failure now says what is wrong instead of leaving you to guess at 6:40 in the morning. In a Lakeflow Declarative Pipeline you can express the same idea as an expectation and let the pipeline record how many rows failed.
The social half matters too. Tell the upstream team which columns you rely on. Most people will happily give you a week of notice for a rename. They just did not know anyone was watching.
A short checklist, in order.
Read the error and decide which of the three cases you have: added, renamed, or retyped.
Look at the table history to see what the last successful write looked like. DESCRIBE HISTORY bronze.orders is usually faster than reading logs.
Check _rescued_data for the recent files. If the values are sitting there, nothing is lost and you have time to think.
Fix forward for an added column. Coordinate for a rename. Slow down for a type change and confirm the numbers before you accept the new type.
Only then, if the table really is in a bad state, use RESTORE TABLE to go back to the last clean version. Time travel exists so that a bad morning does not become a bad week.
RESTORE TABLE bronze.orders TO VERSION AS OF 412;Schema change is one of those topics that feels like an edge case until it happens twice. Then it becomes part of how you design.
Practice it in Free Edition. Load a file, add a column to the next file, and watch what your pipeline does. The lesson lands much harder when you break it yourself.
Continue learning