The backfill. How to reload history in Databricks without breaking downstream tables

A calm plan for rewriting the past

The message arrived on a Thursday afternoon. "The revenue numbers for March look wrong. Can you reload them?"

Nobody says the hard part out loud. Reloading March means touching a table that six dashboards read, two machine learning features depend on, and one finance report was already signed off against. The data is wrong, so it has to change. But the change itself is the risk.

This is the backfill. Every data engineer meets it, usually in a hurry, usually without a plan. And the difference between a calm backfill and a bad weekend is almost entirely decided before you run anything.

A backfill is a rewrite of the past

A normal pipeline run adds new data. A backfill changes data that people have already looked at.

That single difference explains why backfills feel scary. Downstream consumers assumed the past was settled. Reports were exported. Models were trained. Someone quoted a number in a meeting.

So the goal of a good backfill is not just correct data. It is correct data with a clear boundary, so everyone can say exactly what changed and when.

A backfill is not a bigger pipeline run. It is a controlled correction of history.

Step one: name the window

Before writing code, write one sentence. "I am reloading orders_silver for order_date between 2026-03-01 and 2026-03-31, because the source system sent the wrong currency code."

That sentence does three things. It sets the exact range. It sets the exact table. It sets the reason, which is what you will need when someone asks in June why March looks different from the export they saved.

If you cannot write that sentence, you are not ready to run anything.

Step two: make the write replace, not append

The most common backfill bug is duplication. The job reruns, the same March rows land again, and now revenue is double.

Delta Lake gives you a clean way to avoid this. replaceWhere rewrites only the partition range you name, in one atomic commit, and leaves everything outside that range untouched.

from pyspark.sql import functions as F

march_orders = (
    spark.read.format("csv")
    .option("header", "true")
    .option("inferSchema", "true")
    .load("/Volumes/workspace/default/book_data/orders_march.csv")
)

corrected_orders = march_orders.withColumn(
    "order_date", F.to_date("order_date")
)

(
    corrected_orders.write
    .format("delta")
    .mode("overwrite")
    .option("replaceWhere", "order_date >= '2026-03-01' AND order_date <= '2026-03-31'")
    .saveAsTable("workspace.default.orders_silver")
)

Two things matter here. The filter in replaceWhere must match the data you are writing, otherwise the write fails. And the whole operation is one commit, so readers either see the old March or the new March, never a half loaded month.

If you are correcting individual rows rather than a whole window, MERGE is the better tool.

MERGE INTO workspace.default.orders_silver AS target
USING workspace.default.orders_march_fixed AS source
  ON target.order_id = source.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *

MERGE is safe to rerun. Running it twice gives the same result as running it once. That property is what makes a backfill restartable, and it is the same idea we covered in the article on safe reruns.

Step three: rehearse in a copy

You do not have to be brave. Clone the table first, run the backfill against the clone, and compare.

CREATE OR REPLACE TABLE workspace.default.orders_silver_rehearsal
SHALLOW CLONE workspace.default.orders_silver

A shallow clone points at the same files, so it is fast and cheap. Run your backfill against the clone, then compare the numbers that people actually care about.

SELECT
  'before' AS version,
  SUM(amount) AS march_revenue,
  COUNT(*) AS row_count
FROM workspace.default.orders_silver
WHERE order_date BETWEEN '2026-03-01' AND '2026-03-31'

UNION ALL

SELECT
  'after' AS version,
  SUM(amount) AS march_revenue,
  COUNT(*) AS row_count
FROM workspace.default.orders_silver_rehearsal
WHERE order_date BETWEEN '2026-03-01' AND '2026-03-31'

Now you know the size of the change before anyone else feels it. If revenue moves by 2 percent, you tell finance a number. If row count triples, you found your own bug in private.

Step four: work in slices, not one heroic run

Reloading three years of history in a single job is a bad idea. It runs for hours, it holds a large write, and when it fails at hour four you learn nothing except that it failed.

Loop over the window instead, one month at a time, with replaceWhere scoped to each slice.

months = [
    ("2026-01-01", "2026-01-31"),
    ("2026-02-01", "2026-02-28"),
    ("2026-03-01", "2026-03-31"),
]

for start_date, end_date in months:
    slice_df = corrected_orders.filter(
        (F.col("order_date") >= start_date) & (F.col("order_date") <= end_date)
    )

    (
        slice_df.write
        .format("delta")
        .mode("overwrite")
        .option("replaceWhere", f"order_date >= '{start_date}' AND order_date <= '{end_date}'")
        .saveAsTable("workspace.default.orders_silver")
    )

    print(f"Backfilled {start_date} to {end_date}")

Each slice is a commit. If the loop dies at February, January is already correct and you restart from February. Progress is kept, not lost.

Step five: keep the undo button

Delta Lake keeps a history of commits, and that history is your rollback.

DESCRIBE HISTORY workspace.default.orders_silver

Note the version number before you begin. If the backfill is wrong, you can read the old state, or restore it.

RESTORE TABLE workspace.default.orders_silver TO VERSION AS OF 412

This is the calmest part of a backfill. Knowing you can go back changes how it feels to move forward. If time travel is new to you, the Delta Lake lesson walks through versions and restores with runnable examples.

One caution. VACUUM removes old files, and once they are gone, older versions cannot be restored. Do not vacuum a table on the same day you backfill it.

Step six: tell the downstream

Correct data that nobody knows about still causes confusion.

After the backfill, do three small things. Refresh or rerun the downstream tables that read the window you changed. Post the before and after numbers from your rehearsal query. And record the reason somewhere permanent, such as a table comment or a change log table.

COMMENT ON TABLE workspace.default.orders_silver IS
  'Silver orders. March 2026 backfilled on 2026-08-19 to fix source currency codes.'

That comment costs you ten seconds and saves someone an hour in three months. It also helps the AI agents that now read your catalog, which we looked at in agent discovers, agent applies.

The habits that make backfills boring

Most painful backfills are painful because of choices made much earlier.

Keep a date column that reflects the business event, not the load time, so a window is easy to name. Keep raw landed files, so you can rebuild bronze without asking the source team for a resend. Make every write either a replace of a known range or a merge on a key. Add quality checks that catch a wrong currency code the day it arrives instead of five months later.

Those habits are the medallion idea in practice. Bronze holds what arrived, silver holds what is clean, and gold holds what is agreed. When each layer can be rebuilt from the one below it, a backfill stops being an emergency and becomes a scheduled task.

Continue learning

The next time someone asks you to reload a month, you do not need courage. You need a window, a rehearsal, a slice loop, and a version number to fall back to. That is the whole job.