The Delete Your Pipeline Never Saw: Delta Lake Change Data Feed Explained

Why timestamp filters miss deletes, and how CDF passes every change downstream

Picture a nightly job that copies new orders from Silver to Gold. It uses a simple filter: load every row where updated_at is newer than last night.

It runs green for months. Then finance asks why a cancelled order still shows up in the revenue dashboard.

The order was deleted in Silver. But a deleted row has no new updated_at value. It is simply gone. So the filter never saw it, and Gold kept it forever.

This is the quiet weakness of timestamp-based pipelines. Delta Lake Change Data Feed is the fix.

The problem with "newer than last time"

A timestamp filter only sees rows that still exist and carry a fresh timestamp. That leaves two gaps.

The first gap is deletes. Nothing points to a row that is no longer there.

The second gap is timing. If one writer commits a little late with an older timestamp, the next run has already moved past it. That row is skipped for good.

Both bugs are silent. The job does not fail. The numbers are just slightly wrong.

What Change Data Feed does

Change Data Feed, or CDF, asks Delta Lake to record every row-level change as it happens. Inserts, updates and deletes are all written down, commit by commit.

You turn it on with one table property:

ALTER TABLE silver_orders
SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

From that point on, every write to the table also records which rows changed and how. Downstream jobs stop guessing. They read the list of changes instead.

flowchart LR
  A[Write to silver_orders] --> B[Delta commit]
  B --> C[Table data files]
  B --> D[Change records]
  D --> E[table_changes]
  E --> F[MERGE into gold_orders]

The four change types

Each change row carries a column called _change_type. There are only four values.

Two more columns come with it. _commit_version tells you which table version made the change. _commit_timestamp tells you when.

Having both the before and after values is useful. You can see that an order moved from 40 to 55, not just that it is 55 now.

Try it in Free Edition

This runs in a Databricks Free Edition notebook using SQL cells.

CREATE OR REPLACE TABLE cdf_orders (
  order_id INT,
  status STRING,
  amount DOUBLE
)
TBLPROPERTIES (delta.enableChangeDataFeed = true);

INSERT INTO cdf_orders VALUES
  (1, 'placed', 40.0),
  (2, 'placed', 25.0),
  (3, 'placed', 60.0);

UPDATE cdf_orders SET amount = 55.0 WHERE order_id = 1;

DELETE FROM cdf_orders WHERE order_id = 3;

Now ask the table what changed since version 1:

SELECT
  order_id,
  status,
  amount,
  _change_type,
  _commit_version
FROM table_changes('cdf_orders', 1)
ORDER BY _commit_version, order_id;

You will see three inserts, a before and after pair for order 1, and a delete for order 3. The delete is the row a timestamp filter would never have found.

To push these changes into Gold, keep only the latest change per key and drop the preimages:

MERGE INTO gold_orders AS target
USING (
  SELECT *
  FROM table_changes('cdf_orders', 1)
  WHERE _change_type != 'update_preimage'
  QUALIFY ROW_NUMBER() OVER (
    PARTITION BY order_id
    ORDER BY _commit_version DESC
  ) = 1
) AS changes
ON target.order_id = changes.order_id
WHEN MATCHED AND changes._change_type = 'delete' THEN DELETE
WHEN MATCHED THEN UPDATE SET
  target.status = changes.status,
  target.amount = changes.amount
WHEN NOT MATCHED AND changes._change_type != 'delete' THEN INSERT
  (order_id, status, amount)
  VALUES (changes.order_id, changes.status, changes.amount);

Create gold_orders first with the same three columns. In a real job, you save the last version you processed and start from the next one each run.

Streaming reads it too

If you prefer Structured Streaming, CDF works there as well. Spark keeps track of the last version in its checkpoint, so you never store versions by hand.

changes_df = (
    spark.readStream
    .option("readChangeFeed", "true")
    .table("cdf_orders")
)

Pair it with trigger(availableNow=True) and you get an incremental job that runs like a cheap batch.

Three rules for production

Turn it on where rows change. CDF earns its place on tables that receive updates and deletes, like Silver tables fed by MERGE. For append-only logs, a plain streaming read is enough.

Remember that history expires. Change records follow the table's retention. If VACUUM removes old files, very old versions can no longer be read. Run downstream jobs often enough to stay inside that window.

Enable it before you need it. CDF only records changes made after you switch it on. It cannot rebuild the past.

Why this matters for your career

Every company that moves data from one layer to another hits this bug eventually. Knowing why deletes vanish, and how to catch them, is the kind of understanding interviewers look for.

It also shows up on both the Databricks Data Engineer Associate and Professional exams. Questions often ask which _change_type to filter, or why a timestamp filter misses deletes.

Continue learning

• Stream Like Batch: How availableNow Runs Streaming Pipelines Cheaply
• Deletion Vectors in Delta Lake: Why Deleting One Row No Longer Rewrites Your Data
• Delta Lake Best Practices: VACUUM, OPTIMIZE, and Time Travel
• Free Data Engineer Associate Practice Exam