A small mark on the side instead of a full rewrite. Here is how it works and when to clean up.
Imagine a table with 50 million orders. A customer asks you to remove their data. You run one DELETE for one customer.
In older versions of Delta Lake, that simple request could take minutes. Spark had to find every file that held a matching row, read the whole file, and write a brand new copy without that row.
One row out. A whole file rewritten. This is called write amplification.
Delta tables store data in Parquet files. Parquet files are immutable. Once written, they are never edited in place.
So the only way to remove a row was to write a new file without it, then mark the old file as removed in the transaction log.
That works well for big batch changes. It is wasteful for small, frequent ones like GDPR deletes or a few MERGE updates each hour.
Think of a library catalog card. When one entry is wrong, the librarian does not reprint the whole card. They draw a line through the entry and keep going.
A deletion vector is that line. It is a tiny file that says: in data file X, rows 102 and 5,431 are deleted.
The data file stays exactly as it was. Readers check the deletion vector and skip the marked rows.
flowchart LR
A[DELETE one row] --> B[Write small deletion vector]
B --> C[Transaction log points to file + vector]
C --> D[Reader loads data file]
D --> E[Skips rows marked in vector]
E --> F[Correct result]Updates work the same way. The old row is marked as deleted, and the new version of the row is written to a small new file.
Create a small table and check its properties.
CREATE OR REPLACE TABLE workspace.default.dv_orders AS
SELECT
id AS order_id,
id % 100 AS customer_id,
id * 1.5 AS amount
FROM range(100000);
SHOW TBLPROPERTIES workspace.default.dv_orders;On newer runtimes, deletion vectors are often turned on by default. If you do not see delta.enableDeletionVectors set to true, turn it on:
ALTER TABLE workspace.default.dv_orders
SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);Now delete a few rows and look at what happened.
DELETE FROM workspace.default.dv_orders
WHERE customer_id = 42;
DESCRIBE HISTORY workspace.default.dv_orders;Open the operationMetrics column of the DELETE row. Look for numDeletionVectorsAdded. You should also see that very few files, or none, were rewritten.
That is the whole idea in one observation.
Nothing is free. Every reader now has to check the deletion vectors and filter out marked rows.
A few vectors cost almost nothing. Thousands of small deletes over weeks can slowly add up, and the deleted rows still take space on storage.
So you clean up from time to time. These two commands rewrite affected files so the marked rows are truly gone:
OPTIMIZE workspace.default.dv_orders;
REORG TABLE workspace.default.dv_orders APPLY (PURGE);OPTIMIZE compacts files and applies the vectors as it goes. REORG ... APPLY (PURGE) focuses on removing deleted rows from files. After that, VACUUM can clear old files once the retention period passes.
Deletion vectors help most when changes are small and frequent:
Privacy requests that remove one customer at a time.
MERGE jobs that update a few rows in a large table every hour.
Streaming upserts that touch a small slice of data on each run.
For a nightly job that rewrites a full partition, you will barely notice a difference. The benefit shows up when the change is tiny and the table is large.
One more thing to know. Some older readers outside Databricks may not support deletion vectors. If other tools read your table directly, check their Delta reader version before turning the feature on.
To see how the transaction log tracks files, read Delta Lake Fundamentals. For OPTIMIZE and clustering, see File Formats and Optimization. Then test yourself with the free Data Engineer Associate practice exam.