Delta Lake Shallow Clone vs Deep Clone Explained

Safe testing on real data without paying for a second copy

You need to test a risky change. Maybe a new column type, a MERGE rewrite, or a cleanup job that deletes old rows. The real table is large. Copying it would take hours and double your storage bill. Testing on a tiny sample feels safe, but it hides the problems real data would show.

Delta Lake has a quiet answer to this: clone. It comes in two kinds, shallow and deep. Knowing the difference saves time, money and a few bad mornings.

What a Delta table really is

A Delta table is two things. A folder of Parquet data files, and a transaction log that says which files belong to the current version.

The log is small. The data files are big. That split is the whole trick behind cloning.

Shallow clone: copy the map, not the land

A shallow clone copies only the transaction log. The new table points at the same data files as the source. It is created in seconds, no matter how big the source is.

When you change the clone, Delta writes new files for the clone only. The source files are never touched. This is called copy on write.

flowchart LR
  S[Source table log] --> F[(Shared Parquet files)]
  C[Shallow clone log] --> F
  C --> N[(New files written by the clone)]

Deep clone: a full, independent copy

A deep clone copies the log and every data file. It takes longer and uses real storage. In return, the copy stands on its own. If the source is vacuumed or dropped, the deep clone keeps working.

Deep clones can also be rerun. Each run copies only what changed since the last one, which makes them useful for backups and moving data between environments.

Try it in Databricks Free Edition

Run this in a SQL notebook cell. It uses the default workspace.default schema.

CREATE OR REPLACE TABLE workspace.default.orders AS
SELECT
  id AS order_id,
  CAST(rand() * 500 AS INT) AS amount
FROM range(100000);

CREATE OR REPLACE TABLE workspace.default.orders_test
SHALLOW CLONE workspace.default.orders;

Now make a destructive change on the clone only.

DELETE FROM workspace.default.orders_test
WHERE amount < 250;

SELECT 'source' AS table_name, COUNT(*) AS row_count FROM workspace.default.orders
UNION ALL
SELECT 'clone' AS table_name, COUNT(*) AS row_count FROM workspace.default.orders_test;

The source still has 100,000 rows. The clone has about half. You tested a delete against realistic data and production never noticed.

Run DESCRIBE HISTORY workspace.default.orders_test and you will see the CLONE operation recorded as the first version.

To make a full copy instead, swap one word:

CREATE OR REPLACE TABLE workspace.default.orders_backup
DEEP CLONE workspace.default.orders;

The VACUUM trap

This is the one rule to remember. A shallow clone depends on the source files. If someone runs VACUUM on the source and removes files the clone still points at, queries on the clone start failing with missing file errors.

So treat shallow clones as short lived. Create one, test, drop it. If you need something that lasts, use a deep clone.

When to use which

That last one is handy. SHALLOW CLONE my_table VERSION AS OF 12 gives you yesterday's table to debug, without restoring anything in production.

Why this matters in real work

Good engineers do not test on production, and they do not test on toy data either. Clones close that gap. They make the safe path also the fast path, which is the only way safety habits survive busy weeks.

If you are learning, this is also a great way to see the Delta log in action. Clone a table, change it, and read the history of both. The idea of files plus a log stops being abstract.

Continue learning

• Deletion Vectors in Delta Lake
• Delta Lake Change Data Feed Explained
• Start the BricksNotes book