Delta Lake vs Apache Iceberg vs Apache Hudi: A Practical Comparison for Data Engineers

Understanding the three dominant open table formats, when to use each, and what it means for your career

Delta Lake vs Apache Iceberg vs Apache Hudi: A Practical Comparison for Data Engineers

If you work with data, you have probably heard the phrase "open table format" more times than you can count. Delta Lake, Apache Iceberg, and Apache Hudi are the three technologies fighting for this space. And the choice between them is not academic. It affects how you build pipelines, how you handle schema changes, and how your data platform evolves over the next few years.

This article breaks down all three formats with honest, practical comparisons. No vendor cheerleading. Just what you need to know as a working data engineer.

What Is an Open Table Format?

Before comparing the three, let us agree on what we are actually talking about.

An open table format is a metadata layer that sits between your query engine and your raw data files. It turns a collection of Parquet files (or ORC, or Avro) into something that behaves like a database table. That means ACID transactions, schema enforcement, time travel, and partition management.

Without a table format, your "table" is really just a folder of files. There is no transaction log. No way to roll back a bad write. No guarantee that a reader will not see a half-written batch.

Here is how the stack fits together:

graph TD
    A[Query Engine] --> B[Table Format]
    B --> C[File Format]
    C --> D[Cloud Storage]
    A1[Spark / Trino / Flink] --> A
    B1[Delta Lake / Iceberg / Hudi] --> B
    C1[Parquet / ORC / Avro] --> C
    D1[S3 / ADLS / GCS] --> D

The table format is the layer that makes raw files behave like a proper table. Each of the three formats does this differently.

If you are new to how data files work at this level, our chapter on File Formats and Storage covers the foundation. Understanding Parquet and columnar storage makes everything in this article click better.

Delta Lake

Origins and Philosophy

Delta Lake was created at Databricks and open-sourced in 2019. It grew out of a simple frustration: data lakes were unreliable. Files would get corrupted, schema mismatches would break pipelines, and there was no way to undo a bad write.

The philosophy is pragmatic. Delta Lake focuses on making data lakes reliable first, and fast second. It is deeply integrated with Apache Spark and the Databricks ecosystem.

How It Works

Delta Lake uses a transaction log (the _delta_log directory) to track every change to a table. Each transaction creates a new JSON file in this log. Periodically, these JSON files get compacted into Parquet checkpoint files for faster reads.

This is a simple, linear log. To know the current state of the table, you replay the log from the last checkpoint forward.

Key Features

• ACID transactions with serializable isolation
• Time travel via version numbers or timestamps
• Schema enforcement that rejects bad writes
• Schema evolution for adding, renaming, or reordering columns
• Z-ORDER clustering for query optimization
• Change Data Feed for downstream consumers
• Liquid clustering (newer feature) for automatic data layout

Code Example: Delta Lake in PySpark

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
    .getOrCreate()

# Create a Delta table
df = spark.createDataFrame([
    (1, "Alice", "Engineering", 95000),
    (2, "Bob", "Marketing", 72000),
    (3, "Carol", "Engineering", 110000)
], ["id", "name", "department", "salary"])

df.write.format("delta").save("/data/employees")

# Time travel: read a previous version
df_v0 = spark.read.format("delta") \
    .option("versionAsOf", 0) \
    .load("/data/employees")

# MERGE (upsert)
from delta.tables import DeltaTable

target = DeltaTable.forPath(spark, "/data/employees")
updates = spark.createDataFrame([
    (2, "Bob", "Sales", 78000),
    (4, "Diana", "Engineering", 88000)
], ["id", "name", "department", "salary"])

target.alias("t").merge(
    updates.alias("u"),
    "t.id = u.id"
).whenMatchedUpdateAll() \
 .whenNotMatchedInsertAll() \
 .execute()

-- SQL equivalent: time travel
SELECT * FROM employees VERSION AS OF 3;

-- Schema evolution
ALTER TABLE employees ADD COLUMNS (bonus DOUBLE);

-- Z-ORDER optimization
OPTIMIZE employees ZORDER BY (department);

We cover Delta Lake architecture in depth, including the transaction log, checkpointing, and VACUUM operations, in our Delta Lake chapter. Schema evolution patterns are explored in Schema Management.

Apache Iceberg

Origins and Philosophy

Apache Iceberg was created at Netflix and donated to the Apache Software Foundation in 2018. It was born from Netflix's need to manage petabyte-scale tables across thousands of partitions without the performance problems of the Hive metastore.

Iceberg's philosophy is engine-agnostic correctness. It was designed from the start to work with any query engine, not just Spark. This is a key difference from Delta Lake's Spark-first approach. As of June 2026, Iceberg v3 is GA, and Databricks provides full support for Managed Iceberg and Foreign Iceberg.

How It Works

Iceberg uses a tree of metadata files. At the top is a metadata file that points to a manifest list, which points to individual manifest files, which track the actual data files. Each snapshot captures the complete state of the table at a point in time.

This tree structure is what enables Iceberg's partition evolution and hidden partitioning. The metadata is rich enough that the engine can plan queries without listing files in storage.

Key Features

• ACID transactions with snapshot isolation
• Hidden partitioning (users query by column, Iceberg handles the partition layout)
• Partition evolution (change partitioning without rewriting data)
• Schema evolution with full column ID tracking
• Time travel via snapshot IDs or timestamps
• Row-level deletes (position deletes and equality deletes)
• Engine-agnostic (Spark, Trino, Flink, Dremio, Snowflake)

Code Example: Apache Iceberg in PySpark

spark = SparkSession.builder \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config("spark.sql.catalog.my_catalog", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.my_catalog.type", "hadoop") \
    .config("spark.sql.catalog.my_catalog.warehouse", "/warehouse") \
    .getOrCreate()

# Create an Iceberg table with hidden partitioning
spark.sql("""
    CREATE TABLE my_catalog.db.events (
        event_id BIGINT,
        event_time TIMESTAMP,
        user_id STRING,
        event_type STRING
    )
    USING iceberg
    PARTITIONED BY (days(event_time), bucket(16, user_id))
""")

# Partition evolution: change strategy without rewriting
spark.sql("""
    ALTER TABLE my_catalog.db.events
    ADD PARTITION FIELD hours(event_time)
""")

# Time travel
spark.sql("""
    SELECT * FROM my_catalog.db.events
    FOR SYSTEM_TIME AS OF '2025-01-15 10:00:00'
""")

Notice the days(event_time) and bucket(16, user_id) syntax. Users never need to know about partition columns in their queries. They just write WHERE event_time > '2025-01-01' and Iceberg handles the rest. This is what "hidden partitioning" means.

The concept of partition pruning and storage optimization connects directly to our Partitioning and Performance chapter. If you want to understand why partition strategy matters so much, start there.

Apache Hudi

Origins and Philosophy

Apache Hudi (Hadoop Upserts Deletes and Incrementals) was created at Uber and open-sourced in 2019. It was built to solve a very specific problem: efficiently updating and deleting records in massive datasets on data lakes.

Hudi's philosophy is incremental processing. While Delta Lake and Iceberg focus on table correctness, Hudi focuses on making updates fast and enabling incremental data pipelines.

How It Works

Hudi organizes data into a timeline of actions. Each action (commit, compaction, cleaning) is recorded on this timeline. The unique feature is Hudi's two storage types:

• Copy-on-Write (CoW): Updates rewrite entire files. Reads are fast, writes are slower.
• Merge-on-Read (MoR): Updates go to delta log files. Writes are fast, reads merge base files with logs.

This gives you explicit control over the read/write performance tradeoff.

Key Features

• ACID transactions with snapshot isolation
• Two storage types (CoW and MoR) for tuning performance
• Incremental queries (only read what changed since last checkpoint)
• Record-level indexing for fast upserts
• Built-in table services (compaction, clustering, cleaning)
• Time travel via instant timestamps
• Schema evolution support

Code Example: Apache Hudi in PySpark

# Write a Hudi table (Copy-on-Write)
df.write.format("hudi") \
    .option("hoodie.table.name", "rides") \
    .option("hoodie.datasource.write.recordkey.field", "ride_id") \
    .option("hoodie.datasource.write.precombine.field", "updated_at") \
    .option("hoodie.datasource.write.operation", "upsert") \
    .option("hoodie.datasource.write.table.type", "COPY_ON_WRITE") \
    .mode("append") \
    .save("/data/rides")

# Incremental query: only read changes
incremental_df = spark.read.format("hudi") \
    .option("hoodie.datasource.query.type", "incremental") \
    .option("hoodie.datasource.read.begin.instanttime", "20250115100000") \
    .load("/data/rides")

# Time travel
history_df = spark.read.format("hudi") \
    .option("as.of.instant", "20250115100000") \
    .load("/data/rides")

Notice how Hudi requires a record key and a precombine field. This is because Hudi is fundamentally designed around record-level operations. Every row has an identity.

Incremental processing is a core concept in modern data engineering. Our Incremental Processing chapter covers the patterns and tradeoffs in detail. The CoW vs MoR decision connects to how you handle Slowly Changing Dimensions.

Head-to-Head Comparison

Here is where we put all three side by side. This table covers the features that matter most when choosing a format.

Performance Characteristics

Performance depends heavily on your workload pattern:

Understanding file compaction and storage optimization connects to our File Formats chapter. The performance tradeoffs here are the same ones you face when designing any data pipeline.

Architecture Comparison

The biggest technical difference between the three formats is how they track metadata. This affects everything from query planning to concurrency.

graph LR
    subgraph Delta Lake
        D1[Transaction Log] --> D2[JSON Entries]
        D2 --> D3[Checkpoint Parquet]
        D3 --> D4[Data Files]
    end
    subgraph Apache Iceberg
        I1[Metadata File] --> I2[Manifest List]
        I2 --> I3[Manifest Files]
        I3 --> I4[Data Files]
    end
    subgraph Apache Hudi
        H1[Timeline] --> H2[Commits]
        H2 --> H3[File Groups]
        H3 --> H4[Base + Log Files]
    end

Delta Lake keeps it simple. A linear log of JSON files, periodically checkpointed. This simplicity is a strength for debugging but can be a bottleneck for tables with thousands of partitions.

Iceberg uses a tree structure. This is more complex but enables faster query planning because the engine can prune entire manifest files without reading individual entries. This is why Iceberg shines with very large tables.

Hudi organizes around file groups. Each record key maps to a file group, and updates either rewrite the base file (CoW) or append to a log file (MoR). This record-centric design is why Hudi excels at upserts.

When to Use What

Choosing a table format is a real decision with real consequences. Here is a framework to help:

graph TD
    A[Start: Choose a Table Format] --> B{Primary platform?}
    B -->|Databricks| C{Need multi-engine?}
    C -->|No| D[Delta Lake]
    C -->|Yes| E[Delta Lake + UniForm]
    B -->|Multi-cloud / Multi-engine| F{Primary workload?}
    F -->|Analytics + Ad-hoc| G[Apache Iceberg]
    F -->|Heavy upserts| H[Apache Hudi]
    F -->|Mixed| I[Apache Iceberg]
    B -->|AWS-native| J{Upsert volume?}
    J -->|High| K[Apache Hudi]
    J -->|Low to moderate| L[Apache Iceberg]

Practical Guidance

Choose Delta Lake if:

• You are on Databricks or Microsoft Fabric
• Your team already knows Spark well
• You want the simplest path to a reliable lakehouse
• Streaming is a core part of your architecture

Choose Apache Iceberg if:

• You need to query from multiple engines (Spark, Trino, Flink, Snowflake)
• You have very large tables with evolving partition strategies
• You want the most portable format
• You value hidden partitioning to prevent query mistakes

Choose Apache Hudi if:

• Your primary workload is record-level updates at scale
• You need true incremental processing pipelines
• You want explicit control over read/write performance tradeoffs
• Your update patterns are CDC-heavy

If you are building a lakehouse architecture, the table format choice shapes everything. Our Medallion Architecture chapter shows how Bronze, Silver, and Gold layers work in practice.

The Convergence Story

Here is the thing that most comparison articles miss: the three formats are converging.

Delta Lake UniForm can now write data that is readable as both Delta Lake and Iceberg. This means you can use Delta Lake for writes and let Iceberg-compatible engines (like Snowflake or Trino) read the same data without conversion.

Iceberg REST Catalog is becoming an industry standard. Even Databricks has adopted it as a protocol for Unity Catalog. This means the catalog layer is separating from the format layer.

Hudi has added support for reading Iceberg metadata, and its newer versions support more Spark SQL syntax that aligns with how Delta Lake and Iceberg work.

The practical implication? The format you choose today matters less than it did two years ago. The ecosystem is moving toward interoperability.

But that does not mean the choice is irrelevant. Each format still has strengths in specific areas. And the tooling, community, and managed service support still differ significantly.

Unity Catalog is playing a major role in this convergence story. Our Unity Catalog chapter covers how governance and catalog management work across formats. OpenSharing, the evolution of Delta Sharing hosted by the Linux Foundation, now provides a vendor-neutral protocol for sharing these assets.

Ecosystem and Community

The health of an open-source project matters for your long-term investment.

Iceberg has seen the fastest growth in enterprise adoption over the past two years, partly because Snowflake adopted it as a native format. Delta Lake remains dominant in the Databricks ecosystem. Hudi has a strong niche in CDC-heavy and streaming workloads.

What This Means for Your Career

As a data engineer, you do not need to become an expert in all three formats. But you should understand the concepts they share and the tradeoffs they make.

Here is what to focus on:

Concepts that transfer across all three:

• ACID transactions on data lakes
• Schema evolution patterns
• Partition strategies and file layout
• Time travel and data versioning
• Merge (upsert) operations
• Compaction and small file management

Format-specific skills worth learning:

The good news is that the foundational concepts are the same across all three. If you understand how Delta Lake handles transactions, you understand 80% of how Iceberg and Hudi handle them too. Modern tools like Lakeflow Declarative Pipelines (formerly Delta Live Tables) further simplify these implementations within the Databricks ecosystem.

This is exactly the approach we take in our book. Start with the fundamentals in Data Engineering Fundamentals, build up through DataFrames and Transformations, and the table format concepts will feel natural by the time you reach the Delta Lake chapter.

Summary

Delta Lake, Apache Iceberg, and Apache Hudi all solve the same core problem: making data lakes reliable and performant. They differ in architecture, philosophy, and ecosystem support.

Delta Lake is the simplest and most mature for Databricks users. Apache Iceberg is the most portable and scales best for very large, multi-engine environments. Apache Hudi excels at record-level updates and incremental processing.

The formats are converging. UniForm, REST Catalog standards, and growing cross-compatibility mean your choice is less of a lock-in decision than it used to be.

What matters most is understanding the underlying concepts. Schema evolution, partition strategy, ACID transactions, and merge patterns are the skills that transfer regardless of which format your organization chooses.

Start with the fundamentals. The format will follow.

Continue Learning

Deepen your Delta Lake skills with Chapter 7: Delta Lake and Chapter 8: Schema Evolution. For production patterns, read Delta Lake Best Practices.