Lakeflow Connect: How Ingestion Is Moving Directly Into the Lakehouse

Databricks expanded Lakeflow to more than 100 native connectors and gave managed ingestion a free tier. Here is what changed, how it works, and what it means for the work data engineers actually do.

Lakeflow Connect: How Ingestion Is Moving Directly Into the Lakehouse

Every data platform starts with the same job. Before anyone can build a dashboard or train a model, someone has to get the data in.

For years, that job meant writing your own ingestion code. A script that reads from a source database. A scheduled job that copies files. A small service that calls an API and lands the results in storage. Every team built this layer by hand, and every team maintained it forever.

This year, that layer started moving into the platform itself. Databricks expanded Lakeflow to more than 100 native connectors and gave Lakeflow Connect, its managed ingestion service, a free tier for getting started. It is a quiet change, but it redraws the line between the work the platform does and the work your team does.

The old way: ingestion as a side project

Think about what a hand-built connector really involves.

You write code that reads from the source. You handle retries when the network drops. You track which rows you already loaded, so you do not load them twice. You notice when the source adds a column, usually because something breaks. Then you repeat all of this for the next source, and the next.

None of this work is visible in a demo. All of it is visible at 2 a.m. when a pipeline fails.

The strange part is that almost every team solves the same problems in slightly different ways. The logic is not special to your business. It is plumbing, and plumbing is exactly the kind of work a platform should absorb.

What Lakeflow Connect is

Lakeflow Connect is a set of managed connectors that bring data from source systems directly into Delta tables in your lakehouse.

You point a connector at a source, such as a relational database or a SaaS application. The connector handles the reading, the retries, the incremental tracking, and the landing. The data arrives as Delta tables, governed by Unity Catalog, ready for the pipelines you actually care about.

For databases, the connectors lean on change data capture. Instead of re-reading a whole table every night, the connector follows the changes: the inserts, the updates, and the deletes. That is what keeps large source tables practical to sync.

Lakeflow Connect carries data from source databases and SaaS apps through managed connectors into Unity Catalog governed Delta tables

Why this is different from scheduling a script

A scheduled script copies data. A managed connector operates a process.

The difference shows up in the boring places. Schema drift is detected instead of discovered. Checkpointing is built in, so a restart continues from where things stopped instead of starting over. Monitoring lives next to every other pipeline you run, instead of inside one engineer's inbox.

This is the same pattern we have seen before in data engineering. We stopped hand-managing file compaction when Delta Lake took it over. We stopped hand-rolling streaming state when Structured Streaming took it over. Managed ingestion is the next layer to move down into the platform.

If you want to compare the options side by side, we wrote about when to use Connect, Auto Loader, and COPY INTO in Lakeflow Connect vs Auto Loader vs COPY INTO.

Governed from the first row

The detail that matters most is where the data lands.

A hand-built connector usually writes into a staging area first, and governance arrives later, if it arrives at all. Lakeflow Connect writes into Unity Catalog from the start. Ownership, permissions, lineage, and audit history apply to the very first row.

That ordering sounds small. It is not. Governance that arrives late is governance that gets skipped under deadline pressure. Governance that is present from the first row is just how the table works.

Try the idea in Databricks Free Edition

Setting up a managed connector to an external database needs more than the Free Edition sandbox, and some connector options require a paid Databricks workspace. The concept is explained here for understanding.

What you can do in Free Edition is observe the pattern a managed connector automates. Create a small source table and an ingestion table, then apply a day of changes the same way change data capture would.

CREATE OR REPLACE TABLE workspace.default.source_customers (
  customer_id INT,
  name STRING,
  city STRING,
  updated_at TIMESTAMP
);

CREATE OR REPLACE TABLE workspace.default.bronze_customers (
  customer_id INT,
  name STRING,
  city STRING,
  updated_at TIMESTAMP
);

-- Day 1: the first full load
INSERT INTO workspace.default.bronze_customers
SELECT * FROM workspace.default.source_customers;

On day 2, the source has new and changed rows. Instead of reloading everything, apply only the changes.

MERGE INTO workspace.default.bronze_customers AS target
USING workspace.default.source_customers AS source
ON target.customer_id = source.customer_id
WHEN MATCHED AND source.updated_at > target.updated_at THEN
  UPDATE SET *
WHEN NOT MATCHED THEN
  INSERT *;

Now run DESCRIBE HISTORY workspace.default.bronze_customers and look at what happened. You can see each load as a version, how many rows were written, and when.

A managed connector does this tracking, merging, and recording for you, on a schedule, against a real source. The lab shows you the shape of the work it removes. The incremental processing lesson in the book goes deeper on why this pattern scales.

When to use it, and when not to

Use Lakeflow Connect when the source is a supported database or SaaS application and the destination is the lakehouse. That is the case it was built for, and it removes the most maintenance.

Keep Auto Loader for files landing in cloud storage. Keep a small custom job for unusual APIs or sources with no connector yet. Managed ingestion does not replace judgment. It replaces repetition.

One honest caution: a managed connector makes ingestion easy to start, which means it is also easy to start without thinking. Decide the target schema, the owner, and the freshness expectation before you connect anything. A connector will happily deliver data nobody owns.

What this means for data engineers

The skill that fades here is connector babysitting. The skills that grow are the ones above the ingestion line: modeling, freshness, quality, and governance.

That is a good trade. Knowing how change data capture works, why merges matter, and how a table's history tells a story is worth more every year. Those are the ideas that transfer no matter which platform absorbs the plumbing next.

If you are preparing for the Databricks Data Engineer Associate exam, ingestion patterns are a core domain. Our free practice exam is a calm way to check where you stand.

Key takeaways

Lakeflow Connect moves ingestion from custom code into the platform, with managed connectors that land data directly in Unity Catalog governed Delta tables. The free tier makes it something you can try, not just read about. The pattern underneath, tracking and applying changes instead of reloading everything, is worth learning by hand once, even if you never run it by hand again.

Continue learning