The data stack is being redesigned for AI agents
What changes when a human is no longer always in the loop.
Most data platforms were built around one quiet assumption.
A human is always in the loop.
A human writes the pipeline. A human opens the dashboard. A human investigates the failure. A human decides what happens next.
That assumption is starting to break.
AI agents are beginning to query data, inspect pipelines, call tools, make recommendations, and trigger actions on their own. Sometimes they are right. Sometimes they are confidently wrong. Either way, they are already inside the stack.
And that means the data platform itself has to change.
The old flow and the new flow
The traditional stack moved data from source systems into pipelines, then into a warehouse or lakehouse, then into dashboards. A human read the output and decided what it meant.
The new flow starts the same way. Data still arrives from applications, databases, files, and events.
But before an agent can act on it, that data needs meaning, permissions, context, and clear boundaries.
An agent needs to know not just what a field contains, but what it means.
If you prefer watching over reading, this short talk covers the same shift well:
https://youtu.be/wb2lk-YibAQ
A one word example
Take something as simple as customer status.
The table says active.
Active according to whom? Thirty days? Ninety days? One year?
Which definition is approved? Which team owns it? What exceptions apply?
A human analyst learns this by asking someone in a hallway. An agent has no hallway. If the definition is not written down somewhere it can read, the agent will invent one.
This is why raw data is not enough in the agentic era. Agents need governed context: business definitions, metadata, lineage, policies, ownership, trusted documents, memory, and workflow state.
If you want the deeper version of this argument, The Context Advantage is the whole book on it. It is the parent thinking behind a lot of what we teach here.
Tool access is the second shift
Reading data is the easy part.
An agent may also need to query a warehouse, inspect a failed pipeline, open a ticket, trigger a job, or update a business system.
Every one of those tools needs a boundary.
The agent needs an identity. It needs permissions. It needs a list of approved actions. And it needs to know when to stop and ask a human.
This is where the agent harness matters. The model provides reasoning. The harness decides what the agent is actually allowed to do.
In Databricks terms, this is where Unity Catalog stops being paperwork and starts being the safety system. Identity, grants, and lineage are what turn an agent from a guess machine into something you can audit. Unity AI Gateway extends the same idea to model and tool calls.
Observability has to answer harder questions
Traditional monitoring asks one thing. Did the job fail?
Agentic observability has to ask much more.
What data did the agent use? Which tool did it call? Why did it choose that action? How much did the decision cost? Was the result correct? Can we reproduce what happened three weeks later?
Observability is no longer only about infrastructure. It is about context, reasoning, tools, and outcomes.
The habits still start in the same place. Our lesson on debugging and monitoring is about learning to ask what actually happened, and data quality is about proving it before something downstream believes it.
Cost behaves differently now
A dashboard is opened a few times a day.
An agent may run continuously. It queries a warehouse, retrieves documents, calls a model, invokes tools, retries failures, and repeats that loop thousands of times.
That is a very different cost shape. Small per call. Large per month.
We wrote about the estimation side of this in Databricks Lakemeter. The engineering side is the same discipline you learn in partitioning and performance and incremental processing. Reading less data is still the cheapest optimisation anyone has ever found.
The question worth asking out loud in planning meetings:
What business value are we creating for every dollar spent?
Agentic does not mean autonomous
This part gets lost in the excitement.
A strong system knows where automation should stop.
A low risk, reversible fix can happen automatically. Retry a failed task. Backfill a partition. Refresh a stale table.
A pricing change, a financial transaction, or a production schema update should still pass through a human.
The goal is controlled autonomy. Not unlimited autonomy.
Feedback closes the loop
A dashboard shows what happened. An agentic system should also learn what happened next.
Did the recommendation improve the outcome? Did the user reject it? Did the fix solve the issue, or create a new one?
That feedback becomes context for the next decision. Observe. Reason. Act. Measure. Improve.
What this means for data engineers
Data engineers will keep building pipelines and models. That work is not going away.
But the job is expanding. You will increasingly build the foundation that agents depend on.
Trusted data. Business definitions. Permissions. Tool interfaces. Evaluation datasets. Lineage. Observability. Cost controls. Human approval paths.
Read that list again. Almost none of it is new technology. It is old discipline, finally taken seriously, because something without judgement is now reading your tables.
Where to practice this
You do not need an agent platform to start. You need the foundation underneath it, and you can build that in Databricks Free Edition today.
- Delta Lake for tables an agent can trust and time travel back through
- Medallion architecture so meaning improves as data moves
- Unity Catalog for identity, grants, and lineage
- Data quality for expectations an agent cannot argue with
- Jobs and pipelines for actions that are repeatable and observable
- Genie spaces to feel what happens when a language interface meets a poorly documented table
If you want the guided order, the learning paths sequence these for you.
The real shift
The traditional data stack helped humans understand data.
The agentic data stack helps AI systems understand context, take governed actions, and learn from outcomes.
The biggest change is not that AI is entering the data stack.
It is that the data stack is being redesigned for AI.
The teams that understand this early will not be the ones with the fanciest models. They will be the ones whose definitions are written down, whose permissions are real, and whose actions can be explained afterwards.
That is unglamorous work. It is also the work.
Keep learning, keep building, keep growing.