The Context Pipeline: Moving Meaning, Not Just Data

Data pipelines move information. AI agents also need meaning: approved definitions, policies, ownership, and current decisions. Here is the pipeline that delivers it, with a lab you can run in Databricks Free Edition.

The Context Pipeline: Moving Meaning, Not Just Data

For years, data engineering asked one question: how do we move data reliably from source to destination?

That question gave us ingestion frameworks, Spark, Delta Lake, and the medallion architecture. It was the right question, and our industry got very good at answering it.

AI agents add a second question. Moving data is no longer enough. We also need to move meaning.

https://youtu.be/msKesD3SjYY

The question that changed

A data pipeline has a clear finish line. The right rows arrive at the right place, in the right shape, on time. When that happens, the pipeline is done, because a human takes it from there.

The human is the missing layer we never talk about. The analyst reading a dashboard carries the meaning in their head. They know which definition of revenue the board uses. They know which rows they are allowed to see. They know whether the number is fresh enough to repeat in a meeting.

An AI agent carries none of that. When the data arrives, the agent's work has not finished. It has not even started. The agent still needs to know what the data means, who owns it, how current it is, and what it is allowed to do with the answer.

That is the new problem. And it is a pipeline problem, which is good news, because pipelines are what data engineers know how to build.

Why a dashboard can answer and an agent cannot

Here is the simplest version of the problem.

Ask a dashboard for last quarter's revenue and it shows a number. A human looks at the number and silently supplies everything around it. Which definition. Which currency. Which regions. Whether refunds are included. Whether the quarter has closed.

Ask an AI agent the same question and it will also produce a number. But nothing around the number exists unless someone engineered it. The agent picks a definition without telling you there were three. It does not know the quarter has not closed. It does not know the answer is draft.

The dashboard works because the human is the context layer. The agent fails because nobody built it one. We wrote about this pattern in Data starts AI, but context ships it, and this article is about the system that fixes it.

Where enterprise context actually lives

Context sounds abstract until you go looking for it. Then you find it everywhere, in places no machine can read.

Definitions live in someone's head, or on a wiki page last edited two years ago. Policies live in PDFs and email threads. Decisions live in meeting notes and closed tickets. Ownership lives in an org chart and a chat channel. Workflow state, the fact that this quarter's numbers are still draft, lives in a conversation between two people.

Most of this knowledge was never engineered. It accumulated. Every company runs on it, and almost none of it is stored where a system can use it.

If a machine cannot read it, it is documentation. Context is documentation that has been engineered.

What a context pipeline is

A context pipeline takes raw organizational knowledge and delivers it to AI agents in a form they can safely use.

The raw material is everything from the section above: definitions, policies, metadata, decisions, ownership, and workflow state. The finished product is governed, current, machine-readable meaning, delivered to the place where an agent is about to act.

It looks a lot like a data pipeline, and that is on purpose. Everything we learned about reliability, testing, freshness, and ownership applies here too. The difference is what flows through it.

Capture, Normalize, Validate

The first three stages will feel familiar, because they mirror what we already do with data.

Capture means pulling knowledge out of the places it hides. Definitions from the wiki. Decisions from tickets. Ownership from the org chart. The goal is not to move every document. The goal is to extract the small, durable facts and put them where machines can read them. One approved definition per business word. One named owner per important table. One written rule per policy.

Normalize means giving that knowledge one shape. A definition always has the words: what it means, the rule in SQL, the owner, and the review date. A policy always states who it applies to and what it allows. When every artifact has the same shape, an agent can read all of them without special cases.

Validate means checking context the way we check data. Does every metric have an owner. Is any definition past its review date. Do two definitions contradict each other. Context can be wrong exactly the way data can be wrong, and it deserves the same discipline we described in why data engineering decides if you can trust an AI answer.

Govern and deliver the right context

Governance applies to context too, for two reasons.

First, context can be sensitive. A pricing policy, a churn definition, or a list of strategic accounts is not something every agent should read. The same permissions that protect your tables should protect the meaning around them. On Databricks, that is Unity Catalog's job: ownership, permissions, tags, and lineage, applied to context artifacts exactly the way they are applied to data.

Second, governance is what makes context safe to act on. An approved definition is different from a suggested one. The difference is that someone with authority signed it, and the system records who.

Delivery is the last step: putting context where agents actually read. That can be the catalog itself, a Genie space, or tools an agent can call while it works. Some delivery options, such as Genie spaces and agent tool integrations, require a paid Databricks workspace. The concept is explained here for understanding. What matters is the principle: context must arrive at the moment of the question, not live in a document the agent never sees.

Freshness: the danger of stale context

Here is the uncomfortable part. Stale context is worse than no context.

An agent with no context hesitates. It asks, or it guesses visibly. An agent with stale context is confident and wrong at the same time. It applies last quarter's discount policy. It uses a definition the business retired. It routes a question to someone who changed teams in May.

Data engineers already know this pattern from data. A dashboard built on a stale table does not look broken. It looks correct, and that is what makes it dangerous.

The fix is also familiar. Every context artifact gets an owner and a review date. The pipeline gets a refresh loop: re-capture, re-validate, re-publish. Freshness for context does not mean new rows arrived. It means the meaning is still true.

A small example: the pricing agent and the company memory

Picture a pricing agent that answers one question: what discount can we offer this customer?

Without a context pipeline, the agent searches what it can find. It discovers an old email where someone approved 20 percent for a similar deal. It quotes 20 percent, confidently. That policy was retired last quarter. Nobody learns about the mistake until the margin report arrives.

With a context pipeline, the same agent reads from engineered memory. A small governed table holds the current discount policy: maximum 12 percent this quarter, owned by revenue operations, reviewed this month. A second table holds the deal-size rule. The agent answers with the number and the rule it used. When finance changes the policy, they update one row, and every agent moves together.

The model did not get smarter. The memory around it got engineered. This is the same idea we explored in AI agent memory versus business state, seen from the pipeline side.

Data pipelines move information. Context pipelines move meaning.

It is worth stating the distinction plainly, because it changes what you build.

Neither replaces the other. An agent needs both pipelines feeding it, which is why the data stack is being redesigned for AI agents and why context engineering is becoming a real job skill for data engineers.

Try the idea in Databricks Free Edition

You can build the smallest honest version of a context pipeline in Databricks Free Edition. No agents required. The point is to engineer one piece of meaning and watch a system read it.

Create a table of approved metric definitions:

CREATE OR REPLACE TABLE workspace.default.metric_definitions (
  metric_name STRING,
  approved_definition STRING,
  rule_sql STRING,
  owner STRING,
  review_date DATE
);

INSERT INTO workspace.default.metric_definitions VALUES
  ('revenue', 'Booked orders only, before refunds', 'SUM(amount) WHERE status = ''booked''', 'finance-team', DATE '2026-10-01'),
  ('active_customer', 'At least one order in the last 90 days', 'COUNT(DISTINCT customer_id) WHERE days_since_order <= 90', 'growth-team', DATE '2026-09-15');

Now ask the question the way an agent would:

SELECT approved_definition, rule_sql, owner, review_date
FROM workspace.default.metric_definitions
WHERE metric_name = 'revenue';

Finally, publish the definition as a view so anything in the workspace reads the same rule:

CREATE OR REPLACE VIEW workspace.default.v_revenue AS
SELECT SUM(amount) AS revenue
FROM workspace.default.context_lab_orders
WHERE status = 'booked';

That is a context pipeline at the smallest possible scale. You captured a definition, gave it one shape, validated it by naming an owner and a review date, governed it through the catalog, and delivered it where any tool can read it. When the definition changes, you update one row and one view, and every consumer moves together.

Enterprise delivery options, such as serving this context through Genie spaces or agent tool integrations, require a paid Databricks workspace. The concept is explained here for understanding.

Where to go next

This article is the pipeline view of an idea we have been building across BricksNotes. Data starts AI, but context ships it explains why context is the missing layer. Genie Ontology shows how shared meaning becomes trusted action. Genie One MCP shows what delivery looks like when every assistant speaks one business language. And the future of data to decisions is autonomous shows where the road leads once the pipeline is in place.

In the book, the natural chapters are Chapter 20: Data Governance and Security for ownership and permissions, Chapter 13: Data Quality Engineering for validation and freshness, Chapter 20.5: Genie Spaces for delivering context where questions are asked, and Chapter 14: Change Data Capture for keeping everything current.

If this way of thinking resonates, the Context Advantage hub collects these essays grouped by the four Cs, and the companion book The Context Advantage goes deeper on Context, Control, Cost, and Choice.

New to BricksNotes? The first three chapters are free, and the full book teaches the data engineering foundation this article builds on.