When AI Agents Reach Trillion-Token Scale, the Architecture Changes

A clinical AI system now reports roughly 1.2 trillion input tokens a year. The deeper lesson is about context, state, control, observability, and cost.

A useful way to understand production AI is to stop looking at the demo and start looking at the traffic.

On September 23, 2026, Databricks published a case study about Concurrence, a company building clinical AI agents. The reported numbers are striking: about 100.8 billion input tokens and 11.2 million model calls every 30 days. That is an annualized rate of roughly 1.2 trillion input tokens.

Those figures come from the companies involved. They are not an independent benchmark. But they still reveal something important.

At this scale, an AI agent is no longer a clever chat window. It is a distributed system that reads sensitive context, maintains state, calls tools, spends money, and can affect a real person.

The central engineering question changes.

It is no longer only, “Which model gives the best answer?”

It becomes, “Can the whole system behave reliably when millions of decisions pass through it?”

A capable model is one part of a reliable agent. The rest is data engineering.

What changed

The Databricks case study describes a clinical AI environment growing quickly across patient and provider workflows.

Concurrence reports that monthly input-token volume grew about five times from its late-2025 baseline to July 2026. Its systems support AI clinicians, nurses, care coordinators, documentation, care-plan summaries, and knowledge retrieval.

The architecture brings several responsibilities together:

This is not a story about one product doing everything.

It is a story about layers working together.

Five layers for reliable AI agents: trusted context, durable state, controlled access, evaluation and tracing, and cost visibility

The five layers behind a reliable agent

A production agent needs at least five kinds of engineering discipline.

1. Trusted context

Imagine a clinical agent preparing a care summary.

It may see a patient statement, a medication list, a recent lab result, and an older note. These sources can disagree. The newest value is not always the most authoritative value.

The model cannot solve that conflict by sounding confident.

The system needs rules for source authority, freshness, lineage, and meaning. It needs to know which facts are observations, which are approved records, and which need human review.

This is the same problem we explored in Enterprise Context: Why AI Needs It Before It Needs Smarter Models. Better reasoning cannot repair missing or unreliable business context.

For a data engineer, trusted context means familiar work:

The AI part may be new. The responsibility is not.

2. Durable state

A conversation and the current state of the world are different things.

The agent may remember that a user asked about a prescription yesterday. That is conversation memory. Whether the prescription is active today is business state.

Important decisions must be checked against an authoritative system before the agent acts.

This is why an operational store matters. It can hold conversation state, workflow status, approvals, retries, and tool results. It gives the application a current record outside the model prompt.

We explain this distinction in AI Agent Memory vs Business State. Memory helps an agent continue a task. State tells it what is actually true.

In the reported architecture, Lakebase provides a managed PostgreSQL layer for operational data. Lakebase is a paid-workspace capability, so Free Edition learners should treat this part as conceptual. The deeper pattern applies anywhere: keep durable state in a database, not only inside a prompt or model response.

3. Controlled access

An agent should not receive broad access simply because it can reason well.

It should have the smallest set of permissions needed for the current task. A summarization agent may need read access. An agent proposing an action may need permission to create a draft, but not permission to approve it.

Control also means applying rules to the traffic itself.

Current Databricks documentation describes Unity Gateway as a control plane for model and MCP requests. It can route requests, apply limits, record usage, and govern access to services registered in Unity Catalog.

Some newer controls need careful labeling. Unity Gateway is generally available, while service-policy guardrails and the unified trace table are documented as Beta capabilities at the time of writing.

That distinction matters. A production design should never treat a preview label as a footnote.

4. Evaluation and tracing

When one request fails, a person may notice.

When one request in ten thousand fails silently, the system may look healthy while causing damage at scale.

This is the danger of a gray failure. The service is available. Most requests succeed. A narrow slice of users, tools, or inputs gets the wrong result.

Agent evaluation should therefore ask more than, “Was the final answer good?”

It should inspect the path:

  1. Which context was retrieved?
  2. Which tools were called?
  3. Were permissions applied correctly?
  4. Did the agent retry?
  5. Did a human override the result?
  6. What outcome followed?

A trace helps reconstruct one run. Aggregated trace data helps find patterns across millions of runs.

That is where data engineering returns to the center. Traces need schemas, retention rules, quality checks, access controls, and useful aggregates. Observability data is still data.

5. Cost visibility

At small scale, an inefficient prompt feels harmless.

At 100.8 billion input tokens per month, small inefficiencies become material.

Suppose an agent sends 2,000 unnecessary tokens on each of 11.2 million monthly calls. That is 22.4 billion avoidable input tokens. The exact financial effect depends on model prices, caching, and routing, but the engineering lesson is simple.

Tiny waste multiplied by agent-scale traffic is not tiny.

Cost visibility should include:

The last item is easy to miss. A cheap call that produces no useful outcome is still waste. A more expensive call that prevents a costly error may be valuable.

Databricks documents a system.ai_gateway.usage table for request counts, token usage, latency, and related service information. Usage tracking is a billed Unity Gateway feature and requires a supported, Unity Catalog-enabled workspace.

The 4Cs at production scale

The architecture becomes easier to reason about through The Context Advantage.

The 4Cs for production AI agents: Context, Control, Cost, and Choice

Context gives the agent the right facts, definitions, relationships, and current state.

Control decides which data, tools, models, and actions are allowed.

Cost makes consumption visible and connects technical activity to useful outcomes.

Choice lets the enterprise change models, tools, and interfaces without rebuilding its business foundation.

These concerns are connected.

More context can improve an answer, but it can also increase token cost and expose unnecessary information. Stronger control can reduce risk, but poorly designed policies can block useful work. More model choice can improve flexibility, but only if evaluation makes the options comparable.

The goal is not to maximize one C.

The goal is to design the balance deliberately.

A practical architecture for agent-scale systems

You do not need trillion-token traffic to apply the lesson.

Start with one workflow and draw five boxes.

Context store: What facts and definitions does the agent need? Which source is authoritative?

State store: What must persist between steps? What must be rechecked before action?

Access layer: Which identity is calling? What can it read, invoke, or change?

Trace and evaluation layer: What evidence will explain each decision and reveal repeated failure?

Cost layer: Which unit will you monitor, and what useful outcome will you compare it with?

Then add a boundary around the system.

Write down what the agent may do automatically, what requires approval, and what it must never do.

That boundary is more valuable than a vague claim that a model is safe.

Practice the observability idea in Free Edition

You cannot reproduce the enterprise architecture in Databricks Free Edition. Unity Gateway, Lakebase, and the production governance controls discussed here require a paid Databricks workspace.

But you can practice the core thinking with a small Delta table.

Create a few fictional agent events:

from pyspark.sql import functions as F

agent_events = [
    ("run-001", "summary-agent", "model-a", 1850, 720, 840, "success"),
    ("run-002", "summary-agent", "model-a", 6200, 710, 1910, "retry"),
    ("run-003", "review-agent", "model-b", 2400, 980, 1260, "human_review"),
    ("run-004", "summary-agent", "model-a", 2100, 0, 930, "tool_error"),
]

columns = [
    "run_id",
    "agent_name",
    "model_name",
    "input_tokens",
    "output_tokens",
    "latency_ms",
    "outcome",
]

events_df = spark.createDataFrame(agent_events, columns)

events_df.write.mode("overwrite").format("delta").saveAsTable(
    "workspace.default.agent_events"
)

Now ask a simple question in SQL:

SELECT
  agent_name,
  COUNT(*) AS request_count,
  SUM(input_tokens + output_tokens) AS total_tokens,
  ROUND(AVG(latency_ms), 0) AS average_latency_ms,
  SUM(CASE WHEN outcome <> 'success' THEN 1 ELSE 0 END) AS non_success_count
FROM workspace.default.agent_events
GROUP BY agent_name
ORDER BY total_tokens DESC;

The numbers are fictional. The habit is real.

Do not collect telemetry only because it is available. Decide which failure, cost, and outcome questions the table must answer. Then shape the event record around those questions.

Who should care

Data engineers should care because agent reliability depends on pipelines, state, quality, and observability.

AI engineers should care because model quality cannot compensate for weak context or invisible tool failures.

Platform teams should care because identity, limits, logs, and model access become shared infrastructure.

Governance teams should care because agents combine data access with action. Policies need to cover both.

Leaders should care because a successful pilot can hide the operating cost and control gaps that appear only at scale.

What to do next

Choose one agent workflow that matters.

Map its trusted context. Separate memory from business state. Give it the minimum permissions it needs. Record every model and tool call. Define two or three failure signals. Track token use beside a real outcome.

Then run a review using real traces, not a demo script.

Ask where the system guessed. Ask which state became stale. Ask which retries were hidden. Ask which expensive calls changed no outcome.

If the team cannot answer those questions, it is not ready to scale the agent yet.

The BricksNotes view

The most important number in the Concurrence story is not one trillion.

It is the number of engineering responsibilities hiding inside each call.

A model receives context. A database holds state. A gateway controls access. An evaluation system checks behavior. A telemetry layer measures cost and failure. People define the boundaries.

At small scale, those parts can look optional because humans quietly fill the gaps.

At large scale, the gaps become the system.

The path from chatbot to production agent is not mainly a model upgrade. It is an architecture upgrade.

That is why data professionals matter in the agentic era. Reliable AI depends on the work they already understand: trustworthy data, explicit state, controlled change, observable systems, and careful trade-offs.

Continue learning

Read Genie One MCP: One Business Language Across Every AI Assistant to see how different assistants can reach shared governed context.

Review AI Agent Memory vs Business State before designing long-running workflows.

Use the Debugging and Monitoring lesson to strengthen the observability foundations behind this article.

Study Unity Catalog for the core governance ideas. Advanced AI governance capabilities require a paid workspace, but the access-control thinking is useful everywhere.

Sources