How AI Is Merging Data Engineering and Analytics Into One Role

Why the analytics engineer is the role in the middle, and the skills to build now

For years, most data teams had a clean split.

Data engineers built the pipes. They moved raw data into the lakehouse, tuned Spark jobs, and kept clusters healthy.

Data analysts sat at the other end. They wrote SQL, built dashboards, and answered questions from the business.

That split is fading. AI is the main reason, and the role growing in the middle has a name: the analytics engineer.

The Old Split Had a Hidden Cost

The handoff between the two roles was where things went wrong.

An engineer loaded a table but did not know what "active customer" meant to the sales team. An analyst wrote a dashboard query on a raw table because the clean one did not exist yet.

The result was familiar. Two dashboards showed two different revenue numbers. Nobody was wrong exactly. They just used different definitions.

Every fix needed a ticket, a meeting, and a wait.

What AI Changed

AI assistants now write a lot of the routine code on both sides.

A standard ingestion notebook takes minutes, not a day. A first draft of a dashboard query appears from one plain sentence. Tools like Databricks Genie let business users ask questions in plain English.

So the mechanical work is shrinking. But one thing did not get easier: deciding what the data means.

An AI tool can write SUM(amount). It cannot decide whether refunds count, which time zone defines a day, or which table is the trusted one. If nobody decides, the AI simply produces wrong numbers faster.

That gap is exactly where the analytics engineer works.

What an Analytics Engineer Actually Does

Think of the analytics engineer as the person who turns raw data into trusted answers.

They define the metrics. Words like revenue, churn, and active customer get one written definition. Humans and AI tools then use the same formula.

They build clean models. They shape Bronze and Silver tables into tested Gold tables that are easy to query.

They write data contracts. They agree on column names, types, and rules with the teams that produce the data, so a renamed column does not quietly break reports.

They prepare data for AI. They add table and column descriptions so tools like Genie understand what each field means.

They still write SQL every day. They also use Git, tests, and code review like software engineers.

Before and After

flowchart LR
  subgraph Before
    A[Raw data] --> B[Data engineer]
    B --> C[Undocumented tables]
    C --> D[Data analyst]
    D --> E[Dashboards]
  end
  subgraph Now
    F[Bronze and Silver] --> G[Analytics engineer]
    G --> H[Gold tables with definitions and tests]
    H --> I[Dashboards]
    H --> J[Genie and AI agents]
  end

The key difference is in the middle. Gold tables now carry meaning, not just data. Both people and AI read from the same trusted layer.

Try It in Databricks Free Edition

Here is a small example of analytics engineering work. You build a Gold table with a clear definition, a quality rule, and descriptions an AI tool can read.

Open a SQL editor or notebook in Free Edition and run:

CREATE OR REPLACE TABLE workspace.default.orders_silver AS
SELECT * FROM VALUES
  (1, 'C1', DATE'2026-10-01', 120.00, 'completed'),
  (2, 'C2', DATE'2026-10-01',  80.00, 'refunded'),
  (3, 'C1', DATE'2026-10-02',  45.50, 'completed'),
  (4, 'C3', DATE'2026-10-02', 200.00, 'completed')
AS orders(order_id, customer_id, order_date, amount, status);

Now build the Gold table. Notice the definition is written into the table itself.

CREATE OR REPLACE TABLE workspace.default.daily_net_revenue (
  order_date DATE COMMENT 'Calendar day of the order in UTC',
  net_revenue DECIMAL(12,2) COMMENT 'Sum of completed order amounts. Refunded orders are excluded.',
  order_count BIGINT COMMENT 'Number of completed orders'
)
COMMENT 'Trusted daily revenue. Use this table for revenue questions, not orders_silver.';

INSERT INTO workspace.default.daily_net_revenue
SELECT
  order_date,
  SUM(amount) AS net_revenue,
  COUNT(*) AS order_count
FROM workspace.default.orders_silver
WHERE status = 'completed'
GROUP BY order_date;

Add a simple rule so bad data cannot slip in:

ALTER TABLE workspace.default.daily_net_revenue
ADD CONSTRAINT revenue_not_negative CHECK (net_revenue >= 0);

Finally, check the result:

SELECT order_date, net_revenue, order_count
FROM workspace.default.daily_net_revenue
ORDER BY order_date;

You should see 120.00 for October 1 and 245.50 for October 2. The refund was excluded because the definition said so.

Open the table in Catalog Explorer and read the comments. This is the context a person, or an AI tool, needs to answer "what was revenue yesterday?" correctly.

Four Skills to Build Now

1. Data modeling. Learn fact and dimension tables, and the Bronze, Silver, Gold pattern. Clean models are the base of everything else.

2. Engineering habits. Put SQL in Git. Review changes. Add tests and constraints. Treat a metric like code, because it is.

3. Metadata and governance. Use Unity Catalog comments, tags, and permissions. Good descriptions are now part of the product.

4. Checking AI answers. When Genie or an agent answers a question, compare it with a known SQL query. Learn to spot when the AI used the wrong table.

Is This Bad News for Engineers or Analysts?

No. It is a shift in where the value sits.

Engineers who learn the business meaning of data become far more useful. Analysts who learn modeling, Git, and testing move closer to the system that produces the numbers.

AI removes some typing. It does not remove the need for someone who understands both the data and the business. That person matters more now, not less.

Where to Go Next

If you are building these skills, start with the basics and practice them by hand.

The title on your badge may change. The work underneath stays the same: make data trustworthy, and make its meaning clear.