Turn a folder of PDFs into queryable Delta tables with a Medallion-shaped document intelligence pipeline.
Every company has a folder full of PDFs nobody wants to open.
Invoices. Contracts. Compliance reports. Vendor agreements. Insurance claims. They sit in some shared drive, growing every quarter, full of information that should be in a database but is trapped in a layout designed for human eyes.
For years, extracting that information meant building fragile OCR pipelines, training custom models, or hiring an army of interns.
This month, that changed. ai_parse_document is now generally available on Databricks. And it is built for data engineers, not ML engineers.
If you can write a SQL query, you can now build a document intelligence pipeline.
Let me show you what that looks like in practice.
ai_parse_document is a SQL function. You point it at a PDF, an image, a Word document, or a PowerPoint file, and it returns structured content.
It handles documents up to 500 pages and 100MB. It preserves layout. It extracts tables. It reads handwriting in many cases. It works on scanned documents and digital PDFs alike.
You do not train a model. You do not fine-tune anything. You do not manage a serving endpoint. You call a function and you get structured data back.
SELECT ai_parse_document(content) AS parsed
FROM main.bronze.raw_invoices
WHERE file_type = 'pdf';That is the entire API surface for the basic case. The function returns a struct with the parsed content, page metadata, detected tables, and extraction confidence scores.
If you have been through the Medallion Architecture lesson, the right place for document parsing should already feel obvious.
Raw documents go into bronze. They are stored as binary files in Unity Catalog volumes, untouched, with metadata about when they arrived and where they came from. Note that Unity Catalog now includes a native FILE type to govern these unstructured assets.
Parsed content goes into silver. The output of ai_parse_document, structured but still close to the original document. Page numbers, raw text, detected tables, layout information.
Extracted business data goes into gold. The fields your business actually cares about. Invoice numbers. Vendor names. Line items. Totals. Due dates.
Each layer has a clear purpose. Each layer is queryable. Each layer is reproducible. If you change how you extract fields next quarter, you do not need to re-fetch the documents. You just rerun the silver-to-gold transformation.
This is exactly the discipline data engineers already apply to event logs, CDC streams, and reference data. Documents are just another source.
Let me walk you through a complete pipeline for processing invoices.
Your invoices arrive in a Unity Catalog volume. You can use Auto Loader to pick them up incrementally as they arrive.
from pyspark.sql import functions as F
raw_invoices_df = (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "binaryFile")
.load("/Volumes/main/bronze/incoming_invoices/")
.withColumn("ingested_at", F.current_timestamp())
.withColumn("source_file", F.col("path"))
)
(
raw_invoices_df.writeStream
.format("delta")
.option("checkpointLocation", "/Volumes/main/bronze/_checkpoints/invoices")
.toTable("main.bronze.raw_invoices")
)Notice we are storing the binary content directly. No parsing yet. Just a faithful record of what arrived and when.
Now we apply ai_parse_document to extract structured content.
CREATE OR REPLACE TABLE main.silver.parsed_invoices AS
SELECT
source_file,
ingested_at,
ai_parse_document(content) AS parsed,
current_timestamp() AS parsed_at
FROM main.bronze.raw_invoices;The parsed column now contains a struct with the document text, detected tables, page metadata, and confidence scores. You can query it like any other nested column.
SELECT
source_file,
parsed.text AS full_text,
parsed.tables AS detected_tables,
parsed.page_count
FROM main.silver.parsed_invoices
LIMIT 10;This is your structured representation of every document in your bronze layer. It is queryable. It is reproducible. It is governed by Unity Catalog.
Parsing gives you structured content. Extraction gives you business meaning. This is where ai_extract comes in.
CREATE OR REPLACE TABLE main.gold.invoice_records AS
SELECT
source_file,
ai_extract(
parsed.text,
array(
'invoice_number',
'vendor_name',
'invoice_date',
'due_date',
'total_amount',
'currency',
'line_items'
)
) AS extracted,
parsed_at
FROM main.silver.parsed_invoices;Now you can query invoices like any other table.
SELECT
extracted.vendor_name,
SUM(CAST(extracted.total_amount AS DECIMAL(18,2))) AS total_spend
FROM main.gold.invoice_records
WHERE extracted.invoice_date >= '2026-01-01'
GROUP BY extracted.vendor_name
ORDER BY total_spend DESC;A folder full of PDFs just became a queryable spend analysis.
You do not want to re-parse every document every night. Parsing costs money, and your bronze layer is going to keep growing.
This is where Incremental Processing earns its keep. Lakeflow Declarative Pipelines (formerly Delta Live Tables) handle this naturally.
import dlt
from pyspark.sql import functions as F
@dlt.table(name="silver_parsed_invoices")
def parse_invoices():
return (
dlt.read_stream("bronze_raw_invoices")
.withColumn("parsed", F.expr("ai_parse_document(content)"))
.withColumn("parsed_at", F.current_timestamp())
)
@dlt.table(name="gold_invoice_records")
def extract_invoice_fields():
return (
dlt.read_stream("silver_parsed_invoices")
.withColumn(
"extracted",
F.expr("""ai_extract(parsed.text, array(
'invoice_number', 'vendor_name', 'invoice_date',
'due_date', 'total_amount', 'currency', 'line_items'
))""")
)
)Only new documents get parsed. Only new parsed records get extracted. Costs stay predictable. The pipeline runs on a schedule and handles failures with checkpoints.
This is the same pattern you would use for any incremental Delta pipeline. The fact that the source is documents instead of events does not change the shape of the solution.
A few things to know before you push this to production.
Cost tracking matters. Each call to ai_parse_document and ai_extract is a billable AI function. You can track spend in your billing dashboard under the AI_FUNCTIONS product. Set up budget alerts before you parse a million-document backlog.
Region availability is uneven. As of this writing, ai_parse_document is available in most major regions, but not all. Check your workspace region before you design around it.
Non-Latin text is improving but imperfect. If your documents contain Mandarin, Arabic, or Cyrillic, accuracy is lower than for Latin scripts. Test on a sample of your real documents before committing.
You cannot bring your own model. ai_parse_document uses the model Databricks provides. You cannot swap it for a custom-trained extraction model. For most use cases, this is fine. For specialized industries with unique document formats, you may still need a custom pipeline.
Confidence scores are advisory. The function returns confidence values, but they are not a substitute for human review on high-stakes documents. For contracts and compliance reports, build a review queue for low-confidence extractions.
Once you have parsed documents flowing into Delta tables, a lot of doors open.
You can feed parsed content into a RAG system, giving your internal chatbots the ability to answer questions grounded in your contracts and policies.
You can build vendor analytics that combine parsed invoices with your existing finance data.
You can automate compliance checks by extracting clauses from contracts and comparing them against a policy library.
You can route documents to the right teams based on extracted fields, replacing manual triage queues.
Each of these is a normal data engineering project. The hard part used to be getting structured data out of the documents. That part is now a SQL function.
If you are new to building pipelines like this, start with the fundamentals.
Read the Delta Lake lesson to understand how Delta tables give you the reliability layer this pipeline depends on. ACID transactions, schema enforcement, and time travel all matter when you are processing documents.
Read the Medallion Architecture lesson to see why separating raw, parsed, and extracted layers is more than aesthetic. It is what makes the pipeline maintainable.
Read the Incremental Processing lesson to understand how Lakeflow Declarative Pipelines keep costs predictable as your document volume grows.
With those three lessons under your belt, building a document intelligence pipeline is no longer a research project. It is a Tuesday afternoon.
That folder full of PDFs has been waiting for you.