Databricks Variant is now generally available. Here is how to use it.
A native SQL type for semi-structured data that finally solves the flexibility versus performance tradeoff.
Every data engineer who has worked with JSON knows the tradeoff. You either parse it into fixed columns at ingestion time for fast queries, or you store the whole payload as a string and keep your flexibility.
The first option is rigid. When the upstream API changes a field name or adds a nested object, your pipeline breaks. You scramble to update schemas, backfill data, and coordinate with teams you have never met.
The second option is flexible but slow. Storing JSON as a string means parsing it at query time. Filters, joins, and aggregations all suffer. A query that should take seconds takes minutes.
Neither option is good. Both are common.
What Variant actually is
On August 3, 2026, Databricks announced that the Variant data type is now Generally Available. Variant is a native SQL type for semi-structured data, available in Databricks Runtime 15.3 and above.
It works simply. You create a column with the VARIANT type. You store your entire JSON, XML, or CSV payload in that single column. No schema definition required upfront.
When you need a specific field, you extract it using colon syntax. For example, `variant_column:field_name pulls a single field from the payload. You can nest as deep as you need: variant_column:user.address.city`.
This means you can land data first and understand it later. You do not need to know what fields matter before the data arrives. You just store it, and your downstream queries pull what they need.
Over 5,000 teams are already writing Variant data in Databricks. They use it most often for JSON payloads from APIs, streaming events from sources like Kinesis or Event Hub, and schemaless data from databases like PostgreSQL and MongoDB.
What Variant Shredding does
Flexibility alone is not the breakthrough. The breakthrough is that Variant does not sacrifice query speed.
This is where Variant Shredding comes in. Shredding is a performance optimization that runs automatically through Predictive Optimization. It looks at your query patterns, identifies the fields you query most often, and extracts those fields into hidden columns inside the underlying Parquet files.
You do not configure this. You do not maintain it. Databricks learns from your workload and handles it for you.
The numbers are significant. Shredded Variant reads are nearly 4 times faster than unshredded Variant. Compared to storing JSON as a string, shredding delivers up to 30 times faster reads.
Databricks reports over 500 million Variant queries per month across 160 terabytes of Variant data. This is not an experiment. It is production scale.
How to use it in practice
Getting started is straightforward. Here is how to create a table with a Variant column.
CREATE TABLE events_raw (
event_id STRING,
payload VARIANT
);To insert data, use `PARSE_JSON` to convert a JSON string into the VARIANT type.
INSERT INTO events_raw (event_id, payload)
SELECT event_id, PARSE_JSON(json_string)
FROM source_data;You can also create a table from existing data using a CTAS statement.
CREATE TABLE events_clean AS
SELECT
json_string:event_id AS event_id,
PARSE_JSON(json_string) AS payload
FROM raw_source;For streaming or batch ingestion, Auto Loader handles Variant natively. You can land JSON files from cloud storage directly into a Variant column without writing parsing logic.
from pyspark.sql.functions import from_json, col
# Auto Loader with Variant schema evolution
raw_df = (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", "/checkpoint/events")
.load("/data/events/")
)
# Write to a Delta table with a Variant column
raw_df.select(
col("event_id"),
col("raw_payload").cast("variant").alias("payload")
).writeStream
.format("delta")
.option("checkpointLocation", "/checkpoint/events_write")
.toTable("events_raw")Once the data is in the table, you query it like any other column.
SELECT
payload:event_type AS event_type,
payload:user_id AS user_id,
payload:metadata:source AS source
FROM events_raw
WHERE payload:event_type = 'purchase'What to know before you adopt it
Variant is useful, but it has limits. Being honest about them helps you use it well.
Variant columns cannot be used as clustering keys, partition keys, or Z-order keys. If you need to cluster or partition by a field, extract it into its own regular column first.
You also cannot directly compare, group by, order by, or perform set operations on Variant columns. You extract the field you need using colon syntax, then use that extracted value in your query.
Databricks recommends extracting frequently queried fields into non-Variant columns to accelerate queries and optimize storage. Variant Shredding automates much of this, but for your most critical fields, explicit columns give you the most control.
The Variant type is available in Free Edition with Databricks Runtime 15.3 and above. You can try every example in this article without a paid workspace.
How a data engineer should think about this
Variant changes the order of operations. Instead of designing your schema before the data arrives, you can land the data first and understand it later. This is a real shift in how pipelines get built.
Use Variant for sources where the schema evolves. APIs that add fields without warning. Streaming sources where the payload shape changes over time. Schemaless databases like MongoDB where every document can be different.
For stable, well-understood data, keep using typed columns. They are faster, simpler, and easier to govern. Variant is not a replacement for good schema design. It is a tool for when the schema is not yours to control.
The practical approach is hybrid. Land everything as Variant. Extract the fields you query often into regular columns. Let Predictive Optimization handle the rest through Shredding. This gives you flexibility at ingestion and speed at query time.
Where to learn the skills behind this
If you want to understand the foundations that make Variant work, the Data Sources chapter covers how Auto Loader and file ingestion work. The Spark SQL chapter explains the SQL syntax for extracting and querying nested data. The Delta Lake chapter covers how Delta tables store and optimize data. The Streaming chapter shows how to build pipelines that handle evolving data.
If you are working toward a Databricks certification, understanding Variant and semi-structured data handling is a practical skill that shows up in real pipelines. And if you want to go deeper into all of these topics in one place, the book covers them chapter by chapter with hands-on examples.