Stop copying your lakehouse into a second database
Many teams built their first AI search feature the same way. They copied rows from a Delta table into a separate vector database, then wrote a nightly job to keep the two in sync.
It worked for a week. Then a product was deleted but still showed up in answers. A new column broke the export. And someone noticed the vector store had no idea who was allowed to see which document.
Databricks Vector Search exists to remove that second copy. This article explains what it does, how it stays in sync, and how you can practice the core idea in Databricks Free Edition.
A vector database stores embeddings. An embedding is a list of numbers that captures the meaning of a piece of text. Similar meanings produce similar numbers.
When your embeddings live outside the lakehouse, you now own two systems. One holds the truth. The other holds a copy that is always a little behind.
That gap causes three real problems. Data drifts, because deletes and updates arrive late. Access rules drift, because permissions are defined twice. And pipelines break, because every schema change must be handled in two places.
Vector Search builds an index on top of a Delta table. The index is a Unity Catalog object, so it has an owner, permissions and lineage like any table.
You do not export data. You point the index at a source table and tell it which column holds the text, or which column already holds the embeddings.
flowchart LR
A[Delta table<br/>with CDF on] --> B[Vector Search index<br/>in Unity Catalog]
B --> C[Similarity query]
C --> D[RAG app or AI agent]
E[Unity Catalog permissions] -.-> BThe secret is Change Data Feed. CDF records every insert, update and delete on a Delta table. We covered it in detail in The Delete Your Pipeline Never Saw.
A Delta Sync index reads that change feed. It only processes rows that changed, so it never rescans the whole table. A deleted row in the source becomes a deleted vector in the index.
This is why CDF must be enabled on the source table before you create the index.
ALTER TABLE workspace.default.support_articles
SET TBLPROPERTIES (delta.enableChangeDataFeed = true);Delta Sync index. Databricks watches the source table for you. It can compute embeddings using a model endpoint, or read a column of embeddings you already computed. This is the right default for most teams.
Direct Access index. You write and delete vectors yourself through an API. Use it only when your embeddings come from a system outside Databricks.
Delta Sync also has two modes. Triggered mode updates when you ask it to, which is cheaper. Continuous mode updates within seconds, which costs more because compute stays running.
Here is the shape of the code on a paid workspace. Read it to understand the flow.
This feature requires a paid Databricks workspace with a Vector Search endpoint. The concept is explained here for understanding.
from databricks.vector_search.client import VectorSearchClient
client = VectorSearchClient()
index = client.create_delta_sync_index(
endpoint_name="docs_endpoint",
index_name="workspace.default.support_articles_index",
source_table_name="workspace.default.support_articles",
pipeline_type="TRIGGERED",
primary_key="article_id",
embedding_source_column="body",
embedding_model_endpoint_name="databricks-gte-large-en",
)
results = index.similarity_search(
query_text="How do I reset my password?",
columns=["article_id", "title"],
num_results=3,
)Notice what is missing. There is no export job, no sync script and no separate permission model.
You can learn what a vector index does without one. Similarity search is just math: find the rows whose vectors point in the closest direction to your query.
Run this in a Free Edition notebook. The vectors are tiny and hand-made so you can see the logic.
from pyspark.sql import functions as F
articles_df = spark.createDataFrame(
[
(1, "Reset your password", [0.9, 0.1, 0.0]),
(2, "Update billing details", [0.1, 0.9, 0.1]),
(3, "Recover a locked account", [0.8, 0.2, 0.1]),
],
["article_id", "title", "embedding"],
)
query_vector = F.array(F.lit(0.85), F.lit(0.15), F.lit(0.05))
# Cosine similarity = dot product / (length A * length B)
dot_product = F.aggregate(
F.zip_with("embedding", query_vector, lambda a, b: a * b),
F.lit(0.0),
lambda acc, x: acc + x,
)
length_a = F.sqrt(F.aggregate("embedding", F.lit(0.0), lambda acc, x: acc + x * x))
length_q = F.sqrt(F.aggregate(query_vector, F.lit(0.0), lambda acc, x: acc + x * x))
scored_df = articles_df.withColumn("similarity", dot_product / (length_a * length_q))
display(scored_df.orderBy(F.col("similarity").desc()))The password and account articles rank highest. Billing ranks last. That is exactly what a vector index does, only across millions of rows and much faster.
Enable Change Data Feed before creating a Delta Sync index. Without it, sync cannot work.
Pick a stable primary key. The index uses it to match updates and deletes.
Start with Triggered mode. Move to Continuous only when users truly need answers within seconds.
Keep chunks small and meaningful. One long document as one vector gives vague results.
Grant access on the index, not just the table. Unity Catalog treats them as separate objects.
AI features are only as good as the data behind them. The hard part of retrieval is rarely the model. It is freshness, deletes, permissions and clean chunks.
Those are data engineering problems. If you understand Delta tables, Change Data Feed and Unity Catalog, you already understand most of Vector Search.