The same architecture that powers your analytics pipelines is now defending enterprises against AI-driven attacks.
Security teams have been drowning in data for years.
Not because they lack tools. Because their tools were never designed for the scale of modern telemetry.
Traditional SIEMs charge per byte ingested. So teams delete logs they cannot afford to keep. They sample when they should retain. They make decisions about what data matters before they even know what questions to ask.
Sound familiar?
It should. Data engineers faced the exact same problem a decade ago. And the lakehouse architecture solved it.
Now Databricks is betting that same architecture can fix security. The product is called Lakewatch, an open, agentic SIEM built entirely on the Databricks Data Intelligence Platform.
And if you understand how lakehouses work, you already understand most of how Lakewatch works.
The core problem with legacy SIEMs is architectural. They couple storage and compute. They use proprietary formats. They force you to choose between retention and cost.
Here is what that looks like in practice:
Meanwhile, the threat landscape has accelerated. The mean time to exploit a vulnerability collapsed from 23 days to just 1.6 days. AI-driven attacks operate at machine speed. Attackers use agents. Defenders are still writing manual queries.
This is not a tooling problem. It is an architecture problem.
If you have read our guide on designing cost-efficient pipelines, you will recognize the pattern. The same compute-storage decoupling that saves 80% on analytics workloads is the foundation of modern security infrastructure.
Lakewatch stores all security telemetry on open formats, Delta Lake tables backed by cloud object storage. With the GA of Lakehouse//RT and the Reyden engine, security teams can now achieve millisecond query latency directly on these governed tables.
This means:
This is not a new architectural pattern. It is the same pattern you learn in our chapters on Delta Lake fundamentals and file formats. The difference is the domain. Instead of transaction logs and clickstream data, you are storing firewall logs, endpoint telemetry, and authentication events.
The engineering principles are identical.
Lakewatch normalizes all incoming data to the Open Cybersecurity Schema Framework (OCSF) standard. Think of it as a universal schema for security events. Firewall logs from Palo Alto Networks, identity events from Okta, cloud audit trails from AWS, all normalized into a consistent structure before landing in your lakehouse.
If you have worked through our chapter on schema evolution, you understand why this matters. A consistent schema across sources is what makes downstream joins, aggregations, and ML models possible.
Governance in security is not optional. It is the entire point.
Lakewatch uses Unity Catalog as its governance layer. The same catalog that manages access to your analytics tables now manages access to security telemetry.
This means:
Our chapter on Unity Catalog covers these governance primitives in depth. The concepts are the same whether you are governing customer PII or security alert data.
What makes this powerful is the convergence. Security teams and data teams share the same governance framework. A SOC analyst querying threat data and a data engineer querying pipeline metrics both operate under the same access policies, the same audit trail, the same lineage graph.
As we explored in our article on Databricks Genie, Unity Catalog metadata and the new Genie Ontology layer are what make AI assistants trustworthy. Genie can only answer questions about data it is authorized to access. The same principle applies in security, an AI agent hunting for threats operates within the same governance boundaries as a human analyst.
Lakewatch introduces Genie One security agents for threat hunting and detection authoring.
Instead of writing complex SQL queries to investigate an alert, an analyst can ask:
"Show me all authentication failures from this IP range in the last 72 hours, correlated with any lateral movement indicators."
Genie translates that into optimized queries against the security lakehouse. It understands the OCSF schema. It knows which tables contain authentication events, which contain network flows, and how to join them.
This is natural-language data access applied to security. The same Genie Spaces concept we covered in our Genie deep dive, but pointed at security telemetry instead of business metrics.
The second agent capability is Genie Code for detection authoring. Security engineers describe a detection rule in natural language, and Genie generates the SQL or Python implementation, complete with test cases.
If you have practiced writing Spark SQL queries and understand how transformations work, you already have the foundation to understand what these agents produce. They generate the same PySpark and SQL code you write daily, just applied to security event tables.
This is where it gets interesting for data engineers.
Lakewatch implements Detection-as-Code, security detection rules defined as YAML configurations with SQL or Python logic. These rules are version-controlled, backtested against historical data, and deployed through CI/CD pipelines.
The data flow follows a pattern you already know:
graph LR
A[Raw Logs - Bronze] --> B[Normalized OCSF - Silver]
B --> C[Detection Results - Gold]
C --> D[Incidents and Alerts]
B --> E[ML Feature Tables]
E --> F[Anomaly Scores]
F --> CRaw security telemetry lands in Bronze tables via Lakeflow Connect. Normalization to OCSF happens in Silver using Lakeflow Declarative Pipelines (formerly Delta Live Tables). Detection rules run against Silver to produce Gold-layer alert tables. ML models consume Silver features and feed anomaly scores back into the detection layer.
This is the Medallion Architecture applied to security operations.
The backtesting capability is particularly elegant. When a security engineer writes a new detection rule, they can run it against months or years of historical data to measure its precision and recall before deploying to production. False positive rates are calculated before the rule ever fires a real alert.
This is the same principle behind unit testing in data engineering. Test your logic against known data before deploying to production.
And the deployment pipeline? That is Jobs & Pipelines, orchestrated Runs that validate, test, and deploy detection rules on a schedule.
Beyond rule-based detections, Lakewatch supports custom machine learning models for:
These models use the same MLflow tracking, Feature Store, and Model Serving infrastructure covered in our chapter on machine learning with Databricks.
The feature engineering pipeline for security ML follows familiar patterns. You compute rolling aggregates over streaming event data, failed login counts, data transfer volumes, access pattern entropy, and serve them as features for real-time scoring.
If you have built a churn prediction model or a recommendation engine on the lakehouse, the architecture for a threat detection model is structurally identical.
Security telemetry generates massive volumes of small events. Thousands of log entries per second from firewalls, endpoints, cloud services, and identity providers.
This is the small file problem at scale.
Without optimization, these small files degrade query performance dramatically. An analyst trying to investigate an incident cannot wait minutes for a query to scan thousands of tiny Parquet files.
Lakewatch addresses this with the same tools data engineers use:
Our detailed guide on the small file problem explains exactly why these optimizations matter and how they work under the hood. The same concepts apply whether your Delta tables contain sales transactions or firewall logs.
Here is the insight that matters most:
Your lakehouse skills are security skills.
Every concept Lakewatch relies on is something BricksNotes readers are already learning or have already mastered.
| Lakehouse Concept | Security Application | BricksNotes Chapter |
|---|---|---|
| Delta Lake tables | Security telemetry storage | Delta Lake |
| Medallion Architecture | Bronze/Silver/Gold security pipeline | Medallion Architecture |
| Unity Catalog | Security data governance | Unity Catalog |
| Streaming ingestion | Real-time log collection | Streaming |
| Schema evolution | OCSF normalization | Schema Evolution |
| Partitioning and performance | Query-efficient log retention | Partitioning |
| Jobs & Pipelines | Detection deployment pipelines | Workflows |
| MLflow and Feature Store | Anomaly detection models | Machine Learning |
| Unit testing | Detection rule backtesting | Unit Testing |
| Spark SQL | Threat hunting queries | Spark SQL |
| Data quality checks | Alert validation | Data Quality |
Lakewatch is not asking data engineers to learn security from scratch. It is asking security teams to adopt the patterns data engineers already use.
The organizations adopting Lakewatch, Adobe, Dropbox, and others, are not building separate security infrastructure. They are extending the lakehouse they already have.
Lakewatch represents something larger than a new security product.
It is evidence that the lakehouse architecture is becoming a general-purpose data platform. Analytics, machine learning, and now security, all running on the same tables, the same governance, the same compute infrastructure.
As we discussed in our article on the Agentic Enterprise, Databricks is building toward a future where AI agents operate across every domain of enterprise data. Lakewatch is the security instantiation of that vision.
The data engineers who understand these fundamentals today are not just building pipelines.
They are building the infrastructure that keeps organizations safe.
And the best part? You can start learning every concept Lakewatch relies on right now. Our PySpark cheat sheet and SQL cheat sheet give you quick reference guides. Our quizzes let you test your understanding. And the chapters linked throughout this article walk you through each concept step by step.
The lakehouse is not just for analytics anymore. It is for everything.