How seven UC Berkeley researchers built the platform that is reshaping data engineering careers
If you work with data today, you have almost certainly encountered Databricks. It powers pipelines at thousands of companies. It shapes how teams think about data lakes, warehouses, and everything in between.
But Databricks did not appear overnight. It grew from a university research project into one of the most valuable private technology companies in the world. Understanding that journey helps you understand why the platform works the way it does, and why learning it now opens real career doors.
This is the story of Databricks, from its origins in a Berkeley lab to its position at the center of the Data + AI revolution.
In the late 2000s, big data was exploding. Companies like Google, Yahoo, and Facebook were generating data at a scale no one had dealt with before. The standard tool for processing this data was Apache Hadoop and its core engine, MapReduce.
MapReduce worked. But it was slow.
Every MapReduce job wrote intermediate results to disk. If your pipeline had five steps, each step read from disk and wrote back to disk. For iterative workloads like machine learning, this was painful. A simple algorithm might take hours because of all the disk I/O.
The bottleneck was not compute power. It was the constant back-and-forth with disk storage.
At UC Berkeley''s AMPLab (Algorithms, Machines, and People Lab), a group of researchers saw this problem clearly. Among them was Matei Zaharia, a PhD student from Romania who had spent time at Facebook working with Hadoop.
Zaharia asked a simple question: what if we kept data in memory between steps instead of writing to disk every time?
That question led to a research paper in 2010 describing a new system called Spark. The paper introduced Resilient Distributed Datasets (RDDs), a way to hold data in memory across a cluster while still recovering from failures.
The results were dramatic. Spark ran certain workloads 10 to 100 times faster than MapReduce.
# What used to take multiple MapReduce jobs
# could now be expressed in a few lines of PySpark
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("SimpleAnalysis").getOrCreate()
df = spark.read.csv("/data/sales.csv", header=True, inferSchema=True)
result = (
df.filter(df.amount > 100)
.groupBy("region")
.sum("amount")
)
result.show()This simple example hides something important. Behind those few lines, Spark is distributing the work across multiple machines, keeping intermediate results in memory, and optimizing the execution plan automatically.
For a deeper understanding of how Spark processes data, the DataFrames chapter walks through the execution model step by step.
By 2013, Spark had gained serious traction in the open-source community. It was faster than Hadoop MapReduce, easier to program, and supported multiple languages including Python, Scala, Java, and SQL.
The team behind Spark decided to build a company around it. Databricks was founded in 2013 by seven co-founders, all connected to UC Berkeley''s AMPLab:
| Co-founder | Role | Background |
|---|---|---|
| Ali Ghodsi | CEO | UC Berkeley researcher |
| Matei Zaharia | CTO (original) | Creator of Apache Spark |
| Ion Stoica | Executive Chairman | UC Berkeley professor, co-founder of Conviva |
| Scott Shenker | Board member | UC Berkeley professor, networking pioneer |
| Patrick Wendell | VP Engineering | Spark core committer |
| Reynold Xin | Chief Architect | Spark contributor, Shark SQL project |
| Andy Konwinski | VP Product | AMPLab researcher |
Their vision was clear: make Spark accessible to every data team, not just those with the resources to manage complex clusters.
The company raised a $14 million Series A led by Andreessen Horowitz. This was just the beginning.
| Year | Round | Amount | Valuation |
|---|---|---|---|
| 2013 | Series A | $14M | Not disclosed |
| 2014 | Series B | $33M | ~$280M |
| 2016 | Series C | $60M | ~$600M |
| 2017 | Series D | $140M | ~$1.9B |
| 2019 | Series E | $250M | ~$2.75B |
| 2019 | Series F | $400M | ~$6.2B |
| 2021 | Series G | $1B | ~$28B |
| 2021 | Series H | $1.6B | ~$38B |
| 2023 | Series I | $500M | ~$43B |
| 2024 | Series J | $10B (closed at $15.3B in early 2025) | ~$62B |
| 2025 | Series J extension | $10B (total ~$15.3B) | ~$62B |
| 2026 | Equity + Debt (February) | $5B equity + $2B debt | ~$134B |
That trajectory tells a story. The jumps in valuation correspond directly to technical milestones that changed what the platform could do. From $14 million to $134 billion in just over a decade.
The first version of the Databricks cloud platform launched on AWS in 2014. It did something that was genuinely new at the time: it let teams run Spark without managing servers.
Before Databricks, running Spark meant:
Databricks abstracted all of this away. You opened a notebook, wrote your code, and the platform handled the rest.
This was not just convenience. It changed who could use Spark. Data analysts who had never managed a server could now run distributed computations. Data scientists could train models on large datasets without waiting for infrastructure tickets.
The collaborative notebook became the core interface. Inspired by Jupyter notebooks but built for teams, it let multiple people work on the same analysis, share results, and schedule jobs.
The notebook was not just a coding tool. It was a collaboration layer that made data work feel more like a shared conversation than a solo engineering task.
For an introduction to how the Databricks workspace is organized, including notebooks, clusters, and file systems, see the Workspace Essentials chapter.
By 2017, the data industry had a growing problem. Companies had been pouring data into data lakes for years, storage systems like Amazon S3 or Azure Blob Storage that held raw files in formats like Parquet, CSV, and JSON.
Data lakes were cheap and flexible. But they had serious reliability problems:
In 2017, Databricks began building Delta Lake to solve these problems. Released as open source in 2019, Delta Lake added a transaction log on top of Parquet files, bringing database-like reliability to data lakes.
# Writing to a Delta table with ACID guarantees
df.write.format("delta").mode("overwrite").save("/data/sales_delta")
# Reading with time travel
spark.read.format("delta").option("timestampAsOf", "2024-01-15").load("/data/sales_delta").show()-- SQL equivalent: query historical data
SELECT * FROM sales_delta TIMESTAMP AS OF '2024-01-15'Delta Lake was more than a storage format. It was the foundation for a new architectural idea.
To understand Delta Lake deeply, including transaction logs, time travel, and optimization, explore the Delta Lake chapter. For how schemas evolve safely over time, see the Schema Evolution chapter.
With Delta Lake providing reliability, Databricks introduced a concept that would define the next era of data platforms: the Lakehouse.
The idea was simple but powerful. For years, companies had been maintaining two separate systems:
This meant data was copied between systems, creating duplication, inconsistency, and extra cost. The lakehouse proposed a single system that combined the best of both.
graph LR
A[Raw Data] --> B[Data Lake]
A --> C[Data Warehouse]
B --> D[Data Science]
C --> E[BI and Analytics]graph LR
F[Raw Data] --> G[Delta Lake]
G --> H[Bronze Layer - Raw]
H --> I[Silver Layer - Cleaned]
I --> J[Gold Layer - Business Ready]
J --> K[BI and Analytics]
J --> L[Data Science and ML]
I --> LThe Medallion Architecture became the standard pattern for organizing data in a lakehouse. Raw data lands in the Bronze layer, gets cleaned in Silver, and is aggregated for business use in Gold.
This is not just theory. It is how most production Databricks environments are organized today.
For a hands-on walkthrough of building a Medallion pipeline, see the Medallion Architecture chapter.
While Delta Lake solved the storage problem, another innovation was happening in how Spark processed data in real time.
Traditional batch processing runs on a schedule. You collect data, wait, then process it all at once. But many use cases need faster results: fraud detection, live dashboards, IoT monitoring.
Databricks invested heavily in Structured Streaming, which treated streaming data as an infinite table that grows over time. This meant you could write streaming code using the same DataFrame API you already knew.
# Read a stream of events
stream_df = (
spark.readStream
.format("delta")
.load("/data/events")
)
# Process and write results
(
stream_df
.groupBy("event_type")
.count()
.writeStream
.format("delta")
.outputMode("complete")
.option("checkpointLocation", "/checkpoints/event_counts")
.start("/data/event_counts")
)The elegance here is that the same code patterns work for both batch and streaming. A data engineer who knows how to write DataFrame transformations already knows 80% of what they need for streaming.
To explore streaming concepts including triggers, watermarks, and checkpointing, see the Streaming chapter.
As companies adopted the lakehouse pattern and stored more data in Delta Lake, a new challenge emerged: governance. Who can access what data? Where did this data come from? Is it compliant with regulations?
In 2022, Databricks launched Unity Catalog, a unified governance layer for all data assets. It provided:
-- Grant read access to a specific table
GRANT SELECT ON TABLE sales.gold.revenue_summary TO data_analysts;Unity Catalog was significant because it acknowledged that managing data is not just about processing it. In production environments, knowing who accessed what data and when is just as important as the data itself.
For an introduction to Unity Catalog concepts including metastores, catalogs, and schemas, see the Unity Catalog chapter.
From its earliest days, Spark included MLlib, a library for distributed machine learning. But Databricks pushed further with MLflow, an open-source platform for managing the entire ML lifecycle.
MLflow tracks experiments, packages models, and manages deployments. It became one of the most widely adopted ML tools in the industry, with over 18 million monthly downloads.
import mlflow
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
# Start tracking an experiment
with mlflow.start_run():
model = RandomForestClassifier(n_estimators=100)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
accuracy = accuracy_score(y_test, predictions)
# Log parameters and metrics
mlflow.log_param("n_estimators", 100)
mlflow.log_metric("accuracy", accuracy)
# Save the model
mlflow.sklearn.log_model(model, "model")The combination of Spark for data processing, Delta Lake for storage, and MLflow for model management created a complete platform for both data engineering and data science.
To explore machine learning concepts on Databricks, including feature engineering and model tracking, see the Machine Learning chapter.
Databricks has made several strategic acquisitions that expanded its capabilities:
| Year | Acquisition | Purpose |
|---|---|---|
| 2020 | Redash | Open-source SQL analytics and dashboarding |
| 2021 | 8080 Labs | Data exploration tools (became Bamboolib) |
| 2023 | MosaicML | Large language model training infrastructure |
| 2023 | Okera | Data governance and access control |
| 2024 | Tabular | Founded by Apache Iceberg creators |
The MosaicML acquisition was particularly significant. It signaled Databricks'' move into the AI space, not just as a platform for running AI workloads, but as a company that builds and trains foundation models.
The Tabular acquisition brought the creators of Apache Iceberg into Databricks, strengthening its position in the open table format space alongside Delta Lake. In 2026, Databricks achieved GA for Managed Iceberg and Iceberg v3.
As of mid-2026, Databricks has reached a $5.4 billion annual revenue run rate, growing approximately 65% year over year. The company is valued at approximately $134 billion, making it one of the most valuable private technology companies in the world.
Databricks formally repositioned itself as a Data + AI platform in 2023. This was not just marketing. The product had genuinely expanded beyond its Spark origins.
Today, the Databricks platform includes:
Databricks is no longer just a Spark company. It is a platform that covers the entire data lifecycle, from ingestion to AI model serving.
Here is what makes this story relevant to you personally.
Data engineering is one of the fastest-growing roles in technology. Companies across every industry need people who can build reliable data pipelines, manage data quality, and enable analytics and AI.
Databricks sits at the center of this demand. Based on industry reports as of 2026:
The opportunity is not just in knowing the tool. It is in understanding the systems behind it. Why does Delta Lake use a transaction log? Why does the Medallion Architecture separate data into layers? Why does Spark process data the way it does?
When you understand these fundamentals, you can solve problems that go beyond any single platform.
If you are ready to build your data engineering skills with Databricks, here is a structured path through the BricksNotes chapters:
| Stage | What You Learn | Chapters |
|---|---|---|
| Foundation | Spark basics, DataFrames, SQL | Start Here, DataFrames, Spark SQL |
| Core Skills | Transformations, joins, UDFs | Transformations, Joins and Aggregations, UDFs |
| Data Storage | Delta Lake, schema evolution, file formats | Delta Lake, Schema Evolution, File Formats |
| Architecture | Medallion pattern, partitioning, performance | Medallion Architecture, Partitioning |
| Production | Streaming, data quality, testing, workflows | Streaming, Data Quality, Unit Testing, Workflows |
| Advanced | Unity Catalog, ML, debugging | Unity Catalog, Machine Learning, Debugging |
Each chapter builds on the previous one. You do not need to rush. The goal is understanding, not speed.
Databricks started because a PhD student noticed that MapReduce was writing to disk too often. That single observation, turned into a research paper, turned into Apache Spark, turned into a company valued at $134 billion as of June 2026.
The lesson is not about Databricks specifically. It is about understanding systems deeply enough to see what can be improved. That is what data engineering is really about.
And that is what this book is designed to help you do.
For readers who want to explore Databricks hands-on, BricksNotes covers all 20+ chapters from workspace setup to machine learning, with practice exercises designed for Databricks Free Edition.