A two-week study plan paired with our free PDF guide and hands-on Databricks lab.
There is a quiet truth about learning data engineering: you do not learn it by reading. You learn it by building something, watching it break, and fixing it.
That is why we built the free Know Your Data Pipelines bundle on BricksNotes. A short visual PDF teaches you to recognise the eight patterns every data engineer ships in production. A hands-on Databricks lab, with 26 notebooks and realistic data, teaches your hands what those patterns feel like to build.
This post is the practical path through both. A two-week plan, a notebook a day, and a few notes on how to use the PDF and the lab together.
You can spend a month watching Databricks tutorials and still freeze on the first day of a real project. The reason is not that the videos are bad. It is that the videos teach you syntax (read this DataFrame, call that function), and the job is not syntax. The job is choosing the right pattern: when to MERGE instead of overwrite, when to keep history, when to add a watermark, when to quarantine instead of drop.
Eight patterns cover almost every pipeline you will ever build. Once you have written each one with your own hands, on real data, the next pipeline you face at work is recognisable. You stop guessing.
01. ETL is the foundation. A batch read, a transform, a write. The skill is doing it idempotently with explicit schemas and Liquid Clustering on the columns you actually filter by. Pair this with the DataFrames lesson for the API basics.
02. Streaming is ETL with a continuous trigger. Auto Loader handles incremental file ingestion. foreachBatch plus MERGE keeps writes idempotent. The mental shift is small but real: you now have to think about late events and checkpoints. Read the Streaming lesson before the lab notebook for context.
03. Medallion architecture is the convention that lets you replay yesterday's pipeline tomorrow. Bronze keeps the raw bytes. Silver cleans and conforms. Gold answers business questions. The Medallion lesson explains where each transformation belongs.
04. CDC and SCD patterns apply inserts, updates, and deletes from a source onto your warehouse copy. Type 1 keeps only the current row. Type 2 keeps history with effective dates. Both are just MERGE with different WHEN clauses. See the SCD patterns lesson.
05. Orchestration turns a notebook into a job. Databricks Lakeflow is now GA, providing unified ingestion and orchestration. Lakeflow Jobs gives you parameters, retries, dependencies, and alerts. The lab notebook also writes a pipeline_runs table so you can grep your own logs. Pair with the Workflows lesson.
06. Data quality lives in every pipeline, not as a separate job. Three checks every pipeline needs: key integrity, reference integrity, and freshness. Bad rows go to a _quarantine table with a fail reason. They are never silently dropped. The Data Quality lesson covers expectations as code.
07. Real-time analytics is windowed aggregations with watermarks. Late events get a deadline. State gets cleared. Materialised views serve sub-second dashboards. The lab notebook walks you from a static aggregation to a streaming one with withWatermark.
08. ML feature pipelines are ETL with two extra rules: same code path for training and serving, and point-in-time correctness. The lab shows the manual version of both, so when you reach for Feature Store you understand what it is doing under the hood. Pair with the Machine Learning lesson.
You can do this in less time. You can take longer. The point is rhythm, not speed.
| Day | Pattern | What to do |
|---|---|---|
| Mon | 01 ETL | Read the PDF page on ETL. Run 01b_pipeline.py. Do all four exercises. |
| Tue | 02 Streaming | Read the PDF on streaming. Run 02b_pipeline.py. Do exercises 1 to 3. |
| Wed | 03 Medallion | Read the PDF. Run 03b_pipeline.py. Stretch goal: rebuild Silver from scratch. |
| Thu | 04 CDC | Read the PDF. Run 04b_pipeline.py. Do all four exercises (especially the SCD2). |
| Fri | Review | Re-run the four pipelines end to end. Note where you got stuck. Write one paragraph in your own words about each pattern. |
| Day | Pattern | What to do |
|---|---|---|
| Mon | 05 Orchestration | Wire 01b into a Lakeflow Job. Add a retry. Watch a failure go through. |
| Tue | 06 Data Quality | Run 06b_pipeline.py. Add one new quality check to the orders quarantine. |
| Wed | 07 Real-Time | Run 07b_pipeline.py. Convert one static window to a streaming window. |
| Thu | 08 ML Features | Run 08b_pipeline.py. Build a third feature of your own choosing. |
| Fri | Capstone | Combine 03 (Medallion) + 06 (Quality) + 05 (Orchestration) into a single job that builds Bronze, Silver, Gold with quality gates between layers. |
By Friday of week two you will have built every pipeline shape you are likely to face in your first job, on real data, on a real Databricks workspace.
The trick is to read, then run, then break, then fix. In that order.
This loop is fast on Databricks Free Edition. Clusters spin up in two minutes. Data is small enough to iterate. Mistakes are cheap.
BricksNotes is a small site with a single goal: help data engineers think clearly about the systems they build. The full book covers twenty chapters from the basics through Unity Catalog, Delta Lake internals, machine learning, and production hardening. The PDF and lab in this bundle are free because the eight pipeline patterns inside should be standard knowledge, not gated behind a paywall.
If you finish the lab and want to go deeper, the full book is where to go next. The lessons that pair most directly with this lab:
Both the visual PDF and the hands-on lab live on one page. One email gets you both.
Open the bundle on /labs/know-your-pipelines
If this helps you, the best thanks is to send a colleague the same link.