Building pipelines is one thing. Keeping them reliable, fast and affordable is another. A field guide to the problems that keep repeating, and the fixes that work.
Building a data pipeline is one thing.
Keeping it reliable, scalable and ready for production is another thing entirely.
After enough years around data teams, you notice something. The problems repeat. Different company, different dataset, different cloud. Same duplicate rows. Same slow join. Same job that ran green and loaded nothing.
The difference between a junior and an experienced data engineer is not that the experienced one avoids these problems. Nobody avoids them. The difference is recognition. The experienced engineer sees the symptom, names the cause, and reaches for the right fix in minutes instead of days.
This guide is built to give you that recognition. It covers 50 problems, grouped into ten families. For each one you get the symptom, why it happens, the fix, and where in the book or on BricksNotes you can practice it.
Save it. You will likely meet most of these in real projects.
How to read this guide: you do not need to read it in order. Jump to the family that is hurting you today. Everything marked as runnable works in Databricks Free Edition. Anything that needs a paid workspace is clearly marked as conceptual.
Before the list, one idea worth holding on to.
Almost every fix in this guide follows the same shape. Make the pipeline safe to rerun. Make the data checked before it spreads. Make the work proportional to what changed. Make failures loud instead of silent.
If you remember those four ideas, you can reason your way to most solutions even when you forget the exact syntax.
PROBLEM -> ROOT CAUSE -> SOLUTION -> PRACTICE
symptom why it the fix where you
you see happens try itSymptom: totals jump after someone reruns yesterday's job.
Cause: the job appends. Running it twice writes the same rows twice.
Fix: write with MERGE on a business key, so a rerun updates instead of inserting. This is called an idempotent write: running it once or five times gives the same result.
MERGE INTO silver.orders AS target
USING staged_orders AS source
ON target.order_id = source.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *Practice: Chapter 14: Change Data Capture and the hands-on First Data Engineering Project.
Symptom: MERGE fails with an error about multiple source rows matching one target row.
Cause: the incoming batch itself contains the same key more than once.
Fix: deduplicate the batch first. Keep the newest row per key with a window function.
from pyspark.sql import functions as F
from pyspark.sql.window import Window
latest_first = Window.partitionBy("order_id").orderBy(F.col("updated_at").desc())
ranked_df = raw_df.withColumn("row_rank", F.row_number().over(latest_first))
deduped_df = ranked_df.filter(F.col("row_rank") == 1).drop("row_rank")Practice: Chapter 4: Data Transformation Techniques.
Symptom: a small correction rewrites months of history and takes an hour.
Cause: the job uses a plain overwrite for everything.
Fix: use replaceWhere to overwrite only the slice you are correcting.
fixed_day_df.write.format("delta") \
.mode("overwrite") \
.option("replaceWhere", "order_date = '2026-10-01'") \
.saveAsTable("silver.orders")Practice: Chapter 8: Delta Lake Architecture.
Symptom: someone ran the wrong script and the table is wrong.
Cause: humans. It always happens eventually.
Fix: Delta keeps history. Look at it with DESCRIBE HISTORY, query an older version, and restore it.
RESTORE TABLE silver.orders TO VERSION AS OF 42Practice: Delta Lake Tutorial.
Symptom: a file dropped twice into a folder appears twice in bronze.
Cause: the ingestion code reads the whole folder each time.
Fix: use Auto Loader or COPY INTO. Both remember which files they already loaded.
Practice: Databricks Auto Loader guide and Chapter 7.6: Ingestion and ETL Pipelines.
Symptom: hourly counts change after the hour has closed, or late events are ignored.
Cause: events arrive after their event time, for example a phone that was offline.
Fix: a watermark. It tells Spark how long to wait for late data before finalising a window.
hourly_counts_df = events_df \
.withWatermark("event_time", "30 minutes") \
.groupBy(F.window("event_time", "1 hour")) \
.count()Practice: Chapter 12: Real-Time Streaming Processing.
Symptom: a long running stream gets slower and eventually runs out of memory.
Cause: stateful operations without a watermark keep every key forever.
Fix: add a watermark so old state can be dropped, and keep stateful keys bounded.
Practice: Chapter 12 and our deep dive on resizing stateful streams without starting over.
Symptom: after a restart, the stream reads from the beginning again.
Cause: no checkpoint, or the checkpoint location changed.
Fix: always set a stable checkpointLocation. It records exactly what was processed. Never share one checkpoint between two streams.
Practice: Chapter 12. In Free Edition, keep checkpoints under a Unity Catalog Volume such as /Volumes/workspace/default/book_data/checkpoints/.
Symptom: a streaming cluster runs all day for a dashboard checked twice.
Cause: treating "streaming" as "always running".
Fix: run the same streaming code with trigger(availableNow=True). It processes everything new, then stops. You keep the incremental logic and lose the idle cost.
Practice: Chapter 7.6: Ingestion and ETL Pipelines.
Symptom: a customer's address reverts to an old value.
Cause: an older change arrived after a newer one and simply won.
Fix: only update when the incoming row is newer.
WHEN MATCHED AND source.updated_at > target.updated_at THEN UPDATE SET *Practice: Chapter 14: Change Data Capture.
Symptom: the job fails because the source added a field.
Cause: Delta enforces the table schema by default. That is a feature, not a bug.
Fix: decide on purpose. Allow additive changes with mergeSchema, and review anything else.
Practice: Chapter 9: Schema Management and Evolution.
Symptom: amounts become strings, or nulls appear everywhere after a cast.
Cause: inferred types changed between files.
Fix: declare the schema explicitly for important sources instead of trusting inference.
Practice: Chapter 9.
Symptom: one broken line in a million row file fails everything.
Cause: strict parsing with no plan for bad records.
Fix: capture bad rows in a rescued data column and send them to a quarantine table. Good rows keep flowing, bad rows are kept for review.
Practice: Chapter 7.6 and Chapter 13: Data Quality Engineering.
Symptom: every new event type needs a code change.
Cause: a fixed struct for semi structured data.
Fix: land the payload as VARIANT in bronze, then extract the fields you trust into typed silver columns.
Practice: our explainer on Databricks Variant and Chapter 4.
Symptom: reports fail on Monday after an upstream rename.
Cause: dashboards read tables directly with no stable contract.
Fix: put a view between gold tables and consumers. Change the table, keep the view's column names stable.
Practice: Chapter 18: Lakehouse Architecture Design.
Symptom: 199 tasks finish in a minute, one takes 40.
Cause: data skew. One key, often a null or a default value, holds most rows.
Fix: keep Adaptive Query Execution (AQE) on. It detects and splits skewed partitions. For extreme cases, filter out the hot key and handle it separately, or salt it.
Practice: Chapter 10: Performance Optimization.
Symptom: joining orders to a 200 row country list takes minutes.
Cause: both sides are shuffled across the network.
Fix: broadcast the small table so every worker has a copy.
from pyspark.sql.functions import broadcast
orders_with_country_df = orders_df.join(broadcast(countries_df), "country_code")Practice: Chapter 6: Data Integration and Analytics.
Symptom: 10,000 orders become 80,000 rows.
Cause: the join key is not unique on one side.
Fix: check uniqueness before joining. Count rows per key on the dimension side. Anything above one needs a decision.
Practice: Chapter 6 and Chapter 16: Pipeline Testing.
Symptom: thousands of tiny tasks, or a handful of huge ones that spill to disk.
Cause: a fixed partition count that does not match the data.
Fix: let AQE coalesce partitions automatically. Tune only after reading the Spark UI.
Practice: Chapter 10.
Symptom: a simple transformation takes ten times longer than expected.
Cause: row by row Python functions move data out of the engine.
Fix: use built in functions first. If you need custom logic, use a pandas UDF that works on batches.
Practice: Chapter 5: Custom Function Development.
Symptom: a table with modest data takes ages to query.
Cause: thousands of tiny files. Opening files costs more than reading them.
Fix: run OPTIMIZE to compact files, and let auto compaction help on frequent writes.
Practice: Chapter 11: Storage and File Optimization, plus our downloadable small file problem lab.
Symptom: a table partitioned by customer id has a million folders.
Cause: partitioning by a high cardinality column.
Fix: for new tables, prefer Liquid Clustering. Partition only by a low cardinality column you always filter on, such as date at large scale.
CREATE TABLE silver.events (event_id STRING, customer_id STRING, event_time TIMESTAMP)
CLUSTER BY (customer_id)Practice: Chapter 10.
Symptom: filtering one customer reads every file.
Cause: the data layout does not match common filters.
Fix: cluster by the columns you filter on most. Delta then skips files that cannot contain matches.
Practice: Chapter 11.
Symptom: the storage bill keeps rising while data volume is flat.
Cause: old file versions kept for time travel are never removed.
Fix: run VACUUM with a sensible retention period. Remember it limits how far back you can time travel.
Practice: Chapter 8.
Symptom: two jobs writing the same table fail with a concurrency error.
Cause: both touched overlapping files.
Fix: make writes target separate slices with clear predicates, or serialise the jobs in one pipeline.
Practice: Chapter 8.
Symptom: negative quantities in a revenue dashboard.
Cause: nothing stopped them at the door.
Fix: add constraints on the table so bad writes fail immediately.
ALTER TABLE silver.orders ADD CONSTRAINT positive_quantity CHECK (quantity > 0)Practice: Chapter 13: Data Quality Engineering.
Symptom: bronze has 10,000 rows, gold has 9,412, and nobody knows why.
Cause: filters and inner joins drop rows with no record.
Fix: count rows at every layer and log the difference. A drop should always have a reason.
Practice: Chapter 16: Pipeline Testing and Validation.
Symptom: every check is green, the business is still confused.
Cause: success was measured as "did not crash", not "did the right amount of work".
Fix: alert on row volume and freshness, not just job status. We wrote a whole piece on this: Every dashboard was green. Customers were still failing.
Practice: Chapter 17: Observability and Debugging.
Symptom: finance and sales disagree on revenue.
Cause: each team calculates it in its own query.
Fix: define the metric once, in a governed gold table or metric view, and point everyone at it.
Practice: Chapter 18, and read Your spreadsheet is becoming an AI interface.
Symptom: orders point to customers that do not exist.
Cause: child data arrived before parent data, or the parent was deleted.
Fix: run a left anti join check after each load and route orphans to review.
orphan_orders_df = orders_df.join(customers_df, "customer_id", "left_anti")Practice: Chapter 6.
Symptom: a daily job reads two billion rows to find a few thousand changes.
Cause: no way to know what changed.
Fix: turn on Change Data Feed (CDF) and read only the changes since the last version.
ALTER TABLE silver.customers SET TBLPROPERTIES (delta.enableChangeDataFeed = true)Practice: Chapter 14: Change Data Capture.
Symptom: last year's sales show this year's sales region.
Cause: the dimension is overwritten in place.
Fix: a Type 2 slowly changing dimension. Close the old row with an end date, insert the new one as current.
Practice: Chapter 15: Historical Data Management.
Symptom: a cancelled account still appears in reports.
Cause: the pipeline only handles inserts and updates.
Fix: carry delete events through, and add WHEN MATCHED AND source.is_deleted THEN DELETE to your merge.
Practice: Chapter 14.
Symptom: bronze, silver and gold jobs run in the wrong order after a change.
Cause: dependencies live in people's heads.
Fix: Lakeflow Declarative Pipelines. You declare tables and their sources, and the dependency order is worked out for you.
Practice: Databricks Lakeflow guide and Medallion Architecture.
Symptom: a privacy request means finding one person across twenty tables.
Cause: personal data copied everywhere.
Fix: keep personal fields in as few tables as possible, delete there, then VACUUM so old versions go too.
Practice: Chapter 20: Data Governance and Security.
Symptom: a database password is visible in a shared notebook.
Cause: convenience on day one, never revisited.
Fix: store credentials as secrets and read them with dbutils.secrets.get. Secrets are hidden in output.
Practice: Chapter 0.5: Your Databricks Workspace.
Symptom: analysts can read salaries and phone numbers they do not need.
Cause: access was granted at the table level and forgotten.
Fix: use Unity Catalog grants, and add column masks and row filters for sensitive data. Masks and filters may need a paid workspace; the concept is explained in the chapter.
Practice: Chapter 20 and the Unity Catalog guide.
Symptom: an executive asks where a figure came from and the room goes quiet.
Cause: no lineage.
Fix: Unity Catalog records lineage automatically for tables and columns. Look it up before guessing.
Practice: Unity Catalog guide.
Symptom: CSVs and PDFs live in random folders with unknown owners.
Cause: files outside governance.
Fix: store files in Unity Catalog Volumes so they have owners and permissions like tables.
Practice: Chapter 7: Enterprise Data Ingestion.
Symptom: a CSV is sent weekly and goes stale instantly.
Cause: no live, governed sharing.
Fix: Delta Sharing gives partners read access to live data without copies. This feature typically requires a paid Databricks workspace; the concept is explained for understanding.
Practice: Chapter 20.
Symptom: a big bill for a quiet weekend.
Cause: interactive clusters with no auto termination.
Fix: set auto termination and prefer serverless where it fits. In Free Edition compute is serverless, which removes this problem by design.
Practice: Databricks pricing explained.
Symptom: the notebook crashes with a driver memory error.
Cause: .collect() or .toPandas() on millions of rows.
Fix: aggregate first, or limit() before collecting, or write the result to a table.
Practice: Chapter 2: Spark DataFrames.
Symptom: a dashboard runs the same heavy aggregation every refresh.
Cause: no precomputed result.
Fix: build a gold table or materialized view refreshed on a schedule.
Practice: Chapter 18 and SQL Warehouse guide.
Symptom: the first alert is an angry message from a stakeholder.
Cause: no alerting on failure or lateness.
Fix: add failure notifications and freshness checks to scheduled jobs.
Practice: Chapter 19: Production Orchestration and our Jobs and Pipelines guide.
Symptom: a 400 line stack trace and no idea where to start.
Cause: reading errors top down instead of looking for the cause.
Fix: search for "Caused by", then open the Spark UI and find the failed stage. Most errors point to data, not code.
Practice: Chapter 17: Observability and Debugging.
Symptom: a small change breaks a calculation nobody noticed for a week.
Cause: no tests on transformation logic.
Fix: move logic into functions and test them on tiny DataFrames.
Practice: Chapter 16 and the unit testing cheat sheet.
Symptom: a natural language tool gives confident, wrong numbers.
Cause: unclear table names, no descriptions, no definitions.
Fix: add table and column comments, and curate a Genie space with trusted example queries.
Practice: Chapter 20.5: Genie Spaces and Databricks Genie explained.
Symptom: one agent says approve, another says reject.
Cause: different context, no shared definitions, no rule for who decides.
Fix: one source of business context and a clear escalation path to a person. Read What happens when two AI agents disagree? and Enterprise context.
Symptom: classifying a table with an LLM takes hours.
Cause: using a text generating model for a simple choice.
Fix: use SQL AI Functions in batch. For choices, ai_decide returns a decision and a confidence. Read Databricks ai_decide explained.
Symptom: a model is great in the notebook and poor in production.
Cause: features calculated two different ways.
Fix: compute features once in governed tables and reuse them in both places, with MLflow tracking versions.
Practice: Chapter 21: ML Pipeline Development and Chapter 21.5: AI Gateway, Serving and Lakebase.
Reading about these fixes is useful. Feeling them is better. Try this short lab that touches four problems at once: duplicates, reruns, constraints and change data.
from pyspark.sql import functions as F
from pyspark.sql.window import Window
# 1. A small batch with a duplicate key (problem 2)
raw_df = spark.createDataFrame(
[(1, "Ana", 3, "2026-10-01 09:00"),
(1, "Ana", 4, "2026-10-01 10:00"),
(2, "Ben", 2, "2026-10-01 09:30")],
["order_id", "customer", "quantity", "updated_at"],
)
latest_first = Window.partitionBy("order_id").orderBy(F.col("updated_at").desc())
deduped_df = raw_df.withColumn("row_rank", F.row_number().over(latest_first)) \
.filter("row_rank = 1").drop("row_rank")
deduped_df.createOrReplaceTempView("staged_orders")
# 2. A target table with change data feed and a constraint (problems 26 and 31)
spark.sql("""
CREATE TABLE IF NOT EXISTS workspace.default.lab_orders
(order_id INT, customer STRING, quantity INT, updated_at STRING)
TBLPROPERTIES (delta.enableChangeDataFeed = true)
""")
spark.sql("ALTER TABLE workspace.default.lab_orders ADD CONSTRAINT positive_qty CHECK (quantity > 0)")
# 3. An idempotent write (problem 1). Run this cell twice.
spark.sql("""
MERGE INTO workspace.default.lab_orders AS target
USING staged_orders AS source
ON target.order_id = source.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *
""")
display(spark.table("workspace.default.lab_orders"))Now observe three things.
Run the merge cell twice. The table still has two rows. That is idempotency.
Try inserting a row with quantity 0. It fails. That is a constraint doing its job.
Run SELECT * FROM table_changes('workspace.default.lab_orders', 0). You see every change with its type. That is Change Data Feed, and it is how incremental pipelines avoid rereading everything.
Nobody gets hired for knowing 50 answers by heart. People get trusted for staying calm when a pipeline breaks at 7am and knowing where to look.
That calm comes from having seen the problem before, ideally in practice rather than in production. Each problem above links to a place where you can meet it safely first.
It also matters more every year. As AI systems read our tables and act on them, a duplicate row is no longer just a wrong number on a dashboard. It can become a wrong decision, made fast. We explored this in AI answers fast. Data engineering decides if you can trust it. and Superintelligence is coming. The foundation will still be data.
If you are new, start with the first free chapters of the book: Start Here, Spark DataFrames and SQL Query Fundamentals. They build the base every fix in this guide relies on.
If you are already working, pick the three problems your team hit last quarter. Read their chapters this week and run the fixes in Free Edition. That is how recognition is built.
And if you want the full path from first notebook to production patterns, Thinking in Data Engineering with Databricks covers all of it, one practical chapter at a time.