A green checkmark means the code finished. It does not mean the data is true.
It is Monday morning. The pipeline finished at 4:15 AM. Every task is green. No alerts fired, and nobody got paged.
At 9:30 AM a message arrives from the finance lead. "Did revenue really jump 42% on Sunday, or is the dashboard wrong?"
You open the job page. Green. You open the logs. Clean. And slowly you realize the hardest truth in data engineering. A failed job is the easy problem. A job that succeeds with the wrong numbers is the real one.
A scheduler checks one thing. Did the code finish without throwing an error?
It does not check whether the answer makes sense. If a join creates ten million extra rows, Spark calls it a success. If a filter quietly drops a third of your customers, Spark calls that a success too.
So a green checkmark means "the computer did what you told it." It does not mean "the data is true." Those are two very different promises.
flowchart LR
A[Raw data arrives] --> B[Transform]
B --> C{Job finished?}
C -->|Yes| D[Green checkmark]
D --> E[Dashboard shows wrong numbers]
B --> F{Data makes sense?}
F -->|Nobody asked| EA customer table should have one row per customer. One day, a customer gets two "active" rows because an update did not close the old one.
Now every order for that customer joins twice. Revenue doubles for that slice, and nothing crashes.
An upstream team makes a field optional. Some rows now arrive with a null region.
Your INNER JOIN on region quietly throws those rows away. No error, no warning. Just fewer customers than yesterday.
Dates get parsed without a clear timezone. Saturday night sales land in Sunday.
Daily totals shift, trends look strange, and people start making decisions on a pattern that never happened.
Upstream sends an empty batch during maintenance. Your job uses .mode("overwrite") and does exactly what it was told.
Yesterday's good table is replaced with nothing. The run is green, and the dashboard is empty.
Experienced teams do not only watch whether jobs finish. They check the data before they let it into an important table.
Three checks catch most silent problems:
If a check fails, the job stops before writing. A loud failure at 4 AM is far kinder than a quiet lie at 9 AM.
This small lab builds a table with a hidden duplicate, then shows a safe write that refuses bad data. It runs in a Free Edition notebook.
from pyspark.sql import functions as F
# Sample orders. Order 3 appears twice by mistake.
orders_df = spark.createDataFrame(
[(1, "alice", 120.0), (2, "bob", 80.0), (3, "chen", 200.0), (3, "chen", 200.0)],
["order_id", "customer", "amount"],
)
def check_before_write(df, key_column, min_rows):
row_count = df.count()
if row_count < min_rows:
raise ValueError(f"Only {row_count} rows. Expected at least {min_rows}. Stopping.")
duplicate_keys_df = df.groupBy(key_column).count().filter(F.col("count") > 1)
duplicate_count = duplicate_keys_df.count()
if duplicate_count > 0:
duplicate_keys_df.show()
raise ValueError(f"{duplicate_count} duplicate {key_column} values. Stopping.")
print(f"Checks passed. {row_count} rows are safe to write.")
check_before_write(orders_df, "order_id", min_rows=1)Run it and you will see order 3 listed, followed by a clear error. Now remove the duplicate and try again:
clean_orders_df = orders_df.dropDuplicates(["order_id"])
check_before_write(clean_orders_df, "order_id", min_rows=1)
clean_orders_df.write.mode("overwrite").saveAsTable("workspace.default.orders_checked")The write only happens after the checks pass. That one habit protects every table downstream.
If you use Lakeflow Declarative Pipelines, expectations give you the same idea in a built-in way. We walk through them in How to Catch Bad Data Early With Lakeflow Pipeline Expectations.
Junior engineers are measured by whether the job runs. Senior engineers are measured by whether people can trust the numbers.
That trust is built quietly, one check at a time. Nobody celebrates the duplicate that never reached the dashboard. But everyone remembers the Monday it did.