A Green Status Does Not Guarantee Correct Data

An ETL workflow completes successfully. The scheduler reports no errors. The expected target tables are populated, and the processing logs indicate that all configured transformations finished.

Everything looks fine until someone compares the data.

A downstream report contains fewer records than expected. Some identifiers are missing, others appear unexpectedly, and the total record count does not explain the discrepancy.

This is a familiar challenge in enterprise data engineering. A pipeline can satisfy its technical execution requirements without satisfying the business requirements associated with its output.

In my work involving enterprise data processing, reconciliation and data-quality validation, I have found it useful to distinguish three questions:

Did the pipeline execute successfully?

Did the expected data arrive?

Does the resulting data satisfy the relevant business rules?

These questions require different forms of validation.

Execution…