Your Airflow DAG ran last night. All tasks: green. All durations: normal. Export job completed at 02:14.
Zero rows exported. Nobody knows.
This is the silent failure Airflow's built-in alerting doesn't catch. on_failure_callback fires when a task crashes. It doesn't fire when a task exits 0 after connecting to a stale database replica and processing nothing. That's the failure mode that eats your Monday morning.
This article shows you two ways to add external monitoring to Airflow DAGs β so you get paged for both kinds of failures.
Why Airflow's built-in alerts aren't enough
Airflow gives you several callback hooks:
on_failure_callback β task or DAG run failed
on_success_callback β task or DAG run succeeded
on_retry_callback β task queued for retry
on_execute_callback β task about to start
on_skipped_callback β task raised AirflowSkipException
These are useful.β¦