Menu

Moving Cloud Composer 3 to an Airflow 3 image? Run dates shift with no errors. What changes and how to keep the old ones

Reddit r/googlecloud·u/ManufacturerKey6917·1 day ago
#A7k6E2y5
#airflow#composer#build#dags#article
Reading 0:00
15s threshold

Airflow 3 has been GA on Cloud Composer 3 since composer-3-airflow-3.1.7-build.4 , and the current builds are composer-3-airflow-3.3.1-build.5 and 3.2.2-build.9 . The default image is still Airflow 2 ( composer-3-airflow-2.11.1-build.22 ), so a lot of environments still have this upgrade ahead. I compared the same DAGs on Airflow 2.11.2 and 3.3.2. The changes below raise no errors: DAGs import, tests pass, and the dates are wrong.

Cron runs get a different logical date. From 3.0, [scheduler] create_cron_data_intervals defaults to False . An @daily DAG running process {{ ds }} , for the run that fires at 2024-05-02 00:00 UTC:

Timetable Task runs

Airflow 2.11.2 CronDataIntervalTimetable

Airflow 3.3.2 CronTriggerTimetable

Weekly DAGs move by a full week. Partition paths like dt={{ ds }} move with them.

The data interval becomes zero-length. Hourly DAG: 00:00 → 01:00 on 2.11, 00:00 → 00:00 on 3.3.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.