Introduction
Most data platforms eventually reach the same problem. Jobs that once ran from a cron entry now depend on each other, run at different frequencies, fail for reasons nobody remembers, and need to be retried, backfilled, and monitored. A shell script and a crontab are no longer enough. What teams need is an orchestrator: a system that knows what should run, in which order, when, and what to do when something breaks.
Apache Airflow is one of the most widely adopted orchestrators for this job. It lets you describe workflows as code, schedule them, visualize their state, and recover from failures in a consistent way. It is written in Python, and its workflow definitions are Python files, which can make it look like a tool that lives outside the world of Java and Spring Boot.…