When stepping into the world of data engineering, Apache Airflow is likely one of the first tools you will encounter. It is the industry standard for programmatically authoring, scheduling, and monitoring workflows.

Understanding the New Airflow 3.1.0 Syntax

Before building our first DAG, it's important to know what has changed in Airflow 3.1.0.

Initially, Airflow users imported DAGs and tasks from airflow.models and airflow.decorators. In Airflow 3.0 and later, versions introduced the airflow.sdk

This means, you will now use:

from airflow.sdk import dag , task

Enter fullscreen mode

Exit fullscreen mode

This ensures your DAGs remain compatible with future Airflow upgrades

The Pipeline Overview

Our data pipeline consists of two main tasks:

1 Extract: Fetch the daily open, high, low, close, and volume data for a list of stock tickers from the Massive API.

2 Transform and Load: Convert the raw data into a structured format, connect to a PostgreSQL database, create the target table if it does not…