Modern data engineering is built around automation, orchestration, and scalable infrastructure. In this project, I built a Dockerized ETL pipeline that collects crypto data from the CoinPaprika API, processes it using Apache Airflow, and stores it in a PostgreSQL database hosted on Aiven Cloud.

The goal of the project is to simulate a real-world data engineering workflow while learning:

Workflow orchestration with Apache Airflow

Containerization with Docker

API data extraction

Cloud database integration(Aiven)

The entire pipeline runs inside Docker containers, making it portable, reproducible, and easy to deploy.

Architecture Overview

The project has four main components:

CoinPaprika API: an API providing real-time cryptocurrency market data

Apache Airflow: to orchestrate the ETL workflow using TaskFlow API

Docker: containerizes the entire Airflow environment for reproducibility

Aiven PostgreSQL: a cloud PostgreSQL instance where processed data is stored

It follows a simple ETL…