Crypto ETL Pipeline (Python + PostgreSQL)
Overview
This project is a simple ETL (Extract, Transform, Load) pipeline that retrieves real-time cryptocurrency market data from the CoinPaprika API, processes it using Python (Pandas), and loads it into a PostgreSQL database.
The pipeline is designed for learning and demonstrating core data engineering concepts such as API ingestion, data transformation, and relational database storage.
Architecture
Extract → Transform → Load
Extract
Data is fetched from the CoinPaprika REST API
Cryptocurrencies include: Bitcoin, Ethereum, XRP, Solana
Transform
JSON response is normalized using Pandas
Unnecessary fields are removed
Columns are renamed for clarity
Timestamp ( ingested_at ) is added
Load
Cleaned data is inserted into PostgreSQL
Table: crypto_market_data
Tech Stack
Python 3.x
Pandas
Requests
SQLAlchemy
Psycopg2
PostgreSQL
python-dotenv
Project Structure
Crypto_etl.py/
│
├── crypto_etl.py Main ETL script
├── requirements.txt Python…