How I combined Apache Flink, Redpanda, Airflow, dbt Cloud, and Grafana to track Bitcoin, Ethereum, Solana, BNB, and Cardano in real time — all running on Google Cloud for less than a cup of coffee per month.

If you've been learning data engineering, you've probably built pipelines that move CSV files from A to B. Nothing wrong with that — but real data engineering interviews are starting to ask a different question: Can you handle data that never stops arriving?

This article walks through CoinPulse, a project I built from scratch to answer that question. It's a hybrid streaming and batch crypto analytics pipeline that:

Consumes live trade events from Binance's WebSocket feed at ~1 tick per second per coin

Processes them with Apache Flink using 1-minute tumbling windows

Enriches the streaming data daily with CoinGecko market metadata via Airflow

Transforms everything with dbt Cloud

Serves a live, auto-refreshing Grafana dashboard — publicly accessible

Total cloud cost: ~$0.01/month.

Here's how I…