So, I’ve been documenting my journey as I fully transition to data engineering and so far, I’ve built two pipelines. The first is a scheduled GitHub ETL pipeline that extracts GitHub repositories , transforms it and stores it in a SQLite database. This was straightforward and I learned a lot of concepts like ETL, idempotency and how to schedule using a cron job.

The second project took things a step further and I built an automated RSS pipeline that extracts tech articles from various RSS feeds and stores it in a PostgreSQL database. The automation/orchestration was handled by Kestra. This was my first time trying an orchestration platform and I also got exposed to Docker which I used to containerize my data pipeline to be easily orchestrated by Kestra and deployed to AWS cloud platform.

Now, I’m taking on a new project. This time, I want a project that allows me to refresh my memory through repetition of certain aspects of my previous projects while allowing me to learn some fresh sets of skills.…