I built a pipeline I was proud of recently, and it solved a problem I was actually facing. How do I keep track of jobs posted in the industry I'm breaking into? - data engineering. I managed to solve that problem to a point. I had a pipeline that extracts data from two sources, Hacker News (to keep myself updated on articles posted in the industry) and RemoteOk (to keep track of relevant remote jobs posted in the industry).
Here was the detailed architectural diagram of what the pipeline looks like
But something was missing. The pipeline could only tell me a job existed and when it opened and close. I couldn't tell what the job was asking for.
When I finished my job market pipeline, I was happy with it. It pulled remote data jobs from RemoteOK, loaded them into DuckDB, cleaned them with dbt, and used a snapshot to remember when a listing opened and closed.…