Hi all,
Teiva here (author of 100 Go Mistakes). I wrote up the architecture of a side project of mine: a Go pipeline that ingests posts from HN, Lobsters, arXiv and 300+ engineering blogs, then classifies, embeds and ranks them.
Each stage is a short-lived Go binary started on a cron: it claims rows with SELECT ... FOR UPDATE SKIP LOCKED across a few goroutine workers, processes them, and exits. Claiming the row, saving the result and enqueuing the next stage happen in one transaction, so a crash just rolls back and the row is picked up on the next run. Retries are a not_before column, dead letters a table, schedules timestamp columns.
Curious how others structure this kind of worker in Go.
submitted by