Menu

Building Barista—Part 1
📰

Building Barista—Part 1

#golang#redditgolang#stage#next#teiva
Reading 0:00
15s threshold
Building Barista—Part 1

Hi all,

Teiva here (author of 100 Go Mistakes). I wrote up the architecture of a side project of mine: a Go pipeline that ingests posts from HN, Lobsters, arXiv and 300+ engineering blogs, then classifies, embeds and ranks them.

Each stage is a short-lived Go binary started on a cron: it claims rows with SELECT ... FOR UPDATE SKIP LOCKED across a few goroutine workers, processes them, and exits. Claiming the row, saving the result and enqueuing the next stage happen in one transaction, so a crash just rolls back and the row is picked up on the next run. Retries are a not_before column, dead letters a table, schedules timestamp columns.

Curious how others structure this kind of worker in Go.

submitted by /u/teivah
[link] [comments]