Menu

1 / 5

Notes from a 2 AM On-Call: The Goroutine Leak Behind Our RAGFlow Ingestion Pipeline OOM

DEV Community: go·Kevin Tian·17 days ago
#xrjIbeH7
#devtogo#devto#dev#ragflow#exit
Reading 0:00
15s threshold

The Failure Symptom

Page fired at 02:14. Our document ingestion service, the Go shim that batches chunks and streams them into a self-hosted RAGFlow instance ( POST /api/v1/datasets/{id}/documents ), started throwing:

fatal error: runtime: out of memory exit status 2 (SIGKILL, container oom_score 987) goroutine 84213 [chan send, 14 minutes]: main.(*ingestWorker).submit(...) channel unbuffered

Enter fullscreen mode

Exit fullscreen mode

docker stats showed RSS pinned at the 12GB cgroup ceiling. The RAGFlow side was fine — Elasticsearch was healthy, the API returned 200s. The leak was entirely in our shim.

The Red Herring

The first instinct from the runbook: bump GOMEMLIMIT and call it a day. That bought us 40 minutes, then the same OOM. The second instinct — standard tutorials everywhere — was "just add workers":

for i := 0 ; i < 64 ; i ++ { go ingestWorker ( ch ) // ch is unbuffered }

Enter fullscreen mode

Exit fullscreen mode

That made it worse.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.