The Failure Symptom
Page fired at 02:14. Our document ingestion service, the Go shim that batches chunks and streams them into a self-hosted RAGFlow instance ( POST /api/v1/datasets/{id}/documents ), started throwing:
fatal error: runtime: out of memory exit status 2 (SIGKILL, container oom_score 987) goroutine 84213 [chan send, 14 minutes]: main.(*ingestWorker).submit(...) channel unbuffered
Enter fullscreen mode
Exit fullscreen mode
docker stats showed RSS pinned at the 12GB cgroup ceiling. The RAGFlow side was fine — Elasticsearch was healthy, the API returned 200s. The leak was entirely in our shim.
The Red Herring
The first instinct from the runbook: bump GOMEMLIMIT and call it a day. That bought us 40 minutes, then the same OOM. The second instinct — standard tutorials everywhere — was "just add workers":
for i := 0 ; i < 64 ; i ++ { go ingestWorker ( ch ) // ch is unbuffered }
Enter fullscreen mode
Exit fullscreen mode
That made it worse.…