Two clusters, running for months, and no logs anywhere except docker service
logs and my own patience. I decided to fix that.
I thought it is a configuration task. Write some YAML, point it at a backend, go
and do something else. It seems it was not.
Every version of that YAML passed otelcol validate . Every one. Including the
version that lost data, the one that shipped every line twice, the one that
filled a disk with its own output, and the one whose regex could never match
anything at all.
Idea
Camelot AI runs on two small Docker Swarm clusters. Test
and production, two nodes each: a 1 GB manager that is always a bit close to the
edge, and a big ARM worker.…