Our status page was green for weeks. Generations were failing the whole time.

That one still bugs me. Not because the check was broken in an obvious way. Because it was answering a question nobody needed answered.

What the check actually did

It sent a request with our own API key and looked at the status code. If we got a success back, we painted the dot green.

The thing is, that request never touched a model. It hit an endpoint that reads a list from our database and returns it. Auth works, database is up, dot goes green.

So we were testing one thing and reporting on another. The dot meant "our keys still work". We were reading it as "the product works".

How we noticed

A customer emailed. Not angry, just confused. They said generation had been failing since the morning, and asked whether we knew.

We did not know. We looked at the dashboard. Green. We looked at the graphs. Flat, quiet, boring.

Then we ran an actual generation by hand.

It failed. Fast.…