On a Wednesday in October our largest carrier rotated our API key. They gave us a new key and twenty four hours before the old one stopped working, and the engineer on duty did it the way our runbook said. He pasted the new key into a file, built a Kubernetes Secret from it with kubectl create secret generic --from-file, sealed it and merged the change. The rollout restarted the booking service's pods with the new key, and for the next thirty five minutes no shipment could be booked with that carrier.

The carrier's dashboard showed no failed requests at all. That was the first clue, because it meant nothing had reached them.

The booking service is written in Go and puts the key in an Authorization header. Every call was failing inside our own pod with an error saying the header field value was invalid.…