Our customer portal runs on sixteen web servers behind one load balancer. When a customer asks for an invoice, the server that handles the request returns a link signed with an HMAC key, and the download is verified by whichever server the load balancer picks next. Every server has to hold the same key. An Ansible playbook generates it and writes it to every server, and we rotate it every quarter.

The October rotation ran on a Tuesday evening and finished green. By Wednesday morning three invoice downloads in four were failing with invalid signature, and support had more than four hundred tickets before nine.

The task that generates the key is a command running openssl rand, marked run_once and registering its output. With run_once, Ansible runs the task on one host and gives its result to every other host it is working on.…