In September we retired two of the six application servers behind our internal API, and the next morning the warehouse scanners at one site started failing about a third of their requests with 502. Part of our nginx tier was still sending traffic to both retired servers. The upstream list in its configuration had not mentioned them since the fifth of August.
The nginx tier is forty virtual machines configured by an Ansible playbook. The task that renders nginx.conf from a template notifies a handler that reloads nginx. Handlers run at the end of the play, once, on each host where a notifying task reported changed. On the fifth of August the template changed on all forty. Later in the same play, a task installing a monitoring agent failed on eleven hosts because our package mirror timed out. By default a host that fails a task drops out of the rest of the play, and the handlers it had notified never run.…