Menu

Ansible updated our servers five at a time and kept going after most of each batch failed
📰

Ansible updated our servers five at a time and kept going after most of each batch failed

DEV Community: ansible·Sergey Shinder·5 days ago
#mY5sAVNG
#devtoansible#devto#dev#servers#batch
Reading 0:00
15s threshold

We deploy our order API to thirty servers with an Ansible playbook that runs with serial: 5. For each batch it takes the servers out of the HAProxy pool with a delegated task, installs the new package, restarts the service, waits for the health port with wait_for and a sixty second timeout, and puts the servers back in the pool. Five at a time, we told ourselves, limited a bad release to a sixth of the fleet.

On a Thursday in October a release added a product cache that is warmed when the service starts. On our servers the warm up took between fifty and ninety seconds. In the first batch one server answered inside sixty seconds and four did not, so wait_for failed on those four. Ansible marked them failed and ran nothing more on them, including the task that would have put them back in the pool. The fifth server went back in, and Ansible moved on to the second batch.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.