Our integration hub is one EC2 instance in a private subnet. It runs the small services that exchange files and API calls with our carriers, and Terraform builds it with a user data script that installs packages, fetches certificates and writes the configuration for each service.
In September one carrier moved its API behind a certificate issued by a new private authority and sent us the root to trust. The engineer handling it did the reasonable thing for a server defined in code. He added two lines to the user data script, one to download the new root and one to run update-ca-certificates, and opened a pull request. The plan showed a single change, user_data updated in place. It was approved and applied at eleven on a Tuesday morning.
Updated in place did not mean what either of us read it to mean. AWS only lets you change an instance's user data while the instance is stopped, so for this change the AWS provider stops the instance, writes the new user data and starts it again.…