Our login endpoint sits behind AWS WAF, and one rule in the web ACL is a rate based statement: any single address that sends more than a set number of login requests in five minutes is blocked. The web ACL is Terraform, and the number is a variable, login_rate_limit. Our pipeline runs plan and apply for production from GitHub Actions, and it passes production's settings to Terraform as environment variables, TF_VAR_login_rate_limit among them, taken from the variables of the production environment in GitHub.
That was deliberate. The runbook for credential stuffing says to lower the variable in GitHub and run the edge workflow, which changes the limit in a few minutes without waiting for a pull request.
At ten past three on a Saturday in October, logins were arriving from about eleven hundred addresses, each sending between seven and nine hundred attempts every five minutes, under our limit of two thousand. The engineer on call followed the runbook and set the variable to three hundred.…