Our checkout Deployment had replicas: 10 in its manifest for two years, and our pipeline applies manifests with kubectl apply. In October we added a HorizontalPodAutoscaler with a minimum of ten and a maximum of forty ahead of Black Friday.
The Kubernetes documentation is clear that once an autoscaler owns a Deployment, the replicas field should come out of the manifest. Otherwise every apply sets the count back to the number in the file, whatever the autoscaler had decided. So a second pull request deleted one line, replicas: 10. It merged at twenty past twelve on a Wednesday. Two minutes later checkout was running on one pod.
kubectl apply decides what to change by comparing three things: the manifest you give it, the live object, and a copy of the previous manifest it keeps in an annotation on the object, last-applied-configuration. A field that was in the previous manifest and is missing from the new one is a field you deleted, so apply removes it from the live object.…