On a Thursday in May one of our Kubernetes nodes failed its health checks and was replaced, and the pod for our customs document store moved to another node. It took forty one minutes to start. For all of that time export shipments could not have their commercial invoices attached, and two hundred pallets waited at the dock for their paperwork.

The store is a StatefulSet with a 600 gigabyte volume holding about nine million scanned documents. The new pod sat in ContainerCreating. Its events showed the volume attached and mounted within a minute, then nothing. On the node, the kubelet was busy and the volume was running at the limit of its provisioned IOPS.

A year earlier we had changed the image to run as a non root user, and so that the user could write to the volume we set fsGroup in the pod's security context. With fsGroup set, Kubernetes by default changes the group ownership and permissions of everything on the volume to match, recursively, every time the volume is mounted.…