Kubernetes manages what runs on your nodes. Managing the nodes themselves is the challenge: kernel settings, system packages, storage layouts, security agents, and the host-level tuning that GPU workloads depend on. Many teams manage this with Ansible playbooks, custom scripts, and manual runbooks. That works until a new cluster comes up in a different region, a kernel upgrade breaks RDMA, or a CVE has to be remediated across the whole fleet this week.

GPU infrastructure makes the challenge harder. You cannot simply discard a GPU node and spin up a fresh one. Hardware is scarce, replacements can take hours, and long-running training jobs cannot simply be rescheduled. So the operational question is not how to change a node; it is how to change all of them without killing the training run.…