A GPU cluster can pass every health check and still fail to run an AI workload. Even when every GPU, network link, and pod reports healthy, a 512-GPU training job can underperform or fail. The cause may be one slow GPU, a link that degrades under load, or a configuration that quietly routes traffic over a slower path. Operators may not discover the problem until hours into the run or until a customer files a ticket. Teams can then spend days bisecting the cluster to find the root cause while the capacity sits idle.

NVIDIA Cluster Readiness Engine (NVCRE) is an open source Kubernetes controller that narrows the search to the specific nodes involved before production workloads land. It runs real distributed workloads across topology-aware node groups, measures the results, and reports which nodes failed each test. Operators no longer need to write NVIDIA Collective Communications Library (NCCL) manifests by hand, bisect racks manually, or learn about degraded hardware from customer tickets.…