I work on CloudGPU. Keep dataset decoding and collation on the CPU, then move each batch to CUDA in the main training process. If a worker initializes CUDA after a fork, adding more workers can expose a process-initialization problem rather than a model problem.
1. Find the CUDA operation inside the worker
Inspect Dataset.__getitem__ , transforms, collate_fn and worker_init_fn . Remove .cuda() , .to("cuda") and accelerator model construction from those paths. Also check whether a tensor was created on CUDA when the dataset object was built.
Use num_workers=0 temporarily to obtain a clearer traceback. It is a diagnostic step, not a recommendation about throughput.
2. Give the loader CPU data and an explicit process context
Save this as loader_boundary.py in an environment with a suitable PyTorch CUDA build:
import torch from torch . utils . data import DataLoader , Dataset
class Samples ( Dataset ) : def __len__ ( self ) : return 12
def __getitem__ ( self , index ) : return torch .…