Four GPUs should train about four times faster than one. On a rented 4x RTX A6000 machine this morning (2026-10-26), plain PyTorch DDP gave me 2x. This is the notebook of finding out why, and the two flags that took it to 3.6x.

Setup

Machine: one Runcrate 4x RTX A6000 (48 GB each) VM, US Midwest, $2.64/hour for the whole machine at launch time. Two separate machines of the same type over the morning: the first for the baseline, the second for the fixes.

Topology ( nvidia-smi topo -m ): every GPU pair is PHB , i.e. GPU to GPU traffic crosses the CPU's PCIe host bridge. No NVLink. PCIe gen 3 x16 per card.

Model: a GPT-2 style decoder written in plain PyTorch, two sizes: 124M parameters (12 layers, 768 wide) and 355M (24 layers, 1024 wide). Random tokens, sequence 1024, micro-batch 8 per GPU, bf16 autocast, fused AdamW.

Software: torch 2.8.0+cu128 (first machine) and 2.11.0+cu128 (second), NCCL as shipped in the wheel.

Metric: tokens/s from the median step time over 40 steps after 10 warm-up steps.…