📰00Read later0Read More4 GPUs, 2x speed: why PyTorch DDP stalled on a PCIe-only 4x RTX A6000, and two flags that took it to 3.6xDEV Community: pytorch·Runcrate·about 6 hours ago#r9P5Piqo#devtopytorch#devto#dev#gpus#124m+5 more🧰Tag tools✨Add tagPlain DDP on a rented 4x RTX A6000 gave 2x. Measured all-reduce bandwidth, then bf16 gradients and accumulation took it to 3.6x. Commands inside.15s
