Does torch.compile make a small transformer train faster? I rented four GPUs on Runcrate today (2026-10-29), an RTX 5090, an L40S, an RTX A6000 and an H100 PCIe, trained the same 468M-parameter Llama-style model in eager mode and compiled, with PyTorch's SDPA attention and with the plain "eager" attention, and deleted every machine when the numbers were saved.

Short version: with SDPA attention, torch.compile changed the step time by between 9% slower (RTX 5090) and 3% faster (L40S), cost 22 to 39 s of compile time on the first step, and cut peak memory by 23% on every GPU (17.3 to 13.3 GB). With the plain attention it was a different story: on the H100 it made the step 2.6x faster and fit the model in 39 GB instead of 56 GB, and on the 48 GB cards it was the only way the run fit at all.…