I work on CloudGPU. When you use PyTorch gradient scaling, clip after scaler.unscale_(optimizer) and before scaler.step(optimizer) . Clipping scaled gradients changes the meaning of the threshold. During gradient accumulation, wait until the effective batch is complete before unscaling.
1. Identify who owns the training step
The function below is for a custom, single-optimizer PyTorch loop using CUDA autocast with float16 . It is not an instruction to add a second scaler inside a trainer that already manages mixed precision. In a managed trainer, use its documented clipping hook instead.
Construct the optimizer from the adapter parameters you actually intend to train. Frozen parameters do not become trainable just because you include them in a clipping call.
2. Put the operations in an explicit order
import torch
def adapter_step ( model , optimizer , scaler , inputs , targets , loss_fn ) : # Inputs, targets and the prepared model already belong to CUDA. optimizer .…