Menu

Where should gradient clipping go in a mixed-precision LoRA training step?
📰

Where should gradient clipping go in a mixed-precision LoRA training step?

DEV Community: pytorch·CloudGPU·5 days ago
#dW8BRvWA
#devtopytorch#devto#dev#scaler#optimizer
Reading 0:00
15s threshold

I work on CloudGPU. When you use PyTorch gradient scaling, clip after scaler.unscale_(optimizer) and before scaler.step(optimizer) . Clipping scaled gradients changes the meaning of the threshold. During gradient accumulation, wait until the effective batch is complete before unscaling.

1. Identify who owns the training step

The function below is for a custom, single-optimizer PyTorch loop using CUDA autocast with float16 . It is not an instruction to add a second scaler inside a trainer that already manages mixed precision. In a managed trainer, use its documented clipping hook instead.

Construct the optimizer from the adapter parameters you actually intend to train. Frozen parameters do not become trainable just because you include them in a clipping call.

2. Put the operations in an explicit order

import torch

def adapter_step ( model , optimizer , scaler , inputs , targets , loss_fn ) : # Inputs, targets and the prepared model already belong to CUDA. optimizer .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.