Menu

A Collapsed Encoder Scores a Perfect 1.0 on Alignment
📰

A Collapsed Encoder Scores a Perfect 1.0 on Alignment

DEV Community: deeplearning·Devanshu Biswas·13 days ago
#lg8lMfor
#devtodeeplearning#education#devto#dev#loss
Reading 0:00
15s threshold

NT-Xent — InfoNCE, the loss behind SimCLR and most embedding models — asks each view of an item to pick its partner out of a line-up. The line-up is the rest of the batch .

👉 Live, every number measured in your browser: https://dev48.in/dl/day82-contrastive-loss.html

That clause is not a footnote. It means the batch size changes the objective , not just the gradient noise.

The loss number is not comparable across batch sizes

batch negatives per row loss collapse floor log(2N−1) partner ranked first

N = 2 4 0.0084 1.0986 100%

N = 8 16 0.0087 2.7081 100%

N = 32 64 0.0379 4.1431 100%

N = 64 128 0.0817 4.8442 100%

Same encoder, same temperature, same data distribution. The loss climbs anyway, because the task got harder .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.