Contrastive Learning: Why Pulling Similar Points Together Learns Representations Without Labels
Labels are expensive. Unlabelled data is not. Contrastive learning is built on that asymmetry: instead of asking what class an example belongs to, it asks a simpler question that turns out to be surprisingly powerful. Which examples should be close together, and which should be far apart?
If you can answer that question with pairs you construct yourself, you can train a representation that transfers to tasks you never trained for. That is the core idea behind SimCLR, MoCo, CLIP, and much of modern self-supervised learning. It is also an idea that is easy to implement badly, and the failure modes are instructive.
The basic setup
Start with an encoder, a neural network that maps an input to a vector. Call the encoder f, so an input x becomes a representation z = f(x). Contrastive learning does not tell the encoder what z should be.…