On the Alignment Between Supervised and
Self-Supervised Contrastive Learning

Overview

Self-supervised contrastive learning (CL) has achieved remarkable success, producing representations that rival supervised pre-training. Recent work has shown that the CL loss closely approximates a supervised surrogate, Negatives-Only Supervised Contrastive Learning (NSCL), as the number of classes grows. But this similarity at the loss-level leaves a crucial question unresolved.

Understanding this dynamic relationship is crucial. A close alignment throughout training would suggest that CL's learning process inherently mimics a supervised signal, providing a stronger foundation for its empirical success and the transferability of its learned features. In this study, we analyze the alignment between CL and NSCL and find a notable divergence in their parameter-space trajectories. Despite this, we demonstrate that their learned representations remain remarkably aligned. We provide theoretical guarantees for this representation alignment and a better understanding of how this alignment depend on number of classes, higher temperatures and varying batch-sizes.

Methodology

To investigate representation alignment, we conduct a controlled study focusing on the self-supervised contrastive loss, $L^{CL}$, and its supervised counterpart, $L^{NSCL}$. We train two models under identical conditions—shared initialization, mini-batches, augmentations, and hyperparameters—to isolate the effect of the loss function itself.

• $\mathrm{sim(\cdot, \cdot)}$ denotes cosine similarity
• $N$: Total number of training samples
• $K$: Total number of augmented versions of each sample
• $z^k_i = f(\alpha_k(x_i))$, where $x_i$ is an input image and $\alpha_k$ is its $k^{th}$ augmentation.

Key Observations

1 · Divergence in Weights vs. Alignment in Representations

Our central finding is a tale of two spaces. When trained with shared randomness (initialization, batches, augmentations), CL and NSCL models take different paths in parameter space, leading to a significant weight gap. However, the representational geometry they induce remains remarkably similar.

(a) Weight Space: Model parameters diverge significantly as training progresses.

(b) Representation Space: Learned features remain closely aligned, maintaining high similarity.

Corresponds to Figure 1 in the paper. See Appendix A for additional details.

The divergence in weight space suggests that directly comparing model parameters isn't the best approach. What truly matters for downstream performance is the geometry of the learned representations. To better quantify this, we analyze the alignment in "similarity space". Instead of tracking millions of parameters, we track the $N \times N$ pairwise similarity matrix, $\Sigma$, whose entries $\Sigma_{ij}$ represent the cosine similarity between the embeddings of two inputs, $x_i$ and $x_j$. This perspective allows us to directly measure how the geometric structure of the representation space evolves via the following bound.

Note on Metrics: Theorem (1) provides a direct bound on the difference between the two similarity matrices. In our experiments, we use standard and widely-accepted metrics—Centered Kernel Alignment (CKA) and Representational Similarity Analysis (RSA) —to empirically measure alignment. These metrics are fundamentally dependent on the underlying similarity matrices; a small distance between the matrices directly implies high CKA and RSA scores. For the formal derivation connecting our bound to these specific metrics, we refer the interested reader to Corollaries 1 and 2 in our paper.

2 · Factors Influencing Alignment

Our theory predicts how alignment should behave under different conditions. We empirically confirm these predictions across multiple datasets by varying key hyperparameters.

Finding: Alignment gets stronger as the number of classes increases. This holds consistently across different datasets.

Mini-ImageNet

Tiny-ImageNet

ImageNet-1k

Corresponds to Figure 3 in the paper. The heatmaps show the linear CKA between CL and NSCL models on both train (top, green) and test (bottom, purple) datasets.

Finding: Higher temperatures lead to better and more stable alignment. A temperature of τ=1.0 consistently outperforms lower values.

CIFAR-100

Tiny-ImageNet

Mini-ImageNet

Corresponds to Figure 4 in the paper. The plots show RSA (top) and CKA (bottom) over 300 epochs between CL and NSCL models trained with different temperatures.

Finding: The effect of batch size is coupled with the learning rate schedule. With appropriate scaling, larger batch sizes generally improve alignment.

$$\mathcal{O}(B)$$

$$\mathcal{O}(B^{1/2})$$

$$\mathcal{O}(B^{1/4})$$

Corresponds to Figure 5 in the paper. Here we vary $\eta$ with $B$ across three cases: $\eta=\tfrac{0.3B}{256}$, $\eta=\tfrac{0.3\sqrt{B}}{256}$, and $\eta=\tfrac{0.3\sqrt[4]{B}}{256}$. Under $\mathcal{O}(B)$ scaling, CL–NSCL alignment decreases as $B$ grows whereas for the other two cases, alignment overall increases with $B$.

So far, we've demonstrated that the alignment between CL and NSCL is not just theoretical but holds up empirically, influenced by factors like class count, temperature, and batch size. But this raises a natural question: why focus specifically on NSCL as the supervised benchmark? Is it truly the best proxy for understanding self-supervised CL, compared to other supervised methods?

3 · NSCL is the Best Supervised Proxy

We find that NSCL is a much better proxy for CL than both standard Supervised Contrastive Learning (SCL) and Cross-Entropy (CE). This positions NSCL as a principled bridge for understanding self-supervised learning.

CIFAR-100

Tiny-ImageNet

Mini-ImageNet

Corresponds to Figure 2 in the paper. We train CL, NSCL, SCL and CE models with ResNet-50 on CIFAR-100, Tiny-ImageNet and Mini-ImageNet datasets. The plots show RSA (top) and CKA (bottom) between CL and the three supervised objectives. NSCL consistently achieves the highest alignment with CL.