Understanding self-supervised Learning Dynamics without Contrastive Pairs

While contrastive approaches of self-supervised learning (SSL) learn by minimizing the distance between two augmented views of the data point (positive pairs) and maximizing views from different data (negative pairs), recent -contrastive SSL (e.g., BYOL and) show remarkable performance {\ without} negative pairs, with an learnable predictor and a stop-gradient operation. A fundamental question: why do these methods not collapse into trivial representations? We this question via a simple theoretical study and propose a novel, DirectPred, that sets the linear predictor based on statistics of its inputs, without gradient training. On ImageNet, it comparably with more complex two-layer non-linear predictors that BatchNorm and outperforms a linear predictor by $2.5\%$ in 300-epoch (and $5\%$ in 60-epoch). DirectPred is motivated by our theoretical of the nonlinear learning dynamics of non-contrastive SSL in simple networks. Our study yields conceptual insights into how non-contrastive methods learn, how they avoid representational collapse, and how multiple, like predictor networks, stop-gradients, exponential moving averages, weight decay all come into play. Our simple theory recapitulates the of real-world ablation studies in both STL-10 and ImageNet. Code is https://github.com/facebookresearch/luckmatters/tree/master/ssl.