Skip to content

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Authors: Jean-Bastien Grill, Florian Strub, Florent Altche, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Remi Munos, Michal Valko

Published: 2020 (Conference Paper)

Source: Advances in Neural Information Processing Systems

Algorithm: BYOL

arXiv: 2006.07733

Summary

BYOL shows that strong image representations can be learned without explicit negative pairs by training an online network to predict the representation of a slowly averaged target network under a different augmentation. The paper is a key non-contrastive self-supervised learning result: its empirical strength forced later work to explain why collapse is avoided and made target-network bootstrapping a standard design pattern for vision SSL.

Abstract

We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches $74.3\%$ top-1 classification accuracy on ImageNet using a linear evaluation with a ResNet-50 architecture and $79.6\%$ with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. Our implementation and pretrained models are given on GitHub.

Primary

Paper arxiv.org

Standard

arXiv Abstract 2006.07733 arXiv PDF 2006.07733 arXiv HTML 2006.07733

Alternate

proceedings.neurips.cc proceedings.neurips.cc GitHub: google-deepmind/deepmind-research github.com

Tags

  • Self-supervised learning

  • Representation learning

  • Computer vision

  • BYOL

  • Bootstrap learning

  • Non-contrastive learning

  • Momentum target network

  • ImageNet

  • Transfer learning