Self-Supervised Learning From Images With A Joint-Embedding Predictive Architecture

Source

@inproceedings{assran_2023_ijepa,
    author={Assran, Mahmoud and Duval, Quentin and Misra, Ishan and Bojanowski, Piotr and Vincent, Pascal and Rabbat, Michael and LeCun, Yann and Ballas, Nicolas},
    booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, 
    title={Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture}, 
    year={2023},
    pages={15619-15629},
    doi={10.1109/CVPR52729.2023.01499}
}

(Meta)

arXiv

TL;DR

… General pipeline

Flash Reading

Abstract: Learn highly semantic image representations without relying on hand-crafted data-augmentations. I-JEPA is a non-generative self-supervised learning method.
Introduction: For self-supervised learning from images, there are two common families: invariance-based methods (encoding similar embeddings for multiple views of the same image, from hand-crafted augmentations) and generative methods. The former introduces strong biases and is difficult to generalize. Inspired by cognitive learning, representation learning is the adaptation of an internal model to predict sensory input responses. In generative models, especially mask-denoising approaches learn representations by reconstructing masked inputs. However, the learned features tend to be local-details other than high-level concepts, and the feature space is not well structured. The idea of I-JEPA is to predict the missing information in an hidden space.
Background: Joint-embedding architecture (JEA) is to learn similar embeddings for compatible inputs and vice versa. Generative architecture directly reconstructs a signal from a compatible signal (like masked signal). JEPA is similar to the generative architecture, but it computes the loss in a hidden space rather than the input space. JEPA predicts the embeddings of a target signal $y$ from a compatible context signal $x$ and additional (latent) variable $z$.
Method: The task is, given a context block $x$, predict the embeddings of various target blocks $y$ in the same image. Context encoder, target encoder, and predictor are ViT models. The loss is simply L2 distance between the predicted target embeddings and the actual target embeddings.