LeCun has consecutively reposted the new work VISReg, which solves the core problem of "representation collapse" of the JEPA world model.

The foundation of the JEPA world model is the Self-Supervised Learning (SSL) paradigm that Yann LeCun has been continuously advocating since 2017.

SSL can learn general representations from massive amounts of data without manual annotation, but it universally faces a core challenge — representation collapse: the model tends to map different inputs to the same or a tiny number of vectors. While this appears to complete the training process, it has not actually learned discriminative representations.

To mitigate collapse, most mainstream methods rely on a series of heuristic tricks (EMA, teacher-student networks, stop-gradient, frozen layers, etc.). These tricks make training fragile, difficult to tune, and also undermine the interpretability and scalability of the methods.

Another approach is to directly constrain the representation distribution through a regularization term.

The VICReg proposed by LeCun's team splits the learning objective into three components: variance, invariance, and covariance, using the covariance term to constrain the correlation between different dimensions; however, covariance only characterizes second-order statistics and cannot distinguish two types of representations that share the same mean and variance but have completely different distribution shapes.

The subsequently proposed SIGReg is based on the Cramér–Wold theorem, using sketching techniques to align the entire embedding distribution to a standard Gaussian, thus constraining the complete distribution shape.

Nevertheless, SIGReg still has two critical flaws:

• Vanishing gradients at collapse: When representations begin to collapse, SIGReg's gradients decay accordingly — the more severe the collapse, the weaker the correction signal, making it difficult for the model to recover on its own;
• Scale-shape coupling: It fails to separate the two independent properties of "magnitude (scale)" and "distribution morphology (shape)", which interfere with each other during optimization, leading to poor adaptability on long-tailed, low-quality, and low-rank data.

In other words, when the model most needs gradient signals to escape the collapsed state, SIGReg's gradients are precisely approaching zero.

This is exactly the core problem that VISReg aims to address.

Recently, the new self-supervised learning work VISReg (Variance-Invariance-Sketching Regularization) has been continuously shared and highly praised by Turing Award winner Yann LeCun — he commented in his repost that "VICReg begat SIGReg which begat VISReg", a statement that clearly points out the technical lineage of this regularization route.

Earning such recognition from LeCun, what exactly makes VISReg so powerful?

The answer lies in: it precisely targets the core challenge of the JEPA world model that LeCun has long bet on — representation collapse.

Paper link: https://arxiv.org/abs/2606.02572

Code / pre-trained weights: https://github.com/HaiyuWu/visreg

Project homepage: https://haiyuwu.github.io/visreg/

VISReg decouples the regularization term for collapse prevention into two independent objectives: "scale" and "shape". Without relying on any heuristic training tricks or massive datasets, it outperforms 7 mainstream self-supervised learning methods across 15 datasets. Notably, using only about 1/10 of the training data, it matches the performance of DINOv2 on out-of-distribution (OOD) benchmarks.

Figure 2: Simulation of gradient magnitude ‖∇ℒ‖ across different stages of representation collapse for different regularization methods. VISReg maintains strong gradients even in the collapsed state, while SIGReg's gradients nearly vanish.

Core Methodology

VISReg combines the strengths of VICReg and SIGReg: it retains VICReg's variance term to control scale, while replacing the covariance term with a sketching objective based on Sliced Wasserstein Distance (SWD) to control shape, and completely decouples the two via stop-gradient. The entire regularization objective consists of three components.

1. Scale Regularization

The first component constrains the variance of each dimension to prevent magnitude collapse:

Its key property is that: when the model collapses, the gradient of this term approaches a constant, ensuring the model can recover stably — this exactly compensates for the vanishing gradient flaw of SIGReg.

2. Shape Regularization

The second component first normalizes to eliminate the influence of scale, then separately constrains the shape. The critical step is normalization with "stop-gradient" (sg):

Here, stop-gradient is applied to the standard deviation σ, so that the optimization of the shape loss will not alter the scale in return — this is the mechanism by which the two objectives of "scale" and "shape" are truly decoupled without interfering with each other.

After normalization, the Sliced Wasserstein Distance is used to align the geometric shape of the distribution to an isotropic Gaussian:

where

is the standard Gaussian quantile,

is a random projection direction (i.e., "slicing / sketching").

The theoretical basis is the Cramér–Wold theorem (Lemma 3.1 in the paper): two distributions are equal if and only if their 1D projections along all directions on the unit sphere are equal. Therefore, aligning the high-dimensional representations along a sufficient number of random 1D directions to the Gaussian one by one is equivalent to aligning the entire distribution in the high-dimensional space — this allows using low-cost 1D sorting operations to characterize the complete distribution shape, rather than just second-order statistics.

3. Combined Objective

The third component is a centering loss that pulls the batch mean μ towards the origin

The three regularization terms are combined with weights:

The prediction loss adopts the invariance objective of JEPA / LeJEPA — making the embeddings of all views (global + local, total V views)

align to the mean of the global view

:

Finally, a single hyperparameter λ is used to balance between prediction and regularization, obtaining the full objective:

Comparison with VICReg: VICReg also decouples regularization into variance + covariance, but covariance only characterizes second-order statistics; VISReg uses a sliced Wasserstein-based sketching objective to fully characterize the distribution shape, while retaining the variance term for scale control — preserving the flexibility of VICReg while achieving rigor at the distribution level.

Only about 15 lines of PyTorch code required

This regularization objective is extremely lightweight to implement, with the core logic requiring only about 15 lines:

Computational Complexity and Scalability

VISReg also has advantages in computation and scalability. The computational complexity of its regularization part is

(N for batch size, D for dimension, K for number of slices), which is linear in all scaling factors; in contrast, the covariance term of VICReg is

, which grows quadratically with the dimension.

Under the same batch size, VISReg outperforms SIGReg in both running speed and memory usage on a single H100 GPU.

More importantly, K random slices can be distributed across multiple GPUs: generating K/M slices on each of M GPUs yields the same effect as generating all K slices on a single GPU.

In experiments, when the number of slices per GPU is insufficient, using 8 GPUs with 128 slices per GPU (1024 total) reduces the accuracy gap from approximately 2.4% to 0.22% compared to "1024 slices on a single GPU". This means K can remain constant when scaling up training, barely increasing the burden on individual GPUs.