One Transformer Block Can Do the Work of Many Layers
A new approach lets a single Vision Transformer block repeat itself at different \\\\\\\\\\\\\\\"depths,\\\\\\\\\\\\\\\" matching bigger models while using far fewer parameters.
Stacking Blocks or Recycling Them?
Most modern vision transformers — neural networks that process images — work like an assembly line. An image passes through a stack of many transformer blocks, each one refining the picture's features 1. Standard Vision Transformers (ViTs) typically use dozens of these blocks, and each block stores its own separate set of weights. This works well for accuracy, but it costs a lot of memory.
Researchers Adrian Bulat, Yassine Ouali, and Georgios Tzimiropoulos asked a simpler question: what if you didn't need dozens of unique blocks at all? What if one block, repeated over and over, could do the same job?
One Block, Many Personalities
Their answer is a system they call reViT — a recurrent Vision Transformer with depth-programmed experts. Instead of giving each layer its own exclusive FFN (feed-forward network, the part of a transformer that processes information), reViT shares a small bank of expert FFNs and mixes them together differently at each repetition 1.
Think of it like a chef who has a limited set of core ingredients but creates entirely different dishes depending on how much of each ingredient they use. A continuous "normalized-depth" number tells the model exactly how to blend the experts at any given step, tracing a smooth path through the model's parameters as it runs 1.
Smaller But Still Strong
The results are striking. When trained from scratch on ImageNet-1k, a reViT-B/16 model reaches the same accuracy as DeiT III but stores roughly 70% fewer parameters 1. That means a much smaller model that still performs at the level of a much larger one.
When the researchers distilled knowledge from a large teacher model called DINOv2 using only output features, an 8-expert version of reViT kept nearly all of the teacher's accuracy on linear-probe tests — meaning you can easily train a simple classifier on top of its learned features 1. This distilled model also transferred well to other tasks like image segmentation and depth prediction.
One Model, Any Depth You Need
Perhaps the most practical feature is something the team calls elastic-depth training. Because the model learns a continuous trajectory through its expert mixtures, a single saved checkpoint can be run at multiple different depths simply by resampling along the same trajectory 1. You get one model that can flexibly trade speed for accuracy on the fly.
For situations where you need fixed performance, the recurrent block can also be "unrolled" into a traditional static graph, removing the need for on-the-fly mixing while keeping the same computational cost per step 1.
The work was posted to arXiv in October 2026 and represents a shift toward more efficient vision architectures that rethink how much repetition and parameter reuse can accomplish 3.
Key facts
- A single recurrent transformer block can match the accuracy of a full-depth vision encoder
- reViT-B/16 achieves DeiT III accuracy with about 70% fewer stored parameters
- An 8-expert distilled model retains nearly all of its DINOv2 teacher's accuracy
- Elastic-depth training lets one checkpoint operate at multiple depths
- Weight-space merging was identified as the strongest Mixture of Experts approach tested
Sources
- Paper page - One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts
- One Block, Multiple Depths: Recurrent Vision Transformers with Depth ...
- 2610.12448 One Block, Multiple Depths: Recurrent Vision Transformers ...
- One Block, Multiple Depths: Recurrent Vision Transformers with Depth ... — arxiv.deeppaper.ai
- One Block, Multiple Depths | OpenTrain AI — opentrain.ai