H-JEPA Lets AI Plan at Multiple Levels of Abstraction

Building a mental map that thinks in layers

H-JEPA is a new AI technique that helps machines plan over long stretches of time by organizing their thinking into separate layers of detail. Instead of trying to handle every small movement and big-picture goal in one giant brain, H-JEPA splits the job into a hierarchy where each level handles its own scale of time.

The idea builds on an earlier system called JEPA, short for Joint-Embedding Predictive Architecture. A JEPA model learns to predict what comes next by comparing encoded versions of the current situation and future states ¹. But standard JEPA works at a single timescale — it either predicts the near future in fine detail or the distant future at a high level, but not both well at the same time.

H-JEPA fixes this by stacking multiple JEPA models on top of each other. Each higher level learns to look farther ahead in its own abstract space. Fast-changing details like leg movements get filtered out at the top, while slower, more important information like overall position stays clear ².

How the hierarchy works

Planning with H-JEPA flows from top to bottom. The highest level sets an abstract goal, like "reach the exit of the maze." Its predictions then become subgoals for the level below, which breaks them into smaller steps. The lowest level finally turns those steps into actual actions ².

All the levels are trained together in one shot. Gradients from the top-down objectives flow back through the lower encoders, so every layer learns to serve the whole hierarchy ³. A regularization method called SIGReg prevents the system from collapsing into a useless representation during training ².

This layered design mirrors how humans plan. You might decide to drive to a city, then figure out which roads to take, then steer the wheel — each step lives at a different level of detail.

Better results in simulated worlds

The researchers tested H-JEPA across four simulated environments, including navigation and manipulation tasks ³. On Visual AntMaze, a three-level hierarchy boosted success rates from 18 percent to 73 percent, while using less planning compute than a flat JEPA model ².

Further analysis showed the improvements came from two sources: breaking planning across time scales and letting higher levels form cleaner goal representations ³. Adding inverse-dynamics supervision also improved performance on real-robot videos from the DROID dataset, with better offline planning fidelity at lower compute cost .

Code for H-JEPA has been released as a PyTorch implementation, including a masked-patch self-supervised learning variant and an end-to-end training option ⁵.

Key facts

• H-JEPA stacks multiple JEPA models into a hierarchy for multi-scale planning
• On Visual AntMaze, a three-level hierarchy raised success from 18% to 73%
• Each higher level predicts farther ahead in its own learned latent space
• The approach also improved planning on real-robot videos from the DROID dataset
• All levels are trained end-to-end with gradients flowing through the hierarchy

Sources

1. H-JEPA teaches world models to plan at multiple levels of abstraction - VentureBeat — venturebeat.com
2. H-JEPA: Hierarchical World Models for Visual Planning
3. 2610.06805 H-JEPA: End-to-End Learning of Hierarchical World Models ...
4. H-JEPA: End-to-End Learning of Hierarchical World Models for Visual ...
5. GitHub - jonwiggins/H-JEPA: PyTorch implementation of Hierarchical ...