TurboMamba combines Mamba and Transformer for faster, smarter language models
A new hybrid architecture boosts speed and accuracy by mixing two kinds of AI designs.
A New Way to Make Language Models Faster
Researchers have created TurboMamba, a new design for language models that runs faster during both training and when generating text. The work was published on arXiv by Hanxin Zhang 1.
The core idea is to mix two well-known types of AI architectures: Mamba and Transformer. Mamba (also called Mamba-3 in this work) is a model that is very efficient at processing long sequences of text. Transformer is the architecture behind most popular language models today, known for its strong prediction ability. TurboMamba tries to get the best of both worlds.
How It Works
Here is the simple version: a global Mamba model runs once for a whole block of tokens — a group of words or word-parts the model processes together. A smaller Transformer then looks at the global context that Mamba produced, plus earlier tokens in the current block, to predict the next token.
Three small tweaks make the local predictions even better. First, the model can directly access the last token from the previous block. Second, it applies a small correction to the scores for the next token. Third, it learns to blend the Transformer's predictions with how often certain recent tokens appeared.
This means the heavy lifting is done by the fast Mamba model, while the smaller Transformer focuses only on fine-tuning the predictions within each block.
Better and Faster
The researchers tested twelve small-scale versions of different architectures on a public dataset called FineWeb-Edu, keeping the number of parameters and training time similar across all of them. The baseline models included Transformer, Mamba-3, Jamba, and Zamba. For Jamba and Zamba, the researchers swapped in Mamba-3 in place of their original modules 1.
TurboMamba came out on top. It achieved the lowest mean test prediction loss among all twelve models. Its prediction error was 9.4 percent lower than Mamba-3, 7.3 percent lower than the modified Jamba version, and 18.3 percent lower than the modified Zamba version 1.
Speed gains were also large. Training ran 2.60 times faster than Mamba-3, 2.81 times faster than the Jamba version, and 4.82 times faster than the Zamba version 1. For generating text, TurboMamba was 3.12 times faster than Mamba-3, 3.53 times faster than the Jamba version, and 6.21 times faster than the Zamba version on a batch of 32 sequences with 256-token prompts and 128 generated tokens .
What Matters Here
The result shows that mixing Mamba's speed with Transformer's accuracy is a promising direction. The gains are measured on small models, so the approach will need to be tested at larger scales. Still, the idea of letting a fast global model do the heavy lifting while a smaller model handles local decisions could influence how future language models are built.
Key facts
- TurboMamba combines a global Mamba-3 model with a smaller Transformer for token prediction
- It achieved 9.4 percent lower prediction loss than Mamba-3 on FineWeb-Edu
- Training throughput was 2.60 times faster than Mamba-3 and 4.82 times faster than Zamba
- Inference speedup reached 6.21 times faster than the Zamba version
- Tests used twelve small-scale models with comparable parameters and training budgets
Sources
Related stories
How it worksAll-transistor transformer learns to do in-context learningHerald of AI
All-transistor transformer learns to do in-context learning
A new physical circuit made entirely of transistors shows that hardware-level AI can perform the same kind of fast, on-the-fly learning as large language models.