All-transistor transformer learns to do in-context learning

A transformer built from real circuit parts

A new research paper shows that a transformer — the architecture behind today's most powerful AI models — can be built using only physical electronic components instead of digital computer chips ¹. The device is trained through a method called equilibrium propagation, which teaches a system by sending information in both directions through the circuit, rather than by running code on a GPU .

Most hardware-trained systems built this way have struggled with tasks that require connecting related pieces of information, like remembering a rule from earlier in a sentence and applying it later. Those tasks are easy for software transformers but hard for the simpler circuit designs used so far ¹.

Solving the reverse-flow problem

The problem comes from how information flows through the hardware. In equilibrium propagation, a forward signal carries input data through the circuit, and a reverse signal carries error information back. In complex circuits that perform multiplication — which is needed for the attention mechanism at the heart of transformers — the reverse flow gets broken ¹.

The researchers, Michael C. Abbott and Benjamin B. Machta, fixed this by adding special circuit elements that restore the two-way flow of information ¹. They built an attention mechanism using the natural exponential current behavior of sub-threshold MOSFET devices, which are a type of transistor that operates at very low voltages .

Learning from context in a physical circuit

In simulations, the all-transistor circuit was trained to perform a simple in-context learning task — the ability to pick up a pattern from examples given in the prompt and apply it to a new question, without any weight updates ¹. This is one of the most useful abilities of large language models.

The training only worked when the researchers used cross-entropy loss, not the more common mean squared error ¹. To generate the "nudges" the algorithm needed for this loss function, they designed a novel floating circuit element based on the same MOSFET sub-circuit used for softmax .

The resistors in the circuit were trained using the Lion optimiser, which can be implemented using just one capacitor per parameter to store a local memory of past gradient estimates — an important detail for making the whole thing work in physical hardware ¹.

Why it matters

The result does not yet beat software transformers on large benchmarks, but it proves a principle: the kinds of reasoning and associative learning that make transformers useful can be reproduced in pure analog hardware ¹. If researchers can scale this approach, it could one day lead to AI devices that learn and adapt on-chip, without constantly sending data back to a server.

The paper was posted to arXiv on 8 October 2026 ¹.

Key facts

The transformer was built entirely from transistors using equilibrium propagation, a bi-directional learning method
Adding special circuit elements restored the reverse information flow that multiplication normally breaks
The circuit successfully performed in-context learning in simulations using cross-entropy loss
The Lion optimizer was used, storing gradient history with one capacitor per parameter
The work was posted to arXiv on 8 October 2026 by Abbott and Machta

Sources

An all-transistor physical-learning transformer can be trained to perform in-context learning