Sakhanda Wire
NVDA $153.93 -1.41% MSFT $506.53 -2.21% GOOGL $280.96 -0.97% META $620.53 -0.94% AMZN $256.87 -0.34%
← Back to the news

Researcher Proposes Transformer That Carries State Across Every Token

Princeton researcher Yifan Zhang has proposed the Recurrent Looped Transformer (RLT), a design in which the decoder carries its final hidden state and each layer’s sliding-window attention cache from one token to the next. The state continues across both the prompt and the response, without resetting at their boundary. RLT combines a causal encoder with a recurrent decoder, and its next-token prediction is read from the decoder’s final output.

The reference configuration uses 48 encoder layers and 48 decoder layers with tied weights, producing 96 logical blocks for each token. The decoder retains at most W – 1 historical entries per layer, while the state path grows structurally with the sequence. The report describes parallel processing opportunities in the encoder and batching across independent sequences, but assumes no exact parallel scan for the nonlinear decoder and claims no reduced-prefill speedup.

The proposal also specifies training and reinforcement-learning replay procedures. Replay reconstructs encoder memory, decoder outputs, and all attention caches under current parameters while retaining recorded behavior log-probabilities. The report presents the architecture and execution rules, but includes no measured results for efficiency, reasoning quality, or scaling.

Written by Sakhanda Wire AI from reporting by MarkTechPost. Facts, quotes and full context are in the original article below.

Originally published by MarkTechPost on

Read the original on MarkTechPost ↗

The reporting and the image belong to MarkTechPost and are credited and linked above; the text on this page is Sakhanda Wire's own summary.

← Back to the news

More stories

All the latest news