Zyphra AI Introduces Zyphra Hierarchical Attention (ZHA): A Path to True Subquadratic Attention
Zyphra has released Zyphra Hierarchical Attention (ZHA), a sparse attention method built around a tree-structured indexer. ZHA targets a known gap in sparse attention: the indexer that picks tokens still scans the whole context. Zyphra’s technical paper shows ZHA can be distilled into frozen pretrained models with about 4B tokens. On Qwen3.6-35B-A3B it stays close to dense attention on reasoning, agentic coding and 256K retrieval benchmarks.
TL;DR
- Size: It is a method that adds 81,920 trainable parameters to Qwen3.6-35B-A3B (about 0.00024%). Tested up to 256K context.
- Runs on: Retrofits frozen pretrained models after a 4B-token KL warmup at 32K context. Speedups come from a modeled H100 BF16 roofline. No optimized kernels released. An FP4 router variant was also tested.
- Performance: Within a few points of dense attention on 7 reasoning, coding and retrieval benchmarks.
- Best: LongBench v2 hits 63.4% (deep tree) vs 58.0% dense, using about 1.1% of keys.
- Worst: On NoPE-based Nemotron-3.5-Lightning-30B-A3B, deep-tree RULER-13 drops 25.3 points below dense.
What is Zyphra Hierarchical Attention (ZHA)?
ZHA is a sparse attention method from Zyphra that finds relevant keys by searching a tree. It keeps the full KV cache. The tree only decides which cached keys each query reads.
Fixed-state methods like SSMs and sliding windows saturate at long contexts. Fine-grained sparse methods like DeepSeek Sparse Attention score every past token with an indexer. The final attention is sparse, but the indexer still grows with the full context.
Near 1M tokens, cited work shows indexing takes 60 to 90% of layer time. ZHA makes the search itself sparse.
How does ZHA’s tree indexer work?
Building the tree
ZHA splits keys into contiguous chunks and compresses each chunk into a summary key. Summaries are grouped and compressed again, level by level. With chunk size 4, 6 levels cover 4,096 tokens per top node.
Summaries come from a small set of learned reference queries. Each reference query gives a tangent-plane estimate of a chunk’s log attention mass. The max over these planes approximates how much attention any future query would give the chunk.
How does top-p routing pick tokens?
At decode time, a query scores the top-level nodes. It keeps the smallest set holding a fraction p of estimated attention mass. It then expands only those children, repeating until it reaches raw KV entries.
Top-p makes the budget adaptive. A single-fact lookup touches few branches. A query that needs many scattered facts expands more. The first token is always kept as an attention sink.
How does ZHA perform against dense attention and MSA?
Zyphra team converted Qwen3.6-35B-A3B with 2 trees: deep (chunk 4, 6 levels) and wide (chunk 256, 1 level). Both use 16 reference queries and p = 0.95. It also trained a MiniMax Sparse Attention (MSA) indexer on 40B tokens, 10x ZHA’s budget.
| Benchmark (Qwen3.6-35B-A3B) | Dense | ZHA deep | ZHA wide | MSA |
|---|---|---|---|---|
| AIME 2026 | 93.1 | 91.2 | 90.8 | 91.7 |
| SWE-bench Verified | 67.4 | 67.4 | 68.4 | 63.6 |
| Terminal-Bench 2.1 | 49.9 | 46.7 | 53.3 | 46.1 |
| MMLU-Pro | 84.9 | 85.3 | 85.2 | 85.3 |
| GPQA Diamond | 84.8 | 85.0 | 85.0 | 83.1 |
| RULER-13 (256K) | 96.0 | 93.3 | 96.2 | 77.1 |
| LongBench v2 (thinking) | 58.0 | 63.4 | 62.4 | 49.3 |
Source: arXiv 2610.42112, Figure 5. All numbers self-reported by Zyphra.
The biggest gap is on retrieval. MSA drops to 77.1% on RULER-13, while both ZHA trees stay above 93%. On LongBench v2, the deep tree and MSA read similar key shares (1.1% vs 1%). The deep tree scores 14.1 points higher.
Across 28 tree geometries, RULER accuracy held steady when attending to 10 to 30% of keys. At fixed compression, deeper trees kept accuracy at higher sparsity. A 4-parameter model predicts RULER accuracy from geometry with 1.44-point RMSE.
Back</button><button class="btn pri" id="znext">Next step
</button><button class="btn" id="zplay">Auto-play</button><div class="dots" id="zdots"></div></div>
</div></div>
<div class="pn" id="zp2"><div class="box">
<div class="sel" id="zbsel"></div>
<div id="zbars"></div>
<p class="note">Qwen3.6-35B-A3B, p = 0.95, R = 16. Deep = chunk 4, 6 levels. Wide = chunk 256, 1 level. MSA = top-16 blocks of 128, trained on 40B tokens. Self-reported by Zyphra.</p>
</div></div>
<div class="pn" id="zp3"><div class="box">
<div class="sel" id="zksel"></div>
<div id="zkbars"></div>
<p class="note">Share of KV keys attended during decoding Lower is sparser. Top-p lets the deep tree read far fewer keys on retrieval than on diffuse reasoning tasks.</p>
</div></div>
<div class="pn" id="zp4"><div class="box">
<svg id="zlat" viewBox="0 0 800 300" role="img" aria-label="Modeled decode latency vs sequence length"></svg>
<div class="grid2" style="margin-top:10px">
<div class="kpi"><div class="n">~48K</div><div class="t">tokens where modeled ZHA overtakes dense attention</div></div>
<div class="kpi"><div class="n">~0.48M</div><div class="t">tokens where modeled ZHA overtakes MSA</div></div>
<div class="kpi"><div class="n">373 to 417x</div><div class="t">projected speedup over dense at 1B tokens</div></div>
<div class="kpi"><div class="n">52 to 58x</div><div class="t">projected speedup over MSA at 1B tokens</div></div>
</div>
<p class="note">Roofline model: H100, BF16, 1 layer, batch size 1, ZHA R = 1, C = 8, p = 0.75. Not measured kernels. Curves are schematic, anchored to the reported crossovers.</p>
</div></div>
<div class="pn" id="zp5"><div class="lim" id="zlim">
<div class="li"><b>Speedups are modeled, not measured.</b><div class="d">Zyphra released no optimized kernels. Latency beyond 256K comes from a roofline projection. At 256K, frontier settings model only 1.93x to 2.28x over dense.</div></div>
<div class="li"><b>NoPE models lose retrieval accuracy.</b><div class="d">On Nemotron-3.5-Lightning-30B-A3B, RULER-13 at p = 0.95 drops 25.3 points (deep) and 10.2 points (wide) below dense.</div></div>
<div class="li"><b>Top-p must scale with depth.</b><div class="d">A fixed per-level p shrinks effective coverage as trees grow deeper. Too high a p removes pruning and returns to quadratic cost.</div></div>
<div class="li"><b>Small models struggle.</b><div class="d">Models under 1B parameters converted poorly. Results improved sharply at 4B and above.</div></div>
<div class="li"><b>Terminal-Bench 2.1 dip.</b><div class="d">The deep tree scores 46.7% vs 49.9% dense. The wide tree scores 53.3%.</div></div>
</div><p class="note">Tap a card to expand.</p></div>
<div class="ft"><span>Sources: <a href="https://arxiv.org/abs/2610.42112" target="_blank" rel="noopener">arXiv 2610.42112</a> · <a href="https://zyphra.com/our-work/zyphra-hierarchical-attention" target="_blank" rel="noopener">Zyphra blog</a></span><span class="mtp">© Marktechpost</span></div>
<script>
(function(){
var R=document.getElementById('mtp-zha');
function post(){try{parent.postMessage({mtpH:R.offsetHeight+40,type:'mtp-zha-h'},'*')}catch(e){}}
R.querySelectorAll('.tb').forEach(function(b){b.addEventListener('click',function(){
R.querySelectorAll('.tb').forEach(function(x){x.classList.remove('on')});b.classList.add('on');
R.querySelectorAll('.pn').forEach(function(x){x.classList.remove('on')});
R.querySelector('#z'+b.dataset.p).classList.add('on');
if(b.dataset.p==='p2')drawB();if(b.dataset.p==='p3')drawK();if(b.dataset.p==='p4')drawL();
setTimeout(post,60);setTimeout(post,500);});});
/* ---------- tree ---------- */
var NS='http://www.w3.org/2000/svg',T=R.querySelector('#ztree');
function el(n,a,p){var e=document.createElementNS(NS,n);for(var k in a)e.setAttribute(k,a[k]);(p||T).appendChild(e);return e}
var L3=[],L2=[],L1=[],ED=[];
var y3=40,y2=125,y1=205,y0=270;
for(var i=0;i<4;i++){L3.push({x:100+i*200,y:y3});}
for(var i=0;i<16;i++){L2.push({x:40+i*48,y:y2,p:Math.floor(i/4)});}
for(var i=0;i<64;i++){L1.push({x:28+i*11.7,y:y1,p:Math.floor(i/4)});}
L2.forEach(function(n,i){ED.push({a:L3[n.p],b:n,lv:2,ci:i,pi:n.p})});
L1.forEach(function(n,i){ED.push({a:L2[n.p],b:n,lv:1,ci:i,pi:n.p})});
ED.forEach(function(e){e.el=el('line',{x1:e.a.x,y1:e.a.y,x2:e.b.x,y2:e.b.y,stroke:'#2a2a2a','stroke-width':1,class:'edge'})});
var kv=[];for(var i=0;i<64;i++){kv.push(el('rect',{x:24+i*11.7,y:y0-8,width:8,height:16,rx:2,fill:'#222',class:'node'}))}
L1.forEach(function(n){n.el=el('circle',{cx:n.x,cy:n.y,r:4,fill:'#333',class:'node'})});
L2.forEach(function(n){n.el=el('circle',{cx:n.x,cy:n.y,r:8,fill:'#333',class:'node'})});
L3.forEach(function(n){n.el=el('circle',{cx:n.x,cy:n.y,r:14,fill:'#333',class:'node'})});
['Level 3','Level 2','Level 1','KV cache'].forEach(function(t,i){var t1=el('text',{x:4,y:[y3,y2,y1,y0][i]-20,fill:'#9a9a96','font-size':11,'font-family':'Inter,sans-serif'});t1.textContent=t;});
var q=el('g',{});var qc=el('circle',{cx:400,cy:-30,r:13,fill:'#E7620F'},q);var qt=el('text',{x:400,y:-26,'text-anchor':'middle','font-size':12,'font-weight':700,fill:'#0A0A0A','font-family':'Inter,sans-serif'},q);qt.textContent='q';
var K3=[1],K2=[5,6],K1=[21,22,25],keep0={};K1.forEach(function(c){for(var j=0;j<4;j++)keep0[c*4+j]=1});
var C={off:'#333',sc:'#5a3a22',kp:'#E7620F',e:'#2a2a2a',ek:'#E7620F'};
var caps=[
'<b>Step 1 · Build.</b> Raw keys are grouped into chunks. Each chunk is compressed into summary keys using 16 learned reference queries. Summaries are compressed again, level by level.',
'<b>Step 2 · Score the top.</b> A new query scores only the top-level summaries, not every token.',
'<b>Step 3 · Top-p prune.</b> It keeps the smallest set of nodes holding a fraction p of estimated attention mass. The rest are dropped as whole regions.',
'<b>Step 4 · Descend.</b> Only children of kept nodes are scored. Branches can sit far apart in the context.',
'<b>Step 5 · Attend.</b> Exact attention runs over the selected raw KV entries, plus the first token (sink) and the current chunk. The full KV cache is still stored.'];
var st=0,dots=R.querySelector('#zdots');caps.forEach(function(){var d=document.createElement('span');d.className='dot';dots.appendChild(d)});
function paint(){
var all=[].concat(L3,L2,L1);all.forEach(function(n){n.el.setAttribute('fill',C.off);n.el.setAttribute('opacity',1)});
ED.forEach(function(e){e.el.setAttribute('stroke',C.e);e.el.setAttribute('opacity',1)});
kv.forEach(function(k,i){k.setAttribute('fill','#222')});
qc.setAttribute('cy',-30);qt.setAttribute('y',-26);
if(st===0){all.forEach(function(n){n.el.setAttribute('fill','#4a4a46')});kv.forEach(function(k){k.setAttribute('fill','#5a5a56')});}
if(st>=1){qc.setAttribute('cy',14);qt.setAttribute('y',18);L3.forEach(function(n){n.el.setAttribute('fill',C.sc)});}
if(st>=2){L3.forEach(function(n,i){if(K3.indexOf(i)>=0)n.el.setAttribute('fill',C.kp);else n.el.setAttribute('opacity',.35)});}
if(st>=3){L2.forEach(function(n,i){if(K3.indexOf(n.p)>=0)n.el.setAttribute('fill',K2.indexOf(i)>=0?C.kp:C.sc);else n.el.setAttribute('opacity',.25)});
ED.forEach(function(e){if(e.lv===2&&K3.indexOf(e.pi)>=0&&K2.indexOf(e.ci)>=0)e.el.setAttribute('stroke',C.ek);});
L1.forEach(function(n,i){if(K2.indexOf(n.p)>=0)n.el.setAttribute('fill',K1.indexOf(i)>=0?C.kp:C.sc);else n.el.setAttribute('opacity',.25)});
ED.forEach(function(e){if(e.lv===1&&K2.indexOf(e.pi)>=0&&K1.indexOf(e.ci)>=0)e.el.setAttribute('stroke',C.ek);});}
if(st>=4){kv.forEach(function(k,i){if(keep0[i]||i===0||i>=60)k.setAttribute('fill',i===0||i>=60?'#F4F4F2':C.kp);else k.setAttribute('fill','#1a1a1a')});}
R.querySelector('#zcap').innerHTML=caps[st];
dots.querySelectorAll('.dot').forEach(function(d,i){d.classList.toggle('on',i===st)});
}
var tm=null;
R.querySelector('#znext').onclick=function(){st=(st+1)%caps.length;paint()};
R.querySelector('#zprev').onclick=function(){st=(st+caps.length-1)%caps.length;paint()};
R.querySelector('#zplay').onclick=function(){var b=this;if(tm){clearInterval(tm);tm=null;b.textContent='Auto-play';return}b.textContent='Pause';tm=setInterval(function(){st=(st+1)%caps.length;paint()},2200)};
paint();
/* ---------- benchmarks ---------- */
var BD={'AIME 2026':[93.1,91.2,90.8,91.7],'SWE-bench Verified':[67.4,67.4,68.4,63.6],'Terminal-Bench 2.1':[49.9,46.7,53.3,46.1],'MMLU-Pro':[84.9,85.3,85.2,85.3],'GPQA Diamond':[84.8,85.0,85.0,83.1],'RULER-13 (256K)':[96.0,93.3,96.2,77.1],'LongBench v2':[58.0,63.4,62.4,49.3]};
var SN=['Dense','ZHA deep','ZHA wide','MSA'],SC=['#6b6b6b','#E7620F','#f2b27a','#4a7bd0'];
var cb='RULER-13 (256K)';
function chips(id,keys,cur,fn){var w=R.querySelector(id);w.innerHTML='';keys.forEach(function(k){var c=document.createElement('button');c.className='chip'+(k===cur?' on':'');c.textContent=k;c.onclick=function(){fn(k)};w.appendChild(c)})}
function bars(id,names,vals,cols,max,suf){var w=R.querySelector(id);w.innerHTML='';names.forEach(function(n,i){var r=document.createElement('div');r.className='row';r.innerHTML='<span class="lbl">'+n+'</span><div class="trk"><div class="bar" style="background:'+cols[i]+'"></div></div><span class="val">'+(suf==='%'&&vals[i]>=10?vals[i].toFixed(1):vals[i])+suf+'</span>';w.appendChild(r)});
requestAnimationFrame(function(){requestAnimationFrame(function(){w.querySelectorAll('.bar').forEach(function(b,i){b.style.width=Math.max(1.5,vals[i]/max*100)+'%'})})})}
function drawB(){chips('#zbsel',Object.keys(BD),cb,function(k){cb=k;drawB()});bars('#zbars',SN,BD[cb],SC,100,'%')}
var KD={'AIME 2026':[5.7,47,8.7],'MMLU-Pro':[16,81,85],'GPQA Diamond':[9,60,21],'RULER-13 (256K)':[2.8,42,0.76],'LongBench v2':[1.1,21,1]};
var ck='RULER-13 (256K)';
function drawK(){chips('#zksel',Object.keys(KD),ck,function(k){ck=k;drawK()});bars('#zkbars',['ZHA deep','ZHA wide','MSA'],KD[ck],['#E7620F','#f2b27a','#4a7bd0'],100,'%')}
/* ---------- latency ---------- */
var LT=R.querySelector('#zlat');
function drawL(){LT.innerHTML='';var x0=60,x1=780,yb=260,yt=20;
function X(l){return x0+(l-3)/6*(x1-x0)}function Y(l){return yb-(l+2)/5*(yb-yt)}
function e(n,a){var t=document.createElementNS(NS,n);for(var k in a)t.setAttribute(k,a[k]);LT.appendChild(t);return t}
for(var i=3;i<=9;i++){e('line',{x1:X(i),y1:yt,x2:X(i),y2:yb,stroke:'#1f1f1f'});var t=e('text',{x:X(i),y:yb+18,'text-anchor':'middle',fill:'#9a9a96','font-size':11,'font-family':'Inter,sans-serif'});t.textContent='10^'+i;}
var t2=e('text',{x:x0-8,y:yt+4,'text-anchor':'end',fill:'#9a9a96','font-size':11,'font-family':'Inter,sans-serif'});t2.textContent='slower';var t3=e('text',{x:x0-8,y:yb,'text-anchor':'end',fill:'#9a9a96','font-size':11,'font-family':'Inter,sans-serif'});t3.textContent='faster';
var lab=e('text',{x:(x0+x1)/2,y:yb+36,'text-anchor':'middle',fill:'#9a9a96','font-size':11,'font-family':'Inter,sans-serif'});lab.textContent='Sequence length (tokens, log scale)';
function curve(f,col,dash,name,ly){var d='';for(var l=3;l<=9;l+=.05){d+=(d?'L':'M')+X(l).toFixed(1)+' '+Y(f(l)).toFixed(1)}var p=e('path',{d:d,fill:'none',stroke:col,'stroke-width':3,'stroke-dasharray':dash||''});var len=p.getTotalLength();if(!dash){p.style.strokeDasharray=len;p.style.strokeDashoffset=len;p.getBoundingClientRect();p.style.transition='stroke-dashoffset 1.4s ease';p.style.strokeDashoffset=0;}var tt=e('text',{x:X(9)-14,y:ly,'text-anchor':'end',fill:col,'font-size':12,'font-weight':700,'font-family':'Inter,sans-serif'});tt.textContent=name;}
var c48=Math.log10(48000),c48m=Math.log10(480000);
function zha(l){return -1.35+0.1*(l-3)+0.035*(l-3)*(l-3)}
function dense(l){return Math.max(-1.75,zha(c48)+0.85*(l-c48))}
function msa(l){return Math.max(-1.6,zha(c48m)+0.75*(l-c48m))}
curve(dense,'#9a9a96','5 5','Dense',Y(dense(9))-10);curve(msa,'#4a7bd0','8 4','MSA',Y(msa(9))-10);curve(zha,'#E7620F','','ZHA',Y(zha(9))+20);
[[c48,'~48K'],[c48m,'~0.48M']].forEach(function(c){e('line',{x1:X(c[0]),y1:yt,x2:X(c[0]),y2:yb,stroke:'#E7620F','stroke-opacity':.5,'stroke-dasharray':'2 4'});var t=e('text',{x:X(c[0])+4,y:yt+12,fill:'#ff8a3d','font-size':11,'font-family':'Inter,sans-serif'});t.textContent=c[1];});
}
/* ---------- caveats ---------- */
R.querySelectorAll('.li').forEach(function(li){li.addEventListener('click',function(){li.classList.toggle('open');setTimeout(post,30)})});
window.addEventListener('load',post);setTimeout(post,200);setTimeout(post,1200);
})();
</script>
</div>
</body></html>" style="width:100%;max-width:880px;height:600px;border:0;display:block;margin:0 auto;overflow:hidden" scrolling="no" loading="lazy">
How fast is ZHA?
Zyphra released no kernels. It modeled decode latency for one layer at batch size 1 with a roofline. The lightweight setting uses 1 reference query, chunk 8 and p = 0.75.
In that model, ZHA beats dense attention near 48K tokens and MSA near 0.48M tokens. At 1B tokens, projected speedups reach 373 to 417x over dense and 52 to 58x over MSA. The blog post adds 32 to 35x and 4.7 to 5.1x at 10M tokens.
At 256K, the modeled gains are modest. Pareto-frontier settings reach 1.93x to 2.28x over dense, at 92.7 to 96.4% RULER-13. Routing and selection add their own cost.
How is ZHA trained onto existing models?
The base model stays frozen. Only the reference queries train. A KL loss matches dense attention mass per chunk at every level. Qwen and Nemotron runs used 3.98B tokens over 1,900 steps at 32K context.
Zyphra reports models under 1B parameters struggled during conversion. Results improved sharply at 4B and above.
ZHA vs MSA, DSA and NSA
| Feature | ZHA (Zyphra) | MSA (MiniMax) | DSA (DeepSeek) | NSA (DeepSeek) |
|---|---|---|---|---|
| Indexer | Multi-level tree of compressed keys | Flat block indexer per GQA group | Lightning indexer, token level | Compression + selection + sliding window |
| Selection rule | Top-p per level (adaptive) | Top-16 blocks of 128 tokens | Top-2,048 tokens | 16 blocks of 64 tokens, 512 window |
| Indexer cost | Sublinear per query (claimed) | Quadratic index branch | Scores every cached key | Not disclosed |
| Added to pretrained model | Yes, ~4B tokens, backbone frozen | Yes, 400B-token CPT | Continued training of V3.1-Terminus | Native pretraining |
| Largest model shown | 35B MoE (Qwen3.6-35B-A3B) | 109B MoE, 6B active | 685B MoE | 27B MoE, ~3B active |
| Reported speedup | Modeled only (H100 roofline) | 14.2x prefill, 7.6x decode at 1M (H800) | Not disclosed | Speedups at 64K, figure only |
Key Takeaways
- ZHA searches the KV cache with a tree, not a flat scan.
- Only 81,920 parameters train on a frozen 35B model.
- Wide tree hits 96.2% on RULER-13 at 256K vs 96.0% dense.
- MSA falls to 77.1% on the same RULER test.
- Billion-token speedups are projections, not measurements.
Check out the ZHA paper on arXiv and Zyphra’s technical blog. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us