A learned KV handoff cut 8K target time-to-first-token by 87%.
A frozen Minitron 4B to Llama 3.1 8B study made an 8K target handoff 7.9× faster than native prefill while improving held-out cache quality over direct reuse.
Read blogFeatured
The small model already did the reading.
Before a model answers, it pays to read. We are making that prefill step cheaper by letting a smaller model read first, then carrying the work into Qwen and Llama targets.
Read blog All research notes
A rollout residual cut 30B sibling-transfer divergence by 6.5%.
A frozen Qwen3-30B-A3B sibling test improved 20 of 28 paired windows, while a matching-layer ridge map produced the lowest mean divergence.
Read blogAlignment first cut 32-step KV-transfer divergence by 11.1%.
Nine overnight Qwen3 sibling-transfer experiments showed that ridge alignment followed by rollout-aware correction produced our strongest 4B result.
Read blogPartial recomputation reached 100% next-token agreement.
Recomputing two target layers removed the nonlinear ceiling of an affine cache map across 480 held-out toy-model comparisons.
Read blogThe mapped cache retained 91.2% of task quality.
Our Qwen3 1.7B to 4B map retained 91.2% of chance-normalized task quality and reached 65% next-token agreement.
Read blogA ten-point sweep revealed three mapper tradeoffs.
A ten-point sweep picked k=24 on prefix loss while attention similarity peaked at k=8 and mapping time kept climbing.
Read blogMoving a KV cache from Qwen3 0.6B to 1.7B.
A learned map retained 85.1% of chance-normalized task quality and improved next-token agreement 30× over direct injection.
Read blog