The result

A low-rank correction produced the clearest quality and latency tradeoff in our latest cache-transfer experiments.

We transferred the prompt cache from Llama-3.1-Minitron-4B-Width-Base into Llama 3.1 8B, then compared four target paths: direct cache reuse, a ridge map, a validation-selected learned correction, and ordinary native prefill.

The selected learned path reduced held-out 32-step divergence from 0.20051 to 0.18138 KL32, a 9.54% improvement over direct reuse. It won on 54 of 64 paired prompts. At an 8,192-token prefix, it reached the target's first token in 38.25 ms when the source cache was already available, compared with 302.69 ms for native target prefill.

The speed claim has two boundaries

The 7.91× result measures the online handoff after Minitron has already produced a resident cache. Including Minitron's prefill, the learned path took 241.56 ms at 8K, which was still 1.25× faster than native Llama prefill.

The frozen study

Selection ended before the 64 final prompts were loaded.

Source
NVIDIA Llama 3.1 Minitron 4B Width Base
Target
Llama 3.1 8B base
Screen
20 configurations
Confirmation
Top 2 configurations · 3 fixed seeds each
Final quality set
64 untouched WikiText-103 prompts
Quality path
256-token prefix · 32 forced steps
Latency study
6 prefix lengths · 20 warm-ups · 50 repetitions
Measurement
CUDA-synchronized p50 on H100

The screen varied direct and ridge bases, linear and quadratic residuals, ranks 4 through 32, three learning rates, feature choices, and KV-MSE warm-up. The six strongest arms entered checkpoint and residual-scale evaluation. The top two were retrained on a disjoint partition with three fixed seeds. Only then did we freeze one configuration and open the final set.

The learned handoff

The useful correction was a rank-8 residual on direct KV, not a full ridge translation.

The winning method injected the Minitron cache directly, then added a learned linear low-rank correction to each key and value cache. The correction used rank 8, 1,048,576 learned parameters, a 20-step cache-reconstruction warm-up, and a residual scale of 0.75. Its checkpoint was only 12.0 MiB.

This matters because the seemingly more structured baseline failed. A per-head ridge map raised final KL32 to 0.37489 and reduced top-1 agreement to 75.10%. It lost to direct reuse on 60 of 64 prompts and was slower online. For this cross-width pair, direct injection preserved information that the ridge projection destroyed, while a compact residual could correct part of the remaining trajectory error.

Controlled Minitron to Llama screening comparisons across learning rate, rank, base method, and KV-MSE warm-up. Direct-base residuals form the strongest validation cluster, while ridge-base variants are consistently worse.
Controlled slices from the 20-arm screen. Lower KL32 is better. Direct-base residuals, especially the selected rank-8 linear arm, dominate the ridge-base alternatives. Scroll the chart horizontally to inspect each panel.

Held-out quality

The gain over direct reuse was small, paired, and statistically stable.

On the untouched final set, the learned-minus-direct KL32 difference was -0.01913 ± 0.00378 standard error. Its 95% interval ran from -0.02654 to -0.01172, and the learned method improved 54 of 64 paired prompts. Top-1 agreement rose by 0.684 percentage points, from 81.84% to 82.52%.

Native Llama prefill0 KL32100% top-1 reference
Learned direct + residual0.18138 KL3282.52% top-1
Direct cache reuse0.20051 KL3281.84% top-1
Ridge map0.37489 KL3275.10% top-1
Held-out quality chart for ridge, learned, and native paths. The learned method has lower KL32 and higher top-1 agreement than ridge. Native prefill is the exact zero-KL reference.
Quality on 64 untouched WikiText-103 prompts. Bars show mean KL32 with standard-error whiskers, where lower is better. The line shows 32-step top-1 agreement, where higher is better. Native is the exact target-cache reference by definition.

Long-context latency

The handoff crossed native prefill near 1K tokens and widened the lead with context length.

Native target prefill grew from 13.40 ms at 128 tokens to 302.69 ms at 8K. The learned handoff stayed near 30 ms through 4K and reached 38.25 ms at 8K. With a resident source cache, it became faster than native prefill around 1K tokens and delivered a 7.91× speedup at 8K.

Direct reuse was faster still at 16.99 ms, but it gave up the learned method's held-out quality improvement. The result is a measurable frontier: direct reuse for minimum latency, the learned residual for better cache fidelity at modest fixed overhead, and native prefill for exact target behavior.

P50 target time-to-first-token across prefix lengths from 128 to 8192 tokens. Native prefill rises sharply to 302.69 milliseconds. Direct, ridge, and learned cache handoffs remain below 47 milliseconds, with the learned handoff at 38.25 milliseconds at 8K.
CUDA-synchronized p50 target TTFT with the source cache already resident. Each point follows 20 warm-ups and 50 measured repetitions on real FineWeb tokens. Source prefill is excluded from all transfer lines and included separately in the cold-context comparison.
Resident source cache, 8K38.25 ms learned302.69 ms native · 7.91× faster
Cold context, 8K241.56 ms learned302.69 ms native · 1.25× faster

What this establishes

Long-context cache handoff is now a measured systems tradeoff, not a shape-compatibility demo.

This study combines a frozen selection protocol, paired held-out quality, synchronized latency, and both resident-cache and cold-context accounting. It shows that a small learned residual can improve a directly reused cache while preserving a large long-context latency advantage over target prefilling.

The boundary is just as important. KL32 uses 32 oracle-forced tokens, not open-ended generation, task accuracy, or human preference. The source is a width-pruned and distilled derivative of the target lineage, not an unrelated checkpoint. Native prefill remains exact and is faster at short prefixes. The learned path also uses more incremental VRAM at 8K than native prefill, 9.34 GiB versus 7.96 GiB.

The next gate is functional generation on long, diverse prompts, followed by an integrated serving test that includes cache transport across the actual model-routing boundary.