The result
Stop translating where the models stop agreeing.
A KV cache is not portable text memory. It is checkpoint-specific intermediate state. Even when two models have the same architecture and cache shape, their keys and values can mean different things.
Our affine mapper reduced that mismatch substantially, but it could not undo the nonlinear representation drift that accumulated in deeper layers. The strongest result came from changing the strategy: reuse the exact shared state, map the easy boundary, and let the target model recompute the layers where its own representation matters.
That hybrid reached 100% next-token agreement in all 480 held-out comparisons. Here, agreement means matching the target model's own full-prefill next-token choice, not matching a human label. Mean relative logit error fell from 5.20% with affine mapping to 0.00000142%.
The controlled test
Two small models made the failure mechanism visible.
We used two randomly initialized, four-layer decoder-only transformers with identical architecture. The target began as a copy of the source, then layers 1 through 3 and the language-model head were perturbed. Embeddings and layer 0 remained byte-identical, creating a known boundary where reuse should be exact.
- Architecture
- 4 layers · 24 hidden dimensions
- Model difference
- Layers 1–3 perturbed at α=0.12
- Mapper fit
- 192 random prompts per seed
- Evaluation
- 48 disjoint prompts per seed
- Validation
- 10 seeds · 480 comparisons
- Behavior checked
- One-token target decode
At each layer, separate affine maps translated source keys and values into the target basis. Keys were first moved out of their rotary-position basis, mapped in position-independent space, then rotated back at the original position.
Why mapping stalled
The first changed layer was linear. The deeper mismatch was not.
Layer 1 receives the same hidden state in both models because everything below it is shared. Its source and target keys are just two linear projections of identical input, so a linear change of basis can recover the target cache almost exactly.
After that layer runs, the paths diverge. Changed attention, GELU, normalization, and residual mixing create different hidden states. Later maps are no longer translating two projections of one representation. They are trying to reconcile representations that have followed different nonlinear trajectories.
The L2 anchor improved continuous logit error on every seed, although top-1 agreement regressed by one prompt on one seed. Training and held-out errors were similar, changing the ridge penalty did not help, and even the target's exact K/V projections could not repair the wrong source hidden states. Together, those controls point to a representation ceiling rather than an underfit regression.
The recompute anchor
Two target blocks turned an approximation into a faithful handoff.
The successful path resumes target-side prefix computation at the exact output of shared layer 0. It runs target layers 1 and 2, computes the exact layer-3 K/V projections, and then begins decoding. It does not run layer 3's prefix attention, MLP, or language-model head.
This reconstructs the target hidden state that enters layer 3. Once that state is exact, the final K/V projections are exact too, and every changed-layer cache is compatible with the target model. Across ten seeds, the residual relative logit error ranged from 0.000000211% to 0.00000994%.
Pure mapping does the least target-side prefix work and accepts approximation error. A later recompute anchor buys fidelity by giving back some of the hoped-for prefill savings. The right operating point must satisfy both a quality threshold and a measured latency budget.
What this proves
The mechanism works. The economics remain an open test.
This experiment shows that selective recomputation can remove the quality ceiling of a layer-local cache map when related models expose an exact shared boundary. It also gives us a practical diagnostic: when translated state degrades in deeper layers, do not assume a more elaborate mapper is the only answer. Measure where representations diverge, then recompute from the latest trustworthy anchor.
The result does not show that arbitrary models can exchange KV caches. These toy models share a token space, architecture, embeddings, and one full layer, and their square attention projections make the first changed layer unusually easy to align. The evaluation covers one-token continuation, not a long rollout where errors can compound. It also does not measure wall-clock latency, transfer cost, kernel overhead, or distributed communication.
The next useful test is therefore not another quality-only victory. It is an end-to-end benchmark on a real model pair that sweeps the recompute boundary and measures fidelity and time to first token together.