Question
Can the larger model skip prefill?
Inference normally makes every model compute the prompt for itself. V0 tested a different path: prefill with Qwen3 0.6B, remove its source positional encoding, map the cached keys and values head by head, apply the target positions, and inject that state into Qwen3 1.7B before decode.
The useful result is not whether a tensor can be moved. It is whether the target model still behaves like the target model after the handoff.
Experimental setup
One H100, disjoint fitting and evaluation data.
- Source
- Qwen3 0.6B
- Target
- Qwen3 1.7B
- Hardware
- 1× NVIDIA H100 SXM5 80 GB
- Precision
- BF16
- Mapper fit
- 50 training sequences
- Selection
- 25 disjoint sequences
- Selected map
- Per-head linear map, k=1
- Task checks
- HellaSwag and PIQA
Before evaluating the transfer, the harness checked model and cache compatibility, cache serialization, BF16 cached decode, corruption detection, RoPE round trips, and deterministic mapper serialization. Those controls passed.
Results
The learned map recovered most task performance.
| Variant | Perplexity | Next-token top-1 | HellaSwag | PIQA |
|---|---|---|---|---|
| Target full-prefill oracle | 15.2157 | 100% | 59.4% | 74.0% |
| Direct source-cache injection | 44.2169 | 2% | 36.6% | 60.0% |
| Learned mapped transfer, k=1 | 17.3496 | 61% | 54.6% | 70.2% |
| Stabilized mapped transfer, k=4 | 17.0744 | 62% | 51.2% | 68.4% |
The selected k=1 map reduced perplexity from 44.2169 with direct injection to 17.3496 and raised next-token agreement from 2% to 61%, a 30× gain. It also recovered much of the oracle’s held-out HellaSwag and PIQA accuracy. A subsequent stabilized k=4 run reached 62% agreement, 17.0744 perplexity, and 76.4% normalized task retention, giving the next study two distinct mapper configurations to compare on fresh samples.
Interpretation
What “85.1% retained” means.
The headline is the mean task retention after normalizing each benchmark against its chance floor. A model does not get credit for performance that random guessing would produce. On this run, the mapped cache scored 54.6% on HellaSwag against the target oracle’s 59.4%, and 70.2% on PIQA against 74.0%.
This makes the number more comparable across multiple-choice tasks with different random baselines. Token-level agreement is reported separately so task retention and immediate fidelity remain visible together.
Result and roadmap
85.1% retained with a clear fidelity roadmap.
V0 reached the predeclared task-retention floor, stayed within the 1.25× perplexity limit, and improved every metric over direct injection. Next-token agreement reached 61%, establishing a 19-point path to the 80% milestone and a focused objective for multistep rollout work.
The protocol schedules formal latency acceptance after fidelity optimization. V0 establishes that cross-model transfer can preserve useful task performance and identifies the measurements the next model-pair study will optimize.
Next
Expand fidelity, then quantify savings.
The next iteration expands calibration data, compares k=1 and k=4 on fresh disjoint samples, and tests richer mapping strategies. Larger compatible model pairs will make the latency study more representative of the inference savings this technique is intended to unlock.
The experiment sequence is task-level non-inferiority on representative traffic, stronger next-token agreement, stable rollouts, and then a synchronized latency comparison against ordinary target prefill.