Nvidia researchers have developed a method to transfer the key-value cache between different large language models using simple linear regression, avoiding the costly recomputation that currently happens when agentic AI systems switch between models mid-conversation. The technique addresses a major performance bottleneck for enterprises running long-horizon workflows that involve multiple LLMs, where each model switch forces the receiving model to re-execute its entire prefill stage, driving up compute costs and latency. The researchers presented their findings in a study that could make multi-model AI pipelines significantly more efficient for U.S. companies deploying these systems at scale.
The problem stems from how LLMs handle memory. When a model receives a prompt, it runs a prefill stage that computes keys and values for all input tokens and populates the key-value cache, which it then reads during the decode phase to generate new tokens. Because different models have different architectures and expect cache inputs in different formats, swapping models invalidates the cache and forces the new model to recompute the entire context from scratch. This becomes increasingly expensive as conversations grow longer, as prefill costs scale with both model size and input length.
The Nvidia team found that cross-model key-value cache transfer is a highly linear structure, meaning the mapping can be done with basic algebra rather than expensive deep learning. For example, when transferring from a 14-billion parameter Qwen3 model to a 32-billion parameter version, a simple linear regression from one source layer to one target layer recovered 56% of the variance in target keys and 32% in target values, with combined source layers pushing those figures to 79% and 65%. The researchers built a closed-form per-head ridge mapper with three components: per-head linear regression fitted on a few hundred calibration sequences, cross-layer source selection to pick the most predictive layers, and content-space mapping that strips positional encodings to generalize to longer sequences.
The study evaluated the technique across six matched-key-value model pairs, including Qwen3, Llama 3.1, and Ministral 3 families, with parameter sizes ranging from 3 billion to 70 billion. Tests included a large leap from Llama 3.1 8B to 70B, an 8.8x parameter increase. The researchers used five accuracy benchmarks, a language modeling perplexity test, and a multi-turn conversation dataset, training the mapper on just 500 text sequences of 1,024 tokens each.
Results showed that on four of the six pairs, the linear ridge mapper retained between 73% and 98% of the target model鈥檚 standalone prefill accuracy, including the 8B-to-70B transfer, which retained 72.8% of target accuracy. The linear mapping process ran 2.7 to 25 times faster than recomputing the conversation. The technique enables both small-to-large transfers to improve output quality and large-to-small transfers to cut compute costs, such as using a large model to process a dense system prompt then mapping its cache down to a smaller model for rapid conversational turns.
Previous attempts at solving this problem required expensive gradient-based training or strict architectural constraints. The current study limited itself to within-family transfers, where models share tokenizers and core architectural styles, but the researchers noted the approach could eventually extend to cross-family transfers, mismatched key-value head counts, or hybrid architectures.
More AI news from TechManNews.
