Borrowed memory

1. Long context fills the GPU with history it barely reads

Serve DeepSeek-V4.1-Flash on a 500,000-token session and watch what the GPU actually touches while it generates. At each step, the model’s sparse indexer picks 512 positions out of the whole history, and attention reads only those: about 0.1% of the session. The other 99.9% sits in HBM in case it is picked next.

That idle 99.9% is not small. The model’s global cache costs 890 bytes per token, so a 500K session carries about 445 MB of it, step after step, read or not. In our setup, one B300 decode GPU holds 46 such sessions with conventional serving. Past that, the engine stops admitting requests: not out of compute, out of memory.

In The token gap we wrote that the cache agentic inference runs on lives in HBM, the part of the GPU that got scarcer and dearer this year. This post is about making that memory go further. It is the short version of our next paper, Adaptive HiSparse.

2. HiSparse: the history on the host, the hot part on the GPU

Sparse attention makes an old systems idea work again. HiSparse (Xie et al., 2026) keeps every session’s complete KV history in host memory and gives each request a small, fixed hot cache on the GPU. The compact index the model uses to choose positions stays on the GPU, so the model still searches its whole history. When the indexer picks its 512 entries, the ones already in the hot cache are used in place and the missing ones are pulled from host memory before attention runs. The model computes what it would have computed with everything resident; only the bytes live somewhere else.

The catch is the budget. The hot cache size is a deployment-time choice: small, and you admit more sessions; large, and you keep more history close and fetch less. Whatever you pick, you pay for it at every load. At two in the morning, with a handful of sessions and most of HBM empty, a fixed budget still sends reads to the host that it has no need to send.

3. Borrow the memory while it is free, give it back when it is not

Adaptive HiSparse keeps the fixed hot cache and adds an elastic tier on top: whatever HBM is not spoken for is lent out to active requests as extra history pages. A request that holds its whole history runs on a direct path. Every position the model can select already has a GPU address, so there is no cache to miss, no replacement policy to run, and no host read to wait for. At low load, it runs close to conventional serving; we measure the remaining gap below.

The loan is revocable. When a new request needs room, the allocator takes borrowed pages back, oldest history first and in proportion to how much each request holds, and waits for any reader in flight before reusing them. Nothing is lost: the host copy was there all along, so a request that shrinks goes back to fetching the pages it no longer holds. When a request finishes, its pages are handed back to the others in bounded chunks between serving steps. Load goes up, residency goes down; load goes down, residency comes back. Nobody retunes a cache budget.

One detail makes the elastic tier free to consult. Every selected position resolves through a logical-to-physical map, so lookup time does not depend on how many pages a request has borrowed, and the replacement machinery of the hot cache stays the same fixed size whether a request holds a few pages of history or all of it.

4. Three things that had to be true for this to pay off

Lookup had to stay cheap. Elastic residency only helps if resolving 512 positions per query costs about nothing when the history is spread across a hot cache, borrowed pages, scratch space and the fresh tail of the session. One fused GPU kernel does the whole job: it checks each selected position against the tail, the borrowed map and the hot cache, deduplicates the misses within a request, pulls the missing bytes from pinned host memory, and writes the final addresses straight into attention’s metadata. At matched batch geometry, the adaptive path adds 1.2 to 1.4 ms to a 22 ms decode step.

Layers had to share, and fetch ahead. DeepSeek-V4.1-Flash shares its KV cache and its sparse indices across groups of layers. When two layers read the same entries from the same source, we resolve the addresses once and hand them to both. The model’s hierarchical indexer also announces, at the first layer of a group, a candidate pool of up to 16,384 positions from which the later layers will pick their own 512. We start copying the candidates that are not resident while the model is busy with the layers in between. In our non-speculative runs, that removed every demand miss at those later layers: across the whole sweep they found what they needed already there, and in the profiled peak rounds the copies landed before the first later indexer ran, 24 rounds out of 24.

Speculative decoding had to fit. DSpark, the model’s speculative decoder, verifies a different number of draft tokens for each request in one forward pass. A batch therefore holds requests with different numbers of queries, each query with its own selection and its own causal boundary. The retrieval path deduplicates history reads across a request’s queries, so a position selected by four of them is fetched once. The KV written during verification stays in request-owned scratch until the tokens are accepted, and, for the compressed layers, until a whole compression group is complete. Only then does it become committed history, and only committed history is eligible for the elastic tier.

5. What we measured

We ran DeepSeek-V4.1-Flash on four B300 GPUs, two for prefill and two for decode, on 500,000-token inputs with the prefix cached (499,968 of the 500,000 tokens hit the prefix cache) and generation capped at 800 tokens. Every request is the same size, so the numbers isolate one thing: what happens when a fixed pool of GPU memory meets more sessions than it can hold. The baseline is conventional serving with every history in HBM.

At 256 concurrent requests, without speculation, output throughput went from 3,563 to 4,470 tokens/s in the first run and from 3,555 to 4,500 in the repeat: +25.4% and +26.6%. With DSpark, it went from 4,058 to 4,991 and from 4,133 to 4,980: +23.0% and +20.5%. The mechanism is visible in the decode batches. The median batch per decode GPU went from 44 to 78 requests without speculation and from about 30 to 64 with it; the largest batches went from 46 to 82. The GPU that held 46 sessions now holds 82.

At 128 concurrent requests the gains are 8 to 9% without speculation and 15 to 16% with it. Below that there is nothing to gain: at 16, 32 and 64 concurrent requests every history already fits in HBM, and adaptive serving runs 3 to 7% slower than the resident baseline. That overhead is real, and section 6 is about it.

A second cohort re-tuned DSpark’s cost model, the table it uses to decide how many draft tokens to verify per request, with measurements at the larger batches that adaptive serving makes possible. With that table, the same runtime reaches 5,228 to 5,302 tokens/s at 256 concurrent requests, 26.5 to 30.7% above the archived resident DSpark runs. That number mixes two changes, residency and calibration, so we report it on its own rather than folding it into the headline.

6. What it costs, and what we have not shown

Bigger batches mean slower individual streams. In the first non-speculative run at 256 concurrent requests, the median per-request rate fell from 45.6 to 32.5 tokens/s; with DSpark, from 79.9 to 44.7. In exchange, requests waited far less for a decode slot: time to first token at the 90th percentile fell from 34.3 to 22.5 seconds without speculation and from 35.6 to 20.8 seconds with it. The gain is more sessions in flight, not the same sessions running faster. It is a different point on the throughput-versus-interactivity frontier, not a free multiplier, and which point you want depends on the workload.

At low load, the adaptive runtime is slower than plain resident serving, by 3 to 7% at 16 to 64 concurrent requests. Part of that is the admission of a new session, which copies its input history onto the GPU: about 12 ms of serialized work per admission, most of it one 7.6 ms copy. Part is steady-state bookkeeping. We tried moving the population copy to auxiliary streams and bounding its launches; neither improved the serving scores, and both negative controls are in the paper.

Speculative decoding can still miss. A verification pass with several queries can name more candidates than the finite candidate bank holds, so with DSpark the later layers still saw 1 to 2.6% of their selections go to host memory on demand. The fallback is exact and the model is unaffected; the fetch is just not hidden.

And the scope is what it is: one model revision, one eight-GPU node, and a nearly fully cached 500K-token workload with capped output. We have not measured uncached long prefills, multi-node deployments, or live agent trajectories. What we can say without a quality study is that this is a placement change, not an approximation: every query attends over the same selected positions and the same bytes as it would with everything resident, and the paper states the conditions under which that holds.

7. Memory that follows the load

The 445 MB a 500K session carries has not shrunk. It has moved to where it is cheap to keep, and it comes back to the GPU whenever the GPU has room. That is the whole idea: the model’s own sparsity says which bytes matter at each step, and the load says how many you can afford to keep close. Serving efficiency is what closes the token gap, and memory is where the GPU bill rose this year. Making each gigabyte of HBM carry more sessions is a direct lever on it.

The paper, Adaptive HiSparse: Elastic KV Residency for Sparse-Attention Serving, has the full mechanism, the correctness conditions, and every scored run, repeats and negative controls included.

Sources

• HiSparse: Xie et al., HiSparse: Scaling sparse-attention decoding with hierarchical KV cache management, 2026.
• The model: DeepSeek-AI, DeepSeek-V4.1-Flash technical report, 2026.
• Speculative decoding: Cheng et al., DSpark: Confidence-scheduled speculative decoding with semi-autoregressive generation, 2026.
• Our earlier posts: The token gap and Tokenomics, behind the scenes.