The model remembers attention state
A language model generates an answer through successive steps. For attention-based autoregressive generation, each step can reuse keys and values calculated for earlier tokens. A key-value cache retains that attention state instead of recomputing the same projections repeatedly. Hugging Face describes this state as a per-layer collection of key and value tensors. It is separate from the model weights, the application's conversation history and any database used for retrieval.
That distinction changes the sizing question. Loading a checkpoint establishes a weight-storage requirement; accepting a request creates a changing runtime-state requirement. A useful planning record therefore states both what is loaded and what work is allowed to remain active. Aguasto's approach here is a proposed evaluation method, with no measured deployment or model-service availability implied.
How caching works — Hugging Face Transformers (external source)Derive a shape estimate with visible assumptions
For conventional full-context attention, equal key and value widths and no sharing, a preliminary estimate is: cache bytes ≈ 2 × layers × KV heads × head width × retained tokens × concurrent sequences × bytes per element. The factor of two represents keys and values. Use the number of key-value heads rather than automatically substituting query heads: grouped-query attention can use fewer distinct key-value groups. NVIDIA's cache documentation describes that allocation distinction.
For an explicitly hypothetical configuration of 32 layers, 8 KV heads, head width 128, 8,192 retained tokens, 4 sequences and 2 bytes per element, the estimate is 4,294,967,296 bytes, or 4 GiB. This is arithmetic, not a GPU-fit result. It excludes block metadata, padding, allocator reservation, temporary workspace and any replicated or separately retained state.
How caching works — Hugging Face Transformers (external source) · KV Cache System — NVIDIA TensorRT-LLM (external source)Architecture decides how the estimate grows
The simple expression assumes every listed layer retains the same length of full attention state. Real checkpoint and engine configurations can break that assumption. A sliding window bounds retained state differently from a global layer; hybrid and recurrent layers may maintain another representation. Hugging Face documents dynamic, static and sliding storage behavior, while NVIDIA documents model-dependent cache pools and layouts.
For planning, replace one aggregate estimate with a layer-group worksheet. List each state's shape, representation, retention limit and placement. Keep configured maximum context, presently retained tokens and reserved capacity in separate columns. Do not divide an aggregate result by the GPU count until the engine's actual sharding and replication rules are known. A shorter prompt can still coexist with a larger reserved pool.
How caching works — Hugging Face Transformers (external source) · KV Cache System — NVIDIA TensorRT-LLM (external source)Paging changes allocation, not the model's meaning
PagedAttention introduced a mechanism for managing cache in blocks rather than requiring one large contiguous allocation for each sequence. Current TensorRT-LLM documentation likewise distinguishes a monolithic cache from pooled blocks that are allocated and recycled as requests progress. The engineering purpose is to manage changing request lengths and sharing; it does not make every retained key or value intrinsically smaller.
An original block diagram should show two views: the request's ordered logical tokens and the physical blocks assigned to hold their state. That view helps explain why available memory and a request's logical maximum length are different quantities. Historical performance results from the PagedAttention paper belong to its documented experiment. They are not a current serving-engine ranking or evidence of Aguasto throughput.
Efficient Memory Management for Large Language Model Serving with PagedAttention (external source) · Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM (external source)A reusable prefix needs an exact identity
Cross-request reuse is another decision. A prefix that has already been computed may reduce repeated prefill work when another eligible request starts with that same state. vLLM's design includes preceding block identity, exact tokens and additional identifiers such as adapters or multimodal inputs. Text that looks similar to a human is not sufficient evidence that all computational inputs match.
Treat reuse eligibility and retention policy as separate fields. A stored prefix can be correctly identified yet evicted before it is needed, or exist on a different replica. A cache hit therefore needs a measured scope: which request, which state, which engine and which location. Reusing prefill state does not establish a universal decode-speed improvement. Compare cold and eligible warm-prefix workloads rather than reporting only the most favorable path.
KV Cache System — NVIDIA TensorRT-LLM (external source) · Automatic Prefix Caching — vLLM (external source)Decide who may share state
Reuse has a trust boundary as well as a performance boundary. vLLM documents request salting that confines prefix sharing to requests carrying the same salt. Its security guidance discusses timing side channels from cross-tenant reuse. A salt is one cache-sharing control; it does not authenticate a customer, encrypt model state or provide a complete isolation architecture.
A proposed enterprise design should derive the allowed sharing group from trusted application identity and document how that identity reaches the engine. Verify that unrelated tenants cannot choose another group's identity through an unchecked request field. Record cache scope, eviction behavior and logging boundaries alongside the access policy. This is an architecture question for a future service, not a statement that Aguasto has deployed those controls.
Automatic Prefix Caching — vLLM (external source) · Security — vLLM 0.31.0 (external source)Make admission and measurement reproducible
The final worksheet should identify the model and tokenizer revision, engine build, attention layout, cache dtype, prompt and output limits, concurrency, block allocation, parallelism and reuse policy. Define what happens when a request cannot obtain state: waiting, rejection or an engine-supported preemption path. vLLM documents that insufficient cache can cause preemption and recomputation, which can affect latency.
A future evaluation should retain active and reserved memory, prefix-hit distribution, evictions, recomputation, completed work and client latency under mixed lengths and bursts. Link the results to a frozen workload and retain failed or cancelled requests. The existing weight-memory and benchmark guides provide the adjacent storage and measurement boundaries. No command, memory number or performance result in this guide is an executed Aguasto test.
Optimization and Tuning — vLLM (external source)Sources & verification
The register preserves each source's publication and verification date. Manufacturer claims and documentation are attributed; Aguasto has not run the cited products in production.
- How caching works — Hugging Face Transformers ↗
Per-layer keys and values; tensor dimensions; Dynamic, static and sliding cache behavior. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- KV Cache System — NVIDIA TensorRT-LLM ↗
Grouped-query key-value allocation; Model-dependent cache layouts and reuse management. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Efficient Memory Management for Large Language Model Serving with PagedAttention ↗
Original block-based cache allocation and sharing mechanism; Historical experiment boundary. Publication: 2023-09-12. Verified: 2026-10-13.
- Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM ↗
Contiguous versus paged cache allocation; Cache blocks allocated and recycled during processing. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Automatic Prefix Caching — vLLM ↗
Exact prefix-block identity, adapters and multimodal inputs; Optional cache salting and trust-group sharing. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Security — vLLM 0.31.0 ↗
Prefix-cache timing side-channel mitigation; Cache salting as an isolation control. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Optimization and Tuning — vLLM ↗
Insufficient cache can cause preemption and recomputation; Related latency and monitoring boundaries. Publication: date not stated in the reviewed source. Verified: 2026-10-13.