--- title: "What Is KV Cache? Capacity, Reuse and Isolation | Aguasto" description: "Connect attention-state shape, request length, allocation policy and exact-prefix reuse to a transparent inference memory and admission plan." canonical: "https://www.aguasto.com/learn/kv-cache-capacity-and-policy" last-updated: "2026-10-13" --- # What Is KV Cache? Capacity, Reuse and Isolation | Aguasto Canonical HTML: [What Is KV Cache? Capacity, Reuse and Isolation](https://www.aguasto.com/learn/kv-cache-capacity-and-policy) > Connect attention-state shape, request length, allocation policy and exact-prefix reuse to a transparent inference memory and admission plan. [Aguasto](https://www.aguasto.com/) [learn](https://www.aguasto.com/learn) What Is KV Cache? Capacity, Reuse and Isolation [Resources](https://www.aguasto.com/resources) [GPU Library](https://www.aguasto.com/gpus) [Model Index](https://www.aguasto.com/models) [Inference Stack](https://www.aguasto.com/inference-stack) [Learn](https://www.aguasto.com/learn) [Insights](https://www.aguasto.com/insights) [Research](https://www.aguasto.com/research) [Engineering tools](https://www.aguasto.com/tools) Aguasto / learn ## What Is KV Cache? Capacity, Reuse and Isolation Connect attention-state shape, request length, allocation policy and exact-prefix reuse to a transparent inference memory and admission plan. Original Aguasto conceptual illustration. Not manufacturer geometry, an operating Aguasto asset, or a measured benchmark. By Aguasto Research · Published October 13, 2026 · Revision 1 Aguasto Research documents third-party hardware, models and software. References do not represent Aguasto inventory, evaluated deployments or available inference services. Illustration: What Is KV Cache? Capacity, Reuse and Isolation: original technical study. Illustrative cache allocation separates exact shared prefix blocks, private sequence tails and free blocks. Reuse depends on identity and isolation policy, not approximate text similarity. Illustrative cache allocation separates exact shared prefix blocks, private sequence tails and free blocks. Reuse depends on identity and isolation policy, not approximate text similarity. Original Aguasto illustration. Read the guide for configuration and measurement limitations. ### The model remembers attention state A language model generates an answer through successive steps. For attention-based autoregressive generation, each step can reuse keys and values calculated for earlier tokens. A key-value cache retains that attention state instead of recomputing the same projections repeatedly. Hugging Face describes this state as a per-layer collection of key and value tensors. It is separate from the model weights, the application's conversation history and any database used for retrieval. That distinction changes the sizing question. Loading a checkpoint establishes a weight-storage requirement; accepting a request creates a changing runtime-state requirement. A useful planning record therefore states both what is loaded and what work is allowed to remain active. Aguasto's approach here is a proposed evaluation method, with no measured deployment or model-service availability implied. [How caching works — Hugging Face Transformers (external source)](https://huggingface.co/docs/transformers/cache_explanation) ### Derive a shape estimate with visible assumptions For conventional full-context attention, equal key and value widths and no sharing, a preliminary estimate is: cache bytes ≈ 2 × layers × KV heads × head width × retained tokens × concurrent sequences × bytes per element. The factor of two represents keys and values. Use the number of key-value heads rather than automatically substituting query heads: grouped-query attention can use fewer distinct key-value groups. NVIDIA's cache documentation describes that allocation distinction. For an explicitly hypothetical configuration of 32 layers, 8 KV heads, head width 128, 8,192 retained tokens, 4 sequences and 2 bytes per element, the estimate is 4,294,967,296 bytes, or 4 GiB. This is arithmetic, not a GPU-fit result. It excludes block metadata, padding, allocator reservation, temporary workspace and any replicated or separately retained state. [How caching works — Hugging Face Transformers (external source)](https://huggingface.co/docs/transformers/cache_explanation) · [KV Cache System — NVIDIA TensorRT-LLM (external source)](https://nvidia.github.io/TensorRT-LLM/features/kvcache.html) ### Architecture decides how the estimate grows The simple expression assumes every listed layer retains the same length of full attention state. Real checkpoint and engine configurations can break that assumption. A sliding window bounds retained state differently from a global layer; hybrid and recurrent layers may maintain another representation. Hugging Face documents dynamic, static and sliding storage behavior, while NVIDIA documents model-dependent cache pools and layouts. For planning, replace one aggregate estimate with a layer-group worksheet. List each state's shape, representation, retention limit and placement. Keep configured maximum context, presently retained tokens and reserved capacity in separate columns. Do not divide an aggregate result by the GPU count until the engine's actual sharding and replication rules are known. A shorter prompt can still coexist with a larger reserved pool. [How caching works — Hugging Face Transformers (external source)](https://huggingface.co/docs/transformers/cache_explanation) · [KV Cache System — NVIDIA TensorRT-LLM (external source)](https://nvidia.github.io/TensorRT-LLM/features/kvcache.html) ### Paging changes allocation, not the model's meaning PagedAttention introduced a mechanism for managing cache in blocks rather than requiring one large contiguous allocation for each sequence. Current TensorRT-LLM documentation likewise distinguishes a monolithic cache from pooled blocks that are allocated and recycled as requests progress. The engineering purpose is to manage changing request lengths and sharing; it does not make every retained key or value intrinsically smaller. An original block diagram should show two views: the request's ordered logical tokens and the physical blocks assigned to hold their state. That view helps explain why available memory and a request's logical maximum length are different quantities. Historical performance results from the PagedAttention paper belong to its documented experiment. They are not a current serving-engine ranking or evidence of Aguasto throughput. [Efficient Memory Management for Large Language Model Serving with PagedAttention (external source)](https://arxiv.org/abs/2309.06180) · [Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM (external source)](https://nvidia.github.io/TensorRT-LLM/features/paged-attention-ifb-scheduler.html) ### A reusable prefix needs an exact identity Cross-request reuse is another decision. A prefix that has already been computed may reduce repeated prefill work when another eligible request starts with that same state. vLLM's design includes preceding block identity, exact tokens and additional identifiers such as adapters or multimodal inputs. Text that looks similar to a human is not sufficient evidence that all computational inputs match. Treat reuse eligibility and retention policy as separate fields. A stored prefix can be correctly identified yet evicted before it is needed, or exist on a different replica. A cache hit therefore needs a measured scope: which request, which state, which engine and which location. Reusing prefill state does not establish a universal decode-speed improvement. Compare cold and eligible warm-prefix workloads rather than reporting only the most favorable path. [KV Cache System — NVIDIA TensorRT-LLM (external source)](https://nvidia.github.io/TensorRT-LLM/features/kvcache.html) · [Automatic Prefix Caching — vLLM (external source)](https://docs.vllm.ai/en/stable/design/prefix_caching/) ### Decide who may share state Reuse has a trust boundary as well as a performance boundary. vLLM documents request salting that confines prefix sharing to requests carrying the same salt. Its security guidance discusses timing side channels from cross-tenant reuse. A salt is one cache-sharing control; it does not authenticate a customer, encrypt model state or provide a complete isolation architecture. A proposed enterprise design should derive the allowed sharing group from trusted application identity and document how that identity reaches the engine. Verify that unrelated tenants cannot choose another group's identity through an unchecked request field. Record cache scope, eviction behavior and logging boundaries alongside the access policy. This is an architecture question for a future service, not a statement that Aguasto has deployed those controls. [Automatic Prefix Caching — vLLM (external source)](https://docs.vllm.ai/en/stable/design/prefix_caching/) · [Security — vLLM 0.31.0 (external source)](https://docs.vllm.ai/en/v0.31.0/usage/security/) ### Make admission and measurement reproducible The final worksheet should identify the model and tokenizer revision, engine build, attention layout, cache dtype, prompt and output limits, concurrency, block allocation, parallelism and reuse policy. Define what happens when a request cannot obtain state: waiting, rejection or an engine-supported preemption path. vLLM documents that insufficient cache can cause preemption and recomputation, which can affect latency. A future evaluation should retain active and reserved memory, prefix-hit distribution, evictions, recomputation, completed work and client latency under mixed lengths and bursts. Link the results to a frozen workload and retain failed or cancelled requests. The existing weight-memory and benchmark guides provide the adjacent storage and measurement boundaries. No command, memory number or performance result in this guide is an executed Aguasto test. [Optimization and Tuning — vLLM (external source)](https://docs.vllm.ai/en/latest/configuration/optimization/) ### Scope & limitations - The 4 GiB example is hypothetical shape arithmetic, not a measured deployment or a complete GPU-memory requirement. - The formula assumes conventional attention and must be adapted to actual layer layouts, state formats, sharing and device placement. - Living software documentation can change; a deployment must pin its checkpoint, engine and configuration. - Cache salting is one isolation mechanism and does not establish full service security. - Aguasto has not run these serving commands, measured cache allocations or made any model API available. ### Continue the research [Model weights are only the beginning of the memory budget](https://www.aguasto.com/learn/model-weights-and-runtime-memory) [Why tokens per second is not enough to benchmark inference](https://www.aguasto.com/research/inference-benchmark-methodology) [Prefill, decode and the cost of separating them](https://www.aguasto.com/learn/prefill-decode-and-disaggregated-serving) [NVIDIA H100 SXM](https://www.aguasto.com/gpus/h100) [NVIDIA H200 SXM](https://www.aguasto.com/gpus/h200) [NVIDIA B200 SXM](https://www.aguasto.com/gpus/b200) [Qwen3-235B-A22B](https://www.aguasto.com/models/qwen3-235b-a22b) [gpt-oss-120b](https://www.aguasto.com/models/gpt-oss-120b) [vLLM](https://www.aguasto.com/inference-stack/vllm) [TensorRT-LLM](https://www.aguasto.com/inference-stack/tensorrt-llm) [SGLang](https://www.aguasto.com/inference-stack/sglang) [The Aguasto modular thesis](https://www.aguasto.com/why-modular) [Potential OEM and strategic collaboration](https://www.aguasto.com/partners) ### Sources & verification The register preserves each source's publication and verification date. Manufacturer claims and documentation are attributed; Aguasto has not run the cited products in production. - [How caching works — Hugging Face Transformers ↗](https://huggingface.co/docs/transformers/cache_explanation) Per-layer keys and values; tensor dimensions; Dynamic, static and sliding cache behavior. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [KV Cache System — NVIDIA TensorRT-LLM ↗](https://nvidia.github.io/TensorRT-LLM/features/kvcache.html) Grouped-query key-value allocation; Model-dependent cache layouts and reuse management. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Efficient Memory Management for Large Language Model Serving with PagedAttention ↗](https://arxiv.org/abs/2309.06180) Original block-based cache allocation and sharing mechanism; Historical experiment boundary. Publication: 2023-09-12. Verified: 2026-10-13. - [Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM ↗](https://nvidia.github.io/TensorRT-LLM/features/paged-attention-ifb-scheduler.html) Contiguous versus paged cache allocation; Cache blocks allocated and recycled during processing. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Automatic Prefix Caching — vLLM ↗](https://docs.vllm.ai/en/stable/design/prefix_caching/) Exact prefix-block identity, adapters and multimodal inputs; Optional cache salting and trust-group sharing. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Security — vLLM 0.31.0 ↗](https://docs.vllm.ai/en/v0.31.0/usage/security/) Prefix-cache timing side-channel mitigation; Cache salting as an isolation control. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Optimization and Tuning — vLLM ↗](https://docs.vllm.ai/en/latest/configuration/optimization/) Insufficient cache can cause preemption and recomputation; Related latency and monitoring boundaries. Publication: date not stated in the reviewed source. Verified: 2026-10-13. ### In this publication [The model remembers attention state](https://www.aguasto.com/learn/kv-cache-capacity-and-policy#section-1) [Derive a shape estimate with visible assumptions](https://www.aguasto.com/learn/kv-cache-capacity-and-policy#section-2) [Architecture decides how the estimate grows](https://www.aguasto.com/learn/kv-cache-capacity-and-policy#section-3) [Paging changes allocation, not the model's meaning](https://www.aguasto.com/learn/kv-cache-capacity-and-policy#section-4) [A reusable prefix needs an exact identity](https://www.aguasto.com/learn/kv-cache-capacity-and-policy#section-5) [Decide who may share state](https://www.aguasto.com/learn/kv-cache-capacity-and-policy#section-6) [Make admission and measurement reproducible](https://www.aguasto.com/learn/kv-cache-capacity-and-policy#section-7) [More learn ↗](https://www.aguasto.com/learn) [Source register ↓](https://www.aguasto.com/learn/kv-cache-capacity-and-policy#sources) ## Company navigation - [Home](https://www.aguasto.com/) - [Infrastructure](https://www.aguasto.com/infrastructure) - [GPU Cloud](https://www.aguasto.com/gpu-cloud) - [AI Inference](https://www.aguasto.com/ai-inference) - [Development & Investment](https://www.aguasto.com/development) - [Company](https://www.aguasto.com/company) - [Contact](https://www.aguasto.com/contact) - [Privacy](https://www.aguasto.com/privacy) - [Resources](https://www.aguasto.com/resources) - [GPU Library](https://www.aguasto.com/gpus) - [Model Index](https://www.aguasto.com/models) - [Inference Stack](https://www.aguasto.com/inference-stack) - [Learn](https://www.aguasto.com/learn) - [Insights](https://www.aguasto.com/insights) - [Research](https://www.aguasto.com/research) - [Industry cases](https://www.aguasto.com/research/cases) - [Why Modular](https://www.aguasto.com/why-modular) - [Partners](https://www.aguasto.com/partners) - [GPU comparison](https://www.aguasto.com/gpus/compare) - [Model comparison](https://www.aguasto.com/models/compare) - [Engineering tools](https://www.aguasto.com/tools) - [Development friction](https://www.aguasto.com/research/hyperscale-development-friction) - [NVIDIA H100 SXM](https://www.aguasto.com/gpus/h100) - [NVIDIA H200 SXM](https://www.aguasto.com/gpus/h200) - [NVIDIA B200 SXM](https://www.aguasto.com/gpus/b200) - [NVIDIA B300](https://www.aguasto.com/gpus/b300) - [NVIDIA GB200 NVL72](https://www.aguasto.com/gpus/gb200-nvl72) - [NVIDIA GB300 NVL72](https://www.aguasto.com/gpus/gb300-nvl72) - [NVIDIA Vera Rubin NVL72](https://www.aguasto.com/gpus/vera-rubin-nvl72) - [AMD Instinct MI300X](https://www.aguasto.com/gpus/mi300x) - [AMD Instinct MI325X](https://www.aguasto.com/gpus/mi325x) - [AMD Instinct MI355X](https://www.aguasto.com/gpus/mi355x) - [AMD Instinct MI455X](https://www.aguasto.com/gpus/mi455x) - [AMD Helios](https://www.aguasto.com/gpus/helios) - [GLM-4.5](https://www.aguasto.com/models/glm-4-5) - [Qwen3-235B-A22B](https://www.aguasto.com/models/qwen3-235b-a22b) - [DeepSeek-R1](https://www.aguasto.com/models/deepseek-r1) - [Llama 3.3 70B Instruct](https://www.aguasto.com/models/llama-3-3-70b-instruct) - [Gemma 3 27B IT](https://www.aguasto.com/models/gemma-3-27b-it) - [gpt-oss-120b](https://www.aguasto.com/models/gpt-oss-120b) - [Kimi-K2-Instruct](https://www.aguasto.com/models/kimi-k2-instruct) - [GLM-5.3](https://www.aguasto.com/models/glm-5-3) - [Qwen3.8-27B](https://www.aguasto.com/models/qwen3-8-27b) - [DeepSeek-V4.1-Flash](https://www.aguasto.com/models/deepseek-v4-1-flash) - [Llama 4 Scout 17B-16E Instruct](https://www.aguasto.com/models/llama-4-scout-17b-16e-instruct) - [Gemma 4 31B IT](https://www.aguasto.com/models/gemma-4-31b-it) - [Kimi K3](https://www.aguasto.com/models/kimi-k3) - [vLLM](https://www.aguasto.com/inference-stack/vllm) - [SGLang](https://www.aguasto.com/inference-stack/sglang) - [TensorRT-LLM](https://www.aguasto.com/inference-stack/tensorrt-llm) - [NVIDIA Dynamo](https://www.aguasto.com/inference-stack/dynamo) - [NVIDIA Triton Inference Server](https://www.aguasto.com/inference-stack/triton) - [AMD ROCm](https://www.aguasto.com/inference-stack/rocm) - [Ray Serve](https://www.aguasto.com/inference-stack/ray-serve) - [Kubernetes GPU orchestration — NVIDIA device plugin](https://www.aguasto.com/inference-stack/kubernetes) - [Model weights are only the beginning of the memory budget](https://www.aguasto.com/learn/model-weights-and-runtime-memory) - [Why tokens per second is not enough to benchmark inference](https://www.aguasto.com/research/inference-benchmark-methodology) - [Prefill, decode and the cost of separating them](https://www.aguasto.com/learn/prefill-decode-and-disaggregated-serving) - [Choosing between vLLM, SGLang and TensorRT-LLM](https://www.aguasto.com/learn/choosing-an-inference-serving-framework) - [From Power to Inference—Building a Distributed AI Compute Network](https://www.aguasto.com/learn/from-power-to-inference) - [Dedicated compute and shared inference: compare the whole workload](https://www.aguasto.com/insights/dedicated-and-shared-inference-economics) - [Distributed inference reliability begins with the failure boundary](https://www.aguasto.com/learn/distributed-inference-reliability) - [Core Scientific: reading contracted, energized and billed capacity](https://www.aguasto.com/research/cases/core-scientific-capacity-disclosures) - [Why Modular AI Infrastructure Matters Now](https://www.aguasto.com/research/why-modular-ai-infrastructure-matters-now) - [What Is a Modular AI Data Center?](https://www.aguasto.com/learn/what-is-a-modular-ai-data-center) - [How Modular AI Data Centers Are Built and Commissioned](https://www.aguasto.com/learn/modular-data-center-factory-to-commissioning) - [Liquid Cooling for High-Density GPU Systems: Follow the Interfaces](https://www.aguasto.com/learn/liquid-cooling-gpu-rack-interfaces) - [Why Electricity Availability Shapes AI Infrastructure Deployment](https://www.aguasto.com/research/electricity-availability-and-time-to-power) - [What Is KV Cache? Capacity, Reuse and Isolation](https://www.aguasto.com/learn/kv-cache-capacity-and-policy) - [Continuous Batching: Throughput Without Hiding Latency](https://www.aguasto.com/learn/continuous-batching-and-latency) - [Applied Digital: Reading the Polaris Forge 1 Delivery Milestones](https://www.aguasto.com/research/cases/applied-digital-polaris-forge-capacity-milestones) - [Crusoe Abilene: Separate Delivered Infrastructure from Expansion Plans](https://www.aguasto.com/research/cases/crusoe-abilene-delivery-and-expansion) - [Official LinkedIn](https://www.linkedin.com/company/aguasto) This is a generated representation of the public HTML page. Diagrams are conceptual. GPU and inference services are planned and are not currently available to book or call. The contact page describes the available inquiry flow.