--- title: "Model weights are only the beginning of the memory budget | Aguasto" description: "Help engineers separate a transparent weight-storage estimate from the workload-dependent memory required to serve a model." canonical: "https://www.aguasto.com/learn/model-weights-and-runtime-memory" last-updated: "2026-10-12" --- # Model weights are only the beginning of the memory budget | Aguasto Canonical HTML: [Model weights are only the beginning of the memory budget](https://www.aguasto.com/learn/model-weights-and-runtime-memory) > Help engineers separate a transparent weight-storage estimate from the workload-dependent memory required to serve a model. [Aguasto](https://www.aguasto.com/) [learn](https://www.aguasto.com/learn) Model weights are only the beginning of the memory budget [Resources](https://www.aguasto.com/resources) [GPU Library](https://www.aguasto.com/gpus) [Model Index](https://www.aguasto.com/models) [Inference Stack](https://www.aguasto.com/inference-stack) [Learn](https://www.aguasto.com/learn) [Insights](https://www.aguasto.com/insights) [Research](https://www.aguasto.com/research) [Engineering tools](https://www.aguasto.com/tools) Aguasto / learn ## Model weights are only the beginning of the memory budget Help engineers separate a transparent weight-storage estimate from the workload-dependent memory required to serve a model. Original Aguasto conceptual illustration. Not manufacturer geometry, an operating Aguasto asset, or a measured benchmark. By Aguasto Research · Published October 12, 2026 · Revision 1 Aguasto Research documents third-party hardware, models and software. References do not represent Aguasto inventory, evaluated deployments or available inference services. ### Calculate the storage floor For a preliminary weight-storage estimate, multiply parameter count by the storage bits per parameter and divide by eight. Expressed in decimal gigabytes, weight GB = parameters in billions × bits ÷ 8. A hypothetical 70-billion-parameter model at 16 bits therefore has a 140 GB storage floor, or about 130.4 GiB after converting bytes with GB × 10^9 ÷ 2^30. This is arithmetic, not a measured deployment. Hugging Face explains the underlying parameter-storage relationship; its older hardware examples are not current accelerator rankings. Use the documented checkpoint parameter count and storage format. Keep GB and GiB visible rather than silently treating them as equal. A result labeled estimated weight storage is useful; a result labeled required GPU memory would overstate what this calculation establishes. [Optimizing LLMs for Speed and Memory — Hugging Face (external source)](https://huggingface.co/docs/transformers/main/en/llm_tutorial_optimization) ### Budget for the changing cache During autoregressive generation, attention models can retain keys and values from earlier tokens. Hugging Face documents dynamic, static, offloaded and quantized cache strategies, with different memory and latency trade-offs. Static allocation may reserve space for a maximum sequence; offloading moves cache state between devices; cache quantization is a separate choice from weight quantization. Some architectures use model-specific state rather than the same attention-cache layout. Build the runtime budget around the intended prompt and output lengths, concurrent sequences and cache representation. Do not infer cache size from total parameters alone. A long-context service and a short-answer service using the same weights can require different allocation policies. [Cache strategies — Hugging Face Transformers (external source)](https://huggingface.co/docs/transformers/kv_cache) ### Separate storage precision from execution A four-bit checkpoint does not mean every tensor or operation uses four bits. Hugging Face's bitsandbytes documentation describes quantized linear layers, other modules retaining another dtype, and separate compute-dtype settings. Quantization metadata, scales and unconverted tensors make the ideal parameter-times-bits calculation a floor rather than an exact file or runtime size. Record the checkpoint format, method, group or block settings when available, and the serving engine's support for that specific combination. Evaluate output quality on the actual task before changing precision for a capacity target. The memory estimator should never promise identical answers or a speed improvement from a smaller storage representation. [Bitsandbytes — Hugging Face Transformers (external source)](https://huggingface.co/docs/transformers/quantization/bitsandbytes) ### Measure allocation at the right boundary PyTorch distinguishes memory occupied by live tensors from memory reserved by its caching allocator. Unused reserved blocks may still appear occupied in device monitoring. CUDA graphs can retain private pools across replays. Those mechanisms explain why a simple checkpoint calculation and an observed device-memory value need not match. For a later executed evaluation, capture idle loaded-model memory, steady-state serving memory and peak memory during a controlled burst. State whether a chart uses allocated tensors, allocator reservation or total device usage. Include the host-memory and transfer implications if any state is offloaded. Aguasto has not executed this GPU evaluation. [CUDA semantics: Memory management — PyTorch 2.14 (external source)](https://docs.pytorch.org/docs/2.14/notes/cuda.html) ### Treat multiple devices as a topology decision After estimating the storage floor, evaluate how the chosen engine places weights and runtime state across devices. A planning worksheet should list per-device allocation, replication or sharding policy, interconnect, supported kernels and expected communication boundaries. Adding device capacities together is only an initial capacity calculation; it does not verify that a particular distributed layout works. Use a comparison in stages: documented model format, supported engine path, memory estimate, representative workload test and then an operational acceptance decision. Keep any hypothetical fit labeled preliminary. Reject the shortcut from aggregate memory to a claim that the model is available through Aguasto. ### A reproducible sizing record Save a sizing record containing the official model identifier and revision, total parameter count, storage precision, engine version, prompt/output limits, concurrency target, cache settings, parallelism and hardware category. Attach measured allocations only after a real run and keep them distinct from calculator output. The immediate decision is whether a candidate configuration deserves testing, not whether it is production ready. Publish unknowns explicitly: unverified quantization support, unmeasured cache growth, missing interconnect evidence or unknown task-quality impact. This makes the same model reference useful for both a hardware buyer and an engineer without turning an estimate into a capacity commitment. ### Scope & limitations - The illustrative 70B example is hypothetical and is not an Aguasto deployment. - Weight arithmetic excludes cache, activations, workspace, allocator overhead and format metadata. - Living documentation is not an executable pinned environment. ### Continue the research [NVIDIA H100 SXM](https://www.aguasto.com/gpus/h100) [NVIDIA H200 SXM](https://www.aguasto.com/gpus/h200) [NVIDIA B200 SXM](https://www.aguasto.com/gpus/b200) [AMD Instinct MI300X](https://www.aguasto.com/gpus/mi300x) [AMD Instinct MI355X](https://www.aguasto.com/gpus/mi355x) [vLLM](https://www.aguasto.com/inference-stack/vllm) [SGLang](https://www.aguasto.com/inference-stack/sglang) [TensorRT-LLM](https://www.aguasto.com/inference-stack/tensorrt-llm) [The Aguasto modular thesis](https://www.aguasto.com/why-modular) [Potential OEM and strategic collaboration](https://www.aguasto.com/partners) ### Sources & verification Reference revision 1 · Reviewed October 12, 2026. Manufacturer claims and documentation are attributed; Aguasto has not run the cited products in production. - [Optimizing LLMs for Speed and Memory — Hugging Face ↗](https://huggingface.co/docs/transformers/main/en/llm_tutorial_optimization) Parameter storage at float32 and BF16/FP16; Weight storage differs from long-context inference memory. Publication: date not stated in the reviewed source. Verified: 2026-10-12. - [Cache strategies — Hugging Face Transformers ↗](https://huggingface.co/docs/transformers/kv_cache) KV cache grows during generation; Static cache allocation, offloading and cache quantization trade-offs; Model-specific cache limitations. Publication: date not stated in the reviewed source. Verified: 2026-10-12. - [Bitsandbytes — Hugging Face Transformers ↗](https://huggingface.co/docs/transformers/quantization/bitsandbytes) Quantized linear modules and mixed compute/storage precision; Nonquantized modules and offloading constraints. Publication: date not stated in the reviewed source. Verified: 2026-10-12. - [CUDA semantics: Memory management — PyTorch 2.14 ↗](https://docs.pytorch.org/docs/2.14/notes/cuda.html) Allocated tensor memory versus allocator-reserved memory; Graph-private memory pools and caching allocator. Publication: date not stated in the reviewed source. Verified: 2026-10-12. ### In this publication [Calculate the storage floor](https://www.aguasto.com/learn/model-weights-and-runtime-memory#section-1) [Budget for the changing cache](https://www.aguasto.com/learn/model-weights-and-runtime-memory#section-2) [Separate storage precision from execution](https://www.aguasto.com/learn/model-weights-and-runtime-memory#section-3) [Measure allocation at the right boundary](https://www.aguasto.com/learn/model-weights-and-runtime-memory#section-4) [Treat multiple devices as a topology decision](https://www.aguasto.com/learn/model-weights-and-runtime-memory#section-5) [A reproducible sizing record](https://www.aguasto.com/learn/model-weights-and-runtime-memory#section-6) [More learn ↗](https://www.aguasto.com/learn) [Source register ↓](https://www.aguasto.com/learn/model-weights-and-runtime-memory#sources) ## Company navigation - [Home](https://www.aguasto.com/) - [Infrastructure](https://www.aguasto.com/infrastructure) - [GPU Cloud](https://www.aguasto.com/gpu-cloud) - [AI Inference](https://www.aguasto.com/ai-inference) - [Development & Investment](https://www.aguasto.com/development) - [Company](https://www.aguasto.com/company) - [Contact](https://www.aguasto.com/contact) - [Privacy](https://www.aguasto.com/privacy) - [Resources](https://www.aguasto.com/resources) - [GPU Library](https://www.aguasto.com/gpus) - [Model Index](https://www.aguasto.com/models) - [Inference Stack](https://www.aguasto.com/inference-stack) - [Learn](https://www.aguasto.com/learn) - [Insights](https://www.aguasto.com/insights) - [Research](https://www.aguasto.com/research) - [Industry cases](https://www.aguasto.com/research/cases) - [Why Modular](https://www.aguasto.com/why-modular) - [Partners](https://www.aguasto.com/partners) - [GPU comparison](https://www.aguasto.com/gpus/compare) - [Model comparison](https://www.aguasto.com/models/compare) - [Engineering tools](https://www.aguasto.com/tools) - [Development friction](https://www.aguasto.com/research/hyperscale-development-friction) - [NVIDIA H100 SXM](https://www.aguasto.com/gpus/h100) - [NVIDIA H200 SXM](https://www.aguasto.com/gpus/h200) - [NVIDIA B200 SXM](https://www.aguasto.com/gpus/b200) - [NVIDIA B300](https://www.aguasto.com/gpus/b300) - [NVIDIA GB200 NVL72](https://www.aguasto.com/gpus/gb200-nvl72) - [NVIDIA GB300 NVL72](https://www.aguasto.com/gpus/gb300-nvl72) - [NVIDIA Vera Rubin NVL72](https://www.aguasto.com/gpus/vera-rubin-nvl72) - [AMD Instinct MI300X](https://www.aguasto.com/gpus/mi300x) - [AMD Instinct MI325X](https://www.aguasto.com/gpus/mi325x) - [AMD Instinct MI355X](https://www.aguasto.com/gpus/mi355x) - [AMD Instinct MI455X](https://www.aguasto.com/gpus/mi455x) - [AMD Helios](https://www.aguasto.com/gpus/helios) - [GLM-4.5](https://www.aguasto.com/models/glm-4-5) - [Qwen3-235B-A22B](https://www.aguasto.com/models/qwen3-235b-a22b) - [DeepSeek-R1](https://www.aguasto.com/models/deepseek-r1) - [Llama 3.3 70B Instruct](https://www.aguasto.com/models/llama-3-3-70b-instruct) - [Gemma 3 27B IT](https://www.aguasto.com/models/gemma-3-27b-it) - [gpt-oss-120b](https://www.aguasto.com/models/gpt-oss-120b) - [Kimi-K2-Instruct](https://www.aguasto.com/models/kimi-k2-instruct) - [vLLM](https://www.aguasto.com/inference-stack/vllm) - [SGLang](https://www.aguasto.com/inference-stack/sglang) - [TensorRT-LLM](https://www.aguasto.com/inference-stack/tensorrt-llm) - [NVIDIA Dynamo](https://www.aguasto.com/inference-stack/dynamo) - [NVIDIA Triton Inference Server](https://www.aguasto.com/inference-stack/triton) - [AMD ROCm](https://www.aguasto.com/inference-stack/rocm) - [Ray Serve](https://www.aguasto.com/inference-stack/ray-serve) - [Kubernetes GPU orchestration — NVIDIA device plugin](https://www.aguasto.com/inference-stack/kubernetes) - [Model weights are only the beginning of the memory budget](https://www.aguasto.com/learn/model-weights-and-runtime-memory) - [Why tokens per second is not enough to benchmark inference](https://www.aguasto.com/research/inference-benchmark-methodology) - [Prefill, decode and the cost of separating them](https://www.aguasto.com/learn/prefill-decode-and-disaggregated-serving) - [Choosing between vLLM, SGLang and TensorRT-LLM](https://www.aguasto.com/learn/choosing-an-inference-serving-framework) - [From Power to Inference—Building a Distributed AI Compute Network](https://www.aguasto.com/learn/from-power-to-inference) - [Dedicated compute and shared inference: compare the whole workload](https://www.aguasto.com/insights/dedicated-and-shared-inference-economics) - [Distributed inference reliability begins with the failure boundary](https://www.aguasto.com/learn/distributed-inference-reliability) - [Core Scientific: reading contracted, energized and billed capacity](https://www.aguasto.com/research/cases/core-scientific-capacity-disclosures) - [Why Modular AI Infrastructure Matters Now](https://www.aguasto.com/research/why-modular-ai-infrastructure-matters-now) - [Official LinkedIn](https://www.linkedin.com/company/aguasto) This is a generated representation of the public HTML page. Diagrams are conceptual. GPU and inference services are planned and are not currently available to book or call. The contact page describes the available inquiry flow.