--- title: "Model serving & inference | Aguasto" description: "Connect checkpoint identity, memory policy, scheduling and measured request behavior to a defined application boundary." canonical: "https://www.aguasto.com/topics/model-serving" last-updated: "2026-10-13" --- # Model serving & inference | Aguasto Canonical HTML: [Model serving & inference](https://www.aguasto.com/topics/model-serving) > Connect checkpoint identity, memory policy, scheduling and measured request behavior to a defined application boundary. [Aguasto](https://www.aguasto.com/) [Research topics](https://www.aguasto.com/topics) Model serving & inference [Resources](https://www.aguasto.com/resources) [Topics](https://www.aguasto.com/topics) [GPU Library](https://www.aguasto.com/gpus) [Model Index](https://www.aguasto.com/models) [Inference Stack](https://www.aguasto.com/inference-stack) [Learn](https://www.aguasto.com/learn) [Insights](https://www.aguasto.com/insights) [Research](https://www.aguasto.com/research) [Engineering tools](https://www.aguasto.com/tools) [Glossary](https://www.aguasto.com/glossary) [Methods](https://www.aguasto.com/methodologies) Aguasto / Research path ## Model serving & inference Connect checkpoint identity, memory policy, scheduling and measured request behavior to a defined application boundary. Evidence before conclusions Read the mechanism, inspect its sources and carry the stated boundaries into your own evaluation. [How Aguasto Research works](https://www.aguasto.com/methodologies) Aguasto Research · Revision 1 Editorial record: 2026-10-13 [Review and correction policy ↗](https://www.aguasto.com/methodologies#review) ### What must be true before this model becomes a useful service? An open-weight checkpoint is a research artifact with its own license and provenance. A serving runtime supplies part of the execution path, while application access, request handling, observability and operating support introduce additional responsibilities. A compatible product name is not proof that a specific model, hardware backend and optimization work together. Choose a quality task before optimizing throughput. Record the exact prompt and output distributions, context policy, cache rules, concurrency, errors and latency limits. Keep prefill, decoding and application routing distinct. A separate replica may support eligible requests, but geography does not turn independent facilities into one tightly synchronized GPU fabric. ### Follow the evidence. - #### Establish checkpoint and rights Use the publisher’s exact identifier, model card and license. Parameter counts do not establish task quality, commercial rights or the cost of a full serving environment. A research-reference profile is a starting point for evaluation, not an Aguasto-hosted endpoint. [Model Index ↗](https://www.aguasto.com/models) Checkpoint identity, license and deployment context.[Compare checkpoints ↗](https://www.aguasto.com/models/compare) Inspect disclosed differences without a fabricated model ranking. - #### Understand the serving mechanism Prefill and decoding stress a system differently. Cache policies, scheduling and quantization interact with workload shape and quality. Inspect the relevant runtime documentation and expose the assumptions behind a memory estimate before treating an optimization as a result. [KV-cache policy ↗](https://www.aguasto.com/learn/kv-cache-capacity-and-policy) Capacity, reuse and isolation boundaries.[Batching and latency ↗](https://www.aguasto.com/learn/continuous-batching-and-latency) Explain scheduling before interpreting throughput. - #### Measure a service boundary Evaluate successful, quality-qualified requests under a reproducible load. Report failures and tail latency with throughput; isolate warm-up and cache conditions. Routing between replicas additionally needs spare capacity, compatible versions and an approved data-location policy. An untested failover diagram is not an SLA. [Benchmark methodology ↗](https://www.aguasto.com/research/inference-benchmark-methodology) A reproducible test record and honest measurement boundary.[Distributed inference reliability ↗](https://www.aguasto.com/learn/distributed-inference-reliability) State the failure scope and recovery conditions. ### Cornerstone resources learn / Source-backed #### What Is KV Cache? Capacity, Reuse and Isolation Connect attention-state shape, request length, allocation policy and exact-prefix reuse to a transparent inference memory and admission plan. Source dates and scope in the reference [What Is KV Cache? Capacity, Reuse and Isolation](https://www.aguasto.com/learn/kv-cache-capacity-and-policy) learn / Source-backed #### Continuous Batching: Throughput Without Hiding Latency Explain iteration scheduling, token budgets, cache admission and workload measurement without treating larger batches or utilization as proof of a better inference service. Source dates and scope in the reference [Continuous Batching: Throughput Without Hiding Latency](https://www.aguasto.com/learn/continuous-batching-and-latency) research / Source-backed #### Why tokens per second is not enough to benchmark inference Define a reproducible future Aguasto test protocol that evaluates completed useful work, latency, errors, workload and measurement boundaries. Source dates and scope in the reference [Why tokens per second is not enough to benchmark inference](https://www.aguasto.com/research/inference-benchmark-methodology) learn / Source-backed #### Distributed inference reliability begins with the failure boundary Distinguish replica recovery, control-plane recovery and regional failover, and specify the evidence needed before publishing availability claims. Source dates and scope in the reference [Distributed inference reliability begins with the failure boundary](https://www.aguasto.com/learn/distributed-inference-reliability) ### Continue the research [Prefill, decode and the cost of separating them](https://www.aguasto.com/learn/prefill-decode-and-disaggregated-serving) [Choosing between vLLM, SGLang and TensorRT-LLM](https://www.aguasto.com/learn/choosing-an-inference-serving-framework) ### Keep these boundaries close. [Model checkpoint ↗](https://www.aguasto.com/glossary#checkpoint) [KV cache ↗](https://www.aguasto.com/glossary#kv-cache) [Prefill ↗](https://www.aguasto.com/glossary#prefill) [Decode ↗](https://www.aguasto.com/glossary#decode) [Inference replica ↗](https://www.aguasto.com/glossary#replica) [Time to first token ↗](https://www.aguasto.com/glossary#ttft) ### From research to qualification Model hosting and early-interest qualification; no live API. Aguasto’s commercial offerings remain in development. No available capacity, live model API, commercial reservation or procurement relationship is implied by this research. [Explore planned managed inference ↗](https://www.aguasto.com/ai-inference) ### Sources & verification The register preserves each source's publication and verification date. Manufacturer claims and documentation are attributed; Aguasto has not run the cited products in production. - [How caching works — Hugging Face Transformers ↗](https://huggingface.co/docs/transformers/cache_explanation) Per-layer keys and values; tensor dimensions; Dynamic, static and sliding cache behavior. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [KV Cache System — NVIDIA TensorRT-LLM ↗](https://nvidia.github.io/TensorRT-LLM/features/kvcache.html) Grouped-query key-value allocation; Model-dependent cache layouts and reuse management. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Efficient Memory Management for Large Language Model Serving with PagedAttention ↗](https://arxiv.org/abs/2309.06180) Original block-based cache allocation and sharing mechanism; Historical experiment boundary. Publication: 2023-09-12. Verified: 2026-10-13. - [Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM ↗](https://nvidia.github.io/TensorRT-LLM/features/paged-attention-ifb-scheduler.html) Token budget, workspace and cache interactions; Context chunking; Utilization saturation and latency tradeoffs. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Automatic Prefix Caching — vLLM ↗](https://docs.vllm.ai/en/stable/design/prefix_caching/) Exact prefix-block identity, adapters and multimodal inputs; Optional cache salting and trust-group sharing. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Security — vLLM 0.31.0 ↗](https://docs.vllm.ai/en/v0.31.0/usage/security/) Prefix-cache timing side-channel mitigation; Cache salting as an isolation control. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Optimization and Tuning — vLLM ↗](https://docs.vllm.ai/en/latest/configuration/optimization/) Eligible chunked prefill and decode prioritization in V1; Cache pressure, preemption and recomputation. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Orca: A Distributed Serving System for Transformer-Based Generative Models — USENIX ↗](https://www.usenix.org/conference/osdi22/presentation/yu) Historical iteration-level scheduling mechanism; Whole-request versus changing batch membership. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Continuous batching — Hugging Face Transformers 5.11.0 ↗](https://huggingface.co/docs/transformers/v5.11.0/en/continuous_batching) Request handling, cache and scheduling controls; Cancellation and manager lifecycle. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Metrics — NVIDIA NIM Benchmarking ↗](https://docs.nvidia.com/nim/benchmarking/llm/metrics) Client-observed first-token and generation timing; Queueing and network measurement boundaries. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Metrics — NVIDIA NIM Benchmarking 2.0.0 ↗](https://docs.nvidia.com/nim/benchmarking/llm/2.0.0/metrics) Client TTFT and end-to-end latency boundaries; ITL excludes first-token latency; denominator output tokens minus one; Different tools use different definitions. Publication: date not stated in the reviewed source. Verified: 2026-10-12. - [Parameters and Best Practices — NVIDIA NIM Benchmarking ↗](https://docs.nvidia.com/nim/benchmarking/llm/parameters) Concurrency and request-rate load modes; Arrival rate can exceed capacity and grow queues; Prompt/output lengths and batch size matter. Publication: date not stated in the reviewed source. Verified: 2026-10-12. - [Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving ↗](https://arxiv.org/abs/2410.14257v2) Primary authors discuss incentive problems with SLO/system metrics; Smooth-goodput framework; no Aguasto replication. Last modified: 2025-10-29. Verified: 2026-10-12. - [MLPerf Inference: Datacenter — MLCommons ↗](https://mlcommons.org/benchmarks/inference-datacenter/) Results include submitter, software, accelerator count and metadata; Power measured at full-system AC boundary. Publication: date not stated in the reviewed source. Verified: 2026-10-12. - [MLPerf Results Messaging Guidelines — MLCommons ↗](https://github.com/mlcommons/policies/blob/master/MLPerf_Results_Messaging_Guidelines.adoc) Identify benchmark version, division, category, verification status, scenario and chip count; Do not compare MLPerf with arbitrary non-MLPerf results. Publication: date not stated in the reviewed source. Verified: 2026-10-12. - [Add End-to-End Fault Tolerance — Ray Serve ↗](https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html) Replica recovery differs from head/control-plane recovery; KubeRay and external GCS state required for documented head recovery; Spread limits and recovery constraints. Publication: date not stated in the reviewed source. Verified: 2026-10-12. - [Disruptions — Kubernetes official documentation ↗](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) Voluntary versus involuntary failures; PodDisruptionBudgets do not prevent involuntary failure; Spread replicas across failure domains. Publication: date not stated in the reviewed source. Verified: 2026-10-12. - [Inside Fireworks Multi-region Deployments: One Deployment, Global Scale ↗](https://fireworks.ai/blog/multi-region-deployment) Provider-authored distinction between placement and request-path routing; Data-residency and eligible-capacity boundaries; Does not eliminate physical network distance. Publication: 2026-09-28. Verified: 2026-10-12. ### In this research path [The decision](https://www.aguasto.com/topics/model-serving#decision) [Reading sequence](https://www.aguasto.com/topics/model-serving#reading-path) [Cornerstone resources](https://www.aguasto.com/topics/model-serving#cornerstones) [Useful definitions](https://www.aguasto.com/topics/model-serving#definitions) [Sources and verification](https://www.aguasto.com/topics/model-serving#sources) [All research paths ↗](https://www.aguasto.com/topics) ## Company navigation - [Home](https://www.aguasto.com/) - [Infrastructure](https://www.aguasto.com/infrastructure) - [GPU Cloud](https://www.aguasto.com/gpu-cloud) - [AI Inference](https://www.aguasto.com/ai-inference) - [Development & Investment](https://www.aguasto.com/development) - [Company](https://www.aguasto.com/company) - [Contact](https://www.aguasto.com/contact) - [Privacy](https://www.aguasto.com/privacy) - [Research topics](https://www.aguasto.com/topics) - [GPU systems & local clusters](https://www.aguasto.com/topics/gpu-systems) - [Model serving & inference](https://www.aguasto.com/topics/model-serving) - [Modular infrastructure & resources](https://www.aguasto.com/topics/modular-infrastructure) - [Compute economics & evidence](https://www.aguasto.com/topics/compute-economics) - [Glossary](https://www.aguasto.com/glossary) - [Methodologies](https://www.aguasto.com/methodologies) - [Resources](https://www.aguasto.com/resources) - [GPU Library](https://www.aguasto.com/gpus) - [Model Index](https://www.aguasto.com/models) - [Inference Stack](https://www.aguasto.com/inference-stack) - [Learn](https://www.aguasto.com/learn) - [Insights](https://www.aguasto.com/insights) - [Research](https://www.aguasto.com/research) - [Industry cases](https://www.aguasto.com/research/cases) - [Why Modular](https://www.aguasto.com/infrastructure/why-modular) - [Cooling & Resource Efficiency](https://www.aguasto.com/infrastructure/cooling-resource-efficiency) - [Partners](https://www.aguasto.com/partners) - [GPU comparison](https://www.aguasto.com/gpus/compare) - [Model comparison](https://www.aguasto.com/models/compare) - [Engineering tools](https://www.aguasto.com/tools) - [Development friction](https://www.aguasto.com/research/hyperscale-development-friction) - [NVIDIA H100 SXM](https://www.aguasto.com/gpus/h100) - [NVIDIA H200 SXM](https://www.aguasto.com/gpus/h200) - [NVIDIA B200 SXM](https://www.aguasto.com/gpus/b200) - [NVIDIA B300](https://www.aguasto.com/gpus/b300) - [NVIDIA GB200 NVL72](https://www.aguasto.com/gpus/gb200-nvl72) - [NVIDIA GB300 NVL72](https://www.aguasto.com/gpus/gb300-nvl72) - [NVIDIA Vera Rubin NVL72](https://www.aguasto.com/gpus/vera-rubin-nvl72) - [AMD Instinct MI300X](https://www.aguasto.com/gpus/mi300x) - [AMD Instinct MI325X](https://www.aguasto.com/gpus/mi325x) - [AMD Instinct MI355X](https://www.aguasto.com/gpus/mi355x) - [AMD Instinct MI455X](https://www.aguasto.com/gpus/mi455x) - [AMD Helios](https://www.aguasto.com/gpus/helios) - [GLM-4.5](https://www.aguasto.com/models/glm-4-5) - [Qwen3-235B-A22B](https://www.aguasto.com/models/qwen3-235b-a22b) - [DeepSeek-R1](https://www.aguasto.com/models/deepseek-r1) - [Llama 3.3 70B Instruct](https://www.aguasto.com/models/llama-3-3-70b-instruct) - [Gemma 3 27B IT](https://www.aguasto.com/models/gemma-3-27b-it) - [gpt-oss-120b](https://www.aguasto.com/models/gpt-oss-120b) - [Kimi-K2-Instruct](https://www.aguasto.com/models/kimi-k2-instruct) - [GLM-5.3](https://www.aguasto.com/models/glm-5-3) - [Qwen3.8-27B](https://www.aguasto.com/models/qwen3-8-27b) - [DeepSeek-V4.1-Flash](https://www.aguasto.com/models/deepseek-v4-1-flash) - [Llama 4 Scout 17B-16E Instruct](https://www.aguasto.com/models/llama-4-scout-17b-16e-instruct) - [Gemma 4 31B IT](https://www.aguasto.com/models/gemma-4-31b-it) - [Kimi K3](https://www.aguasto.com/models/kimi-k3) - [vLLM](https://www.aguasto.com/inference-stack/vllm) - [SGLang](https://www.aguasto.com/inference-stack/sglang) - [TensorRT-LLM](https://www.aguasto.com/inference-stack/tensorrt-llm) - [NVIDIA Dynamo](https://www.aguasto.com/inference-stack/dynamo) - [NVIDIA Triton Inference Server](https://www.aguasto.com/inference-stack/triton) - [AMD ROCm](https://www.aguasto.com/inference-stack/rocm) - [Ray Serve](https://www.aguasto.com/inference-stack/ray-serve) - [Kubernetes GPU orchestration — NVIDIA device plugin](https://www.aguasto.com/inference-stack/kubernetes) - [Model weights are only the beginning of the memory budget](https://www.aguasto.com/learn/model-weights-and-runtime-memory) - [Why tokens per second is not enough to benchmark inference](https://www.aguasto.com/research/inference-benchmark-methodology) - [Prefill, decode and the cost of separating them](https://www.aguasto.com/learn/prefill-decode-and-disaggregated-serving) - [Choosing between vLLM, SGLang and TensorRT-LLM](https://www.aguasto.com/learn/choosing-an-inference-serving-framework) - [From Power to Inference—Building a Distributed AI Compute Network](https://www.aguasto.com/learn/from-power-to-inference) - [Dedicated compute and shared inference: compare the whole workload](https://www.aguasto.com/insights/dedicated-and-shared-inference-economics) - [Distributed inference reliability begins with the failure boundary](https://www.aguasto.com/learn/distributed-inference-reliability) - [Core Scientific: reading contracted, energized and billed capacity](https://www.aguasto.com/research/cases/core-scientific-capacity-disclosures) - [Why Modular AI Infrastructure Matters Now](https://www.aguasto.com/research/why-modular-ai-infrastructure-matters-now) - [What Is a Modular AI Data Center?](https://www.aguasto.com/learn/what-is-a-modular-ai-data-center) - [How Modular AI Data Centers Are Built and Commissioned](https://www.aguasto.com/learn/modular-data-center-factory-to-commissioning) - [Liquid Cooling for High-Density GPU Systems: Follow the Interfaces](https://www.aguasto.com/learn/liquid-cooling-gpu-rack-interfaces) - [Why Electricity Availability Shapes AI Infrastructure Deployment](https://www.aguasto.com/research/electricity-availability-and-time-to-power) - [What Is KV Cache? Capacity, Reuse and Isolation](https://www.aguasto.com/learn/kv-cache-capacity-and-policy) - [Continuous Batching: Throughput Without Hiding Latency](https://www.aguasto.com/learn/continuous-batching-and-latency) - [Applied Digital: Reading the Polaris Forge 1 Delivery Milestones](https://www.aguasto.com/research/cases/applied-digital-polaris-forge-capacity-milestones) - [Crusoe Abilene: Separate Delivered Infrastructure from Expansion Plans](https://www.aguasto.com/research/cases/crusoe-abilene-delivery-and-expansion) - [Data Center Cooling: Choosing Air, Liquid, Dry and Hybrid Architectures](https://www.aguasto.com/learn/data-center-cooling-air-liquid-dry-hybrid) - [Can Data Centers Reduce Water Consumption? Define the Water Boundary First](https://www.aguasto.com/research/data-center-water-consumption-and-withdrawal) - [Measuring PUE, WUE and AI Compute Efficiency at the Right Boundary](https://www.aguasto.com/learn/measuring-pue-wue-and-ai-compute-efficiency) - [The Environmental Trade-offs of High-Density AI Infrastructure](https://www.aguasto.com/research/environmental-tradeoffs-and-infrastructure-reuse) - [Selecting GPU Systems for Enterprise Inference](https://www.aguasto.com/learn/selecting-enterprise-inference-gpu-systems) - [Networking Rack-Scale AI Systems: Define the Communication Boundary](https://www.aguasto.com/learn/rack-scale-ai-networking-boundaries) - [Official LinkedIn](https://www.linkedin.com/company/aguasto) This is a generated representation of the public HTML page. Diagrams are conceptual. GPU and inference services are planned and are not currently available to book or call. The contact page describes the available inquiry flow.