Aguasto Research · Revision 1Editorial record: Review and correction policy ↗
What must be true before this model becomes a useful service?
An open-weight checkpoint is a research artifact with its own license and provenance. A serving runtime supplies part of the execution path, while application access, request handling, observability and operating support introduce additional responsibilities. A compatible product name is not proof that a specific model, hardware backend and optimization work together.
Choose a quality task before optimizing throughput. Record the exact prompt and output distributions, context policy, cache rules, concurrency, errors and latency limits. Keep prefill, decoding and application routing distinct. A separate replica may support eligible requests, but geography does not turn independent facilities into one tightly synchronized GPU fabric.
Follow the evidence.
Establish checkpoint and rights
Use the publisher’s exact identifier, model card and license. Parameter counts do not establish task quality, commercial rights or the cost of a full serving environment. A research-reference profile is a starting point for evaluation, not an Aguasto-hosted endpoint.
Model Index ↗
Checkpoint identity, license and deployment context.Compare checkpoints ↗
Inspect disclosed differences without a fabricated model ranking.Understand the serving mechanism
Prefill and decoding stress a system differently. Cache policies, scheduling and quantization interact with workload shape and quality. Inspect the relevant runtime documentation and expose the assumptions behind a memory estimate before treating an optimization as a result.
KV-cache policy ↗
Capacity, reuse and isolation boundaries.Batching and latency ↗
Explain scheduling before interpreting throughput.Measure a service boundary
Evaluate successful, quality-qualified requests under a reproducible load. Report failures and tail latency with throughput; isolate warm-up and cache conditions. Routing between replicas additionally needs spare capacity, compatible versions and an approved data-location policy. An untested failover diagram is not an SLA.
Benchmark methodology ↗
A reproducible test record and honest measurement boundary.Distributed inference reliability ↗
State the failure scope and recovery conditions.
Cornerstone resources
What Is KV Cache? Capacity, Reuse and Isolation
Connect attention-state shape, request length, allocation policy and exact-prefix reuse to a transparent inference memory and admission plan.
Continuous Batching: Throughput Without Hiding Latency
Explain iteration scheduling, token budgets, cache admission and workload measurement without treating larger batches or utilization as proof of a better inference service.
Why tokens per second is not enough to benchmark inference
Define a reproducible future Aguasto test protocol that evaluates completed useful work, latency, errors, workload and measurement boundaries.
Distributed inference reliability begins with the failure boundary
Distinguish replica recovery, control-plane recovery and regional failover, and specify the evidence needed before publishing availability claims.
Keep these boundaries close.
Sources & verification
The register preserves each source's publication and verification date. Manufacturer claims and documentation are attributed; Aguasto has not run the cited products in production.
- How caching works — Hugging Face Transformers ↗
Per-layer keys and values; tensor dimensions; Dynamic, static and sliding cache behavior. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- KV Cache System — NVIDIA TensorRT-LLM ↗
Grouped-query key-value allocation; Model-dependent cache layouts and reuse management. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Efficient Memory Management for Large Language Model Serving with PagedAttention ↗
Original block-based cache allocation and sharing mechanism; Historical experiment boundary. Publication: 2023-09-12. Verified: 2026-10-13.
- Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM ↗
Token budget, workspace and cache interactions; Context chunking; Utilization saturation and latency tradeoffs. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Automatic Prefix Caching — vLLM ↗
Exact prefix-block identity, adapters and multimodal inputs; Optional cache salting and trust-group sharing. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Security — vLLM 0.31.0 ↗
Prefix-cache timing side-channel mitigation; Cache salting as an isolation control. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Optimization and Tuning — vLLM ↗
Eligible chunked prefill and decode prioritization in V1; Cache pressure, preemption and recomputation. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Orca: A Distributed Serving System for Transformer-Based Generative Models — USENIX ↗
Historical iteration-level scheduling mechanism; Whole-request versus changing batch membership. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Continuous batching — Hugging Face Transformers 5.11.0 ↗
Request handling, cache and scheduling controls; Cancellation and manager lifecycle. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Metrics — NVIDIA NIM Benchmarking ↗
Client-observed first-token and generation timing; Queueing and network measurement boundaries. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Metrics — NVIDIA NIM Benchmarking 2.0.0 ↗
Client TTFT and end-to-end latency boundaries; ITL excludes first-token latency; denominator output tokens minus one; Different tools use different definitions. Publication: date not stated in the reviewed source. Verified: 2026-10-12.
- Parameters and Best Practices — NVIDIA NIM Benchmarking ↗
Concurrency and request-rate load modes; Arrival rate can exceed capacity and grow queues; Prompt/output lengths and batch size matter. Publication: date not stated in the reviewed source. Verified: 2026-10-12.
- Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving ↗
Primary authors discuss incentive problems with SLO/system metrics; Smooth-goodput framework; no Aguasto replication. Last modified: 2025-10-29. Verified: 2026-10-12.
- MLPerf Inference: Datacenter — MLCommons ↗
Results include submitter, software, accelerator count and metadata; Power measured at full-system AC boundary. Publication: date not stated in the reviewed source. Verified: 2026-10-12.
- MLPerf Results Messaging Guidelines — MLCommons ↗
Identify benchmark version, division, category, verification status, scenario and chip count; Do not compare MLPerf with arbitrary non-MLPerf results. Publication: date not stated in the reviewed source. Verified: 2026-10-12.
- Add End-to-End Fault Tolerance — Ray Serve ↗
Replica recovery differs from head/control-plane recovery; KubeRay and external GCS state required for documented head recovery; Spread limits and recovery constraints. Publication: date not stated in the reviewed source. Verified: 2026-10-12.
- Disruptions — Kubernetes official documentation ↗
Voluntary versus involuntary failures; PodDisruptionBudgets do not prevent involuntary failure; Spread replicas across failure domains. Publication: date not stated in the reviewed source. Verified: 2026-10-12.
- Inside Fireworks Multi-region Deployments: One Deployment, Global Scale ↗
Provider-authored distinction between placement and request-path routing; Data-residency and eligible-capacity boundaries; Does not eliminate physical network distance. Publication: 2026-09-28. Verified: 2026-10-12.