Measure what the user waits for
NVIDIA's benchmarking documentation defines time to first token as the interval from query submission to the first received content token. It can include queueing, prefill and network delay. End-to-end latency ends at the final token. Its AIPerf inter-token latency excludes the first token and uses (end-to-end latency − TTFT) ÷ (output tokens − 1). A one-token response therefore needs separate handling rather than division by zero.
Aguasto's proposed record will retain these boundaries and name the benchmark client. A server-side kernel timer, a client-side streaming measurement and a nonstreaming API measurement answer different questions; publish them in separate columns.
Metrics — NVIDIA NIM Benchmarking 2.0.0 (external source)Keep the offered load visible
Concurrency and arrival rate are distinct controls. NVIDIA documents both and notes that queues can grow when arrivals exceed completed throughput. A concurrency-controlled test feeds work as outstanding requests finish, while an arrival-rate test asks whether the system sustains a specified demand pattern. Input length, output length and engine batch constraints must accompany either result.
For the proposed protocol, test a small matrix of short and long prompts, short and long outputs, and increasing load. Use the same tokenized workload and model revision across configurations. Publish the load setting and completed request count, not just the best point found in a sweep.
Parameters and Best Practices — NVIDIA NIM Benchmarking (external source)Define goodput before the run
A useful operational metric should count work that meets a declared service objective. Primary research on LLM SLOs warns that some aggregate metrics can be improved by delaying tokens or abandoning difficult requests, and proposes a smooth-goodput framework. Aguasto does not reproduce that research or claim its reported results.
For an initial Aguasto methodology, declare request goodput as successfully completed requests meeting both the chosen TTFT and end-to-end limits divided by elapsed measurement seconds. If using a streaming smoothness objective, report per-token stalls separately. Publish accepted, rejected, timed-out and cancelled requests, so dropping work cannot silently improve the headline.
Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving (external source)Freeze the configuration and quality gate
The proposed run manifest includes hardware and device count, host CPU and memory, interconnect, driver/runtime versions, engine commit or container digest, model/tokenizer revision, precision, parallelism, cache policy and decoding settings. Save a workload hash and client location. Define the task-quality check before comparing a quantized variant.
Separate cold-start observations from warmed steady state. Record warmup duration, repeated-run count, test duration and raw request timestamps; these are protocol choices, not claims that testing occurred. Keep the full latency distribution and failure counts. A median alone does not establish behavior under bursts or at the tail.
Use external benchmark evidence faithfully
MLPerf result pages provide system, software, accelerator count and submission metadata. MLCommons' result-use guidance requires differences such as version, division, category, verification status, scenario and chip count to be identified, and prohibits arbitrary non-MLPerf results being presented as directly comparable MLPerf results.
Aguasto should link the precise submission and explain which question it answers. Do not divide a rack submission by its GPU count and label the result a measured standalone accelerator score. Keep vendor-run, independently submitted and future Aguasto-run evidence distinct, even when all are technically useful.
MLPerf Inference: Datacenter — MLCommons (external source) · MLPerf Results Messaging Guidelines — MLCommons (external source)Publish a decision, with its limits
The proposed result table pairs latency-qualified throughput with errors, actual completed output tokens and measured cost over the same interval. Add confidence notes from repeat runs and disclose whether prefix caches were warm or empty. Choose a configuration based on the application's objective rather than a universal winner.
Until dedicated test hardware and reproducible run artifacts exist, this page is a methodology. It contains no Aguasto performance score, synthetic leaderboard or claimed service level. Its immediate value is a test plan buyers can inspect: what would be measured, how successful work is counted, and what evidence would justify the next configuration decision.
Sources & verification
Reference revision 1 · Reviewed . Manufacturer claims and documentation are attributed; Aguasto has not run the cited products in production.
- Metrics — NVIDIA NIM Benchmarking 2.0.0 ↗
Client TTFT and end-to-end latency boundaries; ITL excludes first-token latency; denominator output tokens minus one; Different tools use different definitions. Publication: date not stated in the reviewed source. Verified: 2026-10-12.
- Parameters and Best Practices — NVIDIA NIM Benchmarking ↗
Concurrency and request-rate load modes; Arrival rate can exceed capacity and grow queues; Prompt/output lengths and batch size matter. Publication: date not stated in the reviewed source. Verified: 2026-10-12.
- Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving ↗
Primary authors discuss incentive problems with SLO/system metrics; Smooth-goodput framework; no Aguasto replication. Publication: 2025-10-29. Verified: 2026-10-12.
- MLPerf Inference: Datacenter — MLCommons ↗
Results include submitter, software, accelerator count and metadata; Power measured at full-system AC boundary. Publication: date not stated in the reviewed source. Verified: 2026-10-12.
- MLPerf Results Messaging Guidelines — MLCommons ↗
Identify benchmark version, division, category, verification status, scenario and chip count; Do not compare MLPerf with arbitrary non-MLPerf results. Publication: date not stated in the reviewed source. Verified: 2026-10-12.