---
title: "Why tokens per second is not enough to benchmark inference | Aguasto"
description: "Define a reproducible future Aguasto test protocol that evaluates completed useful work, latency, errors, workload and measurement boundaries."
canonical: "https://www.aguasto.com/research/inference-benchmark-methodology"
last-updated: "2026-10-12"
---

# Why tokens per second is not enough to benchmark inference | Aguasto

Canonical HTML: [Why tokens per second is not enough to benchmark inference](https://www.aguasto.com/research/inference-benchmark-methodology)

> Define a reproducible future Aguasto test protocol that evaluates completed useful work, latency, errors, workload and measurement boundaries.

[Aguasto](https://www.aguasto.com/) [research](https://www.aguasto.com/research)  Why tokens per second is not enough to benchmark inference

[Resources](https://www.aguasto.com/resources) [GPU Library](https://www.aguasto.com/gpus) [Model Index](https://www.aguasto.com/models) [Inference Stack](https://www.aguasto.com/inference-stack) [Learn](https://www.aguasto.com/learn) [Insights](https://www.aguasto.com/insights) [Research](https://www.aguasto.com/research) [Engineering tools](https://www.aguasto.com/tools)

Aguasto / research

## Why tokens per second is not enough to benchmark inference

Define a reproducible future Aguasto test protocol that evaluates completed useful work, latency, errors, workload and measurement boundaries.

Original Aguasto conceptual illustration. Not manufacturer geometry, an operating Aguasto asset, or a measured benchmark.

By Aguasto Research · Published October 12, 2026 · Revision 1

Aguasto Research documents third-party hardware, models and software. References do not represent Aguasto inventory, evaluated deployments or available inference services.

### Measure what the user waits for

NVIDIA's benchmarking documentation defines time to first token as the interval from query submission to the first received content token. It can include queueing, prefill and network delay. End-to-end latency ends at the final token. Its AIPerf inter-token latency excludes the first token and uses (end-to-end latency − TTFT) ÷ (output tokens − 1). A one-token response therefore needs separate handling rather than division by zero.

Aguasto's proposed record will retain these boundaries and name the benchmark client. A server-side kernel timer, a client-side streaming measurement and a nonstreaming API measurement answer different questions; publish them in separate columns.

[Metrics — NVIDIA NIM Benchmarking 2.0.0 (external source)](https://docs.nvidia.com/nim/benchmarking/llm/2.0.0/metrics)

### Keep the offered load visible

Concurrency and arrival rate are distinct controls. NVIDIA documents both and notes that queues can grow when arrivals exceed completed throughput. A concurrency-controlled test feeds work as outstanding requests finish, while an arrival-rate test asks whether the system sustains a specified demand pattern. Input length, output length and engine batch constraints must accompany either result.

For the proposed protocol, test a small matrix of short and long prompts, short and long outputs, and increasing load. Use the same tokenized workload and model revision across configurations. Publish the load setting and completed request count, not just the best point found in a sweep.

[Parameters and Best Practices — NVIDIA NIM Benchmarking (external source)](https://docs.nvidia.com/nim/benchmarking/llm/parameters)

### Define goodput before the run

A useful operational metric should count work that meets a declared service objective. Primary research on LLM SLOs warns that some aggregate metrics can be improved by delaying tokens or abandoning difficult requests, and proposes a smooth-goodput framework. Aguasto does not reproduce that research or claim its reported results.

For an initial Aguasto methodology, declare request goodput as successfully completed requests meeting both the chosen TTFT and end-to-end limits divided by elapsed measurement seconds. If using a streaming smoothness objective, report per-token stalls separately. Publish accepted, rejected, timed-out and cancelled requests, so dropping work cannot silently improve the headline.

[Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving (external source)](https://arxiv.org/abs/2410.14257)

### Freeze the configuration and quality gate

The proposed run manifest includes hardware and device count, host CPU and memory, interconnect, driver/runtime versions, engine commit or container digest, model/tokenizer revision, precision, parallelism, cache policy and decoding settings. Save a workload hash and client location. Define the task-quality check before comparing a quantized variant.

Separate cold-start observations from warmed steady state. Record warmup duration, repeated-run count, test duration and raw request timestamps; these are protocol choices, not claims that testing occurred. Keep the full latency distribution and failure counts. A median alone does not establish behavior under bursts or at the tail.

### Use external benchmark evidence faithfully

MLPerf result pages provide system, software, accelerator count and submission metadata. MLCommons' result-use guidance requires differences such as version, division, category, verification status, scenario and chip count to be identified, and prohibits arbitrary non-MLPerf results being presented as directly comparable MLPerf results.

Aguasto should link the precise submission and explain which question it answers. Do not divide a rack submission by its GPU count and label the result a measured standalone accelerator score. Keep vendor-run, independently submitted and future Aguasto-run evidence distinct, even when all are technically useful.

[MLPerf Inference: Datacenter — MLCommons (external source)](https://mlcommons.org/benchmarks/inference-datacenter/)   · [MLPerf Results Messaging Guidelines — MLCommons (external source)](https://github.com/mlcommons/policies/blob/master/MLPerf_Results_Messaging_Guidelines.adoc)

### Publish a decision, with its limits

The proposed result table pairs latency-qualified throughput with errors, actual completed output tokens and measured cost over the same interval. Add confidence notes from repeat runs and disclose whether prefix caches were warm or empty. Choose a configuration based on the application's objective rather than a universal winner.

Until dedicated test hardware and reproducible run artifacts exist, this page is a methodology. It contains no Aguasto performance score, synthetic leaderboard or claimed service level. Its immediate value is a test plan buyers can inspect: what would be measured, how successful work is counted, and what evidence would justify the next configuration decision.

### Scope & limitations

- Proposed protocol; no Aguasto GPU benchmark was executed.

- Goodput thresholds are workload choices, not Aguasto service guarantees.

- External benchmark conditions must be retained.

### Continue the research

[NVIDIA H100 SXM](https://www.aguasto.com/gpus/h100) [NVIDIA H200 SXM](https://www.aguasto.com/gpus/h200) [NVIDIA B200 SXM](https://www.aguasto.com/gpus/b200) [AMD Instinct MI355X](https://www.aguasto.com/gpus/mi355x) [vLLM](https://www.aguasto.com/inference-stack/vllm) [SGLang](https://www.aguasto.com/inference-stack/sglang) [TensorRT-LLM](https://www.aguasto.com/inference-stack/tensorrt-llm) [NVIDIA Dynamo](https://www.aguasto.com/inference-stack/dynamo) [The Aguasto modular thesis](https://www.aguasto.com/why-modular) [Potential OEM and strategic collaboration](https://www.aguasto.com/partners)

### Sources & verification

Reference revision 1 · Reviewed October 12, 2026. Manufacturer claims and documentation are attributed; Aguasto has not run the cited products in production.

- [Metrics — NVIDIA NIM Benchmarking 2.0.0 ↗](https://docs.nvidia.com/nim/benchmarking/llm/2.0.0/metrics)

Client TTFT and end-to-end latency boundaries; ITL excludes first-token latency; denominator output tokens minus one; Different tools use different definitions. Publication: date not stated in the reviewed source. Verified: 2026-10-12.

- [Parameters and Best Practices — NVIDIA NIM Benchmarking ↗](https://docs.nvidia.com/nim/benchmarking/llm/parameters)

Concurrency and request-rate load modes; Arrival rate can exceed capacity and grow queues; Prompt/output lengths and batch size matter. Publication: date not stated in the reviewed source. Verified: 2026-10-12.

- [Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving ↗](https://arxiv.org/abs/2410.14257)

Primary authors discuss incentive problems with SLO/system metrics; Smooth-goodput framework; no Aguasto replication. Publication: 2025-10-29. Verified: 2026-10-12.

- [MLPerf Inference: Datacenter — MLCommons ↗](https://mlcommons.org/benchmarks/inference-datacenter/)

Results include submitter, software, accelerator count and metadata; Power measured at full-system AC boundary. Publication: date not stated in the reviewed source. Verified: 2026-10-12.

- [MLPerf Results Messaging Guidelines — MLCommons ↗](https://github.com/mlcommons/policies/blob/master/MLPerf_Results_Messaging_Guidelines.adoc)

Identify benchmark version, division, category, verification status, scenario and chip count; Do not compare MLPerf with arbitrary non-MLPerf results. Publication: date not stated in the reviewed source. Verified: 2026-10-12.

### In this publication

[Measure what the user waits for](https://www.aguasto.com/research/inference-benchmark-methodology#section-1) [Keep the offered load visible](https://www.aguasto.com/research/inference-benchmark-methodology#section-2) [Define goodput before the run](https://www.aguasto.com/research/inference-benchmark-methodology#section-3) [Freeze the configuration and quality gate](https://www.aguasto.com/research/inference-benchmark-methodology#section-4) [Use external benchmark evidence faithfully](https://www.aguasto.com/research/inference-benchmark-methodology#section-5) [Publish a decision, with its limits](https://www.aguasto.com/research/inference-benchmark-methodology#section-6) [More research ↗](https://www.aguasto.com/research) [Source register ↓](https://www.aguasto.com/research/inference-benchmark-methodology#sources)

## Company navigation

- [Home](https://www.aguasto.com/)
- [Infrastructure](https://www.aguasto.com/infrastructure)
- [GPU Cloud](https://www.aguasto.com/gpu-cloud)
- [AI Inference](https://www.aguasto.com/ai-inference)
- [Development & Investment](https://www.aguasto.com/development)
- [Company](https://www.aguasto.com/company)
- [Contact](https://www.aguasto.com/contact)
- [Privacy](https://www.aguasto.com/privacy)
- [Resources](https://www.aguasto.com/resources)
- [GPU Library](https://www.aguasto.com/gpus)
- [Model Index](https://www.aguasto.com/models)
- [Inference Stack](https://www.aguasto.com/inference-stack)
- [Learn](https://www.aguasto.com/learn)
- [Insights](https://www.aguasto.com/insights)
- [Research](https://www.aguasto.com/research)
- [Industry cases](https://www.aguasto.com/research/cases)
- [Why Modular](https://www.aguasto.com/why-modular)
- [Partners](https://www.aguasto.com/partners)
- [GPU comparison](https://www.aguasto.com/gpus/compare)
- [Model comparison](https://www.aguasto.com/models/compare)
- [Engineering tools](https://www.aguasto.com/tools)
- [Development friction](https://www.aguasto.com/research/hyperscale-development-friction)
- [NVIDIA H100 SXM](https://www.aguasto.com/gpus/h100)
- [NVIDIA H200 SXM](https://www.aguasto.com/gpus/h200)
- [NVIDIA B200 SXM](https://www.aguasto.com/gpus/b200)
- [NVIDIA B300](https://www.aguasto.com/gpus/b300)
- [NVIDIA GB200 NVL72](https://www.aguasto.com/gpus/gb200-nvl72)
- [NVIDIA GB300 NVL72](https://www.aguasto.com/gpus/gb300-nvl72)
- [NVIDIA Vera Rubin NVL72](https://www.aguasto.com/gpus/vera-rubin-nvl72)
- [AMD Instinct MI300X](https://www.aguasto.com/gpus/mi300x)
- [AMD Instinct MI325X](https://www.aguasto.com/gpus/mi325x)
- [AMD Instinct MI355X](https://www.aguasto.com/gpus/mi355x)
- [AMD Instinct MI455X](https://www.aguasto.com/gpus/mi455x)
- [AMD Helios](https://www.aguasto.com/gpus/helios)
- [GLM-4.5](https://www.aguasto.com/models/glm-4-5)
- [Qwen3-235B-A22B](https://www.aguasto.com/models/qwen3-235b-a22b)
- [DeepSeek-R1](https://www.aguasto.com/models/deepseek-r1)
- [Llama 3.3 70B Instruct](https://www.aguasto.com/models/llama-3-3-70b-instruct)
- [Gemma 3 27B IT](https://www.aguasto.com/models/gemma-3-27b-it)
- [gpt-oss-120b](https://www.aguasto.com/models/gpt-oss-120b)
- [Kimi-K2-Instruct](https://www.aguasto.com/models/kimi-k2-instruct)
- [vLLM](https://www.aguasto.com/inference-stack/vllm)
- [SGLang](https://www.aguasto.com/inference-stack/sglang)
- [TensorRT-LLM](https://www.aguasto.com/inference-stack/tensorrt-llm)
- [NVIDIA Dynamo](https://www.aguasto.com/inference-stack/dynamo)
- [NVIDIA Triton Inference Server](https://www.aguasto.com/inference-stack/triton)
- [AMD ROCm](https://www.aguasto.com/inference-stack/rocm)
- [Ray Serve](https://www.aguasto.com/inference-stack/ray-serve)
- [Kubernetes GPU orchestration — NVIDIA device plugin](https://www.aguasto.com/inference-stack/kubernetes)
- [Model weights are only the beginning of the memory budget](https://www.aguasto.com/learn/model-weights-and-runtime-memory)
- [Why tokens per second is not enough to benchmark inference](https://www.aguasto.com/research/inference-benchmark-methodology)
- [Prefill, decode and the cost of separating them](https://www.aguasto.com/learn/prefill-decode-and-disaggregated-serving)
- [Choosing between vLLM, SGLang and TensorRT-LLM](https://www.aguasto.com/learn/choosing-an-inference-serving-framework)
- [From Power to Inference—Building a Distributed AI Compute Network](https://www.aguasto.com/learn/from-power-to-inference)
- [Dedicated compute and shared inference: compare the whole workload](https://www.aguasto.com/insights/dedicated-and-shared-inference-economics)
- [Distributed inference reliability begins with the failure boundary](https://www.aguasto.com/learn/distributed-inference-reliability)
- [Core Scientific: reading contracted, energized and billed capacity](https://www.aguasto.com/research/cases/core-scientific-capacity-disclosures)
- [Why Modular AI Infrastructure Matters Now](https://www.aguasto.com/research/why-modular-ai-infrastructure-matters-now)
- [Official LinkedIn](https://www.linkedin.com/company/aguasto)

This is a generated representation of the public HTML page. Diagrams are conceptual. GPU and inference services are planned and are not currently available to book or call. The contact page describes the available inquiry flow.