Requests finish at different times
A serving system receives prompts at different moments and generates answers of different lengths. Whole-request grouping can leave a newly arrived request waiting while an existing group completes. The ORCA research introduced iteration-level scheduling: invoke a generation iteration on the current batch, then change its membership as work progresses. Current continuous-batching documentation describes related request handling in a maintained serving implementation.
Consider an explicitly hypothetical sequence in which A needs six decode steps, B needs two, and C arrives while both are active. After B finishes, a scheduler may admit C at an eligible iteration instead of waiting for A's entire answer. This illustrates membership, not speed. Prefill work, available cache, admission policy and engine constraints still decide whether that slot can actually accept C.
Orca: A Distributed Serving System for Transformer-Based Generative Models — USENIX (external source) · Continuous batching — Hugging Face Transformers 5.11.0 (external source)Separate requests from tokens and state
The number of active requests is only one scheduling quantity. A token budget describes work placed into an iteration; retained cache describes state kept across iterations; the response stream describes what a client receives. A long prompt can consume substantial prefill work before its first answer token, while an already active decode request contributes a different amount of work to a step.
A useful conceptual diagram labels these quantities independently. Give each request an arrival, retained-state block and pending-work segment. Show the scheduler selecting work under a token budget rather than drawing one identical box per customer. Hugging Face exposes separate scheduling and cache configuration, and NVIDIA discusses how token limits affect workspace and cache allocation. Their settings are implementation-specific, not interchangeable knobs with universal values.
Continuous batching — Hugging Face Transformers 5.11.0 (external source) · Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM (external source)Balance prefill against ongoing generation
Chunked prefill splits eligible prompt processing into portions that can be scheduled alongside generation. Current vLLM V1 documentation describes prioritizing pending decode work and using the remaining token budget for eligible prefills, splitting a prefill that cannot fit. NVIDIA also documents context chunking and its relationship to paged cache. The precise behavior depends on the engine and selected configuration.
The evaluation question is whose waiting time a policy improves and what it costs elsewhere. A policy that protects an ongoing stream may make a newly admitted long prompt wait; another choice may favor prompt intake while changing token cadence. Define the intended service objective before selecting a policy. Keep prompt-length distribution, output lengths and arrival pattern visible so a favorable homogeneous test does not stand in for the actual customer workload.
Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM (external source) · Optimization and Tuning — vLLM (external source)More scheduled work is not always better
NVIDIA's scheduling documentation notes that utilization can plateau and that increasing scheduled tokens beyond that point may harm first-token and complete-request latency. A high token-throughput total can therefore coexist with a slower experience for an individual request. NVIDIA's benchmarking documentation separately defines client-observed first-token and generation timing, including relevant queueing and network effects.
For a proposed acceptance test, declare the objectives before the run. Retain first-token latency, streaming cadence and completion latency as distributions, not just averages. State whether throughput counts input processing, generated output or both. Keep the benchmark client's location and measurement interval fixed. A server-side step timer and a customer's received stream describe different boundaries; presenting one as the other obscures the decision.
Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM (external source) · Metrics — NVIDIA NIM Benchmarking (external source)Memory pressure changes scheduling behavior
Continuous admission also needs a policy for runtime state. vLLM documents preemption and recomputation when cache space is insufficient, with effects on end-to-end latency. A request may be eligible by token budget yet unable to obtain the required state. Consequently, concurrency, queueing, cache reservation and prefill scheduling should be evaluated together rather than tuned in isolation.
A proposed service record should explain waiting, rejection and cancellation behavior and preserve the reason a request left the system. Hugging Face's continuous-batching manager exposes cancellation and lifecycle operations, but an API product still needs its own contract for those events. Do not improve a headline by silently excluding requests that timed out, were cancelled or failed admission. Record offered demand as well as completed work.
Continuous batching — Hugging Face Transformers 5.11.0 (external source) · Optimization and Tuning — vLLM (external source)Compare policies on a frozen workload
Aguasto's proposed comparison manifest contains checkpoint and tokenizer revisions, engine build, devices and interconnect, precision, cache policy, parallelism, token budget and decoding settings. Specify an arrival pattern and a reproducible mix of prompt and output lengths. Include cold and eligible warm-prefix requests, and a burst case that exceeds steady demand. These are proposed test conditions; no GPU benchmark was executed for this article.
Evaluate completed requests meeting the declared objectives, output-token throughput, error counts, queue depth and state pressure over the same interval. Preserve request-level samples so a later reviewer can examine tails and failures. The existing inference-benchmark methodology explains the detailed latency boundaries; this guide adds the scheduler decision that those measurements are intended to evaluate. Changing a policy should not also change the workload unnoticed.
Metrics — NVIDIA NIM Benchmarking (external source)Use the result to choose a service boundary
The practical output is an acceptance envelope: the workload and offered demand for which a tested configuration meets a stated objective, plus the observed limits. It is not a permanent capacity promise. A later model, quantized artifact, context extension or customer mix can change the envelope and require another evaluation. Preserve the rejected configurations and unresolved tradeoffs so the final recommendation remains reviewable.
This also helps commercial comparisons. Reservation hours describe paid capacity; completed acceptable requests describe useful service delivery. A sampled device-utilization percentage does not establish either revenue utilization or cost per useful token. Link the scheduler evaluation to the existing dedicated/shared economics guide while keeping its cost assumptions separate. Aguasto is developing its platform, and no provisioning, available inventory or service-level commitment is implied by this research.
Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM (external source)Sources & verification
The register preserves each source's publication and verification date. Manufacturer claims and documentation are attributed; Aguasto has not run the cited products in production.
- Orca: A Distributed Serving System for Transformer-Based Generative Models — USENIX ↗
Historical iteration-level scheduling mechanism; Whole-request versus changing batch membership. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Continuous batching — Hugging Face Transformers 5.11.0 ↗
Request handling, cache and scheduling controls; Cancellation and manager lifecycle. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM ↗
Token budget, workspace and cache interactions; Context chunking; Utilization saturation and latency tradeoffs. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Optimization and Tuning — vLLM ↗
Eligible chunked prefill and decode prioritization in V1; Cache pressure, preemption and recomputation. Publication: date not stated in the reviewed source. Verified: 2026-10-13.
- Metrics — NVIDIA NIM Benchmarking ↗
Client-observed first-token and generation timing; Queueing and network measurement boundaries. Publication: date not stated in the reviewed source. Verified: 2026-10-13.