--- title: "Continuous Batching: Throughput Without Hiding Latency | Aguasto" description: "Explain iteration scheduling, token budgets, cache admission and workload measurement without treating larger batches or utilization as proof of a better inference service." canonical: "https://www.aguasto.com/learn/continuous-batching-and-latency" last-updated: "2026-10-13" --- # Continuous Batching: Throughput Without Hiding Latency | Aguasto Canonical HTML: [Continuous Batching: Throughput Without Hiding Latency](https://www.aguasto.com/learn/continuous-batching-and-latency) > Explain iteration scheduling, token budgets, cache admission and workload measurement without treating larger batches or utilization as proof of a better inference service. [Aguasto](https://www.aguasto.com/) [learn](https://www.aguasto.com/learn) Continuous Batching: Throughput Without Hiding Latency [Resources](https://www.aguasto.com/resources) [GPU Library](https://www.aguasto.com/gpus) [Model Index](https://www.aguasto.com/models) [Inference Stack](https://www.aguasto.com/inference-stack) [Learn](https://www.aguasto.com/learn) [Insights](https://www.aguasto.com/insights) [Research](https://www.aguasto.com/research) [Engineering tools](https://www.aguasto.com/tools) Aguasto / learn ## Continuous Batching: Throughput Without Hiding Latency Explain iteration scheduling, token budgets, cache admission and workload measurement without treating larger batches or utilization as proof of a better inference service. Original Aguasto conceptual illustration. Not manufacturer geometry, an operating Aguasto asset, or a measured benchmark. By Aguasto Research · Published October 13, 2026 · Revision 1 Aguasto Research documents third-party hardware, models and software. References do not represent Aguasto inventory, evaluated deployments or available inference services. Illustration: Continuous Batching: Throughput Without Hiding Latency: original technical study. Three illustrative scheduling iterations show requests joining and leaving independently. The membership diagram is not a throughput or latency measurement. Three illustrative scheduling iterations show requests joining and leaving independently. The membership diagram is not a throughput or latency measurement. Original Aguasto illustration. Read the guide for configuration and measurement limitations. ### Requests finish at different times A serving system receives prompts at different moments and generates answers of different lengths. Whole-request grouping can leave a newly arrived request waiting while an existing group completes. The ORCA research introduced iteration-level scheduling: invoke a generation iteration on the current batch, then change its membership as work progresses. Current continuous-batching documentation describes related request handling in a maintained serving implementation. Consider an explicitly hypothetical sequence in which A needs six decode steps, B needs two, and C arrives while both are active. After B finishes, a scheduler may admit C at an eligible iteration instead of waiting for A's entire answer. This illustrates membership, not speed. Prefill work, available cache, admission policy and engine constraints still decide whether that slot can actually accept C. [Orca: A Distributed Serving System for Transformer-Based Generative Models — USENIX (external source)](https://www.usenix.org/conference/osdi22/presentation/yu) · [Continuous batching — Hugging Face Transformers 5.11.0 (external source)](https://huggingface.co/docs/transformers/v5.11.0/en/continuous_batching) ### Separate requests from tokens and state The number of active requests is only one scheduling quantity. A token budget describes work placed into an iteration; retained cache describes state kept across iterations; the response stream describes what a client receives. A long prompt can consume substantial prefill work before its first answer token, while an already active decode request contributes a different amount of work to a step. A useful conceptual diagram labels these quantities independently. Give each request an arrival, retained-state block and pending-work segment. Show the scheduler selecting work under a token budget rather than drawing one identical box per customer. Hugging Face exposes separate scheduling and cache configuration, and NVIDIA discusses how token limits affect workspace and cache allocation. Their settings are implementation-specific, not interchangeable knobs with universal values. [Continuous batching — Hugging Face Transformers 5.11.0 (external source)](https://huggingface.co/docs/transformers/v5.11.0/en/continuous_batching) · [Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM (external source)](https://nvidia.github.io/TensorRT-LLM/features/paged-attention-ifb-scheduler.html) ### Balance prefill against ongoing generation Chunked prefill splits eligible prompt processing into portions that can be scheduled alongside generation. Current vLLM V1 documentation describes prioritizing pending decode work and using the remaining token budget for eligible prefills, splitting a prefill that cannot fit. NVIDIA also documents context chunking and its relationship to paged cache. The precise behavior depends on the engine and selected configuration. The evaluation question is whose waiting time a policy improves and what it costs elsewhere. A policy that protects an ongoing stream may make a newly admitted long prompt wait; another choice may favor prompt intake while changing token cadence. Define the intended service objective before selecting a policy. Keep prompt-length distribution, output lengths and arrival pattern visible so a favorable homogeneous test does not stand in for the actual customer workload. [Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM (external source)](https://nvidia.github.io/TensorRT-LLM/features/paged-attention-ifb-scheduler.html) · [Optimization and Tuning — vLLM (external source)](https://docs.vllm.ai/en/latest/configuration/optimization/) ### More scheduled work is not always better NVIDIA's scheduling documentation notes that utilization can plateau and that increasing scheduled tokens beyond that point may harm first-token and complete-request latency. A high token-throughput total can therefore coexist with a slower experience for an individual request. NVIDIA's benchmarking documentation separately defines client-observed first-token and generation timing, including relevant queueing and network effects. For a proposed acceptance test, declare the objectives before the run. Retain first-token latency, streaming cadence and completion latency as distributions, not just averages. State whether throughput counts input processing, generated output or both. Keep the benchmark client's location and measurement interval fixed. A server-side step timer and a customer's received stream describe different boundaries; presenting one as the other obscures the decision. [Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM (external source)](https://nvidia.github.io/TensorRT-LLM/features/paged-attention-ifb-scheduler.html) · [Metrics — NVIDIA NIM Benchmarking (external source)](https://docs.nvidia.com/nim/benchmarking/llm/metrics) ### Memory pressure changes scheduling behavior Continuous admission also needs a policy for runtime state. vLLM documents preemption and recomputation when cache space is insufficient, with effects on end-to-end latency. A request may be eligible by token budget yet unable to obtain the required state. Consequently, concurrency, queueing, cache reservation and prefill scheduling should be evaluated together rather than tuned in isolation. A proposed service record should explain waiting, rejection and cancellation behavior and preserve the reason a request left the system. Hugging Face's continuous-batching manager exposes cancellation and lifecycle operations, but an API product still needs its own contract for those events. Do not improve a headline by silently excluding requests that timed out, were cancelled or failed admission. Record offered demand as well as completed work. [Continuous batching — Hugging Face Transformers 5.11.0 (external source)](https://huggingface.co/docs/transformers/v5.11.0/en/continuous_batching) · [Optimization and Tuning — vLLM (external source)](https://docs.vllm.ai/en/latest/configuration/optimization/) ### Compare policies on a frozen workload Aguasto's proposed comparison manifest contains checkpoint and tokenizer revisions, engine build, devices and interconnect, precision, cache policy, parallelism, token budget and decoding settings. Specify an arrival pattern and a reproducible mix of prompt and output lengths. Include cold and eligible warm-prefix requests, and a burst case that exceeds steady demand. These are proposed test conditions; no GPU benchmark was executed for this article. Evaluate completed requests meeting the declared objectives, output-token throughput, error counts, queue depth and state pressure over the same interval. Preserve request-level samples so a later reviewer can examine tails and failures. The existing inference-benchmark methodology explains the detailed latency boundaries; this guide adds the scheduler decision that those measurements are intended to evaluate. Changing a policy should not also change the workload unnoticed. [Metrics — NVIDIA NIM Benchmarking (external source)](https://docs.nvidia.com/nim/benchmarking/llm/metrics) ### Use the result to choose a service boundary The practical output is an acceptance envelope: the workload and offered demand for which a tested configuration meets a stated objective, plus the observed limits. It is not a permanent capacity promise. A later model, quantized artifact, context extension or customer mix can change the envelope and require another evaluation. Preserve the rejected configurations and unresolved tradeoffs so the final recommendation remains reviewable. This also helps commercial comparisons. Reservation hours describe paid capacity; completed acceptable requests describe useful service delivery. A sampled device-utilization percentage does not establish either revenue utilization or cost per useful token. Link the scheduler evaluation to the existing dedicated/shared economics guide while keeping its cost assumptions separate. Aguasto is developing its platform, and no provisioning, available inventory or service-level commitment is implied by this research. [Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM (external source)](https://nvidia.github.io/TensorRT-LLM/features/paged-attention-ifb-scheduler.html) ### Scope & limitations - The A/B/C sequence is a hypothetical scheduling illustration, not a timing simulation or benchmark. - Historical ORCA results do not establish a performance ranking for current hardware or engines. - There is no universal best token budget or scheduler policy; exact model, engine and workload matter. - The proposed evaluation manifest has not been executed on Aguasto GPU infrastructure. - No current prices, available capacity, self-service provisioning or Aguasto service-level commitment is represented. ### Continue the research [Why tokens per second is not enough to benchmark inference](https://www.aguasto.com/research/inference-benchmark-methodology) [Prefill, decode and the cost of separating them](https://www.aguasto.com/learn/prefill-decode-and-disaggregated-serving) [Dedicated compute and shared inference: compare the whole workload](https://www.aguasto.com/insights/dedicated-and-shared-inference-economics) [Model weights are only the beginning of the memory budget](https://www.aguasto.com/learn/model-weights-and-runtime-memory) [NVIDIA H100 SXM](https://www.aguasto.com/gpus/h100) [NVIDIA H200 SXM](https://www.aguasto.com/gpus/h200) [NVIDIA B200 SXM](https://www.aguasto.com/gpus/b200) [AMD Instinct MI355X](https://www.aguasto.com/gpus/mi355x) [GLM-4.5](https://www.aguasto.com/models/glm-4-5) [Qwen3-235B-A22B](https://www.aguasto.com/models/qwen3-235b-a22b) [vLLM](https://www.aguasto.com/inference-stack/vllm) [SGLang](https://www.aguasto.com/inference-stack/sglang) [TensorRT-LLM](https://www.aguasto.com/inference-stack/tensorrt-llm) [The Aguasto modular thesis](https://www.aguasto.com/why-modular) [Potential OEM and strategic collaboration](https://www.aguasto.com/partners) ### Sources & verification The register preserves each source's publication and verification date. Manufacturer claims and documentation are attributed; Aguasto has not run the cited products in production. - [Orca: A Distributed Serving System for Transformer-Based Generative Models — USENIX ↗](https://www.usenix.org/conference/osdi22/presentation/yu) Historical iteration-level scheduling mechanism; Whole-request versus changing batch membership. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Continuous batching — Hugging Face Transformers 5.11.0 ↗](https://huggingface.co/docs/transformers/v5.11.0/en/continuous_batching) Request handling, cache and scheduling controls; Cancellation and manager lifecycle. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Paged Attention, IFB, and Request Scheduling — NVIDIA TensorRT-LLM ↗](https://nvidia.github.io/TensorRT-LLM/features/paged-attention-ifb-scheduler.html) Token budget, workspace and cache interactions; Context chunking; Utilization saturation and latency tradeoffs. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Optimization and Tuning — vLLM ↗](https://docs.vllm.ai/en/latest/configuration/optimization/) Eligible chunked prefill and decode prioritization in V1; Cache pressure, preemption and recomputation. Publication: date not stated in the reviewed source. Verified: 2026-10-13. - [Metrics — NVIDIA NIM Benchmarking ↗](https://docs.nvidia.com/nim/benchmarking/llm/metrics) Client-observed first-token and generation timing; Queueing and network measurement boundaries. Publication: date not stated in the reviewed source. Verified: 2026-10-13. ### In this publication [Requests finish at different times](https://www.aguasto.com/learn/continuous-batching-and-latency#section-1) [Separate requests from tokens and state](https://www.aguasto.com/learn/continuous-batching-and-latency#section-2) [Balance prefill against ongoing generation](https://www.aguasto.com/learn/continuous-batching-and-latency#section-3) [More scheduled work is not always better](https://www.aguasto.com/learn/continuous-batching-and-latency#section-4) [Memory pressure changes scheduling behavior](https://www.aguasto.com/learn/continuous-batching-and-latency#section-5) [Compare policies on a frozen workload](https://www.aguasto.com/learn/continuous-batching-and-latency#section-6) [Use the result to choose a service boundary](https://www.aguasto.com/learn/continuous-batching-and-latency#section-7) [More learn ↗](https://www.aguasto.com/learn) [Source register ↓](https://www.aguasto.com/learn/continuous-batching-and-latency#sources) ## Company navigation - [Home](https://www.aguasto.com/) - [Infrastructure](https://www.aguasto.com/infrastructure) - [GPU Cloud](https://www.aguasto.com/gpu-cloud) - [AI Inference](https://www.aguasto.com/ai-inference) - [Development & Investment](https://www.aguasto.com/development) - [Company](https://www.aguasto.com/company) - [Contact](https://www.aguasto.com/contact) - [Privacy](https://www.aguasto.com/privacy) - [Resources](https://www.aguasto.com/resources) - [GPU Library](https://www.aguasto.com/gpus) - [Model Index](https://www.aguasto.com/models) - [Inference Stack](https://www.aguasto.com/inference-stack) - [Learn](https://www.aguasto.com/learn) - [Insights](https://www.aguasto.com/insights) - [Research](https://www.aguasto.com/research) - [Industry cases](https://www.aguasto.com/research/cases) - [Why Modular](https://www.aguasto.com/why-modular) - [Partners](https://www.aguasto.com/partners) - [GPU comparison](https://www.aguasto.com/gpus/compare) - [Model comparison](https://www.aguasto.com/models/compare) - [Engineering tools](https://www.aguasto.com/tools) - [Development friction](https://www.aguasto.com/research/hyperscale-development-friction) - [NVIDIA H100 SXM](https://www.aguasto.com/gpus/h100) - [NVIDIA H200 SXM](https://www.aguasto.com/gpus/h200) - [NVIDIA B200 SXM](https://www.aguasto.com/gpus/b200) - [NVIDIA B300](https://www.aguasto.com/gpus/b300) - [NVIDIA GB200 NVL72](https://www.aguasto.com/gpus/gb200-nvl72) - [NVIDIA GB300 NVL72](https://www.aguasto.com/gpus/gb300-nvl72) - [NVIDIA Vera Rubin NVL72](https://www.aguasto.com/gpus/vera-rubin-nvl72) - [AMD Instinct MI300X](https://www.aguasto.com/gpus/mi300x) - [AMD Instinct MI325X](https://www.aguasto.com/gpus/mi325x) - [AMD Instinct MI355X](https://www.aguasto.com/gpus/mi355x) - [AMD Instinct MI455X](https://www.aguasto.com/gpus/mi455x) - [AMD Helios](https://www.aguasto.com/gpus/helios) - [GLM-4.5](https://www.aguasto.com/models/glm-4-5) - [Qwen3-235B-A22B](https://www.aguasto.com/models/qwen3-235b-a22b) - [DeepSeek-R1](https://www.aguasto.com/models/deepseek-r1) - [Llama 3.3 70B Instruct](https://www.aguasto.com/models/llama-3-3-70b-instruct) - [Gemma 3 27B IT](https://www.aguasto.com/models/gemma-3-27b-it) - [gpt-oss-120b](https://www.aguasto.com/models/gpt-oss-120b) - [Kimi-K2-Instruct](https://www.aguasto.com/models/kimi-k2-instruct) - [GLM-5.3](https://www.aguasto.com/models/glm-5-3) - [Qwen3.8-27B](https://www.aguasto.com/models/qwen3-8-27b) - [DeepSeek-V4.1-Flash](https://www.aguasto.com/models/deepseek-v4-1-flash) - [Llama 4 Scout 17B-16E Instruct](https://www.aguasto.com/models/llama-4-scout-17b-16e-instruct) - [Gemma 4 31B IT](https://www.aguasto.com/models/gemma-4-31b-it) - [Kimi K3](https://www.aguasto.com/models/kimi-k3) - [vLLM](https://www.aguasto.com/inference-stack/vllm) - [SGLang](https://www.aguasto.com/inference-stack/sglang) - [TensorRT-LLM](https://www.aguasto.com/inference-stack/tensorrt-llm) - [NVIDIA Dynamo](https://www.aguasto.com/inference-stack/dynamo) - [NVIDIA Triton Inference Server](https://www.aguasto.com/inference-stack/triton) - [AMD ROCm](https://www.aguasto.com/inference-stack/rocm) - [Ray Serve](https://www.aguasto.com/inference-stack/ray-serve) - [Kubernetes GPU orchestration — NVIDIA device plugin](https://www.aguasto.com/inference-stack/kubernetes) - [Model weights are only the beginning of the memory budget](https://www.aguasto.com/learn/model-weights-and-runtime-memory) - [Why tokens per second is not enough to benchmark inference](https://www.aguasto.com/research/inference-benchmark-methodology) - [Prefill, decode and the cost of separating them](https://www.aguasto.com/learn/prefill-decode-and-disaggregated-serving) - [Choosing between vLLM, SGLang and TensorRT-LLM](https://www.aguasto.com/learn/choosing-an-inference-serving-framework) - [From Power to Inference—Building a Distributed AI Compute Network](https://www.aguasto.com/learn/from-power-to-inference) - [Dedicated compute and shared inference: compare the whole workload](https://www.aguasto.com/insights/dedicated-and-shared-inference-economics) - [Distributed inference reliability begins with the failure boundary](https://www.aguasto.com/learn/distributed-inference-reliability) - [Core Scientific: reading contracted, energized and billed capacity](https://www.aguasto.com/research/cases/core-scientific-capacity-disclosures) - [Why Modular AI Infrastructure Matters Now](https://www.aguasto.com/research/why-modular-ai-infrastructure-matters-now) - [What Is a Modular AI Data Center?](https://www.aguasto.com/learn/what-is-a-modular-ai-data-center) - [How Modular AI Data Centers Are Built and Commissioned](https://www.aguasto.com/learn/modular-data-center-factory-to-commissioning) - [Liquid Cooling for High-Density GPU Systems: Follow the Interfaces](https://www.aguasto.com/learn/liquid-cooling-gpu-rack-interfaces) - [Why Electricity Availability Shapes AI Infrastructure Deployment](https://www.aguasto.com/research/electricity-availability-and-time-to-power) - [What Is KV Cache? Capacity, Reuse and Isolation](https://www.aguasto.com/learn/kv-cache-capacity-and-policy) - [Continuous Batching: Throughput Without Hiding Latency](https://www.aguasto.com/learn/continuous-batching-and-latency) - [Applied Digital: Reading the Polaris Forge 1 Delivery Milestones](https://www.aguasto.com/research/cases/applied-digital-polaris-forge-capacity-milestones) - [Crusoe Abilene: Separate Delivered Infrastructure from Expansion Plans](https://www.aguasto.com/research/cases/crusoe-abilene-delivery-and-expansion) - [Official LinkedIn](https://www.linkedin.com/company/aguasto) This is a generated representation of the public HTML page. Diagrams are conceptual. GPU and inference services are planned and are not currently available to book or call. The contact page describes the available inquiry flow.