Two phases, different work
NVIDIA Dynamo describes prefill as processing the prompt and producing its key-value cache; decode then generates output tokens using that state. In a disaggregated layout, separate worker pools perform those phases and the cache moves between them. The distinction gives an infrastructure designer separate questions: how quickly can a prompt be processed, and how many active generations can remain responsive?
Use input length and output length as different workload variables. A document-reading application with a short answer is not equivalent to a short prompt that produces a long report. Start by characterizing those distributions, rather than selecting a topology from a model's parameter count.
Disaggregated Serving — NVIDIA Dynamo architecture (external source)Begin with the aggregated baseline
An aggregated worker avoids an additional handoff between the prompt and generation phases. NVIDIA's development guide cautions that disaggregation is not automatically better and can be unattractive for small models, short prompts, low concurrency or an inadequate transfer fabric. That is guidance for evaluation, not a result measured by Aguasto.
For a first engineering comparison, retain one baseline with the same model, precision, output-quality gate and offered load. Confirm whether tail latency comes from queueing, prompt processing, generation or transport. A change that moves a bottleneck to another pool may improve one chart while increasing the system's total cost.
Disaggregated Serving — NVIDIA Dynamo development guide (external source)Tune latency objectives separately
vLLM's disaggregated-prefill documentation describes separate instances as a way to tune TTFT and inter-token latency independently and control decode tail latency. It explicitly labels the feature experimental and warns that disaggregated prefill itself does not improve throughput. That warning should remain visible beside any architectural diagram.
Treat the feature as a candidate response to a measured interference problem. If long prefills interrupt ongoing generation, evaluate scheduling alternatives and split pools against the same latency objective. The useful outcome is an explained trade-off under a defined workload, not a claim that two pools are universally faster than one.
Disaggregated Prefilling (experimental) — vLLM (external source)The cache transfer is part of the design
In the documented Dynamo architecture, routing and transfer metadata coordinate the movement of KV state, with NIXL used for transfer. Backend metadata differs. The development guide also requires compatible model, dtype and cache layout between prefill and decode workers. Its current development examples should not be copied into a production guide without version checks.
A proposed experiment should measure transferred bytes per request, transfer latency, failure behavior and capacity used at both ends. Include the actual local or inter-node link and the placement of workers. A diagram arrow is a real dependency: it needs bandwidth, compatible state and a recovery strategy.
Disaggregated Serving — NVIDIA Dynamo architecture (external source) · Disaggregated Serving — NVIDIA Dynamo development guide (external source)Do not confuse reuse with faster generation
vLLM's automatic prefix caching reuses prefill computation when requests share an existing token prefix. The documentation states that this optimization does not shorten the decode phase itself. Its benefit therefore depends on the workload's actual reuse pattern.
Run cached and uncached variants deliberately, and disclose the cache-hit distribution. Test a shared system prompt separately from unrelated long documents. A benchmark populated with repeated prefixes can be useful for that application, but it cannot stand in for every enterprise workload. Keep cache affinity, eviction pressure and cold-start behavior in the evaluation record.
Automatic Prefix Caching — vLLM (external source)An acceptance test for disaggregation
The proposed acceptance criteria should require the selected TTFT and streaming-latency objectives at representative arrival rates, unchanged task quality, explicit error counts and a defensible cost per completed useful response. Retest with bursty traffic and a worker loss. Record whether admission control protects generation when prompt demand exceeds the prefill pool's capacity.
Aguasto documents these concepts as research. It has not operated these pools, benchmarked a transfer path or launched an inference API. Until measured evidence exists, hardware-to-engine relationships are evaluation paths. The practical recommendation is to add the architectural split only when an observed workload problem and a complete comparison justify it.
Sources & verification
Reference revision 1 · Reviewed . Manufacturer claims and documentation are attributed; Aguasto has not run the cited products in production.
- Disaggregated Serving — NVIDIA Dynamo architecture ↗
Prefill computes cache; cache transferred to decode; NIXL and KV-aware routing; Backend-specific transfer metadata. Publication: date not stated in the reviewed source. Verified: 2026-10-12.
- Disaggregated Serving — NVIDIA Dynamo development guide ↗
Independent worker pools and fast-transfer fabric assumptions; Matching model/cache layout required; Development documentation, not a pinned production recipe. Publication: date not stated in the reviewed source. Verified: 2026-10-12.
- Disaggregated Prefilling (experimental) — vLLM ↗
Separate instances allow TTFT/ITL tuning; Feature explicitly experimental; Documented feature does not itself promise throughput improvement. Publication: date not stated in the reviewed source. Verified: 2026-10-12.
- Automatic Prefix Caching — vLLM ↗
Shared prefixes reuse prefill cache; Prefix caching does not accelerate decode itself. Publication: date not stated in the reviewed source. Verified: 2026-10-12.