Aguasto / inference-stack
Between hardware and answers.
Understand the serving engines, runtime ecosystems and distributed systems that connect GPU infrastructure to useful inference. Compatibility and operating trade-offs matter as much as feature lists.
8 of 8 references
vLLM
Serve supported language and multimodal models using continuous batching, attention-cache management and distributed inference.
SGLang
Serve supported language and multimodal models with prefix caching and multi-GPU inference.
TensorRT-LLM
Optimize and serve supported generative models using NVIDIA GPU kernels, cache management and parallel inference.
NVIDIA Dynamo
Coordinate inference engines through routing, separate prefill and decode workers, cache management and scaling.
NVIDIA Triton Inference Server
Expose model inference through supported backends, scheduling, batching and model pipelines.
AMD ROCm
Provide GPU runtimes, compilers, math and communication libraries for supported AMD workloads.
Ray Serve
Build Python inference applications that compose models and business logic and scale replicas across Ray resources.
Kubernetes GPU orchestration — NVIDIA device plugin
Expose NVIDIA GPU devices as Kubernetes resources and allocate them to container workloads.