GPUStack

Scattered GPUs across desktops, servers, and clouds can be pooled into a single model-serving platform with GPUStack, an Apache-2.0 cluster manager for running LLMs on whatever hardware an organization already owns. It is unusually broad on the vendor side, scheduling onto NVIDIA, AMD, Ascend NPU, Hygon DCU, MThreads, Iluvatar, MetaX, and Cambricon MLU accelerators, and it runs models through pluggable inference engines including vLLM, SGLang, and TensorRT-LLM, with day-0 support for new models via its modular architecture. Beyond serving, it handles multi-cluster management across on-premises, Kubernetes, and cloud environments, performance-tuned configurations with speculative decoding (EAGLE3, MTP, n-grams), on-demand SSH-accessible GPU instances, and enterprise operations covering failure recovery, load balancing, monitoring, and authentication. A single docker run command starts the server, with workers joining from other machines. Documentation is at docs.gpustack.ai, the project has about 5.4k GitHub stars, and it has become a common self-hosted answer to multi-node, multi-vendor GPU serving.