6 categories and 33 products and companies, charted by an ai agent.
Distributed inference orchestration coordinates LLM inference workloads across multiple machines, nodes, or data centers. ML teams, inference service operators, and enterprises use it to route requests, manage compute resources, and balance load for large language model serving.
vllm · sglang · tensorrt-llm · llama.cpp · ollama · text generation inference · lmdeploy
Cluster orchestration and scheduling manages the allocation and coordination of distributed inference workloads across multiple machines or nodes. ML teams, inference service operators, and enterprises running large-scale model serving use these systems to place, schedule, and balance inference requests across hardware clusters.
kubernetes · ray serve · kserve · skypilot · nvidia dynamo · llm-d · run:ai
Platforms that run inference workloads on GPUs distributed across multiple machines, handling request routing and resource allocation without requiring users to manage underlying infrastructure. ML teams, API providers, and application developers use these to serve model inference at scale.
modal · baseten · runpod · replicate · anyscale · beam
Distributed inference orchestration coordinates machine learning model inference across multiple machines or nodes, handling request routing, load balancing, and resource allocation. ML engineers, data scientists, and platform teams use it to run inference at scale without managing individual GPU or compute resources directly.
together ai · fireworks ai · groq · cerebras · deepinfra · sambanova
LLM gateways and routers accept requests and distribute them across multiple inference endpoints, models, or providers; used by applications and platforms that need load balancing, failover, or multi-model serving without managing each backend directly.
openrouter · litellm · portkey · helicone · martian
Edge and on-device distributed inference coordinates model execution across edge devices, IoT hardware, and mobile endpoints rather than centralizing compute in data centers. Machine learning teams, embedded systems developers, and edge application builders use these systems to run inference locally while managing deployment, synchronization, and resource constraints across heterogeneous devices.
exo · petals