6 categories and 33 products and companies, charted by an ai agent.
Inference serving engines deploy and run large language models in production, handling request batching, token generation, and resource allocation. Data centers, cloud platforms, and research labs use them to serve LLM APIs and internal applications.
vllm · sglang · tensorrt-llm · text generation inference · lmdeploy · nvidia dynamo · llm-d
Local inference runtimes are software frameworks that execute language models and other AI models on a user's own machine without sending data to external servers. Data scientists, developers, and enterprises use them to run inference with privacy and without cloud dependencies.
llama.cpp · ollama · lm studio · mlx · jan · llamafile
Serverless inference clouds run model inference on managed cloud infrastructure without users managing servers or containers. ML engineers, application developers, and data scientists use them to call models via API without provisioning compute.
together ai · fireworks ai · baseten · modal · deepinfra · replicate · novita ai
Custom silicon inference refers to specialized hardware accelerators (ASICs, TPUs, NPUs) and their associated software stacks designed to run ML model inference at scale. Used by cloud providers, enterprises, and inference platforms to serve models with optimized latency and cost per token.
groq · cerebras · sambanova · etched
Model gateways and routers direct API requests to inference endpoints, load-balance across multiple inference providers, and manage routing logic. ML engineers, platform teams, and enterprises use them to orchestrate inference across different models and backends.
openrouter · litellm · portkey · cloudflare ai gateway · helicone
Edge browser inference runs machine learning models directly in web browsers or on edge devices without sending data to remote servers. Web developers, browser vendors, and device manufacturers use these systems to perform inference locally on user hardware.
webllm · transformers.js · onnx runtime · executorch