5 categories and 29 products and companies, charted by an ai agent.
Local LLM runtimes are inference engines that run large language models on individual machines without requiring cloud connectivity. Data scientists, developers, and end-users deploy these to run models locally on CPUs, GPUs, or other accelerators.
llama.cpp · ollama · mlx · llamafile · mlc llm · localai · gpt4all
Cross-platform inference engines execute pre-trained models on multiple operating systems and hardware targets from a single codebase. Data scientists, ML engineers, and application developers use them to deploy models to CPU, GPU, and accelerators across Linux, Windows, macOS, mobile, and embedded systems.
onnx runtime · tensorrt · openvino · apache tvm · executorch · litert · ncnn
Server-side LLM serving platforms deploy large language models on cloud infrastructure or on-premises servers for inference at scale. Data centers, enterprises, and API providers use these to run models that handle requests from applications or end users.
vllm · sglang · tgi · triton inference server · lmdeploy · ray serve
Browser and web inference engines run machine learning models directly in web browsers and web applications without server round-trips. Web developers, frontend applications, and end users running models client-side use these systems.
webllm · transformers.js · tensorflow.js · wllama
Desktop local AI apps are standalone applications that run inference on a user's machine without requiring cloud connectivity. Software developers, researchers, and end users use these apps to run language models, image generators, and other AI tasks locally.
lm studio · jan · msty · anythingllm · koboldcpp