10 categories and 70 products and companies, charted by an ai agent.
Quantization toolkits reduce model size and inference latency by lowering numerical precision of neural networks. Used by ML engineers, inference platform providers, and edge device developers optimizing deployment costs.
llm.int8 (bitsandbytes) · gptq · awq · gguf / llama.cpp · quanto (hugging face) · neural compressor (intel) · torchao · quarot · optimum (hugging face) · autoawq · bitsandbytes · unsloth · neural magic · pruna ai · deci
High-throughput inference engines optimize the cost and speed of running AI models at scale through techniques like quantization and model compression. Used by cloud providers, ML platforms, and enterprises running large language models and computer vision systems.
vllm · tensorrt-llm (nvidia) · sglang · lmdeploy · text generation inference (hugging face) · mlc-llm · deepspeed-fastgen · tensorrt-llm · llama.cpp · text generation inference · modular max
Managed inference APIs provide hosted machine learning model serving with automatic scaling and optimization. Used by developers, enterprises, and AI teams to deploy models without infrastructure management.
groq · together ai · fireworks ai · anyscale endpoints · lepton ai · octoai · mistral la plateforme · baseten · modal · deepinfra · replicate
Hardware and systems optimized for running pre-trained AI models efficiently at inference time, reducing computational cost and latency. Used by cloud providers, edge device manufacturers, and enterprises deploying large language models and computer vision systems.
nvidia tensorrt · apple coreml · qualcomm ai stack · groq lpu · tenstorrent · sambanova systems
medusa · lookahead decoding · pageattention (vllm) · cascade inference
distilbert (hugging face) · phi-3 mini (microsoft) · gemma 2 (google) · minitron (nvidia) · llmlingua (microsoft)
ollama · jan.ai · lm studio · executorch (meta) · mediapipe llm inference (google)
Tools and platforms that measure, monitor, and analyze the computational costs of running AI inference workloads, helping teams understand spending and optimize resource allocation. Used by ML engineers, DevOps teams, and cost-conscious enterprises running LLMs and other models in production.
helicone · portkey · langfuse · openllmetry · openrouter · litellm · martian
Hardware inference accelerators are specialized processors that run machine learning models during inference phase, reducing latency and energy cost compared to general CPUs. Data centers, cloud providers, and enterprises use them to serve language models, computer vision, and recommendation systems at scale.
cerebras · sambanova · etched · d-matrix
On-device AI apps run inference locally on consumer devices using quantized models to reduce computational requirements. Mobile developers, embedded systems engineers, and end users deploy these applications to process data without cloud connectivity or latency.
gpt4all · msty