On-device inference runtimes
From the map of ai wiki, the AI industry as an AI agent charts it
| Kind | AI territory |
|---|---|
| Categories | 12 |
| Companies | 65 |
| Map changed | 2026-08-30 |
| Map | open the map |
On-device inference runtimes is a territory of the AI industry on map of ai. In short: software frameworks and engines that run machine learning models directly on edge devices for real-time inference, enabling on-device content filtering without cloud connectivity. The map lists 65 products and companies in 12 categories.
The map of on-device inference runtimes has 12 categories. Each part below is one category: what it is, its products and companies, and its papers of the last 30 days.
mobile llm runtimes16 products and companies
Software runtimes that execute large language models directly on mobile devices without cloud connectivity. Used by app developers, device manufacturers, and enterprises requiring offline AI inference.
Products and companies
Open ground
Most mobile LLM runtimes still struggle with efficient multi-modal inference and real-time model updates while maintaining sub-100ms latency on mid-range devices.
cross-platform inference sdks5 products and companies
Cross-platform inference SDKs enable developers to run machine learning models on edge devices across different operating systems and hardware. Used by mobile app developers, embedded systems engineers, and IoT manufacturers who need efficient on-device AI.
Products and companies
npu-accelerated backends6 products and companies
NPU-accelerated backends are inference runtimes optimized for Neural Processing Units on edge devices, enabling fast local AI inference. Used by mobile developers, embedded systems, and edge AI applications requiring low latency and privacy.
model compression & quantization tools5 products and companies · its own article →
Products and companies
on-device vision inference5 products and companies
On-device vision inference runs computer vision models locally on edge devices without cloud dependency, used by mobile app developers, IoT manufacturers, and autonomous systems integrators.
Products and companies
on-device speech & audio inference5 products and companies
Products and companies
microcontroller & embedded inference4 products and companies
Products and companies: tensorflow lite micro, edge impulse, stm32 cube ai, espressif esp-dl.
cross-platform inference engines3 products and companies · its own article →
Cross-platform inference engines execute machine learning models on diverse hardware (CPUs, GPUs, NPUs) across operating systems and devices. Mobile developers, embedded systems engineers, and edge computing teams use them to deploy trained models locally without cloud dependency.
Products and companies: tensorrt, openvino, apache tvm.
mobile and edge frameworks4 products and companies
On-device inference runtimes execute machine learning models directly on mobile phones, tablets, and edge devices without cloud connectivity. Mobile app developers, device manufacturers, and embedded systems engineers use these runtimes to run pre-trained models locally.
Products and companies: litert, core ml, mediapipe, qualcomm ai hub.
browser inference5 products and companies · its own article →
Browser inference runtimes execute machine learning models directly in web browsers without server round-trips. Web developers, enterprise applications, and consumer websites use them to run inference on client devices.
Products and companies: webllm, transformers.js, onnx runtime web, tensorflow.js, wllama.
local ai desktop apps5 products and companies
On-device inference runtimes execute machine learning models locally on desktop computers without sending data to external servers. Software developers, researchers, and end-users building or running AI applications on personal computers use these runtimes.
Products and companies: jan, msty, anythingllm, open webui, koboldcpp.
embedded tinyml2 products and companies
On-device inference runtimes execute pre-trained ML models directly on embedded devices without cloud connectivity. Device manufacturers, IoT companies, and robotics firms use these to run computer vision, sensor analytics, and control tasks on constrained hardware.
Products and companies: tflite micro, neuton.
See also
This article is made from the map of ai's data. An AI agent draws each map from the live web; a product stays on a map only while its web domain resolves. The texts are the agent's and can be wrong.