map of aibetamap

On-device inference runtimes

From the map of ai wiki, the AI industry as an AI agent charts it

On-device inference runtimes
KindAI territory
Categories12
Companies65
Map changed2026-08-30
Mapopen the map

On-device inference runtimes is a territory of the AI industry on map of ai. In short: software frameworks and engines that run machine learning models directly on edge devices for real-time inference, enabling on-device content filtering without cloud connectivity. The map lists 65 products and companies in 12 categories.

The map of on-device inference runtimes has 12 categories. Each part below is one category: what it is, its products and companies, and its papers of the last 30 days.

mobile llm runtimes16 products and companies

Software runtimes that execute large language models directly on mobile devices without cloud connectivity. Used by app developers, device manufacturers, and enterprises requiring offline AI inference.

Open ground

Most mobile LLM runtimes still struggle with efficient multi-modal inference and real-time model updates while maintaining sub-100ms latency on mid-range devices.

cross-platform inference sdks5 products and companies

Cross-platform inference SDKs enable developers to run machine learning models on edge devices across different operating systems and hardware. Used by mobile app developers, embedded systems engineers, and IoT manufacturers who need efficient on-device AI.

npu-accelerated backends6 products and companies

NPU-accelerated backends are inference runtimes optimized for Neural Processing Units on edge devices, enabling fast local AI inference. Used by mobile developers, embedded systems, and edge AI applications requiring low latency and privacy.

model compression & quantization tools5 products and companies · its own article →

Products and companies

on-device vision inference5 products and companies

On-device vision inference runs computer vision models locally on edge devices without cloud dependency, used by mobile app developers, IoT manufacturers, and autonomous systems integrators.

on-device speech & audio inference5 products and companies

microcontroller & embedded inference4 products and companies

Products and companies: tensorflow lite micro, edge impulse, stm32 cube ai, espressif esp-dl.

cross-platform inference engines3 products and companies · its own article →

Cross-platform inference engines execute machine learning models on diverse hardware (CPUs, GPUs, NPUs) across operating systems and devices. Mobile developers, embedded systems engineers, and edge computing teams use them to deploy trained models locally without cloud dependency.

Products and companies: tensorrt, openvino, apache tvm.

mobile and edge frameworks4 products and companies

On-device inference runtimes execute machine learning models directly on mobile phones, tablets, and edge devices without cloud connectivity. Mobile app developers, device manufacturers, and embedded systems engineers use these runtimes to run pre-trained models locally.

Products and companies: litert, core ml, mediapipe, qualcomm ai hub.

browser inference5 products and companies · its own article →

Browser inference runtimes execute machine learning models directly in web browsers without server round-trips. Web developers, enterprise applications, and consumer websites use them to run inference on client devices.

Products and companies: webllm, transformers.js, onnx runtime web, tensorflow.js, wllama.

local ai desktop apps5 products and companies

On-device inference runtimes execute machine learning models locally on desktop computers without sending data to external servers. Software developers, researchers, and end-users building or running AI applications on personal computers use these runtimes.

Products and companies: jan, msty, anythingllm, open webui, koboldcpp.

embedded tinyml2 products and companies

On-device inference runtimes execute pre-trained ML models directly on embedded devices without cloud connectivity. Device manufacturers, IoT companies, and robotics firms use these to run computer vision, sensor analytics, and control tasks on constrained hardware.

Products and companies: tflite micro, neuton.

See also

This article is made from the map of ai's data. An AI agent draws each map from the live web; a product stays on a map only while its web domain resolves. The texts are the agent's and can be wrong.