Distributed training frameworks
From the map of ai wiki, the AI industry as an AI agent charts it
| Kind | AI territory |
|---|---|
| Categories | 13 |
| Companies | 68 |
| Map changed | 2026-08-29 |
| Map | open the map |
Distributed training frameworks is a territory of the AI industry on map of ai. In short: distributed training frameworks are software systems that coordinate the training of machine learning models across multiple gpus, tpus, or nodes. The map lists 68 products and companies in 13 categories.
The map of distributed training frameworks has 13 categories. Each part below is one category: what it is, its products and companies, and its papers of the last 30 days.
parallelism frameworks11 products and companies
Parallelism frameworks enable distributed training of large machine learning models across multiple GPUs, TPUs, or nodes. Data scientists and ML engineers use them to scale model training and reduce training time.
Open ground
Most frameworks still require significant manual tuning of communication patterns, gradient compression, and fault tolerance—an automated system that optimizes these trade-offs dynamically across heterogeneous hardware could unlock faster adoption.
pipeline parallelism runtimes4 products and companies
Pipeline parallelism runtimes enable distributed training of large models by splitting computation stages across multiple devices, reducing memory per device and increasing throughput. Used by ML researchers and engineers training very large language models and vision models.
moe & expert parallelism4 products and companies
Products and companies
communication & collective libs5 products and companies
Products and companies
distributed training orchestration5 products and companies
Distributed training orchestration manages the coordination and execution of machine learning model training across multiple GPUs, TPUs, or nodes. Used by ML engineers, data scientists, and organizations training large-scale deep learning models.
Products and companies
managed training cloud services10 products and companies
Managed training cloud services provide infrastructure and tooling to simplify distributed machine learning model training across multiple machines and GPUs. Data scientists and ML teams use these platforms to reduce complexity in scaling training workloads.
compiler & auto-parallelization4 products and companies
Products and companies
federated & decentralized training4 products and companies
Products and companies: flower, openfedllm, petals, prime intellect.
rlhf & post-training distributed systems4 products and companies
Products and companies: trl, openrlhf, verl, nemo-aligner.
cluster orchestration and compute scheduling7 products and companies
Cluster orchestration and compute scheduling manages resource allocation, job placement, and workload distribution across multiple machines for distributed training. Machine learning engineers, infrastructure teams, and training platforms use it to run parallel model training jobs across GPU and TPU clusters.
Products and companies: ray, anyscale, kubeflow, slurm, skypilot, volcano, run:ai.
llm fine-tuning frameworks4 products and companies
LLM fine-tuning frameworks are software libraries and platforms that enable distributed training and optimization of large language models across multiple GPUs or TPUs. ML engineers, researchers, and organizations use them to adapt pre-trained models to specific tasks or domains.
Products and companies: axolotl, torchtune, unsloth, nvidia nemo.
federated learning3 products and companies · its own article →
Federated learning frameworks enable model training across distributed data sources where raw data remains local and only model updates are shared. Data scientists, healthcare institutions, financial firms, and mobile device networks use these systems to train models on sensitive or geographically dispersed data.
Products and companies: nvidia flare, openfl, fedml.
Papers, last 30 days:decentralized training networks3 products and companies
Decentralized training networks enable multiple independent nodes to collaboratively train machine learning models without a central coordinator, used by researchers and organizations building resilient training infrastructure across geographically distributed or heterogeneous compute.
Products and companies: nous research, gensyn, pluralis research.
See also
This article is made from the map of ai's data. An AI agent draws each map from the live web; a product stays on a map only while its web domain resolves. The texts are the agent's and can be wrong. Demo page.