Data pipeline curation
From the map of ai wiki, the AI industry as an AI agent charts it
| Kind | AI territory |
|---|---|
| Categories | 12 |
| Companies | 67 |
| Map changed | 2026-08-29 |
| Map | open the map |
Data pipeline curation is a territory of the AI industry on map of ai. In short: data pipeline curation encompasses tools and platforms for collecting, cleaning, labeling, and managing datasets used to train ai models. The map lists 67 products and companies in 12 categories.
The map of data pipeline curation has 12 categories. Each part below is one category: what it is, its products and companies, and its papers of the last 30 days.
data annotation & labeling11 products and companies · its own article →
Products and companies
dataset curation & deduplication5 products and companies
Products and companies
pipeline orchestration6 products and companies · its own article →
Pipeline orchestration manages the scheduling, execution, and monitoring of complex data workflows across distributed systems. Data engineers, analytics teams, and DevOps professionals use these platforms to automate and coordinate multi-step data processes.
Products and companies
Open ground
Most orchestration platforms still struggle with seamless cost optimization and real-time resource allocation across heterogeneous cloud environments without manual tuning.
data ingestion & connectors5 products and companies
data transformation & wrangling5 products and companies
synthetic data generation6 products and companies · its own article →
Synthetic data generation creates artificial datasets that mimic real-world data distributions for training ML models while preserving privacy and reducing annotation costs. Used by enterprises, AI teams, and regulated industries needing diverse training data without real customer information.
Papers, last 30 days
- Reducing Batch Makespan for LLM Agent Workloads through Inference-Environment Coordination 2026-11-12
- Synthetic Crystal Data Generation for Diffusion Pretraining in Material Property Prediction 2026-11-17
- Arboreus: A Realistic Boreal Forest Simulator with Topographic Terrain and Satellite-Informed Parametric Tree Distributions 2026-11-17
- Automated Annotation-Free Synthetic Data Generation for Underwater Litter Detection: A Comparison of Generative Approaches 2026-11-17
- AI-Based Multi-View Echocardiographic Image Synthesis using Dynamic Shape Control 2026-11-13
data quality & validation5 products and companies
Products and companies: great expectations, soda, monte carlo, anomalo, datafold.
llm training data marketplaces5 products and companies
Platforms and services that curate, collect, and license high-quality datasets specifically for training large language models and AI systems. Used by AI labs, researchers, and companies building or fine-tuning LLMs.
Products and companies: hugging face datasets, scale data engine, coactive ai, surge ai, sieve.
rlhf & human feedback pipelines5 products and companies
Products and companies: rlhf.com, argilla, predibase, labelstudio, openpipe.
streaming & real-time ingestion4 products and companies · its own article →
Products and companies: apache kafka, confluent, redpanda, materialize.
vector & embedding pipelines5 products and companies
Products and companies: weaviate, pinecone, chroma, qdrant, milvus.
unstructured data parsing5 products and companies · its own article →
Unstructured data parsing converts raw files (PDFs, images, documents, logs) into structured formats that downstream systems can process. Data engineers, ML teams, and analytics organizations use it to prepare diverse document types for pipelines.
Products and companies: unstructured, llamaparse, reducto, docling, firecrawl.
See also
This article is made from the map of ai's data. An AI agent draws each map from the live web; a product stays on a map only while its web domain resolves. The texts are the agent's and can be wrong. Demo page.