README.md

---
title: Curation
emoji: 🧹
colorFrom: blue
colorTo: indigo
---

# Curation

### Selecting, filtering and refining high-quality data for better AI systems

**Curation** is an independent Hugging Face organization focused on the methods, infrastructure and evaluation practices used to transform raw data into reliable, useful and well-governed datasets for **pretraining, post-training, fine-tuning, retrieval, evaluation, agents, multimodal AI and Physical AI**.

The organization treats curation as a first-class AI systems discipline.

> **Better models do not begin with more data. They begin with better data.**

---

# What Is AI Data Curation?

AI data curation is the process of **selecting, cleaning, filtering, deduplicating, labeling, balancing, validating and documenting data** so that it becomes more useful for machine learning.

A simplified pipeline:

```text
Raw Data
   │
   ▼
Collection / Ingestion
   │
   ▼
Normalization
   │
   ▼
Filtering
   │
   ▼
Deduplication
   │
   ▼
Quality Scoring
   │
   ▼
Decontamination
   │
   ▼
PII / Safety / Policy Filtering
   │
   ▼
Selection & Balancing
   │
   ▼
Metadata & Provenance
   │
   ▼
Curated Dataset
   │
   ├── Pretraining
   ├── Post-Training
   ├── Fine-Tuning
   ├── Evaluation
   └── Retrieval / Agents
```

Curation is not a single cleaning step.

It is a **system of decisions about what data should remain, what should be removed, how examples should be weighted, how provenance should be tracked and whether a dataset is fit for a specific AI objective**.

---

# Why Curation Matters

Modern AI pipelines are often constrained less by raw data availability than by **data usefulness**.

Large uncurated corpora can contain:

- duplicates
- near-duplicates
- boilerplate
- spam
- malformed text
- low-information content
- benchmark contamination
- personal information
- unsafe content
- licensing uncertainty
- inconsistent formatting
- language noise
- machine-generated repetition
- irrelevant domains
- incorrect labels
- skewed distributions
- stale information

Curation attempts to reduce these problems before they become model problems.

---

# Curation as an AI Infrastructure Layer

A useful way to view the AI stack:

```text
Sources
  ↓
Raw Data
  ↓
CURATION
  ↓
Training / Post-Training Data
  ↓
Models
  ↓
Evaluation
  ↓
Deployment
```

Curation sits between **data acquisition** and **model optimization**.

It affects:

- model quality
- reasoning performance
- factuality
- style
- domain specialization
- robustness
- safety
- memorization
- privacy
- evaluation reliability
- training efficiency
- cost

---

# Core Curation Operations

## 1. Ingestion

The first stage collects or imports data from one or more sources.

Possible sources include:

- web documents
- code repositories
- books
- research papers
- documentation
- enterprise knowledge bases
- support logs
- conversations
- sensor data
- images
- audio
- video
- synthetic data
- agent trajectories

Important ingestion questions:

- Where did the data come from?
- When was it collected?
- Under what terms can it be used?
- Has it already been transformed?
- Can its provenance be preserved?

---

## 2. Normalization

Raw data often arrives in inconsistent formats.

Normalization may include:

- encoding repair
- whitespace cleanup
- Unicode normalization
- document segmentation
- metadata normalization
- timestamp normalization
- field mapping
- schema conversion
- text extraction
- media conversion

The goal is not to remove information unnecessarily, but to create a stable representation for downstream processing.

---

## 3. Rule-Based Filtering

Rule-based filters remove examples that clearly fail predefined requirements.

Examples:

- minimum or maximum length
- excessive symbol ratios
- broken encoding
- repeated characters
- malformed markup
- missing fields
- unsupported languages
- unwanted file types
- boilerplate patterns

Rule-based filtering is often inexpensive and interpretable.

---

## 4. Language Identification

Multilingual corpora require reliable language detection.

Potential decisions include:

- include only target languages
- balance language proportions
- separate code-switching
- preserve low-resource languages
- remove uncertain examples

Language identification itself can introduce bias, especially for:

- dialects
- mixed-language text
- transliterated text
- low-resource languages

---

# Quality Filtering

Quality is not one universal score.

A high-quality example for one task may be irrelevant for another.

Possible quality signals include:

- linguistic coherence
- information density
- domain relevance
- factual structure
- formatting quality
- source credibility
- educational value
- reasoning depth
- code validity
- conversational usefulness

A generic quality pipeline:

```text
Document
   │
   ├── Heuristic Signals
   ├── Statistical Signals
   ├── Model-Based Score
   ├── Source Signals
   └── Task Relevance
          │
          ▼
      Quality Score
          │
          ▼
 Keep / Downweight / Remove
```

---

# Heuristic Quality Signals

Useful heuristics may include:

- average sentence length
- punctuation ratios
- duplicate-line frequency
- stop-word ratios
- alphabetic character ratios
- link density
- markup density
- document length
- paragraph structure

Heuristics are useful because they are:

- fast
- explainable
- cheap to compute
- easy to audit

But they should not be treated as universal measures of quality.

---

# Model-Based Quality Scoring

Learned quality models can estimate whether an example resembles a target distribution.

Possible approaches include:

- binary classifiers
- reward models
- embedding similarity
- LLM-based scoring
- pairwise preference models
- domain relevance models

Example:

```text
Raw Example
    │
    ▼
Quality Model
    │
    ├── 0.91 → Keep
    ├── 0.62 → Review / Downweight
    └── 0.18 → Remove
```

Important questions:

- What was the scorer trained on?
- What biases does it encode?
- Does it favor particular writing styles?
- Does it penalize rare but valuable data?
- Can the score be reproduced?

---

# Deduplication

Duplicate data can waste compute and amplify memorization.

Curation may therefore remove:

- exact duplicates
- near-duplicates
- repeated documents
- repeated paragraphs
- repeated code files
- template-generated variants

---

# Exact Deduplication

Exact deduplication can use:

- hashes
- normalized hashes
- canonicalized text

Example:

```text
Document
   ↓
Normalize
   ↓
Hash
   ↓
Duplicate?
   ├── Yes → Remove / Reference
   └── No  → Keep
```

---

# Near-Duplicate Detection

Near-duplicate detection may use:

- MinHash
- locality-sensitive hashing
- n-gram similarity
- embeddings
- edit distance
- semantic similarity

Near-duplicate removal matters when documents differ only slightly.

---

# Semantic Deduplication

Two examples can be semantically redundant without sharing identical text.

Embedding-based approaches can help identify:

- paraphrases
- mirrored content
- repeated explanations
- translated duplicates
- highly similar synthetic examples

Semantic deduplication is more expensive and must be tuned carefully to avoid removing legitimate diversity.

---

# Benchmark Decontamination

Training data should not silently contain evaluation answers.

Decontamination attempts to identify overlap between training corpora and benchmarks.

A simplified process:

```text
Training Corpus
      │
      ├─────────────┐
      │             │
      ▼             ▼
Benchmark Items   Similarity Search
      │             │
      └──────┬──────┘
             ▼
        Overlap Detection
             │
             ▼
        Remove / Flag
```

Possible overlap checks include:

- exact string matches
- n-gram overlap
- fuzzy matching
- code similarity
- semantic similarity

Evaluation contamination can create misleading performance estimates.

---

# PII Detection and Privacy Filtering

Curated datasets may need to identify or remove personally identifiable information.

Examples include:

- names
- email addresses
- phone numbers
- account identifiers
- addresses
- private credentials
- government identifiers
- personal records

Possible actions:

- remove
- mask
- hash
- tokenize
- restrict
- retain only with appropriate justification

Privacy filtering is particularly important for:

- enterprise datasets
- conversational data
- user-generated content
- support logs
- healthcare or financial workflows

---

# Safety Filtering

Safety filtering may target data involving:

- explicit abuse
- illegal content
- extreme violence
- harmful instructions
- malware
- harassment
- exploitative content

However, aggressive safety filtering can also remove valuable examples needed for:

- moderation research
- red teaming
- safety evaluation
- refusal training

The correct decision depends on the intended use.

---

# Licensing and Usage Rights

A technically useful dataset may still be unsuitable if its usage rights are unclear.

Curation should track:

- source
- license
- redistribution rights
- commercial-use restrictions
- attribution requirements
- derivative-work rules
- jurisdictional constraints

Metadata should remain connected to the data throughout the pipeline.

---

# Provenance

Provenance answers:

> **Where did this example come from, and what happened to it before it reached the dataset?**

Useful provenance metadata can include:

```text
Source
  ↓
Collection Date
  ↓
Original Identifier
  ↓
License
  ↓
Transformation History
  ↓
Filtering Decisions
  ↓
Quality Score
  ↓
Dataset Version
```

Provenance supports:

- reproducibility
- auditing
- governance
- deletion requests
- contamination analysis
- licensing review

---

# Data Lineage

Data lineage extends provenance across multiple processing stages.

Example:

```text
raw_000184
   ↓ normalize
norm_000184
   ↓ language-filter
lang_000184
   ↓ deduplicate
dedup_000184
   ↓ quality-score
score_000184
   ↓ final-selection
train_000184
```

A mature curation pipeline should be able to explain why an example was included.

---

# Data Selection

More data is not always better.

Selection decides which examples are most useful for a target objective.

Possible selection signals include:

- quality
- diversity
- domain relevance
- difficulty
- novelty
- uncertainty
- model loss
- gradient influence
- embedding coverage
- source quality
- task utility

---

# Diversity

A highly filtered dataset can become too homogeneous.

Useful diversity dimensions may include:

- topic
- language
- geography
- domain
- writing style
- source type
- difficulty
- reasoning pattern
- code language
- modality

Curation therefore requires balancing **quality** and **coverage**.

---

# Distribution Balancing

Datasets often contain overrepresented sources or domains.

Balancing can use:

- caps
- quotas
- reweighting
- stratified sampling
- temperature sampling
- source-level mixing

Example:

```text
Web          70%
Code         15%
Books         8%
Research      4%
Dialogue      3%
```

A curation system may intentionally adjust this distribution before training.

---

# Mixture Design

Model training often uses several datasets simultaneously.

Mixture design determines:

- which datasets are included
- how much each contributes
- how sampling changes over time
- whether difficult examples are introduced later
- whether domain data is oversampled

Mixture design is therefore a form of curation.

---

# Curriculum Curation

Training data can be ordered or weighted by difficulty.

Possible progression:

```text
Simple Examples
      ↓
General Examples
      ↓
Difficult Examples
      ↓
Specialized Examples
```

Curriculum strategies can be useful when the order of exposure matters.

---

# Data Curation for Pretraining

Pretraining datasets may require:

- large-scale deduplication
- quality filtering
- language balancing
- source filtering
- safety filtering
- PII removal
- contamination checks
- mixture design
- provenance tracking

At pretraining scale, even inexpensive filters can have significant computational consequences.

---

# Data Curation for Post-Training

Post-training curation has different priorities.

Relevant data may include:

- instruction-response pairs
- preference data
- critiques
- reasoning traces
- tool-use examples
- multi-turn conversations
- agent trajectories
- verification examples

Key questions include:

- Is the instruction clear?
- Is the answer correct?
- Is the preference reliable?
- Is the reasoning useful?
- Is the task representative?
- Is the example too easy?
- Is the label consistent?

---

# Instruction Data Curation

Instruction datasets can be filtered for:

- task diversity
- clarity
- response correctness
- difficulty
- style
- duplication
- safety
- domain coverage

Example:

```text
Raw Instruction Pair
        │
        ├── Instruction Quality
        ├── Answer Quality
        ├── Difficulty
        ├── Domain
        ├── Safety
        └── Duplication
                │
                ▼
          Curated SFT Data
```

---

# Preference Data Curation

Preference optimization depends on reliable comparisons.

Potential issues include:

- noisy annotators
- ambiguous pairs
- trivial differences
- style bias
- length bias
- inconsistent criteria
- reward hacking

Preference curation may include:

- pair filtering
- annotator agreement
- judge calibration
- difficulty balancing
- confidence thresholds

---

# Reasoning Data Curation

Reasoning-oriented datasets can be curated by:

- correctness
- intermediate consistency
- verification
- diversity of solution paths
- problem difficulty
- answer uniqueness
- self-correction quality

For reasoning systems, **verified trajectories** can be more valuable than unverified chain-like text.

---

# Agent Data Curation

Agentic AI introduces new data types.

Examples:

- tool calls
- API traces
- browser actions
- multi-agent messages
- planning steps
- environment states
- task outcomes
- retries
- failures
- recoveries

Agent trajectory curation can evaluate:

- task completion
- tool correctness
- efficiency
- unnecessary actions
- recovery behavior
- safety
- cost
- latency

---

# Tool-Use Data

Tool-use examples may include:

```text
User Goal
   ↓
Tool Selection
   ↓
Arguments
   ↓
Tool Result
   ↓
Reasoning / State Update
   ↓
Final Response
```

Curation should identify:

- invalid tool calls
- unnecessary tool calls
- malformed arguments
- incorrect interpretation of tool results
- unsafe actions

---

# Retrieval and RAG Curation

Retrieval systems depend on curated source material.

RAG curation may include:

- document quality
- chunk quality
- metadata quality
- freshness
- duplication
- source trust
- access permissions
- semantic coverage

A retrieval pipeline can fail even when the generation model is strong if the knowledge base is poorly curated.

---

# Evaluation Data Curation

Evaluation data requires especially careful treatment.

Good evaluation datasets should consider:

- benchmark contamination
- task relevance
- difficulty
- clear ground truth
- judge reliability
- distribution coverage
- adversarial cases
- temporal freshness

Evaluation curation determines what “good performance” actually means.

---

# Synthetic Data Curation

Synthetic data still needs curation.

Generation alone does not guarantee quality.

A synthetic pipeline may look like:

```text
Seed Data
   ↓
Generator
   ↓
Synthetic Examples
   ↓
Verification
   ↓
Deduplication
   ↓
Difficulty / Diversity Scoring
   ↓
Curated Synthetic Dataset
```

Potential synthetic-data problems include:

- mode collapse
- stylistic repetition
- factual errors
- contamination
- easy-example bias
- self-reinforcing model artifacts

---

# Human + Model Curation

Modern curation often combines automation with human judgment.

```text
Automated Filters
       ↓
Model Scoring
       ↓
Human Review
       ↓
Final Selection
```

Humans are particularly useful when:

- context is ambiguous
- quality criteria are subjective
- domain expertise is required
- high-risk data is involved
- the model scorer is uncertain

---

# Active Learning

Active learning can prioritize examples where annotation is most valuable.

Typical workflow:

```text
Unlabeled Pool
     ↓
Current Model
     ↓
Uncertainty / Utility Score
     ↓
Select Examples
     ↓
Human Labeling
     ↓
Retrain
```

This connects curation directly to model improvement.

---

# Data Difficulty

Not every example contributes equally.

Difficulty signals may include:

- model loss
- disagreement between models
- human disagreement
- reasoning depth
- number of steps
- rare concepts
- adversarial structure

Difficulty-aware curation can help create more informative training mixtures.

---

# Domain Curation

Enterprise and specialized AI systems often require domain-specific data.

Examples:

- legal
- finance
- medicine
- engineering
- manufacturing
- cybersecurity
- scientific research
- customer support

Domain curation requires:

- terminology preservation
- source validation
- expert review
- provenance
- licensing
- domain-specific quality criteria

---

# Code Data Curation

Code datasets introduce additional signals.

Possible filters include:

- parseability
- compilation
- test success
- repository quality
- license
- generated files
- vendored dependencies
- duplicate code
- comments
- documentation quality

Code curation may operate at:

- file level
- function level
- repository level
- commit level

---

# Multimodal Curation

Multimodal AI expands curation beyond text.

Possible modalities:

- image
- audio
- video
- depth
- LiDAR
- radar
- tactile signals
- sensor streams

Multimodal curation may require:

- alignment checks
- synchronization
- caption quality
- frame quality
- corrupted-media detection
- duplicate detection
- temporal consistency
- metadata validation

---

# Robotics and Physical AI Data Curation

Physical AI datasets may contain:

- camera feeds
- robot states
- actions
- trajectories
- tactile data
- force data
- depth
- LiDAR
- audio
- environment metadata

A robotics trajectory may need to be curated by:

- success
- task relevance
- safety
- trajectory quality
- control smoothness
- state completeness
- sensor synchronization

---

# Curation Metrics

There is no single universal curation metric.

Useful measurements can include:

## Dataset-Level Metrics

- number of examples
- token count
- modality distribution
- language distribution
- domain distribution
- source distribution
- duplicate rate
- rejection rate
- provenance coverage

## Quality Metrics

- quality-score distribution
- human acceptance rate
- classifier confidence
- annotation agreement
- verification success

## Diversity Metrics

- embedding coverage
- topic entropy
- source diversity
- language diversity
- cluster balance

## Model-Based Metrics

The strongest validation is often downstream:

> **Does the curated dataset produce a better model?**

---

# Ablation Testing

Curation decisions should ideally be tested.

Example:

```text
Dataset A
No Quality Filter
      ↓
Model A

Dataset B
Quality Filter
      ↓
Model B

Compare:
- loss
- benchmark scores
- robustness
- memorization
- safety
```

Ablation testing helps distinguish useful filters from filters that merely sound reasonable.

---

# Data Value

An important research direction is estimating the value of individual examples or subsets.

Possible approaches include:

- influence functions
- gradient similarity
- loss changes
- Shapley-inspired methods
- retraining experiments
- proxy models

The goal is to answer:

> **Which data actually improves the target system?**

---

# Curation at Scale

Large datasets require distributed infrastructure.

A production curation pipeline may include:

```text
Object Storage
     │
     ▼
Distributed Processing
     │
     ├── Parsing
     ├── Filtering
     ├── Scoring
     ├── Deduplication
     ├── PII Detection
     └── Metadata Enrichment
             │
             ▼
       Versioned Dataset
```

Operational concerns include:

- throughput
- memory
- storage
- sharding
- checkpointing
- reproducibility
- failure recovery
- cost

---

# Incremental Curation

Datasets do not need to be rebuilt from scratch every time.

Incremental pipelines can process:

- new documents
- updated sources
- revised labels
- deletion requests
- newly detected contamination

This enables continuously maintained datasets.

---

# Dataset Versioning

A curated dataset should evolve transparently.

Useful version metadata includes:

- version number
- date
- added sources
- removed sources
- filter changes
- scoring changes
- schema changes
- known issues

Versioning is essential for reproducible experiments.

---

# Reproducibility

A reproducible curation pipeline should document:

- source list
- processing code
- filter thresholds
- model versions
- random seeds
- deduplication parameters
- selection strategy
- dataset version

Without this information, it can be difficult to understand why results changed.

---

# Governance

Enterprise curation requires governance.

Relevant concerns include:

- ownership
- access control
- retention
- deletion
- jurisdiction
- licensing
- privacy
- auditability
- provenance

Governance should not be added only after the dataset is created.

It should be part of the curation architecture.

---

# Curation Failure Modes

Common failure modes include:

## Over-Filtering

Removing too much data can reduce:

- diversity
- rare knowledge
- dialect coverage
- creativity
- difficult examples

## Under-Filtering

Keeping too much can increase:

- noise
- repetition
- memorization
- safety problems
- compute waste

## Proxy Bias

A quality classifier may reward examples that resemble its own training distribution.

## Source Concentration

A dataset may appear large while being dominated by a small number of sources.

## Benchmark Leakage

Evaluation data may accidentally appear in training.

## Synthetic Collapse

Repeated model-generated data may narrow distributional diversity.

---

# Curation vs. Cleaning

These concepts overlap but are not identical.

**Cleaning** usually focuses on fixing or removing bad data.

**Curation** is broader.

It includes:

- cleaning
- selection
- prioritization
- balancing
- scoring
- governance
- provenance
- versioning
- alignment with a target objective

---

# Curation vs. Annotation

Annotation adds labels or structured information.

Curation decides:

- which examples deserve annotation
- which labels are trustworthy
- which examples should be retained
- how labeled data should be balanced

Annotation is therefore one component inside a larger curation system.

---

# Curation vs. Data Engineering

Data engineering focuses on reliable data movement and transformation.

Curation focuses on **fitness for AI use**.

The two disciplines overlap strongly.

A complete AI data platform may combine:

```text
Data Engineering
      +
Data Curation
      +
Data Governance
      +
ML Evaluation
```

---

# Curation vs. Synthetic Data

Synthetic data generates new examples.

Curation determines which synthetic examples are useful.

These layers are complementary:

```text
Synthetic Data
      ↓
Curation
      ↓
Post-Training
      ↓
Validation
```

---

# Curation and Post-Training

Post-training quality depends heavily on data selection.

Relevant areas include:

- SFT
- preference optimization
- reward modeling
- reasoning data
- tool use
- agent trajectories
- safety data
- domain specialization

Curation is therefore a natural bridge between raw or generated data and post-training.

---

# Curation and Evaluation

Evaluation and curation should inform each other.

```text
Curated Dataset
      ↓
Train Model
      ↓
Evaluate
      ↓
Identify Weakness
      ↓
Curate Better Data
      ↓
Retrain
```

This creates a **data-centric feedback loop**.

---

# Curation and Data-Centric AI

Model-centric development asks:

> How can we improve the model?

Data-centric development also asks:

> How can we improve the data?

Curation is one of the core operational disciplines behind data-centric AI.

---

# A Reference Curation Architecture

```text
                         RAW DATA
                            │
                            ▼
                    ┌──────────────┐
                    │   INGESTION  │
                    └──────┬───────┘
                           ▼
                    ┌──────────────┐
                    │ NORMALIZATION│
                    └──────┬───────┘
                           ▼
             ┌──────────────────────────┐
             │ RULE / FORMAT FILTERING  │
             └────────────┬─────────────┘
                          ▼
                ┌───────────────────┐
                │   DEDUPLICATION   │
                └─────────┬─────────┘
                          ▼
                ┌───────────────────┐
                │ QUALITY SCORING   │
                └─────────┬─────────┘
                          ▼
          ┌────────────────────────────────┐
          │ PRIVACY / SAFETY / COMPLIANCE  │
          └──────────────┬─────────────────┘
                         ▼
               ┌────────────────────┐
               │ DECONTAMINATION    │
               └─────────┬──────────┘
                         ▼
               ┌────────────────────┐
               │ DATA SELECTION     │
               └─────────┬──────────┘
                         ▼
               ┌────────────────────┐
               │ MIXTURE / BALANCE  │
               └─────────┬──────────┘
                         ▼
               ┌────────────────────┐
               │ PROVENANCE / META  │
               └─────────┬──────────┘
                         ▼
                    CURATED DATA
                         │
          ┌──────────────┼──────────────┐
          ▼              ▼              ▼
     PRETRAINING    POST-TRAINING   EVALUATION
```

---

# Practical Curation Questions

Before using a dataset, ask:

### Source
- Where did the data come from?
- Can the source be traced?

### Quality
- What does “high quality” mean for this task?
- Is quality measured or assumed?

### Duplication
- Are exact and near duplicates present?

### Contamination
- Does the dataset overlap with evaluation data?

### Privacy
- Does the dataset contain personal information?

### Rights
- Is the usage license clear?

### Diversity
- Are important domains or languages missing?

### Bias
- Do the filtering rules systematically remove some data?

### Freshness
- Is the dataset still representative?

### Utility
- Does training on the data actually improve the target system?

---

# Areas We Track

## Data Filtering
Rule-based, statistical and learned filtering.

## Quality Scoring
Models and metrics for estimating data usefulness.

## Deduplication
Exact, near-duplicate and semantic deduplication.

## Decontamination
Detection of training/evaluation overlap.

## Privacy
PII detection, redaction and sensitive-data handling.

## Provenance
Origin, lineage and transformation history.

## Data Selection
Choosing examples based on utility, relevance and diversity.

## Mixture Design
Balancing datasets for training and post-training.

## Synthetic Data Curation
Filtering and verifying generated examples.

## Post-Training Data
Instruction, preference, reasoning and agent data.

## Evaluation Curation
Building reliable benchmarks and test sets.

## Multimodal Curation
Images, video, audio, sensor data and robotics trajectories.

---

# Planned Resources

The organization is intended to become a practical reference layer for AI data curation.

## Curation Explorer

A structured explorer covering:

- curation methods
- filtering strategies
- deduplication
- quality scoring
- decontamination
- privacy
- data selection
- use cases

## Data Quality

An interactive dataset-readiness and quality assessment Space.

## Dataset Curation

A practical reference for transforming raw datasets into AI-ready training or evaluation data.

## Curation Collection

A curated Hugging Face collection of:

- datasets
- models
- papers
- tools
- Spaces
- research
- data-quality resources

---

# SEO & GEO Topic Map

This organization is intentionally structured around explicit concepts relevant to traditional search engines and generative retrieval systems:

- AI data curation
- data curation
- dataset curation
- training data curation
- post-training data
- data quality
- data filtering
- deduplication
- semantic deduplication
- decontamination
- benchmark contamination
- quality scoring
- dataset selection
- data mixture
- data provenance
- data lineage
- synthetic data curation
- preference data
- instruction data
- reasoning data
- agent data
- multimodal datasets
- data-centric AI
- AI-ready data
- dataset governance
- enterprise AI data
- LLM training data

---

# Who Is Curation For?

**Curation** is relevant for:

- foundation model teams
- LLM developers
- dataset creators
- data engineers
- ML engineers
- researchers
- post-training teams
- evaluation teams
- enterprise AI teams
- data-platform companies
- annotation providers
- synthetic-data companies
- data-governance teams
- multimodal AI teams
- robotics teams
- open-source communities

---

# Principles

## Quality Is Task-Dependent

There is no universal definition of good data.

Curation criteria should be linked to an explicit objective.

## Provenance Matters

Useful data should remain traceable wherever possible.

## Filtering Should Be Measurable

Filters should be evaluated by their downstream effects, not only by intuition.

## Diversity Has Value

Aggressive filtering should not destroy important distributional coverage.

## Evaluation Must Be Protected

Benchmark contamination should be treated as a curation problem.

## Human Judgment Still Matters

Automation can scale curation, but humans remain important for ambiguous and high-stakes decisions.

## Transparency Improves Reproducibility

Dataset versions, filtering rules and known limitations should be documented.

---

# Collaboration & Partnerships

**Curation is open to collaboration with companies, research teams, universities, open-source projects and organizations working on AI data quality and data-centric machine learning.**

Relevant collaboration areas include:

- training data
- dataset curation
- data quality
- deduplication
- filtering
- decontamination
- provenance
- privacy
- synthetic data
- annotation
- post-training data
- preference data
- evaluation data
- agent trajectories
- multimodal data
- enterprise AI data
- data governance
- data infrastructure

Possible collaboration formats include:

- technical showcases
- dataset curation case studies
- joint Hugging Face Spaces
- open-source integrations
- benchmark projects
- dataset quality analysis
- ecosystem maps
- research collaborations
- methodology comparisons
- clearly disclosed partnerships and sponsorships

## Collaboration Contact

For technical collaborations, research partnerships, ecosystem projects or sponsorship inquiries:

**agenten@magenta.de**

---

# Independent Organization

**Curation is an independent Hugging Face organization.**

It is not an official organization of Hugging Face or of any dataset provider, annotation company, AI laboratory, model provider or technology company that may be referenced in future resources.

Product names and trademarks belong to their respective owners.

---

# Long-Term Vision

AI development is increasingly becoming a data-quality problem.

As models become stronger, the difference between useful and useless data becomes more important.

The long-term objective of **Curation** is to become an open technical reference for understanding how raw information becomes **training-ready, evaluation-ready and AI-ready data**.

> **Collecting data creates volume. Curation creates value.**

---

## Filter. Select. Verify. Refine.

### Curation

**Building better AI through better data.**

For collaborations, research and partnerships:

**agenten@magenta.de**