| --- |
| title: Curation |
| emoji: 🧹 |
| colorFrom: blue |
| colorTo: indigo |
| --- |
| |
| # Curation |
|
|
| ### Selecting, filtering and refining high-quality data for better AI systems |
|
|
| **Curation** is an independent Hugging Face organization focused on the methods, infrastructure and evaluation practices used to transform raw data into reliable, useful and well-governed datasets for **pretraining, post-training, fine-tuning, retrieval, evaluation, agents, multimodal AI and Physical AI**. |
|
|
| The organization treats curation as a first-class AI systems discipline. |
|
|
| > **Better models do not begin with more data. They begin with better data.** |
|
|
| --- |
|
|
| # What Is AI Data Curation? |
|
|
| AI data curation is the process of **selecting, cleaning, filtering, deduplicating, labeling, balancing, validating and documenting data** so that it becomes more useful for machine learning. |
|
|
| A simplified pipeline: |
|
|
| ```text |
| Raw Data |
| │ |
| ▼ |
| Collection / Ingestion |
| │ |
| ▼ |
| Normalization |
| │ |
| ▼ |
| Filtering |
| │ |
| ▼ |
| Deduplication |
| │ |
| ▼ |
| Quality Scoring |
| │ |
| ▼ |
| Decontamination |
| │ |
| ▼ |
| PII / Safety / Policy Filtering |
| │ |
| ▼ |
| Selection & Balancing |
| │ |
| ▼ |
| Metadata & Provenance |
| │ |
| ▼ |
| Curated Dataset |
| │ |
| ├── Pretraining |
| ├── Post-Training |
| ├── Fine-Tuning |
| ├── Evaluation |
| └── Retrieval / Agents |
| ``` |
|
|
| Curation is not a single cleaning step. |
|
|
| It is a **system of decisions about what data should remain, what should be removed, how examples should be weighted, how provenance should be tracked and whether a dataset is fit for a specific AI objective**. |
|
|
| --- |
|
|
| # Why Curation Matters |
|
|
| Modern AI pipelines are often constrained less by raw data availability than by **data usefulness**. |
|
|
| Large uncurated corpora can contain: |
|
|
| - duplicates |
| - near-duplicates |
| - boilerplate |
| - spam |
| - malformed text |
| - low-information content |
| - benchmark contamination |
| - personal information |
| - unsafe content |
| - licensing uncertainty |
| - inconsistent formatting |
| - language noise |
| - machine-generated repetition |
| - irrelevant domains |
| - incorrect labels |
| - skewed distributions |
| - stale information |
|
|
| Curation attempts to reduce these problems before they become model problems. |
|
|
| --- |
|
|
| # Curation as an AI Infrastructure Layer |
|
|
| A useful way to view the AI stack: |
|
|
| ```text |
| Sources |
| ↓ |
| Raw Data |
| ↓ |
| CURATION |
| ↓ |
| Training / Post-Training Data |
| ↓ |
| Models |
| ↓ |
| Evaluation |
| ↓ |
| Deployment |
| ``` |
|
|
| Curation sits between **data acquisition** and **model optimization**. |
|
|
| It affects: |
|
|
| - model quality |
| - reasoning performance |
| - factuality |
| - style |
| - domain specialization |
| - robustness |
| - safety |
| - memorization |
| - privacy |
| - evaluation reliability |
| - training efficiency |
| - cost |
|
|
| --- |
|
|
| # Core Curation Operations |
|
|
| ## 1. Ingestion |
|
|
| The first stage collects or imports data from one or more sources. |
|
|
| Possible sources include: |
|
|
| - web documents |
| - code repositories |
| - books |
| - research papers |
| - documentation |
| - enterprise knowledge bases |
| - support logs |
| - conversations |
| - sensor data |
| - images |
| - audio |
| - video |
| - synthetic data |
| - agent trajectories |
|
|
| Important ingestion questions: |
|
|
| - Where did the data come from? |
| - When was it collected? |
| - Under what terms can it be used? |
| - Has it already been transformed? |
| - Can its provenance be preserved? |
|
|
| --- |
|
|
| ## 2. Normalization |
|
|
| Raw data often arrives in inconsistent formats. |
|
|
| Normalization may include: |
|
|
| - encoding repair |
| - whitespace cleanup |
| - Unicode normalization |
| - document segmentation |
| - metadata normalization |
| - timestamp normalization |
| - field mapping |
| - schema conversion |
| - text extraction |
| - media conversion |
|
|
| The goal is not to remove information unnecessarily, but to create a stable representation for downstream processing. |
|
|
| --- |
|
|
| ## 3. Rule-Based Filtering |
|
|
| Rule-based filters remove examples that clearly fail predefined requirements. |
|
|
| Examples: |
|
|
| - minimum or maximum length |
| - excessive symbol ratios |
| - broken encoding |
| - repeated characters |
| - malformed markup |
| - missing fields |
| - unsupported languages |
| - unwanted file types |
| - boilerplate patterns |
|
|
| Rule-based filtering is often inexpensive and interpretable. |
|
|
| --- |
|
|
| ## 4. Language Identification |
|
|
| Multilingual corpora require reliable language detection. |
|
|
| Potential decisions include: |
|
|
| - include only target languages |
| - balance language proportions |
| - separate code-switching |
| - preserve low-resource languages |
| - remove uncertain examples |
|
|
| Language identification itself can introduce bias, especially for: |
|
|
| - dialects |
| - mixed-language text |
| - transliterated text |
| - low-resource languages |
|
|
| --- |
|
|
| # Quality Filtering |
|
|
| Quality is not one universal score. |
|
|
| A high-quality example for one task may be irrelevant for another. |
|
|
| Possible quality signals include: |
|
|
| - linguistic coherence |
| - information density |
| - domain relevance |
| - factual structure |
| - formatting quality |
| - source credibility |
| - educational value |
| - reasoning depth |
| - code validity |
| - conversational usefulness |
|
|
| A generic quality pipeline: |
|
|
| ```text |
| Document |
| │ |
| ├── Heuristic Signals |
| ├── Statistical Signals |
| ├── Model-Based Score |
| ├── Source Signals |
| └── Task Relevance |
| │ |
| ▼ |
| Quality Score |
| │ |
| ▼ |
| Keep / Downweight / Remove |
| ``` |
|
|
| --- |
|
|
| # Heuristic Quality Signals |
|
|
| Useful heuristics may include: |
|
|
| - average sentence length |
| - punctuation ratios |
| - duplicate-line frequency |
| - stop-word ratios |
| - alphabetic character ratios |
| - link density |
| - markup density |
| - document length |
| - paragraph structure |
|
|
| Heuristics are useful because they are: |
|
|
| - fast |
| - explainable |
| - cheap to compute |
| - easy to audit |
|
|
| But they should not be treated as universal measures of quality. |
|
|
| --- |
|
|
| # Model-Based Quality Scoring |
|
|
| Learned quality models can estimate whether an example resembles a target distribution. |
|
|
| Possible approaches include: |
|
|
| - binary classifiers |
| - reward models |
| - embedding similarity |
| - LLM-based scoring |
| - pairwise preference models |
| - domain relevance models |
|
|
| Example: |
|
|
| ```text |
| Raw Example |
| │ |
| ▼ |
| Quality Model |
| │ |
| ├── 0.91 → Keep |
| ├── 0.62 → Review / Downweight |
| └── 0.18 → Remove |
| ``` |
|
|
| Important questions: |
|
|
| - What was the scorer trained on? |
| - What biases does it encode? |
| - Does it favor particular writing styles? |
| - Does it penalize rare but valuable data? |
| - Can the score be reproduced? |
|
|
| --- |
|
|
| # Deduplication |
|
|
| Duplicate data can waste compute and amplify memorization. |
|
|
| Curation may therefore remove: |
|
|
| - exact duplicates |
| - near-duplicates |
| - repeated documents |
| - repeated paragraphs |
| - repeated code files |
| - template-generated variants |
|
|
| --- |
|
|
| # Exact Deduplication |
|
|
| Exact deduplication can use: |
|
|
| - hashes |
| - normalized hashes |
| - canonicalized text |
|
|
| Example: |
|
|
| ```text |
| Document |
| ↓ |
| Normalize |
| ↓ |
| Hash |
| ↓ |
| Duplicate? |
| ├── Yes → Remove / Reference |
| └── No → Keep |
| ``` |
|
|
| --- |
|
|
| # Near-Duplicate Detection |
|
|
| Near-duplicate detection may use: |
|
|
| - MinHash |
| - locality-sensitive hashing |
| - n-gram similarity |
| - embeddings |
| - edit distance |
| - semantic similarity |
|
|
| Near-duplicate removal matters when documents differ only slightly. |
|
|
| --- |
|
|
| # Semantic Deduplication |
|
|
| Two examples can be semantically redundant without sharing identical text. |
|
|
| Embedding-based approaches can help identify: |
|
|
| - paraphrases |
| - mirrored content |
| - repeated explanations |
| - translated duplicates |
| - highly similar synthetic examples |
|
|
| Semantic deduplication is more expensive and must be tuned carefully to avoid removing legitimate diversity. |
|
|
| --- |
|
|
| # Benchmark Decontamination |
|
|
| Training data should not silently contain evaluation answers. |
|
|
| Decontamination attempts to identify overlap between training corpora and benchmarks. |
|
|
| A simplified process: |
|
|
| ```text |
| Training Corpus |
| │ |
| ├─────────────┐ |
| │ │ |
| ▼ ▼ |
| Benchmark Items Similarity Search |
| │ │ |
| └──────┬──────┘ |
| ▼ |
| Overlap Detection |
| │ |
| ▼ |
| Remove / Flag |
| ``` |
|
|
| Possible overlap checks include: |
|
|
| - exact string matches |
| - n-gram overlap |
| - fuzzy matching |
| - code similarity |
| - semantic similarity |
|
|
| Evaluation contamination can create misleading performance estimates. |
|
|
| --- |
|
|
| # PII Detection and Privacy Filtering |
|
|
| Curated datasets may need to identify or remove personally identifiable information. |
|
|
| Examples include: |
|
|
| - names |
| - email addresses |
| - phone numbers |
| - account identifiers |
| - addresses |
| - private credentials |
| - government identifiers |
| - personal records |
|
|
| Possible actions: |
|
|
| - remove |
| - mask |
| - hash |
| - tokenize |
| - restrict |
| - retain only with appropriate justification |
|
|
| Privacy filtering is particularly important for: |
|
|
| - enterprise datasets |
| - conversational data |
| - user-generated content |
| - support logs |
| - healthcare or financial workflows |
|
|
| --- |
|
|
| # Safety Filtering |
|
|
| Safety filtering may target data involving: |
|
|
| - explicit abuse |
| - illegal content |
| - extreme violence |
| - harmful instructions |
| - malware |
| - harassment |
| - exploitative content |
|
|
| However, aggressive safety filtering can also remove valuable examples needed for: |
|
|
| - moderation research |
| - red teaming |
| - safety evaluation |
| - refusal training |
|
|
| The correct decision depends on the intended use. |
|
|
| --- |
|
|
| # Licensing and Usage Rights |
|
|
| A technically useful dataset may still be unsuitable if its usage rights are unclear. |
|
|
| Curation should track: |
|
|
| - source |
| - license |
| - redistribution rights |
| - commercial-use restrictions |
| - attribution requirements |
| - derivative-work rules |
| - jurisdictional constraints |
|
|
| Metadata should remain connected to the data throughout the pipeline. |
|
|
| --- |
|
|
| # Provenance |
|
|
| Provenance answers: |
|
|
| > **Where did this example come from, and what happened to it before it reached the dataset?** |
|
|
| Useful provenance metadata can include: |
|
|
| ```text |
| Source |
| ↓ |
| Collection Date |
| ↓ |
| Original Identifier |
| ↓ |
| License |
| ↓ |
| Transformation History |
| ↓ |
| Filtering Decisions |
| ↓ |
| Quality Score |
| ↓ |
| Dataset Version |
| ``` |
|
|
| Provenance supports: |
|
|
| - reproducibility |
| - auditing |
| - governance |
| - deletion requests |
| - contamination analysis |
| - licensing review |
|
|
| --- |
|
|
| # Data Lineage |
|
|
| Data lineage extends provenance across multiple processing stages. |
|
|
| Example: |
|
|
| ```text |
| raw_000184 |
| ↓ normalize |
| norm_000184 |
| ↓ language-filter |
| lang_000184 |
| ↓ deduplicate |
| dedup_000184 |
| ↓ quality-score |
| score_000184 |
| ↓ final-selection |
| train_000184 |
| ``` |
|
|
| A mature curation pipeline should be able to explain why an example was included. |
|
|
| --- |
|
|
| # Data Selection |
|
|
| More data is not always better. |
|
|
| Selection decides which examples are most useful for a target objective. |
|
|
| Possible selection signals include: |
|
|
| - quality |
| - diversity |
| - domain relevance |
| - difficulty |
| - novelty |
| - uncertainty |
| - model loss |
| - gradient influence |
| - embedding coverage |
| - source quality |
| - task utility |
|
|
| --- |
|
|
| # Diversity |
|
|
| A highly filtered dataset can become too homogeneous. |
|
|
| Useful diversity dimensions may include: |
|
|
| - topic |
| - language |
| - geography |
| - domain |
| - writing style |
| - source type |
| - difficulty |
| - reasoning pattern |
| - code language |
| - modality |
|
|
| Curation therefore requires balancing **quality** and **coverage**. |
|
|
| --- |
|
|
| # Distribution Balancing |
|
|
| Datasets often contain overrepresented sources or domains. |
|
|
| Balancing can use: |
|
|
| - caps |
| - quotas |
| - reweighting |
| - stratified sampling |
| - temperature sampling |
| - source-level mixing |
|
|
| Example: |
|
|
| ```text |
| Web 70% |
| Code 15% |
| Books 8% |
| Research 4% |
| Dialogue 3% |
| ``` |
|
|
| A curation system may intentionally adjust this distribution before training. |
|
|
| --- |
|
|
| # Mixture Design |
|
|
| Model training often uses several datasets simultaneously. |
|
|
| Mixture design determines: |
|
|
| - which datasets are included |
| - how much each contributes |
| - how sampling changes over time |
| - whether difficult examples are introduced later |
| - whether domain data is oversampled |
|
|
| Mixture design is therefore a form of curation. |
|
|
| --- |
|
|
| # Curriculum Curation |
|
|
| Training data can be ordered or weighted by difficulty. |
|
|
| Possible progression: |
|
|
| ```text |
| Simple Examples |
| ↓ |
| General Examples |
| ↓ |
| Difficult Examples |
| ↓ |
| Specialized Examples |
| ``` |
|
|
| Curriculum strategies can be useful when the order of exposure matters. |
|
|
| --- |
|
|
| # Data Curation for Pretraining |
|
|
| Pretraining datasets may require: |
|
|
| - large-scale deduplication |
| - quality filtering |
| - language balancing |
| - source filtering |
| - safety filtering |
| - PII removal |
| - contamination checks |
| - mixture design |
| - provenance tracking |
|
|
| At pretraining scale, even inexpensive filters can have significant computational consequences. |
|
|
| --- |
|
|
| # Data Curation for Post-Training |
|
|
| Post-training curation has different priorities. |
|
|
| Relevant data may include: |
|
|
| - instruction-response pairs |
| - preference data |
| - critiques |
| - reasoning traces |
| - tool-use examples |
| - multi-turn conversations |
| - agent trajectories |
| - verification examples |
|
|
| Key questions include: |
|
|
| - Is the instruction clear? |
| - Is the answer correct? |
| - Is the preference reliable? |
| - Is the reasoning useful? |
| - Is the task representative? |
| - Is the example too easy? |
| - Is the label consistent? |
|
|
| --- |
|
|
| # Instruction Data Curation |
|
|
| Instruction datasets can be filtered for: |
|
|
| - task diversity |
| - clarity |
| - response correctness |
| - difficulty |
| - style |
| - duplication |
| - safety |
| - domain coverage |
|
|
| Example: |
|
|
| ```text |
| Raw Instruction Pair |
| │ |
| ├── Instruction Quality |
| ├── Answer Quality |
| ├── Difficulty |
| ├── Domain |
| ├── Safety |
| └── Duplication |
| │ |
| ▼ |
| Curated SFT Data |
| ``` |
|
|
| --- |
|
|
| # Preference Data Curation |
|
|
| Preference optimization depends on reliable comparisons. |
|
|
| Potential issues include: |
|
|
| - noisy annotators |
| - ambiguous pairs |
| - trivial differences |
| - style bias |
| - length bias |
| - inconsistent criteria |
| - reward hacking |
|
|
| Preference curation may include: |
|
|
| - pair filtering |
| - annotator agreement |
| - judge calibration |
| - difficulty balancing |
| - confidence thresholds |
|
|
| --- |
|
|
| # Reasoning Data Curation |
|
|
| Reasoning-oriented datasets can be curated by: |
|
|
| - correctness |
| - intermediate consistency |
| - verification |
| - diversity of solution paths |
| - problem difficulty |
| - answer uniqueness |
| - self-correction quality |
|
|
| For reasoning systems, **verified trajectories** can be more valuable than unverified chain-like text. |
|
|
| --- |
|
|
| # Agent Data Curation |
|
|
| Agentic AI introduces new data types. |
|
|
| Examples: |
|
|
| - tool calls |
| - API traces |
| - browser actions |
| - multi-agent messages |
| - planning steps |
| - environment states |
| - task outcomes |
| - retries |
| - failures |
| - recoveries |
|
|
| Agent trajectory curation can evaluate: |
|
|
| - task completion |
| - tool correctness |
| - efficiency |
| - unnecessary actions |
| - recovery behavior |
| - safety |
| - cost |
| - latency |
|
|
| --- |
|
|
| # Tool-Use Data |
|
|
| Tool-use examples may include: |
|
|
| ```text |
| User Goal |
| ↓ |
| Tool Selection |
| ↓ |
| Arguments |
| ↓ |
| Tool Result |
| ↓ |
| Reasoning / State Update |
| ↓ |
| Final Response |
| ``` |
|
|
| Curation should identify: |
|
|
| - invalid tool calls |
| - unnecessary tool calls |
| - malformed arguments |
| - incorrect interpretation of tool results |
| - unsafe actions |
|
|
| --- |
|
|
| # Retrieval and RAG Curation |
|
|
| Retrieval systems depend on curated source material. |
|
|
| RAG curation may include: |
|
|
| - document quality |
| - chunk quality |
| - metadata quality |
| - freshness |
| - duplication |
| - source trust |
| - access permissions |
| - semantic coverage |
|
|
| A retrieval pipeline can fail even when the generation model is strong if the knowledge base is poorly curated. |
|
|
| --- |
|
|
| # Evaluation Data Curation |
|
|
| Evaluation data requires especially careful treatment. |
|
|
| Good evaluation datasets should consider: |
|
|
| - benchmark contamination |
| - task relevance |
| - difficulty |
| - clear ground truth |
| - judge reliability |
| - distribution coverage |
| - adversarial cases |
| - temporal freshness |
|
|
| Evaluation curation determines what “good performance” actually means. |
|
|
| --- |
|
|
| # Synthetic Data Curation |
|
|
| Synthetic data still needs curation. |
|
|
| Generation alone does not guarantee quality. |
|
|
| A synthetic pipeline may look like: |
|
|
| ```text |
| Seed Data |
| ↓ |
| Generator |
| ↓ |
| Synthetic Examples |
| ↓ |
| Verification |
| ↓ |
| Deduplication |
| ↓ |
| Difficulty / Diversity Scoring |
| ↓ |
| Curated Synthetic Dataset |
| ``` |
|
|
| Potential synthetic-data problems include: |
|
|
| - mode collapse |
| - stylistic repetition |
| - factual errors |
| - contamination |
| - easy-example bias |
| - self-reinforcing model artifacts |
|
|
| --- |
|
|
| # Human + Model Curation |
|
|
| Modern curation often combines automation with human judgment. |
|
|
| ```text |
| Automated Filters |
| ↓ |
| Model Scoring |
| ↓ |
| Human Review |
| ↓ |
| Final Selection |
| ``` |
|
|
| Humans are particularly useful when: |
|
|
| - context is ambiguous |
| - quality criteria are subjective |
| - domain expertise is required |
| - high-risk data is involved |
| - the model scorer is uncertain |
|
|
| --- |
|
|
| # Active Learning |
|
|
| Active learning can prioritize examples where annotation is most valuable. |
|
|
| Typical workflow: |
|
|
| ```text |
| Unlabeled Pool |
| ↓ |
| Current Model |
| ↓ |
| Uncertainty / Utility Score |
| ↓ |
| Select Examples |
| ↓ |
| Human Labeling |
| ↓ |
| Retrain |
| ``` |
|
|
| This connects curation directly to model improvement. |
|
|
| --- |
|
|
| # Data Difficulty |
|
|
| Not every example contributes equally. |
|
|
| Difficulty signals may include: |
|
|
| - model loss |
| - disagreement between models |
| - human disagreement |
| - reasoning depth |
| - number of steps |
| - rare concepts |
| - adversarial structure |
|
|
| Difficulty-aware curation can help create more informative training mixtures. |
|
|
| --- |
|
|
| # Domain Curation |
|
|
| Enterprise and specialized AI systems often require domain-specific data. |
|
|
| Examples: |
|
|
| - legal |
| - finance |
| - medicine |
| - engineering |
| - manufacturing |
| - cybersecurity |
| - scientific research |
| - customer support |
|
|
| Domain curation requires: |
|
|
| - terminology preservation |
| - source validation |
| - expert review |
| - provenance |
| - licensing |
| - domain-specific quality criteria |
|
|
| --- |
|
|
| # Code Data Curation |
|
|
| Code datasets introduce additional signals. |
|
|
| Possible filters include: |
|
|
| - parseability |
| - compilation |
| - test success |
| - repository quality |
| - license |
| - generated files |
| - vendored dependencies |
| - duplicate code |
| - comments |
| - documentation quality |
|
|
| Code curation may operate at: |
|
|
| - file level |
| - function level |
| - repository level |
| - commit level |
|
|
| --- |
|
|
| # Multimodal Curation |
|
|
| Multimodal AI expands curation beyond text. |
|
|
| Possible modalities: |
|
|
| - image |
| - audio |
| - video |
| - depth |
| - LiDAR |
| - radar |
| - tactile signals |
| - sensor streams |
|
|
| Multimodal curation may require: |
|
|
| - alignment checks |
| - synchronization |
| - caption quality |
| - frame quality |
| - corrupted-media detection |
| - duplicate detection |
| - temporal consistency |
| - metadata validation |
|
|
| --- |
|
|
| # Robotics and Physical AI Data Curation |
|
|
| Physical AI datasets may contain: |
|
|
| - camera feeds |
| - robot states |
| - actions |
| - trajectories |
| - tactile data |
| - force data |
| - depth |
| - LiDAR |
| - audio |
| - environment metadata |
|
|
| A robotics trajectory may need to be curated by: |
|
|
| - success |
| - task relevance |
| - safety |
| - trajectory quality |
| - control smoothness |
| - state completeness |
| - sensor synchronization |
|
|
| --- |
|
|
| # Curation Metrics |
|
|
| There is no single universal curation metric. |
|
|
| Useful measurements can include: |
|
|
| ## Dataset-Level Metrics |
|
|
| - number of examples |
| - token count |
| - modality distribution |
| - language distribution |
| - domain distribution |
| - source distribution |
| - duplicate rate |
| - rejection rate |
| - provenance coverage |
|
|
| ## Quality Metrics |
|
|
| - quality-score distribution |
| - human acceptance rate |
| - classifier confidence |
| - annotation agreement |
| - verification success |
|
|
| ## Diversity Metrics |
|
|
| - embedding coverage |
| - topic entropy |
| - source diversity |
| - language diversity |
| - cluster balance |
|
|
| ## Model-Based Metrics |
|
|
| The strongest validation is often downstream: |
|
|
| > **Does the curated dataset produce a better model?** |
|
|
| --- |
|
|
| # Ablation Testing |
|
|
| Curation decisions should ideally be tested. |
|
|
| Example: |
|
|
| ```text |
| Dataset A |
| No Quality Filter |
| ↓ |
| Model A |
| |
| Dataset B |
| Quality Filter |
| ↓ |
| Model B |
| |
| Compare: |
| - loss |
| - benchmark scores |
| - robustness |
| - memorization |
| - safety |
| ``` |
|
|
| Ablation testing helps distinguish useful filters from filters that merely sound reasonable. |
|
|
| --- |
|
|
| # Data Value |
|
|
| An important research direction is estimating the value of individual examples or subsets. |
|
|
| Possible approaches include: |
|
|
| - influence functions |
| - gradient similarity |
| - loss changes |
| - Shapley-inspired methods |
| - retraining experiments |
| - proxy models |
|
|
| The goal is to answer: |
|
|
| > **Which data actually improves the target system?** |
|
|
| --- |
|
|
| # Curation at Scale |
|
|
| Large datasets require distributed infrastructure. |
|
|
| A production curation pipeline may include: |
|
|
| ```text |
| Object Storage |
| │ |
| ▼ |
| Distributed Processing |
| │ |
| ├── Parsing |
| ├── Filtering |
| ├── Scoring |
| ├── Deduplication |
| ├── PII Detection |
| └── Metadata Enrichment |
| │ |
| ▼ |
| Versioned Dataset |
| ``` |
|
|
| Operational concerns include: |
|
|
| - throughput |
| - memory |
| - storage |
| - sharding |
| - checkpointing |
| - reproducibility |
| - failure recovery |
| - cost |
|
|
| --- |
|
|
| # Incremental Curation |
|
|
| Datasets do not need to be rebuilt from scratch every time. |
|
|
| Incremental pipelines can process: |
|
|
| - new documents |
| - updated sources |
| - revised labels |
| - deletion requests |
| - newly detected contamination |
|
|
| This enables continuously maintained datasets. |
|
|
| --- |
|
|
| # Dataset Versioning |
|
|
| A curated dataset should evolve transparently. |
|
|
| Useful version metadata includes: |
|
|
| - version number |
| - date |
| - added sources |
| - removed sources |
| - filter changes |
| - scoring changes |
| - schema changes |
| - known issues |
|
|
| Versioning is essential for reproducible experiments. |
|
|
| --- |
|
|
| # Reproducibility |
|
|
| A reproducible curation pipeline should document: |
|
|
| - source list |
| - processing code |
| - filter thresholds |
| - model versions |
| - random seeds |
| - deduplication parameters |
| - selection strategy |
| - dataset version |
|
|
| Without this information, it can be difficult to understand why results changed. |
|
|
| --- |
|
|
| # Governance |
|
|
| Enterprise curation requires governance. |
|
|
| Relevant concerns include: |
|
|
| - ownership |
| - access control |
| - retention |
| - deletion |
| - jurisdiction |
| - licensing |
| - privacy |
| - auditability |
| - provenance |
|
|
| Governance should not be added only after the dataset is created. |
|
|
| It should be part of the curation architecture. |
|
|
| --- |
|
|
| # Curation Failure Modes |
|
|
| Common failure modes include: |
|
|
| ## Over-Filtering |
|
|
| Removing too much data can reduce: |
|
|
| - diversity |
| - rare knowledge |
| - dialect coverage |
| - creativity |
| - difficult examples |
|
|
| ## Under-Filtering |
|
|
| Keeping too much can increase: |
|
|
| - noise |
| - repetition |
| - memorization |
| - safety problems |
| - compute waste |
|
|
| ## Proxy Bias |
|
|
| A quality classifier may reward examples that resemble its own training distribution. |
|
|
| ## Source Concentration |
|
|
| A dataset may appear large while being dominated by a small number of sources. |
|
|
| ## Benchmark Leakage |
|
|
| Evaluation data may accidentally appear in training. |
|
|
| ## Synthetic Collapse |
|
|
| Repeated model-generated data may narrow distributional diversity. |
|
|
| --- |
|
|
| # Curation vs. Cleaning |
|
|
| These concepts overlap but are not identical. |
|
|
| **Cleaning** usually focuses on fixing or removing bad data. |
|
|
| **Curation** is broader. |
|
|
| It includes: |
|
|
| - cleaning |
| - selection |
| - prioritization |
| - balancing |
| - scoring |
| - governance |
| - provenance |
| - versioning |
| - alignment with a target objective |
|
|
| --- |
|
|
| # Curation vs. Annotation |
|
|
| Annotation adds labels or structured information. |
|
|
| Curation decides: |
|
|
| - which examples deserve annotation |
| - which labels are trustworthy |
| - which examples should be retained |
| - how labeled data should be balanced |
|
|
| Annotation is therefore one component inside a larger curation system. |
|
|
| --- |
|
|
| # Curation vs. Data Engineering |
|
|
| Data engineering focuses on reliable data movement and transformation. |
|
|
| Curation focuses on **fitness for AI use**. |
|
|
| The two disciplines overlap strongly. |
|
|
| A complete AI data platform may combine: |
|
|
| ```text |
| Data Engineering |
| + |
| Data Curation |
| + |
| Data Governance |
| + |
| ML Evaluation |
| ``` |
|
|
| --- |
|
|
| # Curation vs. Synthetic Data |
|
|
| Synthetic data generates new examples. |
|
|
| Curation determines which synthetic examples are useful. |
|
|
| These layers are complementary: |
|
|
| ```text |
| Synthetic Data |
| ↓ |
| Curation |
| ↓ |
| Post-Training |
| ↓ |
| Validation |
| ``` |
|
|
| --- |
|
|
| # Curation and Post-Training |
|
|
| Post-training quality depends heavily on data selection. |
|
|
| Relevant areas include: |
|
|
| - SFT |
| - preference optimization |
| - reward modeling |
| - reasoning data |
| - tool use |
| - agent trajectories |
| - safety data |
| - domain specialization |
|
|
| Curation is therefore a natural bridge between raw or generated data and post-training. |
|
|
| --- |
|
|
| # Curation and Evaluation |
|
|
| Evaluation and curation should inform each other. |
|
|
| ```text |
| Curated Dataset |
| ↓ |
| Train Model |
| ↓ |
| Evaluate |
| ↓ |
| Identify Weakness |
| ↓ |
| Curate Better Data |
| ↓ |
| Retrain |
| ``` |
|
|
| This creates a **data-centric feedback loop**. |
|
|
| --- |
|
|
| # Curation and Data-Centric AI |
|
|
| Model-centric development asks: |
|
|
| > How can we improve the model? |
|
|
| Data-centric development also asks: |
|
|
| > How can we improve the data? |
|
|
| Curation is one of the core operational disciplines behind data-centric AI. |
|
|
| --- |
|
|
| # A Reference Curation Architecture |
|
|
| ```text |
| RAW DATA |
| │ |
| ▼ |
| ┌──────────────┐ |
| │ INGESTION │ |
| └──────┬───────┘ |
| ▼ |
| ┌──────────────┐ |
| │ NORMALIZATION│ |
| └──────┬───────┘ |
| ▼ |
| ┌──────────────────────────┐ |
| │ RULE / FORMAT FILTERING │ |
| └────────────┬─────────────┘ |
| ▼ |
| ┌───────────────────┐ |
| │ DEDUPLICATION │ |
| └─────────┬─────────┘ |
| ▼ |
| ┌───────────────────┐ |
| │ QUALITY SCORING │ |
| └─────────┬─────────┘ |
| ▼ |
| ┌────────────────────────────────┐ |
| │ PRIVACY / SAFETY / COMPLIANCE │ |
| └──────────────┬─────────────────┘ |
| ▼ |
| ┌────────────────────┐ |
| │ DECONTAMINATION │ |
| └─────────┬──────────┘ |
| ▼ |
| ┌────────────────────┐ |
| │ DATA SELECTION │ |
| └─────────┬──────────┘ |
| ▼ |
| ┌────────────────────┐ |
| │ MIXTURE / BALANCE │ |
| └─────────┬──────────┘ |
| ▼ |
| ┌────────────────────┐ |
| │ PROVENANCE / META │ |
| └─────────┬──────────┘ |
| ▼ |
| CURATED DATA |
| │ |
| ┌──────────────┼──────────────┐ |
| ▼ ▼ ▼ |
| PRETRAINING POST-TRAINING EVALUATION |
| ``` |
|
|
| --- |
|
|
| # Practical Curation Questions |
|
|
| Before using a dataset, ask: |
|
|
| ### Source |
| - Where did the data come from? |
| - Can the source be traced? |
|
|
| ### Quality |
| - What does “high quality” mean for this task? |
| - Is quality measured or assumed? |
|
|
| ### Duplication |
| - Are exact and near duplicates present? |
|
|
| ### Contamination |
| - Does the dataset overlap with evaluation data? |
|
|
| ### Privacy |
| - Does the dataset contain personal information? |
|
|
| ### Rights |
| - Is the usage license clear? |
|
|
| ### Diversity |
| - Are important domains or languages missing? |
|
|
| ### Bias |
| - Do the filtering rules systematically remove some data? |
|
|
| ### Freshness |
| - Is the dataset still representative? |
|
|
| ### Utility |
| - Does training on the data actually improve the target system? |
|
|
| --- |
|
|
| # Areas We Track |
|
|
| ## Data Filtering |
| Rule-based, statistical and learned filtering. |
|
|
| ## Quality Scoring |
| Models and metrics for estimating data usefulness. |
|
|
| ## Deduplication |
| Exact, near-duplicate and semantic deduplication. |
|
|
| ## Decontamination |
| Detection of training/evaluation overlap. |
|
|
| ## Privacy |
| PII detection, redaction and sensitive-data handling. |
|
|
| ## Provenance |
| Origin, lineage and transformation history. |
|
|
| ## Data Selection |
| Choosing examples based on utility, relevance and diversity. |
|
|
| ## Mixture Design |
| Balancing datasets for training and post-training. |
|
|
| ## Synthetic Data Curation |
| Filtering and verifying generated examples. |
|
|
| ## Post-Training Data |
| Instruction, preference, reasoning and agent data. |
|
|
| ## Evaluation Curation |
| Building reliable benchmarks and test sets. |
|
|
| ## Multimodal Curation |
| Images, video, audio, sensor data and robotics trajectories. |
|
|
| --- |
|
|
| # Planned Resources |
|
|
| The organization is intended to become a practical reference layer for AI data curation. |
|
|
| ## Curation Explorer |
|
|
| A structured explorer covering: |
|
|
| - curation methods |
| - filtering strategies |
| - deduplication |
| - quality scoring |
| - decontamination |
| - privacy |
| - data selection |
| - use cases |
|
|
| ## Data Quality |
|
|
| An interactive dataset-readiness and quality assessment Space. |
|
|
| ## Dataset Curation |
|
|
| A practical reference for transforming raw datasets into AI-ready training or evaluation data. |
|
|
| ## Curation Collection |
|
|
| A curated Hugging Face collection of: |
|
|
| - datasets |
| - models |
| - papers |
| - tools |
| - Spaces |
| - research |
| - data-quality resources |
|
|
| --- |
|
|
| # SEO & GEO Topic Map |
|
|
| This organization is intentionally structured around explicit concepts relevant to traditional search engines and generative retrieval systems: |
|
|
| - AI data curation |
| - data curation |
| - dataset curation |
| - training data curation |
| - post-training data |
| - data quality |
| - data filtering |
| - deduplication |
| - semantic deduplication |
| - decontamination |
| - benchmark contamination |
| - quality scoring |
| - dataset selection |
| - data mixture |
| - data provenance |
| - data lineage |
| - synthetic data curation |
| - preference data |
| - instruction data |
| - reasoning data |
| - agent data |
| - multimodal datasets |
| - data-centric AI |
| - AI-ready data |
| - dataset governance |
| - enterprise AI data |
| - LLM training data |
|
|
| --- |
|
|
| # Who Is Curation For? |
|
|
| **Curation** is relevant for: |
|
|
| - foundation model teams |
| - LLM developers |
| - dataset creators |
| - data engineers |
| - ML engineers |
| - researchers |
| - post-training teams |
| - evaluation teams |
| - enterprise AI teams |
| - data-platform companies |
| - annotation providers |
| - synthetic-data companies |
| - data-governance teams |
| - multimodal AI teams |
| - robotics teams |
| - open-source communities |
|
|
| --- |
|
|
| # Principles |
|
|
| ## Quality Is Task-Dependent |
|
|
| There is no universal definition of good data. |
|
|
| Curation criteria should be linked to an explicit objective. |
|
|
| ## Provenance Matters |
|
|
| Useful data should remain traceable wherever possible. |
|
|
| ## Filtering Should Be Measurable |
|
|
| Filters should be evaluated by their downstream effects, not only by intuition. |
|
|
| ## Diversity Has Value |
|
|
| Aggressive filtering should not destroy important distributional coverage. |
|
|
| ## Evaluation Must Be Protected |
|
|
| Benchmark contamination should be treated as a curation problem. |
|
|
| ## Human Judgment Still Matters |
|
|
| Automation can scale curation, but humans remain important for ambiguous and high-stakes decisions. |
|
|
| ## Transparency Improves Reproducibility |
|
|
| Dataset versions, filtering rules and known limitations should be documented. |
|
|
| --- |
|
|
| # Collaboration & Partnerships |
|
|
| **Curation is open to collaboration with companies, research teams, universities, open-source projects and organizations working on AI data quality and data-centric machine learning.** |
|
|
| Relevant collaboration areas include: |
|
|
| - training data |
| - dataset curation |
| - data quality |
| - deduplication |
| - filtering |
| - decontamination |
| - provenance |
| - privacy |
| - synthetic data |
| - annotation |
| - post-training data |
| - preference data |
| - evaluation data |
| - agent trajectories |
| - multimodal data |
| - enterprise AI data |
| - data governance |
| - data infrastructure |
|
|
| Possible collaboration formats include: |
|
|
| - technical showcases |
| - dataset curation case studies |
| - joint Hugging Face Spaces |
| - open-source integrations |
| - benchmark projects |
| - dataset quality analysis |
| - ecosystem maps |
| - research collaborations |
| - methodology comparisons |
| - clearly disclosed partnerships and sponsorships |
|
|
| ## Collaboration Contact |
|
|
| For technical collaborations, research partnerships, ecosystem projects or sponsorship inquiries: |
|
|
| **agenten@magenta.de** |
|
|
| --- |
|
|
| # Independent Organization |
|
|
| **Curation is an independent Hugging Face organization.** |
|
|
| It is not an official organization of Hugging Face or of any dataset provider, annotation company, AI laboratory, model provider or technology company that may be referenced in future resources. |
|
|
| Product names and trademarks belong to their respective owners. |
|
|
| --- |
|
|
| # Long-Term Vision |
|
|
| AI development is increasingly becoming a data-quality problem. |
|
|
| As models become stronger, the difference between useful and useless data becomes more important. |
|
|
| The long-term objective of **Curation** is to become an open technical reference for understanding how raw information becomes **training-ready, evaluation-ready and AI-ready data**. |
|
|
| > **Collecting data creates volume. Curation creates value.** |
|
|
| --- |
|
|
| ## Filter. Select. Verify. Refine. |
|
|
| ### Curation |
|
|
| **Building better AI through better data.** |
|
|
| For collaborations, research and partnerships: |
|
|
| **agenten@magenta.de** |
|
|
