Blame for README.md

---
title: Dataset Curation
emoji: 🧰
colorFrom: blue
colorTo: indigo
sdk: static
pinned: false
---

# Dataset Curation

### An interactive assistant for turning raw data into AI-ready datasets

**Dataset Curation** is an interactive Hugging Face Space that helps users design a practical curation workflow for a specific dataset and AI objective.

Instead of presenting one universal pipeline, the assistant asks about the intended use, modality, scale, data condition and major risks. It then generates a tailored sequence of curation steps with concrete recommendations.

> **The right curation pipeline depends on what the data is for.**

---

# What the Assistant Covers

The assistant can generate guidance for:

- pretraining
- post-training
- fine-tuning
- evaluation
- RAG / retrieval
- agent trajectories
- multimodal datasets
- synthetic data
- code
- robotics and Physical AI data

It can recommend combinations of:

- schema validation
- normalization
- language identification
- heuristic filtering
- exact deduplication
- near-duplicate detection
- semantic deduplication
- quality scoring
- answer verification
- trajectory verification
- PII detection
- privacy filtering
- benchmark decontamination
- source tracking
- licensing review
- provenance
- distribution analysis
- balancing
- mixture design
- human review
- versioning
- downstream validation

---

# Why a Curation Assistant?

A generic checklist is useful, but different datasets fail in different ways.

### Large Web Corpus

```text
Normalize
   ↓
Language Identification
   ↓
Heuristic Filtering
   ↓
Exact + Near Deduplication
   ↓
Quality Scoring
   ↓
PII / Safety Filtering
   ↓
Decontamination
   ↓
Mixture Design
```

### Post-Training Dataset

```text
Validate Instruction
   ↓
Verify Response
   ↓
Deduplicate
   ↓
Difficulty Scoring
   ↓
Preference / Label Validation
   ↓
Task Balancing
   ↓
Human Review
```

### RAG Knowledge Base

```text
Source Validation
   ↓
Freshness Check
   ↓
Deduplicate
   ↓
Document Quality
   ↓
Chunking Review
   ↓
Metadata Enrichment
   ↓
Access / Privacy Check
```

### Agent Trajectories

```text
Validate Trace
   ↓
Check Tool Calls
   ↓
Verify Outcome
   ↓
Score Efficiency
   ↓
Detect Unsafe Actions
   ↓
Balance Tasks
   ↓
Select High-Utility Trajectories
```

---

# Curation Is Objective-Dependent

There is no universal definition of “clean data.”

A filter that improves one use case can damage another.

Examples:

- aggressive language filtering can remove dialect diversity
- aggressive deduplication can remove useful repeated patterns
- safety filtering can remove examples needed for red-team evaluation
- short-document filtering can remove valuable code or factual records
- source-quality filters can encode hidden bias

The assistant therefore frames recommendations around the selected objective.

---

# AI Objectives

## Pretraining

Focus on:

- scale
- diversity
- deduplication
- source quality
- language balance
- privacy
- contamination
- mixture design

## Post-Training

Focus on:

- instruction clarity
- response correctness
- preference quality
- reasoning difficulty
- domain relevance
- tool-use quality
- task diversity

## Evaluation

Focus on:

- clean ground truth
- contamination control
- representative difficulty
- judge reliability
- coverage
- freshness

## RAG

Focus on:

- source relevance
- freshness
- chunk quality
- metadata
- provenance
- duplication
- permissions

## Agent Data

Focus on:

- task completion
- action correctness
- tool use
- recovery behavior
- efficiency
- safety

---

# Data-Type Specific Curation

## Text

Typical concerns:

- encoding
- boilerplate
- language
- quality
- duplication
- PII
- contamination

## Code

Additional concerns:

- parseability
- compilation
- tests
- licenses
- vendored code
- generated files
- repository-level duplication

## Synthetic Data

Additional concerns:

- generator artifacts
- mode collapse
- factual errors
- low difficulty
- self-reinforcing patterns
- insufficient diversity

## Multimodal Data

Additional concerns:

- modality alignment
- corrupted media
- weak captions
- timestamp mismatch
- missing modalities

## Trajectories

Additional concerns:

- invalid actions
- failed tasks
- missing states
- inefficient behavior
- incorrect tool calls
- unsafe behavior

---

# Reference Architecture

```text
Raw Dataset
     │
     ▼
Structural Validation
     │
     ▼
Normalization
     │
     ▼
Filtering
     │
     ▼
Deduplication
     │
     ▼
Quality / Verification
     │
     ▼
Privacy / Safety
     │
     ▼
Decontamination
     │
     ▼
Selection / Balancing
     │
     ▼
Provenance / Versioning
     │
     ▼
Curated Dataset
```

---

# Human Review

Human review is especially useful for:

- ambiguous quality decisions
- specialist domains
- preference data
- reasoning traces
- safety-sensitive examples
- uncertain licensing
- high-impact evaluation items

The assistant treats human review as a targeted intervention rather than a blanket requirement.

---

# Curation at Scale

Large datasets need scalable processing.

```text
Object Storage
     │
     ▼
Distributed Processing
     │
     ├── Parse
     ├── Normalize
     ├── Filter
     ├── Score
     ├── Deduplicate
     └── Enrich Metadata
             │
             ▼
       Versioned Dataset
```

Important operational concerns:

- throughput
- storage
- checkpointing
- deterministic processing
- cost
- sharding
- observability
- reproducibility

---

# Provenance and Lineage

Useful metadata includes:

- source
- source ID
- collection date
- license
- transformations
- filter outcomes
- quality scores
- dataset version

This supports:

- auditing
- reproducibility
- deletion
- licensing review
- contamination analysis

---

# Downstream Validation

A curation pipeline should not be judged only by how much data it removes.

The important question is:

> **Does the curated dataset improve the target AI system?**

Useful validation can include:

- training loss
- benchmark performance
- robustness
- memorization
- domain performance
- task success
- retrieval quality
- human evaluation
- safety metrics
- cost efficiency

---

# Interactive Assistant

The included `index.html` asks for:

- AI objective
- data modality
- dataset scale
- duplicate level
- quality inconsistency
- provenance gaps
- privacy risk
- contamination risk
- distribution imbalance
- human-review availability

It then produces:

- a recommended ordered pipeline
- priority labels
- concrete actions
- key risks
- validation recommendations
- a concise implementation plan

The interface is educational and vendor-neutral.

---

# SEO & GEO Topic Map

This Space is structured around:

- dataset curation
- AI data curation
- data curation pipeline
- AI-ready data
- training data curation
- post-training data
- dataset quality
- deduplication
- semantic deduplication
- quality scoring
- benchmark decontamination
- data provenance
- data lineage
- PII filtering
- synthetic data curation
- RAG data curation
- agent trajectory curation
- multimodal curation
- data selection
- mixture design
- dataset readiness
- data-centric AI

---

# Collaboration & Partnerships

**Dataset Curation is open to collaboration with companies, research teams, universities and open-source projects working on AI data infrastructure and data quality.**

Relevant areas include:

- dataset curation
- training data
- data quality
- filtering
- deduplication
- decontamination
- provenance
- privacy
- synthetic data
- annotation
- post-training
- evaluation
- RAG
- agent trajectories
- multimodal data
- enterprise AI data
- data governance

Possible collaboration formats include:

- technical case studies
- curation methodology comparisons
- joint Hugging Face Spaces
- open-source integrations
- benchmark projects
- dataset quality studies
- research collaborations
- clearly disclosed partnerships and sponsorships

## Collaboration Contact

**agenten@magenta.de**

---

# Independence

**Dataset Curation is an independent Hugging Face Space.**

It is not an official project of Hugging Face or any dataset provider, annotation company, AI laboratory, model provider or technology company that may be referenced in future resources.

---

# Long-Term Vision

The long-term goal of Dataset Curation is to make the path from raw data to useful AI data more explicit, measurable and reproducible.

> **Raw data is an input. Curated data is an engineered asset.**

### Diagnose. Curate. Validate. Improve.