Spaces:

curation
/

dataset-curation

Running

App Files Files Community

dataset-curation / README.md

Agenten

Upload 2 files

a0ae180 verified about 1 month ago

preview code

Raw History Blame Contribute Delete

8.65 kB

---
title: Dataset Curation
emoji: 🧰
colorFrom: blue
colorTo: indigo
sdk: static
pinned: false
---
# Dataset Curation

### An interactive assistant for turning raw data into AI-ready datasets

**Dataset Curation** is an interactive Hugging Face Space that helps users design a practical curation workflow for a specific dataset and AI objective.

Instead of presenting one universal pipeline, the assistant asks about the intended use, modality, scale, data condition and major risks. It then generates a tailored sequence of curation steps with concrete recommendations.

> **The right curation pipeline depends on what the data is for.**

---

# What the Assistant Covers

The assistant can generate guidance for:

- pretraining
- post-training
- fine-tuning
- evaluation
- RAG / retrieval
- agent trajectories
- multimodal datasets
- synthetic data
- code
- robotics and Physical AI data

It can recommend combinations of:

- schema validation
- normalization
- language identification
- heuristic filtering
- exact deduplication
- near-duplicate detection
- semantic deduplication
- quality scoring
- answer verification
- trajectory verification
- PII detection
- privacy filtering
- benchmark decontamination
- source tracking
- licensing review
- provenance
- distribution analysis
- balancing
- mixture design
- human review
- versioning
- downstream validation

---

# Why a Curation Assistant?

A generic checklist is useful, but different datasets fail in different ways.

### Large Web Corpus

```text
Normalize
   ↓
Language Identification
   ↓
Heuristic Filtering
   ↓
Exact + Near Deduplication
   ↓
Quality Scoring
   ↓
PII / Safety Filtering
   ↓
Decontamination
   ↓
Mixture Design
```

### Post-Training Dataset

```text
Validate Instruction
   ↓
Verify Response
   ↓
Deduplicate
   ↓
Difficulty Scoring
   ↓
Preference / Label Validation
   ↓
Task Balancing
   ↓
Human Review
```

### RAG Knowledge Base

```text
Source Validation
   ↓
Freshness Check
   ↓
Deduplicate
   ↓
Document Quality
   ↓
Chunking Review
   ↓
Metadata Enrichment
   ↓
Access / Privacy Check
```

### Agent Trajectories

```text
Validate Trace
   ↓
Check Tool Calls
   ↓
Verify Outcome
   ↓
Score Efficiency
   ↓
Detect Unsafe Actions
   ↓
Balance Tasks
   ↓
Select High-Utility Trajectories
```

---

# Curation Is Objective-Dependent

There is no universal definition of “clean data.”

A filter that improves one use case can damage another.

Examples:

- aggressive language filtering can remove dialect diversity
- aggressive deduplication can remove useful repeated patterns
- safety filtering can remove examples needed for red-team evaluation
- short-document filtering can remove valuable code or factual records
- source-quality filters can encode hidden bias

The assistant therefore frames recommendations around the selected objective.

---

# AI Objectives

## Pretraining

Focus on:

- scale
- diversity
- deduplication
- source quality
- language balance
- privacy
- contamination
- mixture design

## Post-Training

Focus on:

- instruction clarity
- response correctness
- preference quality
- reasoning difficulty
- domain relevance
- tool-use quality
- task diversity

## Evaluation

Focus on:

- clean ground truth
- contamination control
- representative difficulty
- judge reliability
- coverage
- freshness

## RAG

Focus on:

- source relevance
- freshness
- chunk quality
- metadata
- provenance
- duplication
- permissions

## Agent Data

Focus on:

- task completion
- action correctness
- tool use
- recovery behavior
- efficiency
- safety

---

# Data-Type Specific Curation

## Text

Typical concerns:

- encoding
- boilerplate
- language
- quality
- duplication
- PII
- contamination

## Code

Additional concerns:

- parseability
- compilation
- tests
- licenses
- vendored code
- generated files
- repository-level duplication

## Synthetic Data

Additional concerns:

- generator artifacts
- mode collapse
- factual errors
- low difficulty
- self-reinforcing patterns
- insufficient diversity

## Multimodal Data

Additional concerns:

- modality alignment
- corrupted media
- weak captions
- timestamp mismatch
- missing modalities

## Trajectories

Additional concerns:

- invalid actions
- failed tasks
- missing states
- inefficient behavior
- incorrect tool calls
- unsafe behavior

---

# Reference Architecture

```text
Raw Dataset
     │
     ▼
Structural Validation
     │
     ▼
Normalization
     │
     ▼
Filtering
     │
     ▼
Deduplication
     │
     ▼
Quality / Verification
     │
     ▼
Privacy / Safety
     │
     ▼
Decontamination
     │
     ▼
Selection / Balancing
     │
     ▼
Provenance / Versioning
     │
     ▼
Curated Dataset
```

---

# Human Review

Human review is especially useful for:

- ambiguous quality decisions
- specialist domains
- preference data
- reasoning traces
- safety-sensitive examples
- uncertain licensing
- high-impact evaluation items

The assistant treats human review as a targeted intervention rather than a blanket requirement.

---

# Curation at Scale

Large datasets need scalable processing.

```text
Object Storage
     │
     ▼
Distributed Processing
     │
     ├── Parse
     ├── Normalize
     ├── Filter
     ├── Score
     ├── Deduplicate
     └── Enrich Metadata
             │
             ▼
       Versioned Dataset
```

Important operational concerns:

- throughput
- storage
- checkpointing
- deterministic processing
- cost
- sharding
- observability
- reproducibility

---

# Provenance and Lineage

Useful metadata includes:

- source
- source ID
- collection date
- license
- transformations
- filter outcomes
- quality scores
- dataset version

This supports:

- auditing
- reproducibility
- deletion
- licensing review
- contamination analysis

---

# Downstream Validation

A curation pipeline should not be judged only by how much data it removes.

The important question is:

> **Does the curated dataset improve the target AI system?**

Useful validation can include:

- training loss
- benchmark performance
- robustness
- memorization
- domain performance
- task success
- retrieval quality
- human evaluation
- safety metrics
- cost efficiency

---

# Interactive Assistant

The included `index.html` asks for:

- AI objective
- data modality
- dataset scale
- duplicate level
- quality inconsistency
- provenance gaps
- privacy risk
- contamination risk
- distribution imbalance
- human-review availability

It then produces:

- a recommended ordered pipeline
- priority labels
- concrete actions
- key risks
- validation recommendations
- a concise implementation plan

The interface is educational and vendor-neutral.

---

# SEO & GEO Topic Map

This Space is structured around:

- dataset curation
- AI data curation
- data curation pipeline
- AI-ready data
- training data curation
- post-training data
- dataset quality
- deduplication
- semantic deduplication
- quality scoring
- benchmark decontamination
- data provenance
- data lineage
- PII filtering
- synthetic data curation
- RAG data curation
- agent trajectory curation
- multimodal curation
- data selection
- mixture design
- dataset readiness
- data-centric AI

---

# Collaboration & Partnerships

**Dataset Curation is open to collaboration with companies, research teams, universities and open-source projects working on AI data infrastructure and data quality.**

Relevant areas include:

- dataset curation
- training data
- data quality
- filtering
- deduplication
- decontamination
- provenance
- privacy
- synthetic data
- annotation
- post-training
- evaluation
- RAG
- agent trajectories
- multimodal data
- enterprise AI data
- data governance

Possible collaboration formats include:

- technical case studies
- curation methodology comparisons
- joint Hugging Face Spaces
- open-source integrations
- benchmark projects
- dataset quality studies
- research collaborations
- clearly disclosed partnerships and sponsorships

## Collaboration Contact

**agenten@magenta.de**

---

# Independence

**Dataset Curation is an independent Hugging Face Space.**

It is not an official project of Hugging Face or any dataset provider, annotation company, AI laboratory, model provider or technology company that may be referenced in future resources.

---

# Long-Term Vision

The long-term goal of Dataset Curation is to make the path from raw data to useful AI data more explicit, measurable and reproducible.

> **Raw data is an input. Curated data is an engineered asset.**

### Diagnose. Curate. Validate. Improve.

	---
	title: Dataset Curation
	emoji: 🧰
	colorFrom: blue
	colorTo: indigo
	sdk: static
	pinned: false
	---

	# Dataset Curation

	### An interactive assistant for turning raw data into AI-ready datasets

	Dataset Curation is an interactive Hugging Face Space that helps users design a practical curation workflow for a specific dataset and AI objective.

	Instead of presenting one universal pipeline, the assistant asks about the intended use, modality, scale, data condition and major risks. It then generates a tailored sequence of curation steps with concrete recommendations.

	> The right curation pipeline depends on what the data is for.

	---

	# What the Assistant Covers

	The assistant can generate guidance for:

	- pretraining
	- post-training
	- fine-tuning
	- evaluation
	- RAG / retrieval
	- agent trajectories
	- multimodal datasets
	- synthetic data
	- code
	- robotics and Physical AI data

	It can recommend combinations of:

	- schema validation
	- normalization
	- language identification
	- heuristic filtering
	- exact deduplication
	- near-duplicate detection
	- semantic deduplication
	- quality scoring
	- answer verification
	- trajectory verification
	- PII detection
	- privacy filtering
	- benchmark decontamination
	- source tracking
	- licensing review
	- provenance
	- distribution analysis
	- balancing
	- mixture design
	- human review
	- versioning
	- downstream validation

	---

	# Why a Curation Assistant?

	A generic checklist is useful, but different datasets fail in different ways.

	### Large Web Corpus

	```text
	Normalize
	↓
	Language Identification
	↓
	Heuristic Filtering
	↓
	Exact + Near Deduplication
	↓
	Quality Scoring
	↓
	PII / Safety Filtering
	↓
	Decontamination
	↓
	Mixture Design
	```

	### Post-Training Dataset

	```text
	Validate Instruction
	↓
	Verify Response
	↓
	Deduplicate
	↓
	Difficulty Scoring
	↓
	Preference / Label Validation
	↓
	Task Balancing
	↓
	Human Review
	```

	### RAG Knowledge Base

	```text
	Source Validation
	↓
	Freshness Check
	↓
	Deduplicate
	↓
	Document Quality
	↓
	Chunking Review
	↓
	Metadata Enrichment
	↓
	Access / Privacy Check
	```

	### Agent Trajectories

	```text
	Validate Trace
	↓
	Check Tool Calls
	↓
	Verify Outcome
	↓
	Score Efficiency
	↓
	Detect Unsafe Actions
	↓
	Balance Tasks
	↓
	Select High-Utility Trajectories
	```

	---

	# Curation Is Objective-Dependent

	There is no universal definition of “clean data.”

	A filter that improves one use case can damage another.

	Examples:

	- aggressive language filtering can remove dialect diversity
	- aggressive deduplication can remove useful repeated patterns
	- safety filtering can remove examples needed for red-team evaluation
	- short-document filtering can remove valuable code or factual records
	- source-quality filters can encode hidden bias

	The assistant therefore frames recommendations around the selected objective.

	---

	# AI Objectives

	## Pretraining

	Focus on:

	- scale
	- diversity
	- deduplication
	- source quality
	- language balance
	- privacy
	- contamination
	- mixture design

	## Post-Training

	Focus on:

	- instruction clarity
	- response correctness
	- preference quality
	- reasoning difficulty
	- domain relevance
	- tool-use quality
	- task diversity

	## Evaluation

	Focus on:

	- clean ground truth
	- contamination control
	- representative difficulty
	- judge reliability
	- coverage
	- freshness

	## RAG

	Focus on:

	- source relevance
	- freshness
	- chunk quality
	- metadata
	- provenance
	- duplication
	- permissions

	## Agent Data

	Focus on:

	- task completion
	- action correctness
	- tool use
	- recovery behavior
	- efficiency
	- safety

	---

	# Data-Type Specific Curation

	## Text

	Typical concerns:

	- encoding
	- boilerplate
	- language
	- quality
	- duplication
	- PII
	- contamination

	## Code

	Additional concerns:

	- parseability
	- compilation
	- tests
	- licenses
	- vendored code
	- generated files
	- repository-level duplication

	## Synthetic Data

	Additional concerns:

	- generator artifacts
	- mode collapse
	- factual errors
	- low difficulty
	- self-reinforcing patterns
	- insufficient diversity

	## Multimodal Data

	Additional concerns:

	- modality alignment
	- corrupted media
	- weak captions
	- timestamp mismatch
	- missing modalities

	## Trajectories

	Additional concerns:

	- invalid actions
	- failed tasks
	- missing states
	- inefficient behavior
	- incorrect tool calls
	- unsafe behavior

	---

	# Reference Architecture

	```text
	Raw Dataset
	│
	▼
	Structural Validation
	│
	▼
	Normalization
	│
	▼
	Filtering
	│
	▼
	Deduplication
	│
	▼
	Quality / Verification
	│
	▼
	Privacy / Safety
	│
	▼
	Decontamination
	│
	▼
	Selection / Balancing
	│
	▼
	Provenance / Versioning
	│
	▼
	Curated Dataset
	```

	---

	# Human Review

	Human review is especially useful for:

	- ambiguous quality decisions
	- specialist domains
	- preference data
	- reasoning traces
	- safety-sensitive examples
	- uncertain licensing
	- high-impact evaluation items

	The assistant treats human review as a targeted intervention rather than a blanket requirement.

	---

	# Curation at Scale

	Large datasets need scalable processing.

	```text
	Object Storage
	│
	▼
	Distributed Processing
	│
	├── Parse
	├── Normalize
	├── Filter
	├── Score
	├── Deduplicate
	└── Enrich Metadata
	│
	▼
	Versioned Dataset
	```

	Important operational concerns:

	- throughput
	- storage
	- checkpointing
	- deterministic processing
	- cost
	- sharding
	- observability
	- reproducibility

	---

	# Provenance and Lineage

	Useful metadata includes:

	- source
	- source ID
	- collection date
	- license
	- transformations
	- filter outcomes
	- quality scores
	- dataset version

	This supports:

	- auditing
	- reproducibility
	- deletion
	- licensing review
	- contamination analysis

	---

	# Downstream Validation

	A curation pipeline should not be judged only by how much data it removes.

	The important question is:

	> Does the curated dataset improve the target AI system?

	Useful validation can include:

	- training loss
	- benchmark performance
	- robustness
	- memorization
	- domain performance
	- task success
	- retrieval quality
	- human evaluation
	- safety metrics
	- cost efficiency

	---

	# Interactive Assistant

	The included `index.html` asks for:

	- AI objective
	- data modality
	- dataset scale
	- duplicate level
	- quality inconsistency
	- provenance gaps
	- privacy risk
	- contamination risk
	- distribution imbalance
	- human-review availability

	It then produces:

	- a recommended ordered pipeline
	- priority labels
	- concrete actions
	- key risks
	- validation recommendations
	- a concise implementation plan

	The interface is educational and vendor-neutral.

	---

	# SEO & GEO Topic Map

	This Space is structured around:

	- dataset curation
	- AI data curation
	- data curation pipeline
	- AI-ready data
	- training data curation
	- post-training data
	- dataset quality
	- deduplication
	- semantic deduplication
	- quality scoring
	- benchmark decontamination
	- data provenance
	- data lineage
	- PII filtering
	- synthetic data curation
	- RAG data curation
	- agent trajectory curation
	- multimodal curation
	- data selection
	- mixture design
	- dataset readiness
	- data-centric AI

	---

	# Collaboration & Partnerships

	Dataset Curation is open to collaboration with companies, research teams, universities and open-source projects working on AI data infrastructure and data quality.

	Relevant areas include:

	- dataset curation
	- training data
	- data quality
	- filtering
	- deduplication
	- decontamination
	- provenance
	- privacy
	- synthetic data
	- annotation
	- post-training
	- evaluation
	- RAG
	- agent trajectories
	- multimodal data
	- enterprise AI data
	- data governance

	Possible collaboration formats include:

	- technical case studies
	- curation methodology comparisons
	- joint Hugging Face Spaces
	- open-source integrations
	- benchmark projects
	- dataset quality studies
	- research collaborations
	- clearly disclosed partnerships and sponsorships

	## Collaboration Contact

	agenten@magenta.de

	---

	# Independence

	Dataset Curation is an independent Hugging Face Space.

	It is not an official project of Hugging Face or any dataset provider, annotation company, AI laboratory, model provider or technology company that may be referenced in future resources.

	---

	# Long-Term Vision

	The long-term goal of Dataset Curation is to make the path from raw data to useful AI data more explicit, measurable and reproducible.

	> Raw data is an input. Curated data is an engineered asset.

	### Diagnose. Curate. Validate. Improve.