Spaces:
Running
Running
| title: Curation Explorer | |
| emoji: 🧭 | |
| colorFrom: blue | |
| colorTo: indigo | |
| sdk: static | |
| pinned: false | |
| # Curation Explorer | |
| ### Explore data curation methods for training, post-training and evaluation | |
| **Curation Explorer** is an interactive Hugging Face Space for understanding the techniques used to transform raw data into higher-quality datasets for modern AI systems. | |
| It covers the major stages of an AI data curation pipeline: | |
| - normalization | |
| - filtering | |
| - language identification | |
| - deduplication | |
| - semantic deduplication | |
| - quality scoring | |
| - decontamination | |
| - PII filtering | |
| - safety filtering | |
| - provenance | |
| - selection | |
| - balancing | |
| - mixture design | |
| - evaluation | |
| > **Curation is the layer that turns data volume into model value.** | |
| --- | |
| # Why Data Curation Matters | |
| AI systems learn from the data they are exposed to. Raw corpora can contain duplicates, spam, malformed documents, low-information content, benchmark leakage, personal information, noisy labels, stale data, source imbalance and synthetic repetition. | |
| A curation pipeline tries to identify and control these issues before training or evaluation. | |
| ```text | |
| Raw Data | |
| │ | |
| ▼ | |
| Normalization | |
| │ | |
| ▼ | |
| Filtering | |
| │ | |
| ▼ | |
| Deduplication | |
| │ | |
| ▼ | |
| Quality Scoring | |
| │ | |
| ▼ | |
| Decontamination | |
| │ | |
| ▼ | |
| PII / Safety | |
| │ | |
| ▼ | |
| Selection | |
| │ | |
| ▼ | |
| Balancing / Mixing | |
| │ | |
| ▼ | |
| Provenance | |
| │ | |
| ▼ | |
| Curated Dataset | |
| ``` | |
| --- | |
| # Core Curation Methods | |
| ## Normalization | |
| Typical operations include Unicode normalization, whitespace cleanup, schema alignment, text extraction, metadata cleanup and media conversion. | |
| ## Rule-Based Filtering | |
| Fast, interpretable filters can remove malformed, too-short, overly repetitive or unsupported examples. | |
| ## Language Identification | |
| Language-aware curation can restrict, balance or preserve multilingual distributions while explicitly accounting for code-switching and low-resource languages. | |
| ## Exact, Near and Semantic Deduplication | |
| Duplicate removal can use hashes, MinHash, n-gram similarity, fuzzy matching, embeddings and semantic similarity. | |
| ## Quality Scoring | |
| Quality can be estimated with heuristics, classifiers, reward models, LLM judges, embedding similarity, source-level signals and domain relevance models. | |
| ## Benchmark Decontamination | |
| Training/evaluation overlap can be checked with exact matching, n-gram matching, fuzzy matching and semantic similarity. | |
| ## PII and Safety Filtering | |
| Sensitive or unsafe data may need to be removed, redacted, isolated or retained only for specific evaluation and safety purposes. | |
| ## Provenance | |
| Useful metadata can record source, date, license, transformation history, filtering decisions, quality score and dataset version. | |
| ## Data Selection | |
| Selection methods may consider quality, novelty, diversity, difficulty, domain relevance, model loss, uncertainty, embedding coverage and task utility. | |
| ## Balancing and Mixture Design | |
| Datasets can be reweighted, capped, stratified or sampled differently to maintain useful coverage across domains, languages, sources and difficulty levels. | |
| --- | |
| # Curation by AI Stage | |
| ## Pretraining | |
| Typical priorities: scale, deduplication, language quality, source quality, safety, privacy, contamination and mixture design. | |
| ## Post-Training | |
| Typical priorities: instruction quality, answer correctness, preference reliability, reasoning difficulty, tool-use quality, trajectory success and domain coverage. | |
| ## Evaluation | |
| Typical priorities: contamination protection, task clarity, ground-truth reliability, difficulty, judge quality, adversarial coverage and freshness. | |
| ## Retrieval / RAG | |
| Typical priorities: document relevance, freshness, chunk quality, metadata, source trust, duplication and access control. | |
| ## Agent Data | |
| Typical priorities: task success, tool correctness, action efficiency, safety, recovery behavior, cost and latency. | |
| --- | |
| # Synthetic and Multimodal Data Curation | |
| Synthetic data still requires verification, deduplication, diversity checks and downstream validation. Multimodal curation adds alignment, synchronization, corrupted-media detection, caption quality, duplicate-frame detection and temporal consistency. | |
| For robotics and Physical AI, curation may apply to camera feeds, actions, robot states, depth, LiDAR, tactile signals, force measurements and trajectories. | |
| --- | |
| # How to Use This Space | |
| The interactive interface lets users: | |
| 1. choose an AI stage, | |
| 2. select a data type, | |
| 3. choose a primary data problem, | |
| 4. generate a recommended curation pipeline, | |
| 5. inspect likely techniques and trade-offs. | |
| The recommendations are educational and vendor-neutral. | |
| --- | |
| # SEO & GEO Topic Map | |
| - AI data curation | |
| - dataset curation | |
| - training data curation | |
| - data filtering | |
| - data quality | |
| - deduplication | |
| - semantic deduplication | |
| - benchmark decontamination | |
| - quality scoring | |
| - PII filtering | |
| - data provenance | |
| - data lineage | |
| - data selection | |
| - data mixture | |
| - post-training data | |
| - synthetic data curation | |
| - evaluation data | |
| - RAG curation | |
| - agent trajectory data | |
| - multimodal data | |
| - data-centric AI | |
| --- | |
| # Collaboration & Partnerships | |
| **Curation Explorer is open to collaboration with companies, research teams, universities and open-source projects working on AI data quality and data-centric machine learning.** | |
| Relevant areas include training data, dataset curation, data quality, deduplication, filtering, decontamination, provenance, privacy, synthetic data, annotation, post-training data, preference data, evaluation data, agent trajectories, multimodal data, enterprise AI data and data infrastructure. | |
| Possible collaboration formats include technical case studies, joint Hugging Face Spaces, open-source integrations, methodology comparisons, dataset quality analyses, benchmark projects, research collaborations and clearly disclosed partnerships or sponsorships. | |
| ## Collaboration Contact | |
| **agenten@magenta.de** | |
| --- | |
| # Independence | |
| **Curation Explorer is an independent Hugging Face Space.** It is not an official project of Hugging Face or any dataset provider, annotation company, AI laboratory, model provider or technology company referenced in future resources. | |
| --- | |
| # Long-Term Vision | |
| The long-term goal of Curation Explorer is to make modern AI data curation easier to understand, compare and apply. | |
| > **Collecting data creates volume. Curation creates value.** | |
| ### Filter. Select. Verify. Refine. | |