Blame for README.md

---
title: DataInventor
emoji: 🧪
colorFrom: indigo
colorTo: pink
sdk: docker
app_port: 7860
pinned: false
hf_oauth: true
hf_oauth_scopes:
  - contribute-repos
hf_oauth_expiration_minutes: 120
short_description: Invent, curate, chart and expand training data
---

# DataInventor

DataInventor has four modes.

- **Invent rows.** An open-source recreation of the [Invent a Dataset](https://adaptionlabs.ai/invent-a-dataset) workflow. Describe the data you want, choose a format, domains and languages, preview a blueprint, and get an instruction or preference dataset back as CSV or JSONL.
- **Curate a corpus.** Clean, score, rephrase and mix the pretraining data you already have. Preview every step here on a sample, then download the recipe and run it on all of your data, on your own machines.
- **Expand.** Grow a dataset you own, from as few as ten rows, into a larger one that keeps its task, format and language: more rows like yours, related tasks from your domain, or a planned mix from outside it. The cost is shown before you run.
- **Visualize.** Ask a question about a dataset in plain words and get a chart back, with the rows behind every bar, box or dot one click away. Use the rows you just wrote, a sample, any public Hugging Face dataset, or a CSV/JSONL file.

It works with any OpenAI-compatible endpoint. The default is a free model served through Hugging Face Inference Providers.

## How it writes rows

1. **Blueprint.** The model turns your description into topics, personas, who answers and the shapes questions take.
2. **Sampling.** Unique combinations of topic, persona, shape, difficulty, length and language.
3. **Writing.** One message per combination, then the reply (preference pairs get two replies and a ranking).
4. **Filters.** Cut-off or degenerate text, requests for many items at once, and near-duplicate questions are rejected.

## How curation works

You register Hugging Face datasets as **sources** to curate, and optionally describe or add **targets** that show what the trained model should be good at. Then four steps run in order.

1. **Clean.** Gopher and C4 style heuristics (FineWeb thresholds) drop broken, boilerplate and repetitive documents, with looser rules for math and code. Any document that shares a 13-word run with a test set you name (GSM8K, MATH-500, HumanEval, MMLU, ARC and others, bundled with the package) or with a held-out set of your own is removed.
2. **Calibrate.** Exact and MinHash near-duplicates are dropped. A model grades a sample for teaching value, clarity, accuracy and density, and for relevance to your targets. A small linear scorer learns those grades and scores every document on CPU. Each source keeps its best share, ranked by a blend of quality and relevance that you set.
3. **Create.** Where a source is too small for its share of a mix, its best documents are rephrased into new ones (dialogue, tutorial, questions and answers, dense notes, textbook section, worked problems, code walkthrough). These are the styles from our [Sutra pedagogical pretraining work](https://huggingface.co/blog/codelion/scaling-pedagogical-pretraining-10-billion-tokens). Rewrites that add numbers or names the source lacks are dropped. You can also write new documents on a subject.
4. **Compose.** One prepared corpus becomes one or more mixes, each fitted to a token budget. Short sources repeat up to a cap, then get topped up with rephrased text up to a synthetic cap. The default shares come from our [1B-token mixing study](https://huggingface.co/blog/codelion/optimal-dataset-mixing): 50% textbook-style, 30% web, 20% educational web. That result was measured at 70M parameters, so treat it as a starting point.

The hosted run works on about 200 documents per source and returns the kept and created documents with a report. The runner does the same on every document:

```bash
pip install 'datainventor[full] @ https://huggingface.co/spaces/codelion/data-inventor/resolve/main/dist/datainventor-0.3.0-py3-none-any.whl'
export DI_API_KEY=...                      # key for the endpoint named in the recipe
datainventor init recipe.yaml              # or download one from the app
datainventor preview recipe.yaml           # every step on a sample
datainventor run recipe.yaml --out ./curated --procs 32 --workers 64
```

It writes `curated/<source>/`, `synthetic/<source>/` and `mixes/<mix>/` as JSONL shards, plus `REPORT.md`. Interrupted runs resume from the last finished stage or source. The endpoint is any OpenAI-compatible server: vLLM, SGLang, Ollama or a hosted API.

Two limits to know. Token counts are estimates at four characters per token. The learned scorer is a hashed n-gram model. In our test it reached a rank agreement of about 0.5 with the grading model after 300 grades, so grade more documents, or set `calibrate.scorer: llm` to grade every document with the model.

## How Visualize works

It is built for small datasets first: the 25 to 200 rows a hosted run writes. Every row stays in memory, so each chart is an exact count over all of them.

- **At a glance.** As soon as data loads you see two or three charts that need no model: rows per category, and how long the texts are in each.
- **Ask.** A model turns your question into a chart plan. If an existing column answers it (area, language, length), the chart comes back in a couple of seconds. If it needs something no column has, such as tone, intent or whether a reply gives steps, the model reads the rows in small parallel batches and fills a new column. A regex handles literal marks like links or numbered lists without a model.
- **Drill down.** Click a bar, box, dot or legend entry to read the rows behind it, download them, or ask a follow-up about just those rows.
- **Reuse.** Columns the model made stay with the dataset, so follow-up questions reuse them instead of reading every row again.
- **Honest numbers.** The write-up uses only the chart's exact numbers, says when a group is too small to read much into, and the page notes which columns a model assigned.

## How Expand works

1. **Seed.** At least ten rows: a Hugging Face dataset, a CSV/JSONL file, or the rows you just wrote. Up to 200 are used here.
2. **Read it.** The columns are measured without a model: types, which ones are ids, short categories or text, the values small categories take, lengths, the writing script, and rules every row obeys (a count column that counts a JSON column's entries, or two JSON columns with the same keys). One model call then names the task, the input and output fields, the topics and the faults in the seed that new rows should not copy. You can edit all of it.
3. **Grow.** Pick one way or mix them with shares:
   - **More data like yours**: the seed's topics, weighted by how often they occur. Each row is written from three of your rows, rotated so every one is used.
   - **Related tasks from your domain**: the model proposes six other tasks from the same field that fit your fields.
   - **Outside your domain**: the model plans a mix of outside topics with stated shares and a reason for each. It writes them; it does not fetch outside sources.
4. **Estimate, then run.** Calls, tokens, dollars and minutes are shown before Run, from your seed's measured lengths and the engine's price. Each row takes two calls: the inputs, then the outputs written from that input and full example rows.
5. **Checks.** A row is kept only if it has the seed's fields and types, keeps the cross-column rules and (in "like yours") the small closed categories such as `language`, passes the Invent text filters, is in the seed's script, is not a near-copy of a seed row or another new row (word 3-gram MinHash at about 0.5 Jaccard), and shares no 13-word run with the bundled benchmarks. Every rejection is counted by reason.

Downloads add two columns, `expand_mode` and `expand_topic`. The hosted run writes up to 200 rows. For more, download the recipe and run it anywhere:

```bash
datainventor expand plan seed.jsonl --mode like=0.7 --mode related=0.3 --rows 100000 -o expand.json
datainventor expand run expand.json --out expanded.jsonl --workers 64 --max-usd 20   # resumable
```

The checks catch format faults, not wrong facts. Read a sample before you train on it.

## How it compares

| | [Invent a Dataset](https://adaptionlabs.ai/invent-a-dataset) (Adaption) | [Curation Studio](https://www.datologyai.com/) (DatologyAI) | DataInventor |
|---|---|---|---|
| Job | Training rows from a plain-language goal | Curating pretraining data | Both |
| Access | Hosted | Enterprise, deployed in your cloud | Open source: this Space plus a CLI |
| Model endpoint | Theirs | Theirs | Any OpenAI-compatible endpoint |
| Published quality evaluation | Yes, their own | Yes, their own | Not yet |

## Run the app yourself

```bash
pip install -r requirements.txt
cd app && DI_API_KEY=... uvicorn main:app --port 7860
```

The model endpoint is fixed by the host, so visitors never enter a key or a URL. Configure it with environment variables:

| Variable | Default | What it does |
|---|---|---|
| `DI_API_KEY` | `HF_TOKEN` | Key for the endpoint |
| `DI_BASE_URL` | `https://router.huggingface.co/v1` | Any OpenAI-compatible base URL |
| `DI_MODEL` | `openai/gpt-oss-20b:deepinfra` | Model name |
| `DI_DISABLE_THINKING` | `0` | Set `1` to turn thinking off for reasoning models. The default model does not think, so the setting is off |
| `DI_WORKERS` | `4` | Parallel calls (1 to 8) |
| `MAX_ROWS` | `200` | Rows per run |
| `MAX_RUNNING` | `2` | Runs at once |
| `EXPAND_MAX_ROWS` | `200` | New rows per Expand run |
| `EXPAND_MAX_SEED` | `200` | Seed rows Expand reads |
| `DI_PRICE_IN`, `DI_PRICE_OUT` | known for the default model | USD per million tokens, for the cost estimate |

To use your own model, clone the Space and set `DI_BASE_URL`, `DI_MODEL` and `DI_API_KEY` as Space secrets and variables.

Not affiliated with Adaption Labs.