codelion commited on
Commit
4f740eb
·
verified ·
1 Parent(s): 721a7d3

Curate a corpus: clean, calibrate, create, compose on a sample, plus a recipe and runner for your own machines

Browse files
Dockerfile CHANGED
@@ -6,6 +6,7 @@ ENV PATH="/home/user/.local/bin:$PATH"
6
  WORKDIR /home/user/app
7
  COPY --chown=user requirements.txt .
8
  RUN pip install --no-cache-dir -r requirements.txt
 
9
  COPY --chown=user app ./app
10
  COPY --chown=user static ./static
11
  WORKDIR /home/user/app/app
 
6
  WORKDIR /home/user/app
7
  COPY --chown=user requirements.txt .
8
  RUN pip install --no-cache-dir -r requirements.txt
9
+ COPY --chown=user datainventor ./datainventor
10
  COPY --chown=user app ./app
11
  COPY --chown=user static ./static
12
  WORKDIR /home/user/app/app
README.md CHANGED
@@ -10,12 +10,15 @@ hf_oauth: true
10
  hf_oauth_scopes:
11
  - contribute-repos
12
  hf_oauth_expiration_minutes: 120
13
- short_description: Describe a dataset in plain words, get training rows back.
14
  ---
15
 
16
  # DataInventor
17
 
18
- An open-source recreation of the [Invent a Dataset](https://adaptionlabs.ai/invent-a-dataset) workflow. Describe the data you want, choose a format, domains and languages, preview a blueprint, and get an instruction or preference dataset back as CSV or JSONL.
 
 
 
19
 
20
  It works with any OpenAI-compatible endpoint. The default is a free model served through Hugging Face Inference Providers.
21
 
@@ -26,7 +29,30 @@ It works with any OpenAI-compatible endpoint. The default is a free model served
26
  3. **Writing.** One message per combination, then the reply (preference pairs get two replies and a ranking).
27
  4. **Filters.** Cut-off or degenerate text, requests for many items at once, and near-duplicate questions are rejected.
28
 
29
- ## Run it yourself
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30
 
31
  ```bash
32
  pip install -r requirements.txt
 
10
  hf_oauth_scopes:
11
  - contribute-repos
12
  hf_oauth_expiration_minutes: 120
13
+ short_description: Invent training rows, or curate your pretraining data
14
  ---
15
 
16
  # DataInventor
17
 
18
+ DataInventor has two modes.
19
+
20
+ - **Invent rows.** An open-source recreation of the [Invent a Dataset](https://adaptionlabs.ai/invent-a-dataset) workflow. Describe the data you want, choose a format, domains and languages, preview a blueprint, and get an instruction or preference dataset back as CSV or JSONL.
21
+ - **Curate a corpus.** Clean, score, rephrase and mix the pretraining data you already have. Preview every step here on a sample, then download the recipe and run it on all of your data, on your own machines.
22
 
23
  It works with any OpenAI-compatible endpoint. The default is a free model served through Hugging Face Inference Providers.
24
 
 
29
  3. **Writing.** One message per combination, then the reply (preference pairs get two replies and a ranking).
30
  4. **Filters.** Cut-off or degenerate text, requests for many items at once, and near-duplicate questions are rejected.
31
 
32
+ ## How curation works
33
+
34
+ You register Hugging Face datasets as **sources** to curate, and optionally describe or add **targets** that show what the trained model should be good at. Then four steps run in order.
35
+
36
+ 1. **Clean.** Gopher and C4 style heuristics (FineWeb thresholds) drop broken, boilerplate and repetitive documents, with looser rules for math and code. Any document that shares a 13-word run with a test set you name (GSM8K, MATH-500, HumanEval, MMLU, ARC and others, bundled with the package) or with a held-out set of your own is removed.
37
+ 2. **Calibrate.** Exact and MinHash near-duplicates are dropped. A model grades a sample for teaching value, clarity, accuracy and density, and for relevance to your targets. A small linear scorer learns those grades and scores every document on CPU. Each source keeps its best share, ranked by a blend of quality and relevance that you set.
38
+ 3. **Create.** Where a source is too small for its share of a mix, its best documents are rephrased into new ones (dialogue, tutorial, questions and answers, dense notes, textbook section, worked problems, code walkthrough). These are the styles from the Sutra datasets. Rewrites that add numbers or names the source lacks are dropped. You can also write new documents on a subject.
39
+ 4. **Compose.** One prepared corpus becomes one or more mixes, each fitted to a token budget. Short sources repeat up to a cap, then get topped up with rephrased text up to a synthetic cap. The default shares come from our [1B-token mixing study](https://huggingface.co/blog/codelion/optimal-dataset-mixing): 50% textbook-style, 30% web, 20% educational web. That result was measured at 70M parameters, so treat it as a starting point.
40
+
41
+ The hosted run works on about 200 documents per source and returns the kept and created documents with a report. The runner does the same on every document:
42
+
43
+ ```bash
44
+ pip install 'datainventor[full] @ git+https://huggingface.co/spaces/codelion/data-inventor'
45
+ export DI_API_KEY=... # key for the endpoint named in the recipe
46
+ datainventor init recipe.yaml # or download one from the app
47
+ datainventor preview recipe.yaml # every step on a sample
48
+ datainventor run recipe.yaml --out ./curated --procs 32 --workers 64
49
+ ```
50
+
51
+ It writes `curated/<source>/`, `synthetic/<source>/` and `mixes/<mix>/` as JSONL shards, plus `REPORT.md`. Interrupted runs resume from the last finished stage or source. The endpoint is any OpenAI-compatible server: vLLM, SGLang, Ollama or a hosted API.
52
+
53
+ Two limits to know. Token counts are estimates at four characters per token. The learned scorer is a hashed n-gram model. In our test it reached a rank agreement of about 0.5 with the grading model after 300 grades, so grade more documents, or set `calibrate.scorer: llm` to grade every document with the model.
54
+
55
+ ## Run the app yourself
56
 
57
  ```bash
58
  pip install -r requirements.txt
app/curate_api.py ADDED
@@ -0,0 +1,166 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Curate a corpus: sample previews of the clean, calibrate, create, compose workflow, plus the recipe to run it in full."""
2
+ from __future__ import annotations
3
+
4
+ import json
5
+ import os
6
+ import threading
7
+ import time
8
+ import uuid
9
+
10
+ from fastapi import APIRouter, HTTPException
11
+ from fastapi.responses import Response
12
+ from pydantic import BaseModel
13
+
14
+ from datainventor.clean import BENCHMARKS, DEFAULTS as CLEAN_RULES
15
+ from datainventor.create import DOC_TYPES, STYLES, chunks, rephrase, styles_for
16
+ from datainventor.llm import EndpointError
17
+ from datainventor.preview import Preview, run_preview
18
+ from datainventor.recipe import DEFAULT, dump, validate
19
+ from datainventor.sources import KINDS, inspect
20
+
21
+ router = APIRouter(prefix="/api/curate")
22
+ PER_SOURCE = int(os.environ.get("CURATE_PER_SOURCE", "200"))
23
+ GRADE_N = int(os.environ.get("CURATE_GRADE", "48"))
24
+ REPHRASE_N = int(os.environ.get("CURATE_REPHRASE", "8"))
25
+ MAX_SOURCES, MAX_TARGETS = 4, 2
26
+ INSTALL = "pip install 'datainventor[full] @ git+https://huggingface.co/spaces/codelion/data-inventor'"
27
+ PREVIEWS: dict[str, Preview] = {}
28
+ TRIES: dict[str, list] = {}
29
+ GUARD = threading.Lock()
30
+
31
+
32
+ def _ep():
33
+ from main import make_endpoint # the host's endpoint; visitors never supply one
34
+ return make_endpoint()
35
+
36
+
37
+ def _token() -> str:
38
+ return os.environ.get("HF_TOKEN", "")
39
+
40
+
41
+ @router.get("/defaults")
42
+ def defaults():
43
+ from main import MAX_ROWS
44
+ return {"recipe": DEFAULT, "benchmarks": list(BENCHMARKS), "kinds": list(KINDS), "clean_rules": CLEAN_RULES,
45
+ "styles": {k: v[0] for k, v in STYLES.items()}, "styles_by_kind": {k: styles_for(k) for k in KINDS},
46
+ "doc_types": list(DOC_TYPES), "per_source": PER_SOURCE, "max_sources": MAX_SOURCES, "max_targets": MAX_TARGETS,
47
+ "max_docs": MAX_ROWS, "install": INSTALL}
48
+
49
+
50
+ @router.get("/inspect")
51
+ def do_inspect(dataset: str):
52
+ ds = dataset.strip().removeprefix("https://huggingface.co/datasets/").strip("/")
53
+ if not ds or ds.count("/") != 1 or len(ds) > 120:
54
+ raise HTTPException(400, "Use a Hugging Face dataset id like HuggingFaceFW/fineweb-edu.")
55
+ try:
56
+ return inspect(ds, _token())
57
+ except ValueError as e:
58
+ raise HTTPException(404, str(e))
59
+ except Exception:
60
+ raise HTTPException(404, "The dataset viewer could not open that dataset. It must be public and have the viewer enabled.")
61
+
62
+
63
+ class RecipeIn(BaseModel):
64
+ recipe: dict
65
+
66
+
67
+ def _checked(rec: dict) -> dict:
68
+ for s in rec.get("sources", []) + rec.get("targets", {}).get("datasets", []):
69
+ if s.get("files"):
70
+ raise HTTPException(400, "Local files only work in the runner on your own machines. Use Hugging Face datasets here.")
71
+ if len(rec.get("sources", [])) > MAX_SOURCES or len(rec.get("targets", {}).get("datasets", [])) > MAX_TARGETS:
72
+ raise HTTPException(400, f"The hosted preview takes up to {MAX_SOURCES} sources and {MAX_TARGETS} target datasets.")
73
+ try:
74
+ return validate(rec)
75
+ except ValueError as e:
76
+ raise HTTPException(400, str(e))
77
+
78
+
79
+ @router.post("/recipe")
80
+ def recipe_file(body: RecipeIn):
81
+ """The recipe as YAML, for the runner."""
82
+ r = _checked(body.recipe)
83
+ return Response(dump(r), media_type="application/x-yaml", headers={"Content-Disposition": 'attachment; filename="recipe.yaml"'})
84
+
85
+
86
+ @router.post("/previews")
87
+ def start(body: RecipeIn):
88
+ r = _checked(body.recipe)
89
+ ep = _ep()
90
+ with GUARD:
91
+ running = sum(1 for p in PREVIEWS.values() if p.state in ("queued", "running"))
92
+ if running >= 2:
93
+ raise HTTPException(429, "Two previews are already running. Wait for one to finish.")
94
+ pid = uuid.uuid4().hex[:10]
95
+ p = Preview(r)
96
+ PREVIEWS[pid] = p
97
+ for old in sorted(PREVIEWS, key=lambda k: PREVIEWS[k].started)[:-20]:
98
+ PREVIEWS.pop(old, None)
99
+ from main import MAX_ROWS
100
+ threading.Thread(target=run_preview, args=(p, ep, _token()),
101
+ kwargs={"per_source": PER_SOURCE, "grade_n": GRADE_N, "rephrase_n": REPHRASE_N, "invent_n": 4, "max_docs": MAX_ROWS},
102
+ daemon=True).start()
103
+ return {"id": pid}
104
+
105
+
106
+ def _get(pid: str) -> Preview:
107
+ if pid not in PREVIEWS:
108
+ raise HTTPException(404, "That preview is no longer here. Run it again.")
109
+ return PREVIEWS[pid]
110
+
111
+
112
+ @router.get("/previews/{pid}")
113
+ def status(pid: str):
114
+ p = _get(pid)
115
+ out = {"id": pid, "state": p.state, "stage": p.stage, "message": p.message, "elapsed": round((p.finished or time.time()) - p.started, 1)}
116
+ if p.state in ("done", "stopped"):
117
+ out["report"] = p.report
118
+ return out
119
+
120
+
121
+ @router.post("/previews/{pid}/stop")
122
+ def stop(pid: str):
123
+ _get(pid).stop.set()
124
+ return {"ok": True}
125
+
126
+
127
+ @router.get("/previews/{pid}/download")
128
+ def download(pid: str):
129
+ p = _get(pid)
130
+ if not p.docs:
131
+ raise HTTPException(404, "There are no documents to download yet.")
132
+ body = "\n".join(json.dumps(d, ensure_ascii=False) for d in p.docs) + "\n"
133
+ return Response(body, media_type="application/jsonl", headers={"Content-Disposition": f'attachment; filename="curated-sample-{pid}.jsonl"'})
134
+
135
+
136
+ class TryIn(BaseModel):
137
+ text: str
138
+ kind: str = "web"
139
+ styles: list[str] = []
140
+
141
+
142
+ @router.post("/try")
143
+ def try_styles(body: TryIn):
144
+ """Rewrite one document in up to four styles, side by side."""
145
+ text = body.text.strip()
146
+ if len(text.split()) < 60:
147
+ raise HTTPException(400, "Paste or pick a document of at least 60 words.")
148
+ styles = [s for s in (body.styles or styles_for(body.kind)) if s in STYLES][:4]
149
+ if not styles:
150
+ raise HTTPException(400, "Choose at least one style.")
151
+ now = time.time()
152
+ with GUARD:
153
+ recent = [t for t in TRIES.get("all", []) if now - t < 60]
154
+ if len(recent) >= 6:
155
+ raise HTTPException(429, "Too many rewrites in the last minute. Try again shortly.")
156
+ TRIES["all"] = recent + [now]
157
+ ep = _ep()
158
+ piece = chunks(text)[0]
159
+ from concurrent.futures import ThreadPoolExecutor
160
+ try:
161
+ with ThreadPoolExecutor(len(styles)) as ex:
162
+ outs = list(ex.map(lambda s: rephrase(ep, piece, s), styles))
163
+ except EndpointError as e:
164
+ raise HTTPException(502, str(e))
165
+ return {"source": piece, "results": [{"style": o["style"], "name": STYLES[o["style"]][0], "text": o["text"], "ok": o["ok"],
166
+ "why": o.get("why", ""), "fidelity": o.get("fidelity")} for o in outs]}
app/engine.py CHANGED
@@ -10,73 +10,11 @@ import random
10
  import re
11
  import threading
12
  import time
13
- import urllib.request
14
- import urllib.error
15
  from concurrent.futures import ThreadPoolExecutor
16
  from dataclasses import dataclass, field
17
- from typing import Callable, Optional
18
 
19
- HF_BASE = "https://router.huggingface.co/v1"
20
- HF_MODEL = "Qwen/Qwen3-4B-Instruct-2507:nscale"
21
-
22
-
23
- @dataclass
24
- class Endpoint:
25
- base_url: str = HF_BASE
26
- api_key: str = ""
27
- model: str = HF_MODEL
28
- disable_thinking: bool = False
29
- workers: int = 4
30
-
31
- def chat(self, prompt: str, max_tokens: int = 1500, temperature: float = 0.8, retries: int = 5,
32
- validate: Optional[Callable[[str], bool]] = None) -> str:
33
- body = {"model": self.model, "max_tokens": max_tokens, "temperature": temperature,
34
- "messages": [{"role": "user", "content": prompt}]}
35
- if self.disable_thinking:
36
- body["chat_template_kwargs"] = {"enable_thinking": False}
37
- data = json.dumps(body).encode()
38
- url = self.base_url.rstrip("/") + "/chat/completions"
39
- last = "no attempt"
40
- for attempt in range(retries):
41
- try:
42
- req = urllib.request.Request(url, data, {"Content-Type": "application/json", "User-Agent": "DataInventor/1.0",
43
- "Authorization": "Bearer " + self.api_key})
44
- resp = json.load(urllib.request.urlopen(req, timeout=180))
45
- choice = (resp.get("choices") or [{}])[0] or {}
46
- if choice.get("finish_reason") == "length":
47
- last = "reply cut off at max_tokens"
48
- time.sleep(1.5)
49
- continue
50
- text = ((choice.get("message") or {}).get("content") or "").strip()
51
- text = re.sub(r"<think>.*?</think>", "", text, flags=re.S).strip()
52
- if text and (validate is None or validate(text)):
53
- return text
54
- last = "reply rejected by a quality check"
55
- except urllib.error.HTTPError as e:
56
- detail = ""
57
- try:
58
- detail = e.read().decode()[:200]
59
- except Exception:
60
- pass
61
- last = f"HTTP {e.code} {detail}"
62
- if e.code in (401, 403):
63
- raise EndpointError(f"The endpoint refused the key (HTTP {e.code}). Check the API key.")
64
- if e.code == 404:
65
- raise EndpointError("The endpoint or model was not found (HTTP 404). Check the base URL and model name.")
66
- time.sleep(min(30, 2 ** attempt * 2) if e.code == 429 else 3 * (attempt + 1))
67
- except Exception as e: # timeouts, resets
68
- last = str(e)[:160]
69
- time.sleep(3 * (attempt + 1))
70
- raise EndpointError(last)
71
-
72
- def ping(self) -> dict:
73
- t = time.time()
74
- text = self.chat("Reply with the single word ready.", max_tokens=16, temperature=0, retries=2)
75
- return {"ok": True, "latency_ms": int((time.time() - t) * 1000), "sample": text[:40]}
76
-
77
-
78
- class EndpointError(Exception):
79
- pass
80
 
81
 
82
  # ---------------------------------------------------------------- blueprint
 
10
  import re
11
  import threading
12
  import time
 
 
13
  from concurrent.futures import ThreadPoolExecutor
14
  from dataclasses import dataclass, field
15
+ from typing import Optional
16
 
17
+ from datainventor.llm import HF_BASE, HF_MODEL, Endpoint, EndpointError # noqa: F401 (main imports them from here)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
18
 
19
 
20
  # ---------------------------------------------------------------- blueprint
app/main.py CHANGED
@@ -12,8 +12,11 @@ import urllib.parse
12
  import urllib.request
13
  import time
14
  import uuid
 
15
  from pathlib import Path
16
 
 
 
17
  from fastapi import FastAPI, HTTPException, Request
18
  from fastapi.responses import FileResponse, JSONResponse, RedirectResponse, Response
19
  from fastapi.staticfiles import StaticFiles
@@ -256,6 +259,11 @@ def publish(jid: str, body: PublishIn, request: Request):
256
  return {"url": url}
257
 
258
 
 
 
 
 
 
259
  @app.get("/")
260
  def index():
261
  return FileResponse(STATIC / "index.html")
 
12
  import urllib.request
13
  import time
14
  import uuid
15
+ import sys
16
  from pathlib import Path
17
 
18
+ sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) # the datainventor package sits next to app/
19
+
20
  from fastapi import FastAPI, HTTPException, Request
21
  from fastapi.responses import FileResponse, JSONResponse, RedirectResponse, Response
22
  from fastapi.staticfiles import StaticFiles
 
259
  return {"url": url}
260
 
261
 
262
+ from curate_api import router as curate_router # noqa: E402 (imports make_endpoint from this module)
263
+
264
+ app.include_router(curate_router)
265
+
266
+
267
  @app.get("/")
268
  def index():
269
  return FileResponse(STATIC / "index.html")
datainventor/__init__.py ADDED
File without changes
datainventor/calibrate.py ADDED
@@ -0,0 +1,277 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Calibrate: drop redundant copies, score every document, and reweight toward what you care about.
2
+
3
+ Scoring follows the FineWeb-Edu pattern. A language model grades a few thousand documents against a rubric. A cheap
4
+ linear model learns those grades from hashed n-grams and scores the rest of the corpus on CPU. Task relevance is
5
+ similarity to the target datasets you registered.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import json
10
+ import math
11
+ import random
12
+ import re
13
+ from collections import Counter
14
+ from dataclasses import dataclass, field
15
+
16
+ from .llm import Endpoint, EndpointError
17
+ from .text import NearDupIndex, centroid, cosine, doc_hash, features, h64, norm
18
+
19
+ # ---------------------------------------------------------------- dedup + repetition report
20
+
21
+
22
+ @dataclass
23
+ class Deduper:
24
+ """Exact (normalized hash) and near (MinHash LSH) duplicate removal. Keeps the first copy it sees and counts
25
+ every copy, so the report can show which passages a source repeats hundreds of times."""
26
+ near: bool = True
27
+ threshold: float = 0.8
28
+ exact_seen: dict = field(default_factory=dict)
29
+ copies: Counter = field(default_factory=Counter)
30
+ first_text: dict = field(default_factory=dict)
31
+
32
+ def __post_init__(self):
33
+ self.index = NearDupIndex(threshold=self.threshold) if self.near else None
34
+
35
+ def check(self, doc: dict) -> str:
36
+ h = doc_hash(doc["text"])
37
+ if h in self.exact_seen:
38
+ self.copies[h] += 1
39
+ return "exact duplicate"
40
+ self.exact_seen[h] = doc["id"]
41
+ self.copies[h] += 1
42
+ self.first_text.setdefault(h, doc["text"][:240])
43
+ if self.index is not None:
44
+ hit = self.index.check_add(h64(h), doc["text"])
45
+ if hit is not None:
46
+ return "near duplicate"
47
+ return ""
48
+
49
+ def report(self, top: int = 8) -> list[dict]:
50
+ return [{"copies": c, "text": self.first_text.get(h, "")} for h, c in self.copies.most_common(top) if c > 1]
51
+
52
+
53
+ def repeated_passages(docs: list[dict], n: int = 50, top: int = 8) -> list[dict]:
54
+ """Passages (50-word windows) that recur across documents. Whole-document dedup misses these: the same
55
+ worked solution pasted into many pages. Counts are within the sample, so a rate per 1,000 documents is reported."""
56
+ counts: Counter = Counter()
57
+ example: dict = {}
58
+ for d in docs:
59
+ ws = norm(d["text"]).split()
60
+ seen = set()
61
+ for i in range(0, max(0, len(ws) - n + 1), n // 2):
62
+ g = " ".join(ws[i:i + n])
63
+ k = h64(g)
64
+ if k not in seen:
65
+ seen.add(k)
66
+ counts[k] += 1
67
+ example.setdefault(k, g)
68
+ per = 1000 / max(1, len(docs))
69
+ return [{"docs": c, "per_1000": round(c * per, 1), "text": example[k][:240]} for k, c in counts.most_common(top) if c > 2]
70
+
71
+
72
+ # ---------------------------------------------------------------- language-model rubric
73
+
74
+ RUBRIC = """You are grading documents for a language-model pretraining corpus.
75
+ Score the document below on each axis from 0 to 5:
76
+ - "value": how useful it is for teaching a model knowledge and reasoning, at any level from school to research. 0 = spam, ads, navigation or nothing to learn; 2 = some useful facts but disorganized or shallow; 3 = coherent and informative; 4 = clearly explained, like a good textbook section or tutorial; 5 = outstanding teaching material with reasoning and examples.
77
+ - "clarity": how clear and well written it is.
78
+ - "accuracy": how likely the content is correct (5 = no visible errors).
79
+ - "density": information per word (0 = filler, 5 = every sentence carries content).{relevance}
80
+ Return ONLY JSON: {{"value": int, "clarity": int, "accuracy": int, "density": int{rel_key}, "reason": "under 20 words"}}
81
+
82
+ Document:
83
+ <<<
84
+ {doc}
85
+ >>>"""
86
+ REL_AXIS = """
87
+ - "relevance": how much it would help a model get better at the target capabilities: {targets}. 0 = unrelated; 3 = related background; 5 = directly practices them."""
88
+ AXES = ("value", "clarity", "accuracy", "density")
89
+
90
+
91
+ def rubric_prompt(text: str, targets: str = "") -> str:
92
+ t = targets.strip()
93
+ return RUBRIC.format(relevance=REL_AXIS.format(targets=t) if t else "", rel_key=', "relevance": int' if t else "",
94
+ doc=text[:6000])
95
+
96
+
97
+ def grade(ep: Endpoint, text: str, targets: str = "") -> dict | None:
98
+ """Quality is graded without reference to the targets, so general quality and task relevance stay separate
99
+ signals that `focus` can trade off."""
100
+ axes = AXES + (("relevance",) if targets.strip() else ())
101
+
102
+ def ok(s):
103
+ try:
104
+ g = json.loads(re.search(r"\{.*\}", s, re.S).group(0))
105
+ return all(isinstance(g.get(k), (int, float)) for k in axes)
106
+ except Exception:
107
+ return False
108
+ try:
109
+ s = ep.chat(rubric_prompt(text, targets), max_tokens=200, temperature=0.0, retries=3, validate=ok)
110
+ except EndpointError:
111
+ return None
112
+ g = json.loads(re.search(r"\{.*\}", s, re.S).group(0))
113
+ out = {k: max(0.0, min(5.0, float(g[k]))) for k in axes}
114
+ out["reason"] = str(g.get("reason", ""))[:160]
115
+ return out
116
+
117
+
118
+ def rubric_score(g: dict) -> float:
119
+ """General quality in [0, 1]. Value dominates; the others break ties."""
120
+ return round((0.55 * g["value"] + 0.15 * g["clarity"] + 0.15 * g["accuracy"] + 0.15 * g["density"]) / 5, 4)
121
+
122
+
123
+ # ---------------------------------------------------------------- cheap scorer trained on the grades
124
+
125
+ class LinearScorer:
126
+ """Ridge regression on hashed unigram+bigram features, trained with SGD. Pure Python, so it runs anywhere;
127
+ it scores a few thousand documents per second per core."""
128
+
129
+ def __init__(self, l2: float = 1e-4, epochs: int = 30, lr: float = 0.5, seed: int = 0):
130
+ self.w: dict[int, float] = {}
131
+ self.b = 0.0
132
+ self.l2, self.epochs, self.lr, self.seed = l2, epochs, lr, seed
133
+
134
+ def fit(self, X: list[dict], y: list[float]) -> "LinearScorer":
135
+ self.b = sum(y) / len(y)
136
+ idx = list(range(len(X)))
137
+ rng = random.Random(self.seed)
138
+ for ep in range(self.epochs):
139
+ rng.shuffle(idx)
140
+ lr = self.lr / (1 + ep * 0.3)
141
+ for i in idx:
142
+ err = self.predict_vec(X[i]) - y[i]
143
+ for k, v in X[i].items():
144
+ self.w[k] = self.w.get(k, 0.0) * (1 - lr * self.l2) - lr * err * v
145
+ self.b -= lr * err * 0.1
146
+ return self
147
+
148
+ def predict_vec(self, x: dict) -> float:
149
+ return self.b + sum(self.w.get(k, 0.0) * v for k, v in x.items())
150
+
151
+ def predict(self, text: str) -> float:
152
+ return max(0.0, min(1.0, self.predict_vec(features(text))))
153
+
154
+ def to_dict(self) -> dict:
155
+ return {"b": self.b, "w": {str(k): round(v, 6) for k, v in self.w.items() if abs(v) > 1e-5}}
156
+
157
+ @staticmethod
158
+ def from_dict(d: dict) -> "LinearScorer":
159
+ s = LinearScorer()
160
+ s.b, s.w = d["b"], {int(k): v for k, v in d["w"].items()}
161
+ return s
162
+
163
+
164
+ def spearman(a: list[float], b: list[float]) -> float | None:
165
+ def ranks(v):
166
+ order = sorted(range(len(v)), key=lambda i: v[i])
167
+ r = [0.0] * len(v)
168
+ for pos, i in enumerate(order):
169
+ r[i] = pos
170
+ return r
171
+ if len(a) < 3:
172
+ return None
173
+ ra, rb = ranks(a), ranks(b)
174
+ ma, mb = sum(ra) / len(ra), sum(rb) / len(rb)
175
+ cov = sum((x - ma) * (y - mb) for x, y in zip(ra, rb))
176
+ va = math.sqrt(sum((x - ma) ** 2 for x in ra))
177
+ vb = math.sqrt(sum((y - mb) ** 2 for y in rb))
178
+ return round(cov / (va * vb), 3) if va and vb else None
179
+
180
+
181
+ def train_scorer(graded: list[tuple[str, float]], holdout: float = 0.25, seed: int = 0) -> tuple[LinearScorer, float | None]:
182
+ """Fit on the grades and report rank agreement with the model on held-out documents. Low agreement means the
183
+ scorer needs more graded examples before you trust it on the full corpus."""
184
+ data = list(graded)
185
+ random.Random(seed).shuffle(data)
186
+ cut = max(1, int(len(data) * (1 - holdout))) if len(data) >= 8 else len(data)
187
+ train, test = data[:cut], data[cut:]
188
+ sc = LinearScorer().fit([features(t) for t, _ in train], [y for _, y in train])
189
+ agree = spearman([sc.predict(t) for t, _ in test], [y for _, y in test]) if len(test) >= 3 else None
190
+ if test: # refit on everything once agreement is measured
191
+ sc = LinearScorer().fit([features(t) for t, _ in data], [y for _, y in data])
192
+ return sc, agree
193
+
194
+
195
+ # ---------------------------------------------------------------- task relevance
196
+
197
+
198
+ class Relevance:
199
+ """Cosine similarity to the centroid of the target documents, rescaled against the source sample so 1.0 means
200
+ 'as close to the targets as the closest documents in your sources'."""
201
+
202
+ def __init__(self, target_texts: list[str]):
203
+ self.c = centroid([features(t) for t in target_texts]) if target_texts else {}
204
+ self.lo, self.hi = 0.0, 1.0
205
+
206
+ def __bool__(self):
207
+ return bool(self.c)
208
+
209
+ def raw(self, text: str) -> float:
210
+ return cosine(features(text), self.c) if self.c else 0.0
211
+
212
+ def fit_scale(self, texts: list[str]):
213
+ vals = sorted(self.raw(t) for t in texts)
214
+ if vals:
215
+ self.lo = vals[int(0.05 * (len(vals) - 1))]
216
+ self.hi = vals[int(0.95 * (len(vals) - 1))] or 1.0
217
+
218
+ def score(self, text: str) -> float:
219
+ if not self.c:
220
+ return 0.0
221
+ return round(max(0.0, min(1.0, (self.raw(text) - self.lo) / max(1e-6, self.hi - self.lo))), 4)
222
+
223
+
224
+ def combined(quality: float, relevance: float, focus: float) -> float:
225
+ """focus 0 = general quality only, 1 = target relevance only."""
226
+ return round((1 - focus) * quality + focus * relevance, 4)
227
+
228
+
229
+ class Calibrator:
230
+ """Turns grades into two scorers (general quality, task relevance) and scores any document with them.
231
+ Saved with the run, so every machine in a large run scores the same way."""
232
+
233
+ def __init__(self, focus: float = 0.3, targets: str = "", target_texts: list[str] | None = None):
234
+ self.targets = targets.strip()
235
+ self.centroid = Relevance(target_texts or [])
236
+ self.has_targets = bool(self.targets or self.centroid)
237
+ self.focus = float(focus) if self.has_targets else 0.0
238
+ self.q: LinearScorer | None = None
239
+ self.r: LinearScorer | None = None
240
+ self.agreement = {"quality": None, "relevance": None}
241
+
242
+ def fit(self, graded: list[tuple[str, dict]], sample_texts: list[str]) -> "Calibrator":
243
+ self.q, self.agreement["quality"] = train_scorer([(t, rubric_score(g)) for t, g in graded])
244
+ rel = [(t, g["relevance"] / 5) for t, g in graded if "relevance" in g]
245
+ if len(rel) >= 4:
246
+ self.r, self.agreement["relevance"] = train_scorer(rel)
247
+ if self.centroid:
248
+ self.centroid.fit_scale(sample_texts)
249
+ return self
250
+
251
+ def score(self, text: str, g: dict | None = None) -> dict:
252
+ quality = rubric_score(g) if g else round(self.q.predict(text), 4)
253
+ parts = []
254
+ if self.targets:
255
+ parts.append(g["relevance"] / 5 if g and "relevance" in g else (self.r.predict(text) if self.r else 0.0))
256
+ if self.centroid:
257
+ parts.append(self.centroid.score(text))
258
+ relevance = round(sum(parts) / len(parts), 4) if parts else 0.0
259
+ return {"quality": quality, "relevance": relevance, "score": combined(quality, relevance, self.focus)}
260
+
261
+ def to_dict(self) -> dict:
262
+ return {"focus": self.focus, "targets": self.targets, "q": self.q.to_dict() if self.q else None,
263
+ "r": self.r.to_dict() if self.r else None, "agreement": self.agreement,
264
+ "centroid": {"c": {str(k): v for k, v in self.centroid.c.items()}, "lo": self.centroid.lo, "hi": self.centroid.hi}}
265
+
266
+ @staticmethod
267
+ def from_dict(d: dict) -> "Calibrator":
268
+ c = Calibrator(d["focus"], d.get("targets", ""))
269
+ c.focus = d["focus"]
270
+ c.q = LinearScorer.from_dict(d["q"]) if d.get("q") else None
271
+ c.r = LinearScorer.from_dict(d["r"]) if d.get("r") else None
272
+ c.agreement = d.get("agreement", c.agreement)
273
+ cen = d.get("centroid") or {}
274
+ c.centroid.c = {int(k): v for k, v in (cen.get("c") or {}).items()}
275
+ c.centroid.lo, c.centroid.hi = cen.get("lo", 0.0), cen.get("hi", 1.0)
276
+ c.has_targets = bool(c.targets or c.centroid)
277
+ return c
datainventor/clean.py ADDED
@@ -0,0 +1,233 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Clean: drop low-information documents, then remove anything that overlaps the benchmarks you will test on.
2
+
3
+ The prose rules follow the Gopher and C4 heuristics with FineWeb's thresholds. Math and code sources get their own,
4
+ looser rules, because LaTeX and source code fail prose checks by design.
5
+ """
6
+ from __future__ import annotations
7
+
8
+ import re
9
+ from collections import Counter
10
+ from dataclasses import dataclass, field
11
+
12
+ from .text import ngrams, norm, words
13
+
14
+ BOILER = re.compile(r"(cookie|javascript|privacy policy|terms of (use|service)|all rights reserved|lorem ipsum|"
15
+ r"click here|sign up for our newsletter|accept all|log ?in to (reply|comment)|skip to (main )?content)", re.I)
16
+ AUTOGEN = re.compile(r"(auto-?generated|do not edit|generated by (the )?protoc|minified)", re.I)
17
+
18
+ DEFAULTS = {
19
+ "min_words": 50, "max_words": 100_000, "min_mean_word": 3.0, "max_mean_word": 10.0,
20
+ "max_symbol_ratio": 0.1, "min_alpha_words": 0.8, "max_ellipsis_lines": 0.3,
21
+ "max_dup_lines": 0.3, "max_top2gram": 0.2, "max_dup5gram": 0.15, "max_dup10gram": 0.1, "max_boiler_lines": 0.4,
22
+ "strip_boilerplate_lines": True,
23
+ # code
24
+ "code_max_line": 1000, "code_min_alpha": 0.25,
25
+ }
26
+
27
+
28
+ def dup_ngram_chars(ws: list[str], n: int) -> int:
29
+ """Characters inside repeated n-grams, counting each repeat once and never the first copy (Gopher, as
30
+ implemented in datatrove). Overlapping windows are skipped so a repeated paragraph is not counted n times."""
31
+ seen, dup, i = set(), 0, 0
32
+ while i <= len(ws) - n:
33
+ g = tuple(ws[i:i + n])
34
+ if g in seen:
35
+ dup += sum(len(w) for w in g)
36
+ i += n
37
+ else:
38
+ seen.add(g)
39
+ i += 1
40
+ return dup
41
+
42
+
43
+ @dataclass
44
+ class Cleaner:
45
+ rules: dict = field(default_factory=lambda: dict(DEFAULTS))
46
+
47
+ def __post_init__(self):
48
+ self.rules = {**DEFAULTS, **(self.rules or {})}
49
+
50
+ def apply(self, doc: dict) -> tuple[dict | None, str]:
51
+ """Return (document, "") to keep it, possibly with boilerplate lines removed, or (None, reason) to drop it."""
52
+ text = doc["text"]
53
+ kind = doc.get("kind", "web")
54
+ if not text.strip():
55
+ return None, "empty"
56
+ if kind == "code":
57
+ why = self._code(text)
58
+ return (None, why) if why else (doc, "")
59
+ if self.rules["strip_boilerplate_lines"] and kind in ("web", "educational", "multilingual"):
60
+ lines = text.split("\n")
61
+ kept = [l for l in lines if not (BOILER.search(l) and len(l.split()) < 25)]
62
+ if lines and len(lines) - len(kept) > self.rules["max_boiler_lines"] * len(lines):
63
+ return None, "mostly boilerplate"
64
+ if len(kept) != len(lines):
65
+ text = "\n".join(kept)
66
+ doc = {**doc, "text": text}
67
+ if kind == "textbook": # PDFs repeat running headers and footers on every page; drop those lines, keep the page
68
+ lines = text.split("\n")
69
+ short = Counter(l.strip() for l in lines if l.strip() and len(l.split()) <= 12)
70
+ running = {l for l, c in short.items() if c >= 3}
71
+ if running:
72
+ text = "\n".join(l for l in lines if l.strip() not in running)
73
+ doc = {**doc, "text": text}
74
+ why = self._prose(text, loose=kind in ("math", "textbook", "synthetic"), rep=1.6 if kind == "math" else 1.0)
75
+ return (None, why) if why else (doc, "")
76
+
77
+ def _prose(self, text: str, loose: bool, rep: float = 1.0) -> str:
78
+ """rep scales the repetition caps: worked math repeats symbols and steps by nature."""
79
+ r = self.rules
80
+ toks = text.split()
81
+ n = len(toks)
82
+ if n < r["min_words"]:
83
+ return "too short"
84
+ if n > r["max_words"]:
85
+ return "too long"
86
+ if not loose:
87
+ mean = sum(len(t) for t in toks) / n
88
+ if not r["min_mean_word"] <= mean <= r["max_mean_word"]:
89
+ return "odd word lengths"
90
+ sym = (text.count("#") + text.count("...") + text.count("…")) / n
91
+ if sym > r["max_symbol_ratio"]:
92
+ return "too many symbols"
93
+ alpha = sum(1 for t in toks if any(c.isalpha() for c in t)) / n
94
+ if alpha < r["min_alpha_words"]:
95
+ return "too little real text"
96
+ lines = [l.strip() for l in text.split("\n") if l.strip()]
97
+ if lines:
98
+ ell = sum(1 for l in lines if l.endswith(("...", "…"))) / len(lines)
99
+ if ell > r["max_ellipsis_lines"]:
100
+ return "truncated lines"
101
+ if len(lines) >= 5:
102
+ dup = 1 - len(set(lines)) / len(lines)
103
+ if dup > r["max_dup_lines"] * rep:
104
+ return "repeated lines"
105
+ ws = words(text)
106
+ if len(ws) >= 20:
107
+ bi = Counter(ngrams(ws, 2))
108
+ top = bi.most_common(1)[0]
109
+ if top[1] > 2 and top[1] * len(top[0]) / max(1, sum(len(w) + 1 for w in ws)) > r["max_top2gram"] * rep:
110
+ return "one phrase repeated"
111
+ total = max(1, sum(len(w) for w in ws))
112
+ for n, cap in ((5, r["max_dup5gram"]), (10, r["max_dup10gram"])):
113
+ if dup_ngram_chars(ws, n) / total > cap * rep:
114
+ return "repeated passages"
115
+ return ""
116
+
117
+ def _code(self, text: str) -> str:
118
+ r = self.rules
119
+ lines = text.split("\n")
120
+ if len(text) < 100:
121
+ return "too short"
122
+ if max(len(l) for l in lines) > r["code_max_line"]:
123
+ return "minified or data blob"
124
+ if sum(c.isalpha() for c in text) / len(text) < r["code_min_alpha"]:
125
+ return "too little real code"
126
+ if AUTOGEN.search(text[:600]):
127
+ return "auto-generated"
128
+ return ""
129
+
130
+
131
+ # ---------------------------------------------------------------- decontamination
132
+
133
+ # Benchmarks people usually report. Each entry: (hub id, config, split, fields joined to form the test text).
134
+ BENCHMARKS = {
135
+ "gsm8k": ("openai/gsm8k", "main", "test", ["question"]),
136
+ "math500": ("HuggingFaceH4/MATH-500", "default", "test", ["problem"]),
137
+ "humaneval": ("openai/openai_humaneval", "openai_humaneval", "test", ["prompt"]),
138
+ "mbpp": ("google-research-datasets/mbpp", "full", "test", ["text"]),
139
+ "mmlu": ("cais/mmlu", "all", "test", ["question"]),
140
+ "arc_challenge": ("allenai/ai2_arc", "ARC-Challenge", "test", ["question"]),
141
+ "arc_easy": ("allenai/ai2_arc", "ARC-Easy", "test", ["question"]),
142
+ "hellaswag": ("Rowan/hellaswag", "default", "validation", ["ctx"]),
143
+ "openbookqa": ("allenai/openbookqa", "main", "test", ["question_stem"]),
144
+ "winogrande": ("allenai/winogrande", "winogrande_xl", "validation", ["sentence"]),
145
+ "truthfulqa": ("truthfulqa/truthful_qa", "generation", "validation", ["question"]),
146
+ "commonsenseqa": ("tau/commonsense_qa", "default", "validation", ["question"]),
147
+ }
148
+
149
+
150
+ class Decontaminator:
151
+ """Drop a document when it shares any long n-gram with a held-out test item.
152
+
153
+ 13-grams is the GPT-3 / OLMo convention. Short test items (under n words) are matched as a whole normalized string.
154
+ """
155
+
156
+ def __init__(self, n: int = 13):
157
+ self.n = n
158
+ self.grams: dict[str, str] = {}
159
+ self.short: dict[str, str] = {}
160
+ self.counts: Counter = Counter()
161
+
162
+ def add(self, name: str, texts: list[str]):
163
+ for t in texts:
164
+ ws = norm(t).split()
165
+ if len(ws) < self.n:
166
+ if len(ws) >= 6:
167
+ self.short[" ".join(ws)] = name
168
+ continue
169
+ for g in ngrams(ws, self.n):
170
+ self.grams.setdefault(g, name)
171
+
172
+ def __len__(self):
173
+ return len(self.grams) + len(self.short)
174
+
175
+ def hit(self, text: str) -> str:
176
+ if not self:
177
+ return ""
178
+ nt = norm(text)
179
+ ws = nt.split()
180
+ for g in ngrams(ws, self.n):
181
+ b = self.grams.get(g)
182
+ if b:
183
+ self.counts[b] += 1
184
+ return b
185
+ if self.short:
186
+ for s, b in self.short.items():
187
+ if s in nt:
188
+ self.counts[b] += 1
189
+ return b
190
+ return ""
191
+
192
+
193
+ _BUNDLED: dict | None = None
194
+
195
+
196
+ def bundled_benchmarks() -> dict:
197
+ """The test sets shipped with the package (data/benchmarks.json.gz, built by make_benchmarks.py)."""
198
+ global _BUNDLED
199
+ if _BUNDLED is None:
200
+ import gzip
201
+ import json
202
+ import os
203
+ path = os.path.join(os.path.dirname(__file__), "data", "benchmarks.json.gz")
204
+ try:
205
+ with gzip.open(path, "rt", encoding="utf-8") as f:
206
+ _BUNDLED = json.load(f)
207
+ except OSError:
208
+ _BUNDLED = {}
209
+ return _BUNDLED
210
+
211
+
212
+ def benchmark_texts(key: str, token: str = "") -> list[str]:
213
+ """Bundled copy first; the dataset viewer only when a set is missing from the bundle."""
214
+ return bundled_benchmarks().get(key) or load_benchmark_texts(key, token)
215
+
216
+
217
+ def load_benchmark_texts(key: str, token: str = "", limit: int = 20000) -> list[str]:
218
+ """Test items through the dataset viewer API (no download). Test sets are small, so this is enough."""
219
+ from .sources import _get
220
+ ds, cfg, split, fields = BENCHMARKS[key]
221
+ out, off = [], 0
222
+ while off < limit:
223
+ got = _get("rows", {"dataset": ds, "config": cfg, "split": split, "offset": off, "length": 100}, token)
224
+ rows = got.get("rows", [])
225
+ for r in rows:
226
+ row = r.get("row") or {}
227
+ t = " ".join(str(row.get(f, "")) for f in fields).strip()
228
+ if t:
229
+ out.append(t)
230
+ if len(rows) < 100:
231
+ break
232
+ off += 100
233
+ return out
datainventor/cli.py ADDED
@@ -0,0 +1,98 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """datainventor: curate a pretraining corpus on your own machines.
2
+
3
+ datainventor init recipe.yaml write a starter recipe to edit
4
+ datainventor preview recipe.yaml run every stage on a small sample and print what each did
5
+ datainventor run recipe.yaml --out ./curated run it in full (resumable)
6
+
7
+ The model endpoint is any OpenAI-compatible server: vLLM, SGLang, Ollama, a hosted API. Set its key in the
8
+ environment variable named by endpoint.api_key_env (DI_API_KEY by default).
9
+ """
10
+ from __future__ import annotations
11
+
12
+ import argparse
13
+ import json
14
+ import os
15
+ import sys
16
+
17
+ from .compose import fmt
18
+ from .recipe import DEFAULT, dump, endpoint_of, load
19
+
20
+
21
+ def main(argv=None):
22
+ ap = argparse.ArgumentParser(prog="datainventor", description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
23
+ sub = ap.add_subparsers(dest="cmd", required=True)
24
+ a = sub.add_parser("init", help="write a starter recipe")
25
+ a.add_argument("path")
26
+ a = sub.add_parser("preview", help="run every stage on a sample")
27
+ a.add_argument("recipe")
28
+ a.add_argument("--per-source", type=int, default=300)
29
+ a.add_argument("--grade", type=int, default=64)
30
+ a.add_argument("--rephrase", type=int, default=6)
31
+ a.add_argument("--workers", type=int, default=None)
32
+ a.add_argument("--json", help="also write the full report here")
33
+ a = sub.add_parser("run", help="run the recipe in full")
34
+ a.add_argument("recipe")
35
+ a.add_argument("--out", default=None, help="output folder (default: output.dir in the recipe)")
36
+ a.add_argument("--procs", type=int, default=0, help="CPU processes for cleaning and scoring (default: all but one)")
37
+ a.add_argument("--workers", type=int, default=None, help="parallel model calls (default: endpoint.workers)")
38
+ a.add_argument("--stages", default="grade,curate,create,compose")
39
+ a.add_argument("--limit", type=int, default=0, help="read at most this many documents per source (for a trial run)")
40
+ args = ap.parse_args(argv)
41
+
42
+ if args.cmd == "init":
43
+ if os.path.exists(args.path):
44
+ sys.exit(f"{args.path} exists; not overwriting it.")
45
+ r = json.loads(json.dumps(DEFAULT))
46
+ r["sources"] = [
47
+ {"name": "finepdfs", "dataset": "codelion/finepdfs-1B", "kind": "textbook"},
48
+ {"name": "dclm", "dataset": "codelion/dclm-baseline-1B", "kind": "web"},
49
+ {"name": "fineweb-edu", "dataset": "codelion/fineweb-edu-1B", "kind": "educational"},
50
+ ]
51
+ r["targets"]["description"] = "grade-school to university math and science reasoning"
52
+ r["compose"]["mixes"][0]["budget_tokens"] = 2_000_000_000
53
+ with open(args.path, "w") as f:
54
+ f.write(dump(r))
55
+ print(f"Wrote {args.path}. Edit the sources, targets and budget, then: datainventor preview {args.path}")
56
+ return
57
+
58
+ r = load(args.recipe)
59
+ if args.cmd == "preview":
60
+ from .preview import Preview, run_preview
61
+ from .sources import inspect
62
+ token = os.environ.get("HF_TOKEN", "")
63
+ for s in r["sources"]: # row counts make the token estimates possible
64
+ if s.get("dataset") and not s.get("num_rows"):
65
+ try:
66
+ s["num_rows"] = inspect(s["dataset"], token)["num_rows"] if (s.get("config", "default") == "default") else 0
67
+ except Exception:
68
+ pass
69
+ p = Preview(r)
70
+ ep = endpoint_of(r, args.workers)
71
+ run_preview(p, ep, token, per_source=args.per_source, grade_n=args.grade, rephrase_n=args.rephrase, invent_n=2)
72
+ if p.state != "done":
73
+ sys.exit(f"Preview {p.state}: {p.message}")
74
+ rep = p.report
75
+ for f in rep["sources"]:
76
+ drops = ", ".join(f"{k} {v}" for k, v in sorted(f["dropped"].items(), key=lambda x: -x[1]))
77
+ print(f"{f['name']}: read {f['read']}, selected {f['selected']}; dropped: {drops}; decontaminated: {f['decontaminated'] or 'none'}")
78
+ if f.get("available_tokens"):
79
+ print(f" about {fmt(f['available_tokens'])} tokens available after curation")
80
+ c = rep["stages"]["calibrate"]
81
+ print(f"graded {c['graded']}; scorer agreement {c['agreement']}")
82
+ for plan in rep["stages"]["compose"]["plans"]:
83
+ print(f"mix {plan['name']} ({plan['budget']}): " + "; ".join(f"{x['name']} {int(x['share'] * 100)}% x{x['epochs']}" for x in plan["rows"]))
84
+ for w in plan["warnings"]:
85
+ print(" - " + w)
86
+ if args.json:
87
+ with open(args.json, "w") as fh:
88
+ json.dump({"report": rep, "docs": p.docs}, fh, indent=1, ensure_ascii=False)
89
+ return
90
+
91
+ from .runner import Run
92
+ out = args.out or r["output"]["dir"]
93
+ Run(r, out, procs=args.procs, workers=args.workers, limit=args.limit).go(tuple(args.stages.split(",")))
94
+ print(f"Done. See {os.path.join(out, 'REPORT.md')}")
95
+
96
+
97
+ if __name__ == "__main__":
98
+ main()
datainventor/compose.py ADDED
@@ -0,0 +1,83 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Compose: turn one prepared corpus into one or more training mixes that fit a token budget.
2
+
3
+ Defaults come from our own 1B-token study (huggingface.co/blog/codelion/optimal-dataset-mixing): a static mix of
4
+ 50% textbook-style text, 30% filtered web and 20% educational web beat every curriculum we tried at 70M parameters
5
+ and 1B tokens. That is one model size and one budget, so treat it as a starting point and fork it.
6
+
7
+ When a source is too small for its share, it is repeated up to `max_epochs` (about four epochs of repetition cost
8
+ little in Muennighoff et al., 2023). Past that, the gap is filled by rephrasing that source's best documents,
9
+ up to `max_synthetic_share` of the mix, since a mix that leans on synthetic text generalized worst in our study.
10
+ """
11
+ from __future__ import annotations
12
+
13
+ GROUP_OF_KIND = {"textbook": "textbook", "synthetic": "textbook", "web": "web", "multilingual": "web",
14
+ "educational": "educational", "math": "math", "code": "code", "other": "web"}
15
+ BASE_SHARES = {"textbook": 0.50, "web": 0.30, "educational": 0.20}
16
+
17
+
18
+ def default_weights(sources: list[dict]) -> dict[str, float]:
19
+ """Share per source name. Math and code, when present, take 10% each off the top; the rest follows 50/30/20
20
+ over the groups that exist, split evenly inside each group."""
21
+ groups: dict[str, list[str]] = {}
22
+ for s in sources:
23
+ groups.setdefault(GROUP_OF_KIND.get(s.get("kind", "web"), "web"), []).append(s["name"])
24
+ shares: dict[str, float] = {}
25
+ fixed = {g: 0.10 for g in ("math", "code") if g in groups}
26
+ rest = 1.0 - sum(fixed.values())
27
+ base = {g: v for g, v in BASE_SHARES.items() if g in groups}
28
+ if not base:
29
+ # only math and/or code sources: split evenly
30
+ return {s["name"]: round(1 / len(sources), 4) for s in sources}
31
+ tot = sum(base.values())
32
+ for g, v in base.items():
33
+ shares[g] = rest * v / tot
34
+ shares.update(fixed)
35
+ out = {}
36
+ for g, names in groups.items():
37
+ for n in names:
38
+ out[n] = round(shares[g] / len(names), 4)
39
+ return out
40
+
41
+
42
+ def plan_mix(budget_tokens: int, sources: list[dict], weights: dict[str, float], max_epochs: float = 4.0,
43
+ max_synthetic_share: float = 0.25, rephrase_yield: float = 4.0, avg_out_tokens: int = 700) -> dict:
44
+ """sources: [{name, kind, available_tokens (after cleaning and dedup), high_tokens (top tier)}].
45
+ Returns per-source rows and totals, including how much synthetic text to generate and roughly how many calls."""
46
+ tot_w = sum(weights.get(s["name"], 0) for s in sources) or 1.0
47
+ rows, synth_total, short_total = [], 0, 0
48
+ for s in sources:
49
+ share = weights.get(s["name"], 0) / tot_w
50
+ want = int(budget_tokens * share)
51
+ avail = max(0, int(s.get("available_tokens", 0)))
52
+ fresh = min(want, avail)
53
+ repeat_cap = int(avail * max_epochs)
54
+ from_repeat = min(want, repeat_cap)
55
+ gap = want - from_repeat
56
+ synth = min(gap, int(s.get("high_tokens", avail) * rephrase_yield)) if gap > 0 else 0
57
+ short = gap - synth
58
+ epochs = round(from_repeat / avail, 2) if avail else 0.0
59
+ rows.append({"name": s["name"], "kind": s.get("kind", "web"), "share": round(share, 4), "want_tokens": want,
60
+ "available_tokens": avail, "fresh_tokens": fresh, "epochs": epochs, "synthetic_tokens": synth,
61
+ "short_tokens": short})
62
+ synth_total += synth
63
+ short_total += short
64
+ cap = int(budget_tokens * max_synthetic_share)
65
+ warnings = []
66
+ if synth_total > cap:
67
+ warnings.append(f"The plan needs {fmt(synth_total)} synthetic tokens, over the {int(max_synthetic_share * 100)}% cap "
68
+ f"({fmt(cap)}). Add sources, lower the budget, or raise the cap knowingly.")
69
+ if short_total:
70
+ warnings.append(f"{fmt(short_total)} tokens cannot be filled even with repetition and rephrasing.")
71
+ for r in rows:
72
+ if r["epochs"] > 1.0:
73
+ warnings.append(f"{r['name']} repeats {r['epochs']}x to reach its share.")
74
+ return {"budget_tokens": budget_tokens, "rows": rows, "synthetic_tokens": synth_total,
75
+ "synthetic_calls": synth_total // max(1, avg_out_tokens), "short_tokens": short_total, "warnings": warnings}
76
+
77
+
78
+ def fmt(n: float) -> str:
79
+ n = float(n)
80
+ for unit, div in (("T", 1e12), ("B", 1e9), ("M", 1e6), ("K", 1e3)):
81
+ if abs(n) >= div:
82
+ return f"{n / div:.3g}{unit}"
83
+ return f"{int(n)}"
datainventor/create.py ADDED
@@ -0,0 +1,152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Create: rephrase good source documents into new training documents, or write new ones where sources are thin.
2
+
3
+ Rephrase styles are the five from Sutra (weights 30/25/20/15/10, after BeyondWeb), plus math and code styles.
4
+ A fidelity check flags rephrasings that introduce numbers or names the source does not contain, which matters when the
5
+ training data must respect a knowledge cutoff.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import random
10
+ import re
11
+
12
+ from .llm import Endpoint, EndpointError
13
+
14
+ NO_PREAMBLE = ("Output ONLY the rewritten document. No preamble, no meta-commentary, no phrases like \"Here is\", "
15
+ "\"Let me\", \"Sure\" or \"Okay\". Do not add facts, numbers, names or dates that are not in the original.")
16
+
17
+ STYLES = {
18
+ "conversational": ("Teacher-student dialogue", 30, """Rewrite the content as a natural conversation between a teacher and a student.
19
+ The student asks curious, natural questions; the teacher explains clearly with concrete examples taken from the content.
20
+ Keep all key information and the same technical accuracy. Keep a similar length (within 20%).
21
+ Use the labels **Teacher**: and **Student**: and start directly with "**Teacher**:"."""),
22
+ "tutorial": ("Step-by-step tutorial", 25, """Rewrite the content as a clear step-by-step tutorial.
23
+ Start with "## Tutorial: <topic>", then an overview, prerequisites, numbered steps with what to do and how to check it,
24
+ a short practice exercise and key takeaways. Keep all key information accurate. Keep a similar length (within 20%)."""),
25
+ "qa": ("Questions and answers", 20, """Rewrite the content as 4 to 7 questions and thorough answers that together cover every key point,
26
+ moving from basic (what) to deeper (why, how) to applied. Start with "## Understanding <topic>: Questions and Answers".
27
+ Use **Q:** and **A:** labels. Keep a similar total length (within 20%)."""),
28
+ "dense_reference": ("Dense reference notes", 15, """Rewrite the content as compact reference notes at 70 to 85 percent of the original length.
29
+ Remove filler and repetition. Use headings and bullet points: core definition, key properties, relationships,
30
+ essential examples, formulas or procedures if any, pitfalls."""),
31
+ "pedagogical": ("Textbook section", 10, """Rewrite the content as a well-structured textbook section: learning objectives, the connection to what
32
+ the reader already knows, the core explanation, one worked example with step-by-step reasoning, a common question
33
+ with its answer, and a quick self-check question. Keep a similar length."""),
34
+ "worked_examples": ("Worked problems (math)", 0, """Rewrite the content as 2 to 4 worked problems that use exactly the ideas in it. For each: state the problem,
35
+ solve it step by step with the reasoning for each step, and check the answer. Use LaTeX for mathematics."""),
36
+ "code_walkthrough": ("Code walkthrough (code)", 0, """Rewrite the content as an explained code walkthrough: what the code does, then the code in a fenced block
37
+ (keep it correct and runnable; you may tidy names and add comments), then an explanation of the key lines and one
38
+ example of using it."""),
39
+ }
40
+
41
+ KIND_STYLES = {"math": ["worked_examples", "pedagogical", "qa", "tutorial"],
42
+ "code": ["code_walkthrough", "tutorial", "qa", "dense_reference"]}
43
+
44
+
45
+ def styles_for(kind: str) -> list[str]:
46
+ return KIND_STYLES.get(kind, ["conversational", "tutorial", "qa", "dense_reference", "pedagogical"])
47
+
48
+
49
+ def pick_style(kind: str, rng: random.Random, allowed: list[str] | None = None) -> str:
50
+ opts = [s for s in (allowed or styles_for(kind)) if s in STYLES]
51
+ weights = [max(STYLES[s][1], 10) for s in opts]
52
+ return rng.choices(opts, weights=weights, k=1)[0]
53
+
54
+
55
+ def chunks(text: str, max_chars: int = 6000) -> list[str]:
56
+ """Split a long document at paragraph breaks so each rewrite sees a piece it can keep at a similar length."""
57
+ if len(text) <= max_chars:
58
+ return [text]
59
+ out, cur = [], ""
60
+ for para in re.split(r"\n\s*\n", text):
61
+ if cur and len(cur) + len(para) + 2 > max_chars:
62
+ out.append(cur)
63
+ cur = ""
64
+ cur = (cur + "\n\n" + para) if cur else para
65
+ while len(cur) > max_chars: # one giant paragraph
66
+ out.append(cur[:max_chars])
67
+ cur = cur[max_chars:]
68
+ if cur.strip():
69
+ out.append(cur)
70
+ return [c for c in out if len(c.split()) >= 80] or [text[:max_chars]]
71
+
72
+
73
+ def rephrase_prompt(text: str, style: str, language: str = "") -> str:
74
+ lang = f"\nWrite it in {language}." if language and language.lower() != "same" else ""
75
+ return f"""{STYLES[style][2]}{lang}
76
+ {NO_PREAMBLE}
77
+
78
+ Original content:
79
+ <<<
80
+ {text[:8000]}
81
+ >>>"""
82
+
83
+
84
+ LEAD = re.compile(r"^(here('s| is)|sure|okay|certainly|let me|i('ll| will))\b", re.I)
85
+ NUM = re.compile(r"\b\d[\d,.]*\b")
86
+ # a capitalized word that follows a lowercase word mid-sentence: names, places, products. Words at the start of a
87
+ # sentence, a line or a markdown label are skipped, so "Teacher:" or "Great question" never count.
88
+ HEADINGS = re.compile(r"^#+ .*$|\*\*[^*\n]{1,80}\*\*", re.M)
89
+ NAME = re.compile(r"\b[a-z][a-z'-]*[,;]? ([A-Z][a-zA-Z]{2,}(?: [A-Z][a-zA-Z]{2,})*)")
90
+
91
+
92
+ def fidelity(src: str, out: str, template: str = "") -> dict:
93
+ """Share of numbers and mid-sentence names in the rewrite that never appear in the source. A rough proxy for
94
+ invented facts; the threshold is a choice, not a law."""
95
+ s_low = (src + " " + template).lower()
96
+ body = HEADINGS.sub(" ", out) # headings and bold labels the style asks for are not new facts
97
+ nums = {n.rstrip(".,") for n in NUM.findall(out)}
98
+ names = set(NAME.findall(body))
99
+ new_nums = [n for n in nums if n and n not in src]
100
+ new_names = [n for n in names if n.lower() not in s_low]
101
+ total = len(nums) + len(names)
102
+ share = (len(new_nums) + len(new_names)) / total if total else 0.0
103
+ return {"new_numbers": sorted(new_nums)[:8], "new_names": sorted(new_names)[:8], "new_share": round(share, 3),
104
+ "checked": total}
105
+
106
+
107
+ def rephrase(ep: Endpoint, text: str, style: str, language: str = "", max_new_share: float = 0.35) -> dict:
108
+ """One rewrite. Returns {"text", "style", "fidelity", "ok", "why"}."""
109
+ def ok(s):
110
+ return len(s.split()) >= 60 and not LEAD.search(s.strip()[:40])
111
+ try:
112
+ out = ep.chat(rephrase_prompt(text, style, language), max_tokens=3000, temperature=0.7, retries=3, validate=ok)
113
+ except EndpointError as e:
114
+ return {"text": "", "style": style, "ok": False, "why": f"model call failed: {e}"[:200]}
115
+ fid = fidelity(text, out, STYLES[style][2])
116
+ new = len(fid["new_numbers"]) + len(fid["new_names"])
117
+ # invented numbers are the clearest sign of made-up content; one stray number or name is not a pattern
118
+ good = len(fid["new_numbers"]) < 3 and (new < 2 or fid["new_share"] <= max_new_share)
119
+ return {"text": out, "style": style, "fidelity": fid, "ok": good, "why": "" if good else "adds facts the source does not contain"}
120
+
121
+
122
+ # ---------------------------------------------------------------- new documents where sources are thin
123
+
124
+ DOC_TYPES = { # a subset of Sutra's content types that reads well as pretraining text
125
+ "concept_introduction": "an introduction to the concept: definition, intuition, why it matters, how it works, two examples, a common misconception, and how it connects to related ideas",
126
+ "reasoning_demonstration": "a worked demonstration of reasoning about a concrete problem in this area, thinking step by step and checking the result",
127
+ "worked_examples": "two or three worked examples of increasing difficulty, each solved step by step",
128
+ "common_misconceptions": "an explanation of the most common misconceptions about the concept, why people hold them, and the correct understanding",
129
+ "problem_set": "a short problem set of four problems with full solutions",
130
+ "case_study": "a realistic case study that applies the concept, with the decisions and their reasons",
131
+ "technical_documentation": "clear technical documentation of the concept or tool, with a usage example",
132
+ "historical_context": "the history of how the idea developed, who contributed what, and why it mattered",
133
+ }
134
+
135
+
136
+ def invent_prompt(topic: str, doc_type: str, level: int, request: str, language: str = "English") -> str:
137
+ return f"""You are writing high-quality pretraining text for a language model. Collection: "{request}".
138
+ Write {DOC_TYPES[doc_type]}.
139
+ Concept: {topic}
140
+ Difficulty: {level} on a 1 (introductory) to 10 (expert) scale. Language: {language}.
141
+ Write 350 to 700 words of dense, accurate prose with markdown headings where useful. Every sentence should teach
142
+ something. Do not invent statistics, quotes or citations. {NO_PREAMBLE.split('. Do not add')[0]}."""
143
+
144
+
145
+ def invent(ep: Endpoint, topic: str, doc_type: str, level: int, request: str, language: str = "English") -> dict:
146
+ def ok(s):
147
+ return len(s.split()) >= 150 and not LEAD.search(s.strip()[:40])
148
+ try:
149
+ out = ep.chat(invent_prompt(topic, doc_type, level, request, language), max_tokens=2500, temperature=0.8, retries=3, validate=ok)
150
+ except EndpointError as e:
151
+ return {"text": "", "ok": False, "why": f"model call failed: {e}"[:200]}
152
+ return {"text": out, "ok": True, "doc_type": doc_type, "topic": topic, "level": level}
datainventor/data/benchmarks.json.gz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e1b3b8b79353bdece54384094cc308a52e577dab21e195a9fdd8ba7a66e862de
3
+ size 2370882
datainventor/llm.py ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """One OpenAI-compatible chat endpoint: HF Inference Providers, vLLM, SGLang, Ollama, OptiQ serve or a hosted API."""
2
+ from __future__ import annotations
3
+
4
+ import json
5
+ import re
6
+ import time
7
+ import urllib.error
8
+ import urllib.request
9
+ from dataclasses import dataclass
10
+ from typing import Callable, Optional
11
+
12
+ HF_BASE = "https://router.huggingface.co/v1"
13
+ HF_MODEL = "Qwen/Qwen3-4B-Instruct-2507:nscale"
14
+
15
+
16
+ @dataclass
17
+ class Endpoint:
18
+ base_url: str = HF_BASE
19
+ api_key: str = ""
20
+ model: str = HF_MODEL
21
+ disable_thinking: bool = False
22
+ workers: int = 4
23
+
24
+ def chat(self, prompt: str, max_tokens: int = 1500, temperature: float = 0.8, retries: int = 5,
25
+ validate: Optional[Callable[[str], bool]] = None) -> str:
26
+ body = {"model": self.model, "max_tokens": max_tokens, "temperature": temperature,
27
+ "messages": [{"role": "user", "content": prompt}]}
28
+ if self.disable_thinking:
29
+ body["chat_template_kwargs"] = {"enable_thinking": False}
30
+ data = json.dumps(body).encode()
31
+ url = self.base_url.rstrip("/") + "/chat/completions"
32
+ last = "no attempt"
33
+ for attempt in range(retries):
34
+ try:
35
+ req = urllib.request.Request(url, data, {"Content-Type": "application/json", "User-Agent": "DataInventor/1.0",
36
+ "Authorization": "Bearer " + self.api_key})
37
+ resp = json.load(urllib.request.urlopen(req, timeout=180))
38
+ choice = (resp.get("choices") or [{}])[0] or {}
39
+ if choice.get("finish_reason") == "length":
40
+ last = "reply cut off at max_tokens"
41
+ time.sleep(1.5)
42
+ continue
43
+ text = ((choice.get("message") or {}).get("content") or "").strip()
44
+ text = re.sub(r"<think>.*?</think>", "", text, flags=re.S).strip()
45
+ if text and (validate is None or validate(text)):
46
+ return text
47
+ last = "reply rejected by a quality check"
48
+ except urllib.error.HTTPError as e:
49
+ detail = ""
50
+ try:
51
+ detail = e.read().decode()[:200]
52
+ except Exception:
53
+ pass
54
+ last = f"HTTP {e.code} {detail}"
55
+ if e.code in (401, 403):
56
+ raise EndpointError(f"The endpoint refused the key (HTTP {e.code}). Check the API key.")
57
+ if e.code == 404:
58
+ raise EndpointError("The endpoint or model was not found (HTTP 404). Check the base URL and model name.")
59
+ time.sleep(min(30, 2 ** attempt * 2) if e.code == 429 else 3 * (attempt + 1))
60
+ except Exception as e: # timeouts, resets
61
+ last = str(e)[:160]
62
+ time.sleep(3 * (attempt + 1))
63
+ raise EndpointError(last)
64
+
65
+ def ping(self) -> dict:
66
+ t = time.time()
67
+ text = self.chat("Reply with the single word ready.", max_tokens=16, temperature=0, retries=2)
68
+ return {"ok": True, "latency_ms": int((time.time() - t) * 1000), "sample": text[:40]}
69
+
70
+
71
+ class EndpointError(Exception):
72
+ pass
datainventor/preview.py ADDED
@@ -0,0 +1,268 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Run a recipe on a sample: every stage, small enough for a hosted app.
2
+
3
+ It reads a spread sample of each source through the dataset viewer, applies clean, calibrate, create and compose,
4
+ and reports what each stage did, with estimates scaled up to the full sources. The kept and created documents
5
+ come back too, so a sample run is also a small curated dataset.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import random
10
+ import threading
11
+ import time
12
+ from concurrent.futures import ThreadPoolExecutor
13
+ from dataclasses import dataclass, field
14
+ from typing import Callable
15
+
16
+ from . import calibrate as cal
17
+ from .clean import Cleaner, Decontaminator, benchmark_texts
18
+ from .compose import default_weights, fmt, plan_mix
19
+ from .create import chunks, invent, pick_style, rephrase, styles_for
20
+ from .llm import Endpoint, EndpointError
21
+ from .recipe import held_out_of, sources_of, targets_of
22
+ from .sources import sample
23
+ from .text import est_tokens
24
+
25
+ _BENCH_CACHE: dict[str, list[str]] = {}
26
+ _BENCH_LOCK = threading.Lock()
27
+
28
+
29
+ def bench_texts(key: str, token: str = "") -> list[str]:
30
+ with _BENCH_LOCK:
31
+ if key not in _BENCH_CACHE:
32
+ _BENCH_CACHE[key] = benchmark_texts(key, token)
33
+ return _BENCH_CACHE[key]
34
+
35
+
36
+ @dataclass
37
+ class Preview:
38
+ recipe: dict
39
+ state: str = "queued" # queued | running | done | failed | stopped
40
+ message: str = ""
41
+ stage: str = ""
42
+ report: dict = field(default_factory=dict)
43
+ docs: list = field(default_factory=list)
44
+ started: float = field(default_factory=time.time)
45
+ finished: float = 0.0
46
+ stop: threading.Event = field(default_factory=threading.Event)
47
+
48
+ def say(self, stage: str, msg: str):
49
+ self.stage, self.message = stage, msg
50
+
51
+
52
+ def _pmap(fn: Callable, items: list, workers: int, stop: threading.Event, tick: Callable[[int], None] | None = None) -> list:
53
+ done = [0]
54
+ lock = threading.Lock()
55
+
56
+ def guarded(x):
57
+ if stop.is_set():
58
+ return None
59
+ out = fn(x)
60
+ if tick:
61
+ with lock:
62
+ done[0] += 1
63
+ tick(done[0])
64
+ return out
65
+ with ThreadPoolExecutor(max_workers=max(1, workers)) as pool:
66
+ return list(pool.map(guarded, items))
67
+
68
+
69
+ def run_preview(p: Preview, ep: Endpoint, token: str = "", per_source: int = 200, grade_n: int = 32,
70
+ rephrase_n: int = 8, invent_n: int = 4, max_docs: int = 1000, seed: int = 7):
71
+ r = p.recipe
72
+ rng = random.Random(seed)
73
+ try:
74
+ p.state = "running"
75
+ rep: dict = {"sources": [], "stages": {}}
76
+
77
+ # ---- targets
78
+ targets_text: list[str] = []
79
+ for t in targets_of(r):
80
+ p.say("setup", f"Reading target {t.name}")
81
+ targets_text += [d["text"] for d in sample(t, min(100, per_source), token)]
82
+ calib = cal.Calibrator(float(r["calibrate"]["focus"]), r["targets"]["description"], targets_text)
83
+
84
+ # ---- clean
85
+ cleaner = Cleaner(r["clean"]["rules"])
86
+ dec = Decontaminator(int(r["clean"].get("ngram", 13)))
87
+ for b in r["clean"]["decontaminate"]:
88
+ p.say("clean", f"Loading the {b} test set for decontamination")
89
+ try:
90
+ dec.add(b, bench_texts(b, token))
91
+ except Exception as e:
92
+ rep.setdefault("warnings", []).append(f"Could not load {b} for decontamination: {e}")
93
+ for h in held_out_of(r):
94
+ p.say("clean", f"Loading held-out set {h.name}")
95
+ try:
96
+ dec.add(h.name, [d["text"] for d in sample(h, 2000, token)])
97
+ except Exception as e:
98
+ rep.setdefault("warnings", []).append(f"Could not load held-out set {h.name}: {e}")
99
+ dedup = cal.Deduper(near=r["calibrate"]["dedup"]["near"], threshold=float(r["calibrate"]["dedup"]["threshold"]))
100
+ survivors: list[dict] = []
101
+ for s in sources_of(r):
102
+ if p.stop.is_set():
103
+ break
104
+ p.say("clean", f"Sampling {s.name}")
105
+ raw = sample(s, per_source, token)
106
+ f = {"name": s.name, "dataset": s.dataset or s.files, "kind": s.kind, "num_rows": s.num_rows, "read": len(raw),
107
+ "dropped": {}, "decontaminated": {}, "kept": 0}
108
+ f["repeated_passages"] = cal.repeated_passages(raw)
109
+ f["raw_tokens"] = sum(est_tokens(d["text"]) for d in raw)
110
+ for d in raw:
111
+ d2, why = cleaner.apply(d)
112
+ if not d2:
113
+ f["dropped"][why] = f["dropped"].get(why, 0) + 1
114
+ continue
115
+ b = dec.hit(d2["text"])
116
+ if b:
117
+ f["decontaminated"][b] = f["decontaminated"].get(b, 0) + 1
118
+ continue
119
+ why = dedup.check(d2)
120
+ if why:
121
+ f["dropped"][why] = f["dropped"].get(why, 0) + 1
122
+ continue
123
+ survivors.append(d2)
124
+ f["kept"] += 1
125
+ rep["sources"].append(f)
126
+ rep["stages"]["clean"] = {"benchmarks": r["clean"]["decontaminate"], "test_items": len(dec)}
127
+ if not survivors:
128
+ raise ValueError("Nothing survived cleaning. Loosen the rules, or check the text column of each source.")
129
+
130
+ # ---- calibrate: grade, learn, score
131
+ by_src: dict[str, list[dict]] = {}
132
+ for d in survivors:
133
+ by_src.setdefault(d["source"], []).append(d)
134
+ to_grade = []
135
+ per = max(2, grade_n // max(1, len(by_src)))
136
+ for docs in by_src.values():
137
+ to_grade += rng.sample(docs, min(per, len(docs)))
138
+ p.say("calibrate", f"Grading {len(to_grade)} documents with the model")
139
+ desc = r["targets"]["description"]
140
+ grades = _pmap(lambda d: cal.grade(ep, d["text"], desc), to_grade, ep.workers, p.stop,
141
+ lambda n: p.say("calibrate", f"Graded {n} of {len(to_grade)} documents with the model"))
142
+ graded = [(d, g) for d, g in zip(to_grade, grades) if g]
143
+ if len(graded) < 4:
144
+ raise EndpointError("The model returned too few usable grades. Check the endpoint, or try again.")
145
+ calib.fit([(d["text"], g) for d, g in graded], [d["text"] for d in survivors])
146
+ gmap = {d["id"]: g for d, g in graded}
147
+ for d in survivors:
148
+ g = gmap.get(d["id"])
149
+ if g:
150
+ d["grade"] = g
151
+ d.update(calib.score(d["text"], g))
152
+ d["tokens"] = est_tokens(d["text"])
153
+ keep_top = float(r["calibrate"]["keep_top"])
154
+ for f in rep["sources"]:
155
+ docs = sorted(by_src.get(f["name"], []), key=lambda d: -d["score"])
156
+ k = max(1, round(len(docs) * keep_top)) if docs else 0
157
+ hi = max(1, round(len(docs) * 0.2)) if docs else 0
158
+ for i, d in enumerate(docs):
159
+ d["kept"] = i < k
160
+ d["tier"] = "high" if i < hi else "medium" if i < k else "low"
161
+ kept = [d for d in docs if d["kept"]]
162
+ f["selected"] = len(kept)
163
+ f["dropped"]["below score cut"] = len(docs) - len(kept)
164
+ rate = len(kept) / max(1, f["read"])
165
+ mean_tok = sum(d["tokens"] for d in kept) / max(1, len(kept))
166
+ f["mean_tokens"] = round(mean_tok)
167
+ f["keep_rate"] = round(rate, 3)
168
+ if f["num_rows"]:
169
+ f["available_tokens"] = int(f["num_rows"] * rate * mean_tok)
170
+ f["high_tokens"] = int(f["num_rows"] * (hi / max(1, f["read"])) * mean_tok)
171
+ f["score_hist"] = _hist([d["score"] for d in docs])
172
+ f["examples"] = {"top": [_ex(d) for d in docs[:2]], "bottom": [_ex(d) for d in docs[-2:]]}
173
+ rep["stages"]["calibrate"] = {"graded": len(graded), "agreement": calib.agreement, "focus": calib.focus, "keep_top": keep_top,
174
+ "has_targets": calib.has_targets, "grades": [{"id": d["id"], **g} for d, g in graded][:12]}
175
+
176
+ # ---- create
177
+ created: list[dict] = []
178
+ rcfg, icfg = r["create"]["rephrase"], r["create"]["invent"]
179
+ if rcfg["enabled"] and rephrase_n and not p.stop.is_set():
180
+ queues = [sorted([d for d in docs if d.get("tier") == "high"], key=lambda d: -d["score"]) for docs in by_src.values()]
181
+ picks = []
182
+ while len(picks) < rephrase_n and any(queues): # round-robin so every source is represented
183
+ for q in queues:
184
+ if q and len(picks) < rephrase_n:
185
+ picks.append(q.pop(0))
186
+ jobs = [({**d, "text": chunks(d["text"])[0]}, pick_style(d["kind"], rng, rcfg["styles"] or styles_for(d["kind"]))) for d in picks]
187
+ p.say("create", f"Rephrasing {len(jobs)} of the best documents")
188
+ outs = _pmap(lambda j: rephrase(ep, j[0]["text"], j[1], rcfg.get("language", ""), float(rcfg["max_new_share"])), jobs, ep.workers, p.stop,
189
+ lambda n: p.say("create", f"Rephrased {n} of {len(jobs)} of the best documents"))
190
+ for (d, style), o in zip(jobs, outs):
191
+ if o and o["text"]:
192
+ created.append({"id": f"{d['id']}:{style}", "text": o["text"], "source": d["source"], "kind": "synthetic",
193
+ "origin": d["id"], "origin_text": d["text"], "style": style, "fidelity": o.get("fidelity"), "kept": o["ok"],
194
+ "why": o.get("why", ""), "tokens": est_tokens(o["text"])})
195
+ if icfg["enabled"] and invent_n and icfg["request"].strip() and not p.stop.is_set():
196
+ p.say("create", "Writing new documents")
197
+ topics = plan_topics(ep, icfg["request"], invent_n)
198
+ jobs = [(t, rng.choice(icfg["doc_types"]), rng.choice(icfg["levels"])) for t in topics[:invent_n]]
199
+ outs = _pmap(lambda j: invent(ep, j[0], j[1], j[2], icfg["request"], icfg.get("language", "English")), jobs, ep.workers, p.stop)
200
+ for (t, dt, lv), o in zip(jobs, outs):
201
+ if o and o["text"]:
202
+ created.append({"id": f"invent:{len(created)}", "text": o["text"], "source": "invented", "kind": "synthetic",
203
+ "topic": t, "doc_type": dt, "level": lv, "kept": True, "tokens": est_tokens(o["text"])})
204
+ rep["stages"]["create"] = {"made": len(created), "kept": sum(1 for c in created if c["kept"]),
205
+ "flagged": [{"id": c["id"], "why": c["why"], "fidelity": c.get("fidelity")} for c in created if not c["kept"]]}
206
+
207
+ # ---- compose
208
+ src_rows = [{"name": f["name"], "kind": f["kind"], "available_tokens": f.get("available_tokens", 0),
209
+ "high_tokens": f.get("high_tokens", 0)} for f in rep["sources"]]
210
+ defaults = default_weights(src_rows)
211
+ plans = []
212
+ for m in r["compose"]["mixes"]:
213
+ w = {**defaults, **{k: float(v) for k, v in (m.get("weights") or {}).items()}}
214
+ plan = plan_mix(int(m["budget_tokens"]), src_rows, w, float(r["compose"]["max_epochs"]), float(r["compose"]["max_synthetic_share"]))
215
+ plan["name"] = m["name"]
216
+ plan["budget"] = fmt(m["budget_tokens"])
217
+ plans.append(plan)
218
+ if any(not f["num_rows"] for f in rep["sources"]):
219
+ rep.setdefault("warnings", []).append("Some sources report no row count, so their available tokens are unknown and plans treat them as empty.")
220
+ rep["stages"]["compose"] = {"default_weights": defaults, "plans": plans}
221
+
222
+ # ---- output sample
223
+ kept_docs = [d for d in survivors if d.get("kept")]
224
+ kept_docs.sort(key=lambda d: -d["score"])
225
+ out = [c for c in created if c["kept"]] + kept_docs
226
+ p.docs = [_out(d) for d in out[:max_docs]]
227
+ n_created = min(sum(1 for c in created if c["kept"]), max_docs)
228
+ rep["output"] = {"docs": len(p.docs), "curated": len(p.docs) - n_created, "created": n_created,
229
+ "tokens": sum(d["tokens"] for d in out[:max_docs])}
230
+ rep["created_examples"] = [{k: c.get(k) for k in ("id", "style", "doc_type", "topic", "kept", "why", "fidelity")} | {"text": c["text"][:1600], "origin_text": c.get("origin_text", "")[:1600]} for c in created[:6]]
231
+ p.report = rep
232
+ p.state = "stopped" if p.stop.is_set() else "done"
233
+ p.say("done", f"{len(p.docs)} documents ready" if p.state == "done" else "Stopped")
234
+ except EndpointError as e:
235
+ p.state, p.message = "failed", str(e)
236
+ except Exception as e:
237
+ p.state, p.message = "failed", str(e) or e.__class__.__name__
238
+ finally:
239
+ p.finished = time.time()
240
+
241
+
242
+ def plan_topics(ep: Endpoint, request: str, n: int) -> list[str]:
243
+ import json
244
+ import re
245
+ s = ep.chat(f"""List {n} specific concepts to write teaching documents about, for this collection: "{request}".
246
+ Spread them across the whole subject and across difficulty. Return ONLY a JSON list of strings, each 2 to 6 words.""",
247
+ max_tokens=600, temperature=0.8, validate=lambda t: bool(re.search(r"\[.*\]", t, re.S)))
248
+ try:
249
+ return [str(x) for x in json.loads(re.search(r"\[.*\]", s, re.S).group(0))][:n]
250
+ except Exception:
251
+ return [request] * n
252
+
253
+
254
+ def _hist(vals: list[float], bins: int = 10) -> list[int]:
255
+ h = [0] * bins
256
+ for v in vals:
257
+ h[min(bins - 1, int(v * bins))] += 1
258
+ return h
259
+
260
+
261
+ def _ex(d: dict) -> dict:
262
+ return {"id": d["id"], "score": d["score"], "quality": d["quality"], "relevance": d["relevance"],
263
+ "reason": (d.get("grade") or {}).get("reason", ""), "text": d["text"][:700]}
264
+
265
+
266
+ def _out(d: dict) -> dict:
267
+ keep = ("id", "text", "source", "kind", "quality", "relevance", "score", "tier", "style", "origin", "doc_type", "topic", "level", "tokens")
268
+ return {k: d[k] for k in keep if k in d}
datainventor/recipe.py ADDED
@@ -0,0 +1,108 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """A recipe is the whole curation plan in one file. The hosted app builds and previews it on a sample; the runner
2
+ executes it in full on your own machines. Fork the defaults and change any stage."""
3
+ from __future__ import annotations
4
+
5
+ import copy
6
+ import json
7
+ import os
8
+
9
+ from .clean import BENCHMARKS, DEFAULTS as CLEAN_DEFAULTS
10
+ from .create import DOC_TYPES, STYLES
11
+ from .llm import HF_BASE, HF_MODEL, Endpoint
12
+ from .sources import KINDS, Source
13
+
14
+ DEFAULT = {
15
+ "version": 1,
16
+ "name": "my-corpus",
17
+ "endpoint": {"base_url": HF_BASE, "model": HF_MODEL, "api_key_env": "DI_API_KEY", "disable_thinking": False, "workers": 16},
18
+ "sources": [],
19
+ "targets": {"description": "", "datasets": [], "max_docs": 2000},
20
+ "clean": {"rules": dict(CLEAN_DEFAULTS), "decontaminate": ["gsm8k", "math500", "humaneval", "mmlu", "arc_challenge", "hellaswag"],
21
+ "held_out": [], "ngram": 13},
22
+ "calibrate": {"dedup": {"near": True, "threshold": 0.8}, "grade_docs": 2000, "scorer": "learned", "focus": 0.3, "keep_top": 0.6},
23
+ "create": {"rephrase": {"enabled": True, "styles": [], "max_new_share": 0.35, "language": ""},
24
+ "invent": {"enabled": False, "request": "", "docs": 0, "doc_types": list(DOC_TYPES), "levels": [3, 5, 7], "language": "English"}},
25
+ "compose": {"mixes": [{"name": "main", "budget_tokens": 10_000_000_000, "weights": {}}],
26
+ "max_epochs": 4.0, "max_synthetic_share": 0.25},
27
+ "output": {"dir": "./curated", "shard_docs": 50_000},
28
+ }
29
+
30
+
31
+ def merge(base: dict, over: dict) -> dict:
32
+ out = copy.deepcopy(base)
33
+ for k, v in (over or {}).items():
34
+ if isinstance(v, dict) and isinstance(out.get(k), dict):
35
+ out[k] = merge(out[k], v)
36
+ else:
37
+ out[k] = copy.deepcopy(v)
38
+ return out
39
+
40
+
41
+ def validate(r: dict) -> dict:
42
+ """Fill defaults and check the parts a run cannot recover from. Returns the full recipe."""
43
+ r = merge(DEFAULT, r)
44
+ srcs = r["sources"]
45
+ if not srcs:
46
+ raise ValueError("Add at least one source dataset.")
47
+ names = set()
48
+ for s in srcs + r["targets"]["datasets"] + r["clean"]["held_out"]:
49
+ if not (s.get("dataset") or s.get("files")):
50
+ raise ValueError("Every dataset needs a Hugging Face id or a files glob.")
51
+ s.setdefault("name", (s.get("dataset") or s.get("files")).split("/")[-1])
52
+ if s["name"] in names:
53
+ raise ValueError(f"Two datasets share the name {s['name']}. Give each a unique name.")
54
+ names.add(s["name"])
55
+ if s.get("kind", "web") not in KINDS:
56
+ raise ValueError(f"Kind must be one of {', '.join(KINDS)}.")
57
+ unknown = [b for b in r["clean"]["decontaminate"] if b not in BENCHMARKS]
58
+ if unknown:
59
+ raise ValueError(f"Unknown benchmarks: {', '.join(unknown)}. Known: {', '.join(BENCHMARKS)}.")
60
+ bad = [s for s in r["create"]["rephrase"]["styles"] if s not in STYLES]
61
+ if bad:
62
+ raise ValueError(f"Unknown rephrase styles: {', '.join(bad)}.")
63
+ c = r["calibrate"]
64
+ if c["scorer"] not in ("learned", "llm"):
65
+ raise ValueError('calibrate.scorer must be "learned" (grade a sample, learn the rest) or "llm" (grade everything).')
66
+ if not 0 <= c["focus"] <= 1 or not 0 < c["keep_top"] <= 1:
67
+ raise ValueError("focus must be in [0, 1] and keep_top in (0, 1].")
68
+ for m in r["compose"]["mixes"]:
69
+ if int(m.get("budget_tokens", 0)) <= 0:
70
+ raise ValueError("Each mix needs a positive token budget.")
71
+ return r
72
+
73
+
74
+ def sources_of(r: dict) -> list[Source]:
75
+ return [Source.from_dict({**s, "role": "source"}) for s in r["sources"]]
76
+
77
+
78
+ def held_out_of(r: dict) -> list[Source]:
79
+ """Your own test sets for decontamination, read like any other dataset."""
80
+ return [Source.from_dict({**s, "role": "held_out"}) for s in r["clean"]["held_out"]]
81
+
82
+
83
+ def targets_of(r: dict) -> list[Source]:
84
+ return [Source.from_dict({**s, "role": "target"}) for s in r["targets"]["datasets"]]
85
+
86
+
87
+ def endpoint_of(r: dict, workers: int | None = None) -> Endpoint:
88
+ e = r["endpoint"]
89
+ key = os.environ.get(e.get("api_key_env") or "DI_API_KEY") or os.environ.get("HF_TOKEN", "")
90
+ return Endpoint(base_url=e["base_url"], api_key=key, model=e["model"], disable_thinking=bool(e.get("disable_thinking")),
91
+ workers=int(workers or e.get("workers", 16)))
92
+
93
+
94
+ def dump(r: dict) -> str:
95
+ """YAML when PyYAML is installed (easier to edit), JSON otherwise. Both load with load()."""
96
+ try:
97
+ import yaml
98
+ return yaml.safe_dump(r, sort_keys=False, allow_unicode=True, width=110)
99
+ except ImportError:
100
+ return json.dumps(r, indent=2, ensure_ascii=False)
101
+
102
+
103
+ def load(path: str) -> dict:
104
+ text = open(path, encoding="utf-8").read()
105
+ if path.endswith((".yaml", ".yml")):
106
+ import yaml
107
+ return validate(yaml.safe_load(text))
108
+ return validate(json.loads(text))
datainventor/runner.py ADDED
@@ -0,0 +1,497 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Run a recipe in full on your own machines.
2
+
3
+ datainventor run recipe.yaml --out ./curated --procs 16 --workers 32
4
+
5
+ Stages, each resumable (re-run the same command after an interruption):
6
+ grade grade a sample of every source with your model, fit the scorers -> calibrator.json, grades.jsonl
7
+ curate stream every document: clean, decontaminate, dedup, score -> curated/<source>/part-*.jsonl
8
+ create rephrase the best documents where a mix needs more tokens, invent -> synthetic/<source>/part-*.jsonl
9
+ compose build each mix to its token budget -> mixes/<mix>/part-*.jsonl
10
+ Then REPORT.md and report.json summarize what every stage did.
11
+
12
+ Token counts are estimates (four characters per token). Calibrate them against your tokenizer if the budget is tight.
13
+ """
14
+ from __future__ import annotations
15
+
16
+ import itertools
17
+ import json
18
+ import os
19
+ import random
20
+ import shutil
21
+ import time
22
+ from concurrent.futures import ThreadPoolExecutor
23
+ from multiprocessing import get_context
24
+
25
+ from . import calibrate as cal
26
+ from .clean import Cleaner, Decontaminator, benchmark_texts
27
+ from .compose import default_weights, fmt, plan_mix
28
+ from .create import chunks, invent, rephrase, styles_for
29
+ from .preview import plan_topics
30
+ from .recipe import dump, endpoint_of, held_out_of, sources_of, targets_of
31
+ from .sources import Source, stream
32
+ from .text import MinHasher, NearDupIndex, doc_hash, est_tokens, h64
33
+
34
+ STAGES = ("grade", "curate", "create", "compose")
35
+
36
+
37
+ class Shards:
38
+ """Append-only JSONL shards of a fixed number of documents."""
39
+
40
+ def __init__(self, folder: str, per: int):
41
+ os.makedirs(folder, exist_ok=True)
42
+ self.folder, self.per, self.n, self.f, self.count = folder, per, 0, None, 0
43
+
44
+ def write(self, doc: dict):
45
+ if self.f is None or self.count >= self.per:
46
+ if self.f:
47
+ self.f.close()
48
+ self.f = open(os.path.join(self.folder, f"part-{self.n:05d}.jsonl"), "w", encoding="utf-8")
49
+ self.n += 1
50
+ self.count = 0
51
+ self.f.write(json.dumps(doc, ensure_ascii=False) + "\n")
52
+ self.count += 1
53
+
54
+ def close(self):
55
+ if self.f:
56
+ self.f.close()
57
+ self.f = None
58
+
59
+
60
+ def read_dir(folder: str):
61
+ if not os.path.isdir(folder):
62
+ return
63
+ for name in sorted(os.listdir(folder)):
64
+ if name.endswith(".jsonl"):
65
+ with open(os.path.join(folder, name), encoding="utf-8") as f:
66
+ for line in f:
67
+ if line.strip():
68
+ yield json.loads(line)
69
+
70
+
71
+ class Run:
72
+ def __init__(self, recipe: dict, out: str, procs: int = 0, workers: int | None = None, limit: int = 0, log=print):
73
+ self.r, self.out, self.limit, self.log = recipe, os.path.abspath(out), limit, log
74
+ self.procs = procs or max(1, (os.cpu_count() or 2) - 1)
75
+ self.ep = endpoint_of(recipe, workers)
76
+ os.makedirs(self.out, exist_ok=True)
77
+ self.state_path = os.path.join(self.out, "state.json")
78
+ self.state = json.load(open(self.state_path)) if os.path.exists(self.state_path) else {"done": [], "sources": {}}
79
+ with open(os.path.join(self.out, "recipe.yaml"), "w", encoding="utf-8") as f:
80
+ f.write(dump(recipe))
81
+
82
+ def save(self):
83
+ tmp = self.state_path + ".tmp"
84
+ with open(tmp, "w") as f:
85
+ json.dump(self.state, f, indent=1)
86
+ os.replace(tmp, self.state_path)
87
+
88
+ def path(self, *p) -> str:
89
+ return os.path.join(self.out, *p)
90
+
91
+ def go(self, stages=STAGES):
92
+ if not self.ep.api_key and self.r["endpoint"]["base_url"].startswith("https://router.huggingface.co"):
93
+ raise SystemExit("Set DI_API_KEY (or HF_TOKEN) for the model endpoint.")
94
+ for st in STAGES:
95
+ if st not in stages:
96
+ continue
97
+ if st in self.state["done"]:
98
+ self.log(f"[{st}] already done, skipping (delete {self.state_path} to start over)")
99
+ continue
100
+ t = time.time()
101
+ getattr(self, "stage_" + st)()
102
+ self.state["done"].append(st)
103
+ self.save()
104
+ self.log(f"[{st}] done in {time.time() - t:.0f}s")
105
+ self.write_report()
106
+
107
+ # ------------------------------------------------------------ grade
108
+ def stage_grade(self):
109
+ r = self.r
110
+ n_total = int(r["calibrate"]["grade_docs"])
111
+ srcs = sources_of(r)
112
+ per = max(4, n_total // len(srcs))
113
+ cleaner = Cleaner(r["clean"]["rules"])
114
+ pool: list[dict] = []
115
+ for s in srcs:
116
+ got = []
117
+ for d in stream(s):
118
+ d2, why = cleaner.apply(d)
119
+ if d2:
120
+ got.append(d2)
121
+ if len(got) >= per * 3:
122
+ break
123
+ pool += random.Random(1).sample(got, min(per, len(got)))
124
+ self.log(f"[grade] {s.name}: sampled {min(per, len(got))} documents")
125
+ gpath = self.path("grades.jsonl")
126
+ done = {}
127
+ if os.path.exists(gpath):
128
+ for line in open(gpath, encoding="utf-8"):
129
+ g = json.loads(line)
130
+ done[g["id"]] = g
131
+ todo = [d for d in pool if d["id"] not in done]
132
+ desc = r["targets"]["description"]
133
+ self.log(f"[grade] grading {len(todo)} documents ({len(done)} already graded) with {self.ep.model}")
134
+ with open(gpath, "a", encoding="utf-8") as f, ThreadPoolExecutor(self.ep.workers) as ex:
135
+ for i, (d, g) in enumerate(zip(todo, ex.map(lambda d: cal.grade(self.ep, d["text"], desc), todo))):
136
+ if g:
137
+ rec = {"id": d["id"], "source": d["source"], "text": d["text"][:12000], **g}
138
+ f.write(json.dumps(rec, ensure_ascii=False) + "\n")
139
+ done[d["id"]] = rec
140
+ if (i + 1) % 200 == 0:
141
+ f.flush()
142
+ self.log(f"[grade] {i + 1}/{len(todo)}")
143
+ graded = [(g["text"], g) for g in done.values()]
144
+ target_texts = []
145
+ for t in targets_of(r):
146
+ for i, d in enumerate(stream(t)):
147
+ if i >= int(r["targets"]["max_docs"]):
148
+ break
149
+ target_texts.append(d["text"])
150
+ calib = cal.Calibrator(float(r["calibrate"]["focus"]), desc, target_texts).fit(graded, [d["text"] for d in pool])
151
+ with open(self.path("calibrator.json"), "w") as f:
152
+ json.dump(calib.to_dict(), f)
153
+ self.state["grade"] = {"graded": len(graded), "agreement": calib.agreement}
154
+ self.log(f"[grade] scorer agreement with the model on held-out documents: {calib.agreement}")
155
+
156
+ # ------------------------------------------------------------ curate
157
+ def stage_curate(self):
158
+ r = self.r
159
+ dec_keys = r["clean"]["decontaminate"]
160
+ index = NearDupIndex(MinHasher(), threshold=float(r["calibrate"]["dedup"]["threshold"])) if r["calibrate"]["dedup"]["near"] else None
161
+ exact: set = set()
162
+ ctx = get_context("fork") if hasattr(os, "fork") else get_context("spawn")
163
+ held = {}
164
+ for h in held_out_of(r):
165
+ held[h.name] = [d["text"] for _, d in zip(range(200_000), stream(h))]
166
+ self.log(f"[curate] held-out set {h.name}: {len(held[h.name]):,} items")
167
+ init = (r["clean"]["rules"], dec_keys, held, int(r["clean"].get("ngram", 13)), self.path("calibrator.json"), bool(index))
168
+ with ctx.Pool(self.procs, initializer=_worker_init, initargs=init) as pool:
169
+ for s in sources_of(r):
170
+ st = self.state["sources"].get(s.name, {})
171
+ folder = self.path("curated", s.name)
172
+ if st.get("done"):
173
+ self.log(f"[curate] {s.name}: done earlier, loading it into the dedup index")
174
+ for d in read_dir(folder):
175
+ exact.add(d["h"])
176
+ if index is not None:
177
+ for sig, d in zip(pool.imap(_sig_only, (d["text"] for d in read_dir(folder)), chunksize=64), read_dir(folder)):
178
+ index.check_add_sig(h64(d["h"]), sig)
179
+ continue
180
+ if os.path.isdir(folder):
181
+ shutil.rmtree(folder)
182
+ sh = Shards(folder, int(r["output"]["shard_docs"]))
183
+ c = {"read": 0, "kept": 0, "dropped": {}, "decontaminated": {}, "tokens": 0}
184
+ scores: list[float] = []
185
+ t0 = time.time()
186
+ llm_all = r["calibrate"].get("scorer") == "llm"
187
+ calib = cal.Calibrator.from_dict(json.load(open(self.path("calibrator.json")))) if llm_all else None
188
+ ex = ThreadPoolExecutor(self.ep.workers) if llm_all else None
189
+ desc = r["targets"]["description"]
190
+ for batch in pool.imap(_worker_batch, _batches(stream(s), 256, self.limit), chunksize=1):
191
+ if llm_all: # grade every surviving document with the model instead of the learned scorer
192
+ live = [d for d, why, _ in batch if d]
193
+ for d, g in zip(live, ex.map(lambda d: cal.grade(self.ep, d["text"], desc), live)):
194
+ if g:
195
+ d.update(calib.score(d["text"], g))
196
+ for d, why, sig in batch:
197
+ c["read"] += 1
198
+ if why:
199
+ key = "decontaminated" if why.startswith("bench:") else "dropped"
200
+ name = why[6:] if key == "decontaminated" else why
201
+ c[key][name] = c[key].get(name, 0) + 1
202
+ continue
203
+ if d["h"] in exact:
204
+ c["dropped"]["exact duplicate"] = c["dropped"].get("exact duplicate", 0) + 1
205
+ continue
206
+ exact.add(d["h"])
207
+ if index is not None and index.check_add_sig(h64(d["h"]), sig) is not None:
208
+ c["dropped"]["near duplicate"] = c["dropped"].get("near duplicate", 0) + 1
209
+ continue
210
+ sh.write(d)
211
+ c["kept"] += 1
212
+ c["tokens"] += d["tokens"]
213
+ scores.append(d["score"])
214
+ if c["read"] % 50_000 < 256:
215
+ rate = c["read"] / max(1e-6, time.time() - t0)
216
+ self.log(f"[curate] {s.name}: {c['read']:,} read, {c['kept']:,} kept, {rate:,.0f} docs/s")
217
+ sh.close()
218
+ if ex:
219
+ ex.shutdown()
220
+ scores.sort(reverse=True)
221
+ keep_top, n = float(r["calibrate"]["keep_top"]), len(scores)
222
+ c["cut"] = scores[max(0, int(n * keep_top) - 1)] if n else 1.0
223
+ c["high_cut"] = scores[max(0, int(n * 0.2) - 1)] if n else 1.0
224
+ c["selected_tokens"], c["high_tokens"] = 0, 0
225
+ for d in read_dir(folder):
226
+ if d["score"] >= c["cut"]:
227
+ c["selected_tokens"] += d["tokens"]
228
+ if d["score"] >= c["high_cut"]:
229
+ c["high_tokens"] += d["tokens"]
230
+ c["done"] = True
231
+ self.state["sources"][s.name] = c
232
+ self.save()
233
+ self.log(f"[curate] {s.name}: kept {c['kept']:,} of {c['read']:,}; {fmt(c['selected_tokens'])} tokens above the score cut")
234
+
235
+ # ------------------------------------------------------------ create
236
+ def plans(self, synthetic: dict | None = None) -> list[dict]:
237
+ rows = [{"name": s.name, "kind": s.kind, "available_tokens": self.state["sources"].get(s.name, {}).get("selected_tokens", 0),
238
+ "high_tokens": self.state["sources"].get(s.name, {}).get("high_tokens", 0)} for s in sources_of(self.r)]
239
+ defaults = default_weights(rows)
240
+ out = []
241
+ for m in self.r["compose"]["mixes"]:
242
+ w = {**defaults, **{k: float(v) for k, v in (m.get("weights") or {}).items()}}
243
+ inv = float(m.get("invented_share", 0))
244
+ p = plan_mix(int(int(m["budget_tokens"]) * (1 - inv)), rows, w, float(self.r["compose"]["max_epochs"]), float(self.r["compose"]["max_synthetic_share"]))
245
+ p["name"], p["weights"], p["invented_share"], p["budget_tokens"] = m["name"], w, inv, int(m["budget_tokens"])
246
+ out.append(p)
247
+ return out
248
+
249
+ def stage_create(self):
250
+ r = self.r
251
+ rc, ic = r["create"]["rephrase"], r["create"]["invent"]
252
+ need: dict[str, int] = {}
253
+ for p in self.plans():
254
+ for row in p["rows"]:
255
+ need[row["name"]] = max(need.get(row["name"], 0), row["synthetic_tokens"])
256
+ made = self.state.setdefault("synthetic", {})
257
+ rng = random.Random(3)
258
+ if rc["enabled"]:
259
+ for s in sources_of(r):
260
+ want = need.get(s.name, 0)
261
+ have = made.get(s.name, {}).get("tokens", 0)
262
+ if want <= have:
263
+ continue
264
+ self.log(f"[create] {s.name}: rephrasing to add {fmt(want - have)} tokens")
265
+ self._rephrase_source(s, want, rc, rng)
266
+ if ic["enabled"] and int(ic["docs"]) > 0 and ic["request"].strip():
267
+ self._invent(ic, rng)
268
+
269
+ def _rephrase_source(self, s: Source, want: int, rc: dict, rng: random.Random):
270
+ folder = self.path("synthetic", s.name)
271
+ st = self.state["synthetic"].setdefault(s.name, {"tokens": 0, "docs": 0, "flagged": 0, "seen": 0})
272
+ cut = self.state["sources"][s.name]["high_cut"]
273
+ styles = rc["styles"] or styles_for(s.kind)
274
+
275
+ def gen():
276
+ # each pass over the best documents uses a different style, so later passes add variety, not copies
277
+ k = 0
278
+ for pass_ in range(len(styles)):
279
+ for d in read_dir(self.path("curated", s.name)):
280
+ if d["score"] >= cut:
281
+ for i, ch in enumerate(chunks(d["text"])):
282
+ yield d["id"], i, ch, styles[(pass_ + k) % len(styles)]
283
+ k += 1
284
+ jobs = itertools.islice(gen(), st["seen"], None)
285
+ sh = Shards(folder, int(self.r["output"]["shard_docs"]))
286
+ sh.n = len([f for f in os.listdir(folder)]) if os.path.isdir(folder) else 0
287
+ step = self.ep.workers * 4
288
+ with ThreadPoolExecutor(self.ep.workers) as ex:
289
+ while st["tokens"] < want:
290
+ part = list(itertools.islice(jobs, step))
291
+ if not part:
292
+ break
293
+ for (oid, i, ch, style), o in zip(part, ex.map(lambda j: rephrase(self.ep, j[2], j[3], rc.get("language", ""), float(rc["max_new_share"])), part)):
294
+ st["seen"] += 1
295
+ if not o["text"]:
296
+ continue
297
+ if not o["ok"]:
298
+ st["flagged"] += 1
299
+ continue
300
+ t = est_tokens(o["text"])
301
+ sh.write({"id": f"{oid}:{i}:{style}", "text": o["text"], "source": s.name, "kind": "synthetic", "origin": oid,
302
+ "style": style, "tokens": t, "fidelity": o["fidelity"]})
303
+ st["tokens"] += t
304
+ st["docs"] += 1
305
+ sh.close()
306
+ self.save()
307
+ self.log(f"[create] {s.name}: {fmt(st['tokens'])} of {fmt(want)} tokens, {st['flagged']} flagged")
308
+ if st["tokens"] < want:
309
+ self.log(f"[create] {s.name}: ran out of documents to rephrase at {fmt(st['tokens'])} tokens")
310
+
311
+ def _invent(self, ic: dict, rng: random.Random):
312
+ st = self.state["synthetic"].setdefault("invented", {"tokens": 0, "docs": 0, "flagged": 0, "seen": 0})
313
+ n = int(ic["docs"]) - st["docs"]
314
+ if n <= 0:
315
+ return
316
+ topics = plan_topics(self.ep, ic["request"], min(200, max(10, n // 5)))
317
+ jobs = [(rng.choice(topics), rng.choice(ic["doc_types"]), rng.choice(ic["levels"])) for _ in range(n)]
318
+ sh = Shards(self.path("synthetic", "invented"), int(self.r["output"]["shard_docs"]))
319
+ sh.n = len(os.listdir(self.path("synthetic", "invented")))
320
+ self.log(f"[create] inventing {n} documents over {len(topics)} topics")
321
+ with ThreadPoolExecutor(self.ep.workers) as ex:
322
+ for (t, dt, lv), o in zip(jobs, ex.map(lambda j: invent(self.ep, j[0], j[1], j[2], ic["request"], ic.get("language", "English")), jobs)):
323
+ if o.get("text"):
324
+ tk = est_tokens(o["text"])
325
+ sh.write({"id": f"invented:{st['docs']}", "text": o["text"], "source": "invented", "kind": "synthetic",
326
+ "topic": t, "doc_type": dt, "level": lv, "tokens": tk})
327
+ st["docs"] += 1
328
+ st["tokens"] += tk
329
+ sh.close()
330
+
331
+ # ------------------------------------------------------------ compose
332
+ def stage_compose(self):
333
+ r = self.r
334
+ synth = self.state.get("synthetic", {})
335
+ results = []
336
+ for p in self.plans():
337
+ name = p["name"]
338
+ folder = self.path("mixes", name)
339
+ if os.path.isdir(folder):
340
+ shutil.rmtree(folder)
341
+ sh = Shards(folder, int(r["output"]["shard_docs"]))
342
+ buckets = []
343
+ for row in p["rows"]:
344
+ s = row["name"]
345
+ cut = self.state["sources"].get(s, {}).get("cut", 1.0)
346
+ real_target = row["want_tokens"] - min(row["synthetic_tokens"], synth.get(s, {}).get("tokens", 0))
347
+ buckets.append(_Bucket(s, self.path("curated", s), cut, real_target, float(r["compose"]["max_epochs"]),
348
+ self.state["sources"].get(s, {}).get("selected_tokens", 0)))
349
+ if row["synthetic_tokens"] and synth.get(s, {}).get("tokens"):
350
+ buckets.append(_Bucket(s + ":synthetic", self.path("synthetic", s), -1, min(row["synthetic_tokens"], synth[s]["tokens"]), 1.0,
351
+ synth[s]["tokens"]))
352
+ share = float(p.get("invented_share", 0))
353
+ if share and synth.get("invented", {}).get("tokens"):
354
+ buckets.append(_Bucket("invented", self.path("synthetic", "invented"), -1, int(p["budget_tokens"] * share), 1.0,
355
+ synth["invented"]["tokens"]))
356
+ rng = random.Random(11)
357
+ buf: list[dict] = []
358
+ live = [b for b in buckets if b.target > 0]
359
+ while live:
360
+ # draw the next bucket in proportion to the tokens it still owes, which interleaves sources evenly
361
+ b = rng.choices(live, weights=[x.target - x.taken for x in live], k=1)[0]
362
+ d = b.next()
363
+ if d is None:
364
+ live.remove(b)
365
+ continue
366
+ buf.append({"id": d["id"], "text": d["text"], "source": d["source"], "kind": d.get("kind", "")})
367
+ if len(buf) >= 20_000:
368
+ rng.shuffle(buf)
369
+ for x in buf[:10_000]:
370
+ sh.write(x)
371
+ buf = buf[10_000:]
372
+ if b.taken >= b.target:
373
+ live.remove(b)
374
+ rng.shuffle(buf)
375
+ for x in buf:
376
+ sh.write(x)
377
+ sh.close()
378
+ manifest = {"name": name, "budget_tokens": p["budget_tokens"], "weights": p["weights"],
379
+ "buckets": [{"name": b.name, "target_tokens": b.target, "tokens": b.taken, "docs": b.docs, "epochs": b.epochs_used()} for b in buckets],
380
+ "tokens": sum(b.taken for b in buckets), "warnings": p["warnings"]}
381
+ with open(os.path.join(folder, "manifest.json"), "w") as f:
382
+ json.dump(manifest, f, indent=1)
383
+ results.append(manifest)
384
+ self.log(f"[compose] {name}: {fmt(manifest['tokens'])} tokens of {fmt(p['budget_tokens'])}")
385
+ self.state["mixes"] = results
386
+
387
+ # ------------------------------------------------------------ report
388
+ def write_report(self):
389
+ st = self.state
390
+ lines = [f"# Curation report: {self.r['name']}", "", f"Model: `{self.ep.model}`. Stages done: {', '.join(st['done']) or 'none'}.", ""]
391
+ if st.get("grade"):
392
+ a = st["grade"]["agreement"]
393
+ lines += [f"Graded {st['grade']['graded']:,} documents. Scorer rank agreement with the model on held-out documents: "
394
+ f"quality {a.get('quality')}, relevance {a.get('relevance')}.", ""]
395
+ if st["sources"]:
396
+ lines += ["## Sources", "", "| Source | Read | Kept | Decontaminated | Top drop reasons | Tokens above cut |", "|---|---|---|---|---|---|"]
397
+ for n, c in st["sources"].items():
398
+ top = ", ".join(f"{k} {v:,}" for k, v in sorted(c["dropped"].items(), key=lambda x: -x[1])[:3])
399
+ dec = sum(c["decontaminated"].values())
400
+ lines.append(f"| {n} | {c['read']:,} | {c['kept']:,} | {dec:,} | {top} | {fmt(c.get('selected_tokens', 0))} |")
401
+ lines.append("")
402
+ if st.get("synthetic"):
403
+ lines += ["## Created", "", "| From | Documents | Tokens | Flagged |", "|---|---|---|---|"]
404
+ for n, c in st["synthetic"].items():
405
+ lines.append(f"| {n} | {c['docs']:,} | {fmt(c['tokens'])} | {c['flagged']:,} |")
406
+ lines.append("")
407
+ for m in st.get("mixes", []):
408
+ lines += [f"## Mix `{m['name']}`: {fmt(m['tokens'])} of {fmt(m['budget_tokens'])} tokens", "",
409
+ "| Bucket | Target | Written | Epochs |", "|---|---|---|---|"]
410
+ for b in m["buckets"]:
411
+ lines.append(f"| {b['name']} | {fmt(b['target_tokens'])} | {fmt(b['tokens'])} | {b['epochs']} |")
412
+ lines += [""] + [f"- {w}" for w in m["warnings"]] + [""]
413
+ with open(self.path("REPORT.md"), "w") as f:
414
+ f.write("\n".join(lines))
415
+ with open(self.path("report.json"), "w") as f:
416
+ json.dump(st, f, indent=1)
417
+
418
+
419
+ class _Bucket:
420
+ """Documents of one source above its score cut, cycled up to max_epochs times."""
421
+
422
+ def __init__(self, name: str, folder: str, cut: float, target: int, max_epochs: float, pass_tokens: int = 0):
423
+ self.name, self.folder, self.cut, self.target, self.max_epochs = name, folder, cut, int(target), max_epochs
424
+ self.pass_tokens = pass_tokens
425
+ self.taken, self.docs, self.epoch, self.it, self.epoch_tokens = 0, 0, 0, None, 0
426
+
427
+ def next(self):
428
+ while True:
429
+ if self.it is None:
430
+ if self.epoch >= max(1, int(self.max_epochs + 0.999)):
431
+ return None
432
+ self.it = iter(read_dir(self.folder))
433
+ self.epoch += 1
434
+ for d in self.it:
435
+ if d.get("score", 1.0) >= self.cut:
436
+ self.taken += d.get("tokens") or est_tokens(d["text"])
437
+ self.docs += 1
438
+ return d
439
+ self.it = None
440
+ if self.taken == 0:
441
+ return None
442
+
443
+ def epochs_used(self) -> float:
444
+ return round(self.taken / self.pass_tokens, 2) if self.pass_tokens else float(self.epoch)
445
+
446
+
447
+ # ---------------------------------------------------------------- worker processes
448
+
449
+ _W: dict = {}
450
+
451
+
452
+ def _worker_init(rules, dec_keys, held, ngram, calib_path, near):
453
+ _W["clean"] = Cleaner(rules)
454
+ dec = Decontaminator(ngram)
455
+ for k in dec_keys:
456
+ dec.add(k, benchmark_texts(k))
457
+ for k, texts in held.items():
458
+ dec.add(k, texts)
459
+ _W["dec"] = dec
460
+ _W["calib"] = cal.Calibrator.from_dict(json.load(open(calib_path)))
461
+ _W["mh"] = MinHasher() if near else None
462
+
463
+
464
+ def _worker_batch(batch: list[dict]) -> list[tuple]:
465
+ out = []
466
+ for d in batch:
467
+ d2, why = _W["clean"].apply(d)
468
+ if not d2:
469
+ out.append((None, why, None))
470
+ continue
471
+ b = _W["dec"].hit(d2["text"])
472
+ if b:
473
+ out.append((None, "bench:" + b, None))
474
+ continue
475
+ sc = _W["calib"].score(d2["text"])
476
+ rec = {**d2, **sc, "tokens": est_tokens(d2["text"]), "h": doc_hash(d2["text"])}
477
+ out.append((rec, "", _W["mh"].signature(d2["text"]) if _W["mh"] else None))
478
+ return out
479
+
480
+
481
+ def _sig_only(text: str):
482
+ return _W["mh"].signature(text) if _W.get("mh") else None
483
+
484
+
485
+ def _batches(it, n: int, limit: int = 0):
486
+ buf, seen = [], 0
487
+ for d in it:
488
+ buf.append(d)
489
+ seen += 1
490
+ if len(buf) >= n:
491
+ yield buf
492
+ buf = []
493
+ if limit and seen >= limit:
494
+ break
495
+ if buf:
496
+ yield buf
497
+
datainventor/sources.py ADDED
@@ -0,0 +1,192 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Where documents come from.
2
+
3
+ Two readers share one Source spec:
4
+ - sample(): a spread sample through the Hugging Face dataset viewer API. No downloads, used by the hosted preview.
5
+ - stream(): every document, through `datasets` streaming or local JSONL/Parquet files. Used by the runner on your own machines.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import glob
10
+ import json
11
+ import os
12
+ import time
13
+ import urllib.error
14
+ import urllib.parse
15
+ import urllib.request
16
+ from dataclasses import asdict, dataclass, field
17
+ from typing import Iterator
18
+
19
+ VIEWER = "https://datasets-server.huggingface.co"
20
+ KINDS = ("web", "educational", "textbook", "math", "code", "multilingual", "synthetic", "other")
21
+ TEXT_FIELDS = ("text", "content", "markdown", "body", "document", "raw_content", "code", "problem", "question", "prompt")
22
+
23
+
24
+ @dataclass
25
+ class Source:
26
+ """One dataset, registered as a source to curate or as a target that says what good looks like."""
27
+ name: str
28
+ dataset: str = "" # Hugging Face id, e.g. HuggingFaceFW/fineweb-edu
29
+ config: str = "default"
30
+ split: str = "train"
31
+ text_field: str = "text"
32
+ files: str = "" # local glob instead of a hub id: /data/*.jsonl or /data/*.parquet
33
+ kind: str = "web"
34
+ role: str = "source" # source | target
35
+ num_rows: int = 0 # filled by inspect(); 0 means unknown
36
+ bytes: int = 0
37
+ extra: dict = field(default_factory=dict)
38
+
39
+ def to_dict(self) -> dict:
40
+ return {k: v for k, v in asdict(self).items() if v not in ("", 0, {}, None)}
41
+
42
+ @staticmethod
43
+ def from_dict(d: dict) -> "Source":
44
+ keys = Source.__dataclass_fields__.keys()
45
+ return Source(**{k: v for k, v in d.items() if k in keys})
46
+
47
+
48
+ def _get(path: str, params: dict, token: str = "", timeout: int = 60, retries: int = 7) -> dict:
49
+ url = f"{VIEWER}/{path}?{urllib.parse.urlencode(params)}"
50
+ headers = {"User-Agent": "DataInventor/1.0"}
51
+ if token:
52
+ headers["Authorization"] = "Bearer " + token
53
+ for attempt in range(retries):
54
+ try:
55
+ return json.load(urllib.request.urlopen(urllib.request.Request(url, headers=headers), timeout=timeout))
56
+ except urllib.error.HTTPError as e:
57
+ if e.code not in (429, 500, 502, 503, 504) or attempt == retries - 1:
58
+ raise
59
+ time.sleep(min(30, 2 * 2 ** attempt))
60
+ except (TimeoutError, urllib.error.URLError):
61
+ if attempt == retries - 1:
62
+ raise
63
+ time.sleep(2)
64
+ raise RuntimeError("unreachable")
65
+
66
+
67
+ def inspect(dataset: str, token: str = "") -> dict:
68
+ """Configs, splits, string columns and sizes for a hub dataset, so the user can pick what to read."""
69
+ splits = _get("splits", {"dataset": dataset}, token).get("splits", [])
70
+ if not splits:
71
+ raise ValueError("The dataset viewer has no splits for that dataset. Check the id, or that it is public.")
72
+ first = splits[0]
73
+ cfg, spl = first["config"], first["split"]
74
+ for s in splits: # prefer a train split of the default config
75
+ if s["split"] == "train" and s["config"] in ("default", cfg):
76
+ cfg, spl = s["config"], s["split"]
77
+ break
78
+ cols, sample = [], ""
79
+ try:
80
+ fr = _get("first-rows", {"dataset": dataset, "config": cfg, "split": spl}, token)
81
+ cols = [f["name"] for f in fr.get("features", []) if (f.get("type") or {}).get("dtype") in ("string", "large_string")]
82
+ if fr.get("rows"):
83
+ sample = fr["rows"][0]["row"]
84
+ except Exception:
85
+ pass
86
+ field_ = next((c for c in TEXT_FIELDS if c in cols), cols[0] if cols else "text")
87
+ sizes = {}
88
+ try:
89
+ for c in _get("size", {"dataset": dataset}, token).get("size", {}).get("splits", []):
90
+ sizes[(c["config"], c["split"])] = (c.get("num_rows", 0), c.get("num_bytes_parquet_files", 0))
91
+ except Exception:
92
+ pass
93
+ rows, nbytes = sizes.get((cfg, spl), (0, 0))
94
+ preview = str((sample or {}).get(field_, ""))[:4000] if isinstance(sample, dict) else ""
95
+ return {"dataset": dataset, "config": cfg, "split": spl, "splits": [{"config": s["config"], "split": s["split"]} for s in splits][:40],
96
+ "columns": cols, "text_field": field_, "num_rows": rows, "bytes": nbytes, "preview": preview,
97
+ "kind": guess_kind(dataset)}
98
+
99
+
100
+ def guess_kind(dataset: str) -> str:
101
+ d = dataset.lower().split("/")[-1] # the owner name says nothing about the content
102
+ if any(k in d for k in ("math", "gsm", "proof", "numina")):
103
+ return "math"
104
+ if any(k in d for k in ("stack", "code", "github")):
105
+ return "code"
106
+ if any(k in d for k in ("pdf", "textbook", "sutra", "cosmopedia", "wiki")):
107
+ return "textbook"
108
+ if "edu" in d:
109
+ return "educational"
110
+ if any(k in d for k in ("fineweb-2", "multiling", "culturax", "madlad", "hplt")):
111
+ return "multilingual"
112
+ return "web"
113
+
114
+
115
+ def sample(src: Source, n: int, token: str = "", chunks: int = 4) -> list[dict]:
116
+ """Up to n documents read from `chunks` evenly spaced offsets, so a sample is not just the first shard."""
117
+ if src.files:
118
+ out = []
119
+ for d in stream(src):
120
+ out.append(d)
121
+ if len(out) >= n:
122
+ break
123
+ return out
124
+ total = src.num_rows or 0
125
+ per = max(1, min(100, -(-n // chunks)))
126
+ offsets = [0] if total <= n else [int(total * (i + 0.5) / chunks) for i in range(chunks)]
127
+ out, seen = [], set()
128
+ for off in offsets:
129
+ try:
130
+ got = _get("rows", {"dataset": src.dataset, "config": src.config, "split": src.split, "offset": off, "length": per}, token)
131
+ except Exception as e:
132
+ if not out and off == offsets[-1]:
133
+ raise ValueError(f"Could not read rows from {src.dataset}: {e}")
134
+ continue
135
+ for r in got.get("rows", []):
136
+ idx = r.get("row_idx", off)
137
+ text = (r.get("row") or {}).get(src.text_field)
138
+ if isinstance(text, str) and text.strip() and idx not in seen:
139
+ seen.add(idx)
140
+ out.append({"id": f"{src.name}:{idx}", "text": text, "source": src.name, "kind": src.kind})
141
+ if len(out) >= n:
142
+ break
143
+ return out[:n]
144
+
145
+
146
+ def stream(src: Source, skip: int = 0) -> Iterator[dict]:
147
+ """Every document in the source. Local files are read directly; hub datasets stream through `datasets`."""
148
+ if src.files:
149
+ paths = sorted(glob.glob(os.path.expanduser(src.files)))
150
+ if not paths:
151
+ raise ValueError(f"No files match {src.files}")
152
+ i = 0
153
+ for p in paths:
154
+ for row in _read_file(p):
155
+ text = row.get(src.text_field)
156
+ if isinstance(text, str) and text.strip():
157
+ if i >= skip:
158
+ yield {"id": f"{src.name}:{i}", "text": text, "source": src.name, "kind": src.kind}
159
+ i += 1
160
+ return
161
+ try:
162
+ from datasets import load_dataset
163
+ except ImportError as e: # pragma: no cover
164
+ raise SystemExit("Reading a hub dataset in full needs `pip install datasets`.") from e
165
+ ds = load_dataset(src.dataset, None if src.config == "default" else src.config, split=src.split, streaming=True)
166
+ if skip:
167
+ ds = ds.skip(skip)
168
+ for i, row in enumerate(ds, start=skip):
169
+ text = row.get(src.text_field)
170
+ if isinstance(text, str) and text.strip():
171
+ yield {"id": f"{src.name}:{i}", "text": text, "source": src.name, "kind": src.kind}
172
+
173
+
174
+ def _read_file(path: str) -> Iterator[dict]:
175
+ if path.endswith((".jsonl", ".json")):
176
+ with open(path, encoding="utf-8") as f:
177
+ for line in f:
178
+ line = line.strip()
179
+ if line:
180
+ try:
181
+ yield json.loads(line)
182
+ except json.JSONDecodeError:
183
+ continue
184
+ elif path.endswith(".parquet"):
185
+ import pyarrow.parquet as pq
186
+ for batch in pq.ParquetFile(path).iter_batches(batch_size=2048):
187
+ yield from batch.to_pylist()
188
+ elif path.endswith(".txt"):
189
+ with open(path, encoding="utf-8") as f:
190
+ yield {"text": f.read()}
191
+ else:
192
+ raise ValueError(f"Unsupported file type: {path} (use .jsonl, .parquet or .txt)")
datainventor/text.py ADDED
@@ -0,0 +1,143 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Text primitives shared by every stage: normalization, token estimates, n-grams, MinHash, hashed features.
2
+
3
+ Pure standard library so the hosted preview needs no heavy installs. numpy is used when present to speed up MinHash.
4
+ """
5
+ from __future__ import annotations
6
+
7
+ import hashlib
8
+ import math
9
+ import re
10
+ import struct
11
+ import unicodedata
12
+
13
+ try: # optional speed-up for MinHash on large runs
14
+ import numpy as _np
15
+ except Exception: # pragma: no cover
16
+ _np = None
17
+
18
+ WORD = re.compile(r"\w+", re.U)
19
+ _MERSENNE = (1 << 61) - 1
20
+ _MAXH = (1 << 32) - 1
21
+
22
+
23
+ def norm(text: str) -> str:
24
+ """Lowercase, strip accents and punctuation, squeeze whitespace. Used for exact dedup and n-gram matching."""
25
+ t = unicodedata.normalize("NFKD", text.lower())
26
+ t = "".join(c for c in t if not unicodedata.combining(c))
27
+ return " ".join(WORD.findall(t))
28
+
29
+
30
+ def words(text: str) -> list[str]:
31
+ return WORD.findall(text.lower())
32
+
33
+
34
+ def est_tokens(text: str) -> int:
35
+ """About four characters per token for English prose. Good enough to plan a budget, not to bill one."""
36
+ return max(1, len(text) // 4)
37
+
38
+
39
+ def h64(s: str) -> int:
40
+ return int.from_bytes(hashlib.blake2b(s.encode("utf-8", "ignore"), digest_size=8).digest(), "little")
41
+
42
+
43
+ def doc_hash(text: str) -> str:
44
+ return hashlib.blake2b(norm(text).encode(), digest_size=12).hexdigest()
45
+
46
+
47
+ def ngrams(ws: list[str], n: int) -> list[str]:
48
+ return [" ".join(ws[i:i + n]) for i in range(len(ws) - n + 1)] if len(ws) >= n else ([" ".join(ws)] if ws else [])
49
+
50
+
51
+ # ---------------------------------------------------------------- MinHash + LSH (near-duplicate detection)
52
+
53
+ class MinHasher:
54
+ """MinHash over word 5-gram shingles. 112 permutations in 14 bands of 8 rows find pairs at roughly 0.8 Jaccard,
55
+ the setting FineWeb used for near-duplicate removal."""
56
+
57
+ def __init__(self, num_perm: int = 112, bands: int = 14, shingle: int = 5, seed: int = 1):
58
+ assert num_perm % bands == 0
59
+ self.num_perm, self.bands, self.rows, self.shingle = num_perm, bands, num_perm // bands, shingle
60
+ import random
61
+ rng = random.Random(seed)
62
+ # a, b < 2**32 keep a*x + b inside 64 bits, so the numpy and pure-Python paths agree exactly
63
+ self.a = [rng.randrange(1, _MAXH) for _ in range(num_perm)]
64
+ self.b = [rng.randrange(0, _MAXH) for _ in range(num_perm)]
65
+ if _np is not None:
66
+ self._a = _np.array(self.a, dtype=_np.uint64)
67
+ self._b = _np.array(self.b, dtype=_np.uint64)
68
+
69
+ def signature(self, text: str) -> list[int]:
70
+ sh = {h64(g) & _MAXH for g in ngrams(words(text), self.shingle)} or {0}
71
+ if _np is not None:
72
+ x = _np.fromiter(sh, dtype=_np.uint64)[:, None]
73
+ v = ((x * self._a + self._b) % _np.uint64(_MERSENNE)) & _np.uint64(_MAXH)
74
+ return v.min(axis=0).tolist()
75
+ return [min(((a * x + b) % _MERSENNE) & _MAXH for x in sh) for a, b in zip(self.a, self.b)]
76
+
77
+ def band_keys(self, sig: list[int]) -> list[bytes]:
78
+ r = self.rows
79
+ return [bytes([i]) + struct.pack(f"<{r}I", *sig[i * r:(i + 1) * r]) for i in range(self.bands)]
80
+
81
+
82
+ def jaccard_sig(a: list[int], b: list[int]) -> float:
83
+ return sum(1 for x, y in zip(a, b) if x == y) / len(a)
84
+
85
+
86
+ class NearDupIndex:
87
+ """In-memory LSH index. Holds one band table per band; fine for tens of millions of documents on a large box.
88
+ Past that, shard by a hash prefix and run one index per shard."""
89
+
90
+ def __init__(self, hasher: MinHasher | None = None, threshold: float = 0.8):
91
+ self.h = hasher or MinHasher()
92
+ self.threshold = threshold
93
+ self.tables: list[dict] = [dict() for _ in range(self.h.bands)]
94
+ self.sigs: dict[int, list[int]] = {}
95
+
96
+ def check_add(self, key: int, text: str) -> int | None:
97
+ """Return the key of an earlier near-duplicate, or None after adding this document."""
98
+ return self.check_add_sig(key, self.h.signature(text))
99
+
100
+ def check_add_sig(self, key: int, sig: list[int]) -> int | None:
101
+ """Same, with a signature computed elsewhere (in a worker process)."""
102
+ cands = set()
103
+ for i, k in enumerate(self.h.band_keys(sig)):
104
+ hit = self.tables[i].get(k)
105
+ if hit is not None:
106
+ cands.add(hit)
107
+ for c in cands:
108
+ if jaccard_sig(sig, self.sigs[c]) >= self.threshold:
109
+ return c
110
+ for i, k in enumerate(self.h.band_keys(sig)):
111
+ self.tables[i].setdefault(k, key)
112
+ self.sigs[key] = sig
113
+ return None
114
+
115
+
116
+ # ---------------------------------------------------------------- hashed features (cheap classifier + relevance)
117
+
118
+ def features(text: str, dim: int = 1 << 18) -> dict[int, float]:
119
+ """Hashed unigrams and bigrams, log-scaled and L2-normalized. Works for any language with word characters."""
120
+ ws = words(text[:20000])
121
+ counts: dict[int, float] = {}
122
+ for g in ws + ngrams(ws, 2):
123
+ k = h64(g) % dim
124
+ counts[k] = counts.get(k, 0.0) + 1.0
125
+ for k in counts:
126
+ counts[k] = 1.0 + math.log(counts[k])
127
+ n = math.sqrt(sum(v * v for v in counts.values())) or 1.0
128
+ return {k: v / n for k, v in counts.items()}
129
+
130
+
131
+ def cosine(a: dict[int, float], b: dict[int, float]) -> float:
132
+ if len(a) > len(b):
133
+ a, b = b, a
134
+ return sum(v * b.get(k, 0.0) for k, v in a.items())
135
+
136
+
137
+ def centroid(vecs: list[dict[int, float]]) -> dict[int, float]:
138
+ c: dict[int, float] = {}
139
+ for v in vecs:
140
+ for k, x in v.items():
141
+ c[k] = c.get(k, 0.0) + x
142
+ n = math.sqrt(sum(x * x for x in c.values())) or 1.0
143
+ return {k: x / n for k, x in c.items()}
make_benchmarks.py ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Bundle the decontamination test sets with the package, so runs never depend on the dataset viewer.
2
+ Needs `datasets`. Run again to refresh: python make_benchmarks.py"""
3
+ import gzip
4
+ import json
5
+
6
+ from datasets import load_dataset
7
+
8
+ from datainventor.clean import BENCHMARKS
9
+
10
+ out = {}
11
+ for key, (ds, cfg, split, fields) in BENCHMARKS.items():
12
+ rows = load_dataset(ds, None if cfg == "default" else cfg, split=split)
13
+ out[key] = [t for t in (" ".join(str(r.get(f, "")) for f in fields).strip() for r in rows) if t]
14
+ print(key, len(out[key]), flush=True)
15
+ with gzip.open("datainventor/data/benchmarks.json.gz", "wt", encoding="utf-8") as f:
16
+ json.dump(out, f, ensure_ascii=False)
pyproject.toml ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [build-system]
2
+ requires = ["setuptools>=68"]
3
+ build-backend = "setuptools.build_meta"
4
+
5
+ [project]
6
+ name = "datainventor"
7
+ version = "0.2.0"
8
+ description = "Curate a pretraining corpus on your own machines: clean, calibrate, create, compose."
9
+ readme = "README.md"
10
+ requires-python = ">=3.10"
11
+ license = { text = "Apache-2.0" }
12
+ dependencies = ["pyyaml>=6"]
13
+
14
+ [project.optional-dependencies]
15
+ full = ["datasets>=2.18", "pyarrow>=14", "numpy>=1.24"]
16
+
17
+ [project.scripts]
18
+ datainventor = "datainventor.cli:main"
19
+
20
+ [tool.setuptools]
21
+ packages = ["datainventor"]
22
+
23
+ [tool.setuptools.package-data]
24
+ datainventor = ["data/*.json.gz"]
requirements.txt CHANGED
@@ -2,3 +2,4 @@ fastapi==0.115.*
2
  uvicorn[standard]==0.32.*
3
  pydantic>=2.7
4
  huggingface_hub>=0.30
 
 
2
  uvicorn[standard]==0.32.*
3
  pydantic>=2.7
4
  huggingface_hub>=0.30
5
+ pyyaml>=6
static/app.css CHANGED
@@ -188,3 +188,58 @@ textarea.in { min-height: 118px; resize: vertical; font-family: var(--doc); font
188
  .stats { grid-template-columns: repeat(2, minmax(0, 1fr)); }
189
  .press { height: 250px; }
190
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
188
  .stats { grid-template-columns: repeat(2, minmax(0, 1fr)); }
189
  .press { height: 250px; }
190
  }
191
+
192
+ /* mode switch */
193
+ .modes { display: inline-flex; border: 1.5px solid var(--ink); border-radius: var(--r); overflow: hidden; flex: none; }
194
+ .modes button { background: var(--paper-2); border: 0; padding: 7px 14px; font-weight: 600; cursor: pointer; }
195
+ .modes button + button { border-left: 1.5px solid var(--ink); }
196
+ .modes button[aria-pressed="true"] { background: var(--ink); color: #fff; }
197
+
198
+ /* curate */
199
+ .addrow { display: grid; grid-template-columns: minmax(0, 1fr) auto auto; gap: 10px; margin-top: 8px; }
200
+ .in.sm { width: auto; padding: 7px 10px; font-size: 14.5px; }
201
+ .in.num { width: 7.5em; font-variant-numeric: tabular-nums; }
202
+ .srow { display: grid; grid-template-columns: minmax(0, 1fr) auto; gap: 14px; padding: 12px 0; border-bottom: 1px solid var(--rule); align-items: start; }
203
+ .srow:first-of-type { border-top: 1px solid var(--ink); margin-top: 10px; }
204
+ .sname b { display: block; overflow-wrap: anywhere; }
205
+ .sname .hint { display: block; }
206
+ .sctl { display: flex; gap: 10px; align-items: center; flex-wrap: wrap; justify-content: flex-end; }
207
+ .peek, .ptext { font-family: var(--doc); white-space: pre-wrap; font-size: 14.5px; line-height: 1.5; max-height: 260px; overflow: auto; margin-top: 6px; padding: 10px 12px; background: var(--paper-2); border: 1px solid var(--rule); border-radius: 2px; }
208
+ details summary { cursor: pointer; color: var(--blue); font-weight: 600; font-size: 14px; margin-top: 4px; }
209
+ code { font-size: .92em; background: var(--paper-2); padding: 0 4px; border-radius: 2px; }
210
+ .range { width: 100%; accent-color: var(--blue); }
211
+ .sv { font-weight: 600; color: var(--blue); margin-left: 6px; }
212
+ .pairs { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 14px; margin-top: 16px; }
213
+ .pcard .ph { display: flex; gap: 10px; align-items: baseline; flex-wrap: wrap; }
214
+ .flag { color: var(--bad); font-size: 13px; font-weight: 600; }
215
+ .mix { padding-top: 14px; border-top: 1px solid var(--rule); }
216
+ .mixhead { display: flex; flex-wrap: wrap; gap: 12px; align-items: center; }
217
+ .mixhead .seg { margin-top: 0; }
218
+ .wtable { display: grid; grid-template-columns: repeat(auto-fill, minmax(230px, 1fr)); gap: 10px 22px; margin-top: 14px; }
219
+ .wtable label { display: grid; grid-template-columns: minmax(0, 1fr) auto auto; gap: 6px; align-items: center; }
220
+ .wtable label > span:first-child { overflow-wrap: anywhere; }
221
+ .stages { list-style: none; display: flex; flex-wrap: wrap; gap: 6px 22px; padding: 0; margin: 26px 0 0; }
222
+ .stages li { color: var(--slate); }
223
+ .stages li.done { color: var(--ink); }
224
+ .stages li.on { color: var(--blue); font-weight: 600; }
225
+ .stages li.on::after { content: " …"; }
226
+ .tscroll { overflow-x: auto; margin-top: 8px; }
227
+ .tbl { width: 100%; border-collapse: collapse; font-size: 14.5px; font-variant-numeric: tabular-nums; }
228
+ .tbl th { text-align: left; font-weight: 600; border-bottom: 1px solid var(--ink); padding: 8px 12px 8px 0; white-space: nowrap; }
229
+ .tbl td { border-bottom: 1px solid var(--rule); padding: 9px 12px 9px 0; vertical-align: top; }
230
+ .hists { display: grid; grid-template-columns: repeat(auto-fill, minmax(180px, 1fr)); gap: 18px; margin-top: 12px; }
231
+ .hists span { font-weight: 600; font-size: 14px; }
232
+ .hist { display: flex; align-items: flex-end; gap: 3px; height: 56px; margin-top: 6px; border-bottom: 1px solid var(--ink); }
233
+ .hist i { flex: 1; background: var(--blue); min-height: 1px; }
234
+ .ex { margin-top: 10px; }
235
+ .exd { margin-top: 10px; }
236
+ .quote { font-family: var(--doc); font-size: 14.5px; margin-top: 8px; }
237
+ .code { background: var(--ink); color: #f1f2f4; padding: 14px 16px; border-radius: var(--r); overflow-x: auto; font-size: 13.5px; line-height: 1.6; margin-top: 12px; white-space: pre; }
238
+ @media (max-width: 640px) {
239
+ .pairs { grid-template-columns: 1fr; }
240
+ .addrow { grid-template-columns: 1fr auto; }
241
+ .addrow .btn { grid-column: 1 / -1; }
242
+ .srow { grid-template-columns: 1fr; }
243
+ .sctl { justify-content: flex-start; }
244
+ }
245
+ .in.num.wide { width: 11em; }
static/app.js CHANGED
@@ -209,6 +209,7 @@ function updateRun() {
209
 
210
  /* ------------------------------------------------------------ render + events */
211
  function render() {
 
212
  renderLedger(); renderSheet();
213
  const fn = [stepFormat, stepDomains, stepSubs, stepLangs, stepSummary, stepReview, stepRun][S.step];
214
  const keep = document.activeElement && document.activeElement.id;
@@ -262,6 +263,7 @@ async function poll() {
262
  if (j.stats && j.stats.rows) S.maxStep = 6;
263
  } catch (e) { console.error(e.stack); S.jobState = { state: "failed", message: e.message, count: S.jobRows.length, target: S.rows, elapsed: 0, id }; if (S.step === 6) render(); }
264
  }
 
265
  /* "Start writing" on the review step starts the run */
266
  const _go = go;
267
  go = function (i) { const wasReview = S.step === 5; _go(i); if (i === 6 && wasReview && !S.jobState) startRun(); return true; };
 
209
 
210
  /* ------------------------------------------------------------ render + events */
211
  function render() {
212
+ if (window.DI_MODE === "curate") return;
213
  renderLedger(); renderSheet();
214
  const fn = [stepFormat, stepDomains, stepSubs, stepLangs, stepSummary, stepReview, stepRun][S.step];
215
  const keep = document.activeElement && document.activeElement.id;
 
263
  if (j.stats && j.stats.rows) S.maxStep = 6;
264
  } catch (e) { console.error(e.stack); S.jobState = { state: "failed", message: e.message, count: S.jobRows.length, target: S.rows, elapsed: 0, id }; if (S.step === 6) render(); }
265
  }
266
+ window.DI_renderInvent = () => render();
267
  /* "Start writing" on the review step starts the run */
268
  const _go = go;
269
  go = function (i) { const wasReview = S.step === 5; _go(i); if (i === 6 && wasReview && !S.jobState) startRun(); return true; };
static/curate.js ADDED
@@ -0,0 +1,371 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ "use strict";
2
+ /* Curate a corpus: register sources and targets, then clean, calibrate, create, compose.
3
+ The hosted run works on a sample; the recipe runs in full on your own machines. */
4
+ (function () {
5
+ const CSTEPS = ["Sources", "Clean", "Calibrate", "Create", "Compose", "Run"];
6
+ const BUDGETS = [["1B", 1e9], ["10B", 1e10], ["100B", 1e11], ["1T", 1e12]];
7
+ const BENCH_NAMES = { gsm8k: "GSM8K", math500: "MATH-500", humaneval: "HumanEval", mbpp: "MBPP", mmlu: "MMLU", arc_challenge: "ARC-Challenge",
8
+ arc_easy: "ARC-Easy", hellaswag: "HellaSwag", openbookqa: "OpenBookQA", winogrande: "WinoGrande", truthfulqa: "TruthfulQA", commonsenseqa: "CommonsenseQA" };
9
+ const KIND_NAMES = { web: "Web", educational: "Educational web", textbook: "Textbook-style", math: "Math", code: "Code", multilingual: "Multilingual", synthetic: "Synthetic", other: "Other" };
10
+ const RULES = [["min_words", "Fewest words in a document", 1], ["max_dup_lines", "Most repeated lines (share)", 0.01], ["max_dup10gram", "Most repeated passages (share)", 0.01], ["min_alpha_words", "Fewest words with letters (share)", 0.01]];
11
+ const PRESET = [
12
+ { dataset: "codelion/finepdfs-1B", kind: "textbook" }, { dataset: "codelion/dclm-baseline-1B", kind: "web" }, { dataset: "codelion/fineweb-edu-1B", kind: "educational" }];
13
+
14
+ const C = {
15
+ step: 0, maxStep: 0, d: null, sources: [], targets: [], targetsDesc: "", add: "", addRole: "source", addBusy: false, addErr: "",
16
+ decon: ["gsm8k", "math500", "humaneval", "mmlu", "arc_challenge", "hellaswag"], rules: {},
17
+ focus: 0.3, keepTop: 0.6, near: true, threshold: 0.8, gradeDocs: 2000, scorer: "learned",
18
+ rephraseOn: true, styles: [], maxNew: 0.35, inventOn: false, inventReq: "", inventDocs: 0,
19
+ mixes: [{ name: "main", budget: 1e10, weights: {} }], maxEpochs: 4, maxSynth: 0.25,
20
+ tryText: "", tryBusy: false, tryErr: "", tryRes: null, tryFrom: "",
21
+ pid: null, pv: null, runErr: "",
22
+ };
23
+ const $ = (s, el = document) => el.querySelector(s);
24
+ const esc = (t) => String(t ?? "").replace(/[&<>"']/g, (c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" }[c]));
25
+ const fmt = (n) => { n = +n || 0; for (const [u, d] of [["T", 1e12], ["B", 1e9], ["M", 1e6], ["K", 1e3]]) if (Math.abs(n) >= d) return `${+(n / d).toPrecision(3)}${u}`; return `${Math.round(n)}`; };
26
+ const pct = (x) => `${Math.round((+x || 0) * 100)}%`;
27
+ async function api(path, body, method) {
28
+ const r = await fetch(path, { method: method || (body ? "POST" : "GET"), headers: body ? { "Content-Type": "application/json" } : {}, body: body ? JSON.stringify(body) : undefined });
29
+ let data = null; try { data = await r.json(); } catch (e) { /* not json */ }
30
+ if (!r.ok) throw new Error((data && (data.detail || data.message)) || `Request failed (${r.status})`);
31
+ return data;
32
+ }
33
+
34
+ /* ------------------------------------------------------------ recipe */
35
+ function srcOut(s) { return { name: s.name, dataset: s.dataset, config: s.config, split: s.split, text_field: s.text_field, kind: s.kind, num_rows: s.num_rows || 0 }; }
36
+ function recipe() {
37
+ return {
38
+ name: "my-corpus", sources: C.sources.map(srcOut),
39
+ targets: { description: C.targetsDesc.trim(), datasets: C.targets.map(srcOut) },
40
+ clean: { decontaminate: C.decon, rules: C.rules },
41
+ calibrate: { dedup: { near: C.near, threshold: C.threshold }, grade_docs: C.gradeDocs, scorer: C.scorer, focus: C.focus, keep_top: C.keepTop },
42
+ create: { rephrase: { enabled: C.rephraseOn, styles: C.styles, max_new_share: C.maxNew },
43
+ invent: { enabled: C.inventOn && !!C.inventReq.trim(), request: C.inventReq.trim(), docs: C.inventDocs } },
44
+ compose: { mixes: C.mixes.map((m) => ({ name: m.name, budget_tokens: m.budget, weights: m.weights })), max_epochs: C.maxEpochs, max_synthetic_share: C.maxSynth },
45
+ };
46
+ }
47
+ /* mirrors datainventor.compose.default_weights */
48
+ function defaultWeights() {
49
+ const G = { textbook: "textbook", synthetic: "textbook", web: "web", multilingual: "web", other: "web", educational: "educational", math: "math", code: "code" };
50
+ const BASE = { textbook: 0.5, web: 0.3, educational: 0.2 };
51
+ const groups = {}; C.sources.forEach((s) => (groups[G[s.kind] || "web"] = groups[G[s.kind] || "web"] || []).push(s.name));
52
+ const fixed = {}; ["math", "code"].forEach((g) => { if (groups[g]) fixed[g] = 0.1; });
53
+ const base = Object.fromEntries(Object.entries(BASE).filter(([g]) => groups[g]));
54
+ const out = {};
55
+ if (!Object.keys(base).length) { C.sources.forEach((s) => (out[s.name] = 1 / C.sources.length)); return out; }
56
+ const rest = 1 - Object.values(fixed).reduce((a, b) => a + b, 0), tot = Object.values(base).reduce((a, b) => a + b, 0);
57
+ const shares = { ...Object.fromEntries(Object.entries(base).map(([g, v]) => [g, (rest * v) / tot])), ...fixed };
58
+ Object.entries(groups).forEach(([g, names]) => names.forEach((n) => (out[n] = shares[g] / names.length)));
59
+ return out;
60
+ }
61
+ const weightsOf = (m) => ({ ...defaultWeights(), ...m.weights });
62
+
63
+ /* ------------------------------------------------------------ chrome */
64
+ function ledger() {
65
+ $("#steps").innerHTML = CSTEPS.map((n, i) => {
66
+ const cls = i === C.step ? "on" : i < C.step ? "done" : "";
67
+ return `<li class="${cls}"><button data-cgo="${i}" ${i > C.maxStep ? "disabled" : ""} ${i === C.step ? 'aria-current="step"' : ""}><span class="n">${i + 1}</span><span>${n}</span></button></li>`;
68
+ }).join("");
69
+ }
70
+ function sheet() {
71
+ const line = (k, v) => `<div class="line"><dt>${k}</dt><dd>${v}</dd></div>`;
72
+ const m = C.mixes[0];
73
+ $("#sheet").innerHTML = `<h2>Your recipe</h2><dl>
74
+ ${line("Sources", C.sources.length ? esc(C.sources.map((s) => s.name).join(", ")) : '<span class="hint">None yet</span>')}
75
+ ${line("Targets", C.targetsDesc.trim() || C.targets.length ? esc([C.targetsDesc.trim().slice(0, 80), ...C.targets.map((t) => t.name)].filter(Boolean).join("; ")) : '<span class="hint">General quality only</span>')}
76
+ ${line("Decontam", C.decon.length ? esc(C.decon.map((b) => BENCH_NAMES[b] || b).join(", ")) : "Off")}
77
+ ${line("Focus", C.focus === 0 || !(C.targetsDesc.trim() || C.targets.length) ? "General quality" : `${pct(1 - C.focus)} quality, ${pct(C.focus)} targets`)}
78
+ ${line("Keep", `Top ${pct(C.keepTop)} by score`)}
79
+ ${line("Create", C.rephraseOn ? "Rephrase to fill gaps" + (C.inventOn ? ", invent" : "") : C.inventOn ? "Invent" : "Off")}
80
+ ${line("Mixes", C.mixes.map((x) => `${esc(x.name)} ${fmt(x.budget)}`).join(", "))}
81
+ </dl>
82
+ <p class="est">The hosted run works on about <b>${C.d ? C.d.per_source : 200}</b> documents per source. The recipe runs the same steps on every document, on your machines.</p>
83
+ <div class="actions" style="margin-top:14px"><button class="btn" data-c="recipe" ${C.sources.length ? "" : "disabled"}>Download recipe</button></div>`;
84
+ }
85
+ function actions(back, next, ok = true, extra = "") {
86
+ return `<div class="actions">${back ? `<button class="btn" data-c="back">Back</button>` : ""}${next ? `<button class="btn go" data-c="next" ${ok ? "" : "disabled"}>${next}</button>` : ""}${extra}</div>`;
87
+ }
88
+ function check(on, attr, label, hint = "") {
89
+ return `<button class="choice" ${attr} aria-pressed="${on}"><span class="box"></span><span>${label}${hint ? ` <span class="hint">${hint}</span>` : ""}</span></button>`;
90
+ }
91
+ function slider(id, label, val, min, max, step, show) {
92
+ return `<label class="f"><span>${label} <b class="sv" id="${id}-v">${show(val)}</b></span><input type="range" class="range" id="${id}" min="${min}" max="${max}" step="${step}" value="${val}"></label>`;
93
+ }
94
+
95
+ /* ------------------------------------------------------------ steps */
96
+ function srcRow(s, i, role) {
97
+ const kinds = C.d ? C.d.kinds : Object.keys(KIND_NAMES);
98
+ return `<div class="srow">
99
+ <div class="sname"><b>${esc(s.dataset)}</b><span class="hint">${esc(s.config)} / ${esc(s.split)} / column <code>${esc(s.text_field)}</code>${s.num_rows ? ` / ${fmt(s.num_rows)} rows` : ""}</span>
100
+ ${s.preview ? `<details><summary>Peek at a document</summary><div class="peek">${esc(s.preview)}</div></details>` : ""}</div>
101
+ <div class="sctl">${role === "source" ? `<select class="in sm" id="c-kind-${i}" aria-label="Kind of data">${kinds.map((k) => `<option value="${k}" ${s.kind === k ? "selected" : ""}>${KIND_NAMES[k] || k}</option>`).join("")}</select>` : ""}
102
+ ${s.columns && s.columns.length > 1 ? `<select class="in sm" id="c-col-${role}-${i}" aria-label="Text column">${s.columns.map((c) => `<option ${c === s.text_field ? "selected" : ""}>${esc(c)}</option>`).join("")}</select>` : ""}
103
+ <button class="linkish" data-c="rm" data-role="${role}" data-i="${i}">Remove</button></div></div>`;
104
+ }
105
+ function stepSources() {
106
+ return `<h1>Bring the data you have</h1>
107
+ <p class="lede">Register each Hugging Face dataset as a source to curate, or as a target that shows what good looks like. Targets steer every later step toward your tasks.</p>
108
+ <div class="group"><h3>Add a dataset</h3>
109
+ <div class="addrow"><input class="in" id="c-add" placeholder="HuggingFaceFW/fineweb-edu" value="${esc(C.add)}" spellcheck="false" autocomplete="off">
110
+ <select class="in sm" id="c-addrole" aria-label="Register as"><option value="source" ${C.addRole === "source" ? "selected" : ""}>Source</option><option value="target" ${C.addRole === "target" ? "selected" : ""}>Target</option></select>
111
+ <button class="btn" data-c="add" ${C.addBusy || !C.add.trim() ? "disabled" : ""}>${C.addBusy ? "Reading" : "Add"}</button></div>
112
+ ${C.addErr ? `<p class="note bad">${esc(C.addErr)}</p>` : ""}
113
+ ${C.sources.length ? "" : `<p class="hint" style="margin-top:10px">Or start from <button class="linkish" data-c="preset">the 1B-token mix</button>: textbook-style PDFs, DCLM and FineWeb-Edu.</p>`}</div>
114
+ <div class="group"><h3>Sources <span class="hint">(up to ${C.d ? C.d.max_sources : 4} here)</span></h3>
115
+ ${C.sources.length ? C.sources.map((s, i) => srcRow(s, i, "source")).join("") : '<p class="hint">None yet.</p>'}</div>
116
+ <div class="group"><h3>Targets</h3>
117
+ <label class="f" style="margin-top:6px"><span>What should the trained model be good at? <span class="hint">(optional)</span></span>
118
+ <textarea class="in" id="c-tdesc" style="min-height:80px" placeholder="Reading and drafting commercial contracts. Grade-school math word problems.">${esc(C.targetsDesc)}</textarea></label>
119
+ ${C.targets.map((s, i) => srcRow(s, i, "target")).join("")}
120
+ <p class="hint" style="margin-top:8px">Target datasets are examples of the work you care about, such as a benchmark's training split or your own documents.</p></div>
121
+ ${actions(false, "Continue", C.sources.length > 0)}`;
122
+ }
123
+ function stepClean() {
124
+ const b = C.d ? C.d.benchmarks : Object.keys(BENCH_NAMES), r = { ...(C.d ? C.d.clean_rules : {}), ...C.rules };
125
+ return `<h2>Clean</h2>
126
+ <p class="lede">Heuristic filters drop broken text, boilerplate, empty and repetitive documents. Math and code get their own looser rules. Then any document that shares a 13-word run with a test set you name is removed.</p>
127
+ <div class="group"><h3>Decontaminate against</h3><ul class="choices">${b.map((k) => `<li>${check(C.decon.includes(k), `data-bench="${k}"`, BENCH_NAMES[k] || k)}</li>`).join("")}</ul>
128
+ <p class="hint" style="margin-top:6px">To check against test sets of your own, list them under <code>clean.held_out</code> in the recipe, the same way as sources.</p></div>
129
+ <div class="group"><h3>Filters</h3><div class="row2">${RULES.map(([k, label, step]) => `<label class="f"><span>${label}</span><input class="in" type="number" id="c-rule-${k}" step="${step}" value="${r[k]}"></label>`).join("")}</div>
130
+ <p class="hint" style="margin-top:8px">The defaults follow the Gopher and C4 rules with FineWeb's thresholds. Every rule is in the recipe.</p></div>
131
+ ${actions(true, "Continue")}`;
132
+ }
133
+ function stepCalibrate() {
134
+ const hasT = !!(C.targetsDesc.trim() || C.targets.length);
135
+ return `<h2>Calibrate</h2>
136
+ <p class="lede">A model grades a sample of each source for teaching value, clarity, accuracy and density${hasT ? ", and for relevance to your targets" : ""}. A small scorer learns those grades and scores every document. Redundant copies are dropped first.</p>
137
+ ${hasT ? slider("c-focus", "Balance between general quality and your targets", C.focus, 0, 1, 0.05, (v) => (+v === 0 ? "general quality only" : `${pct(1 - v)} quality / ${pct(v)} targets`))
138
+ : `<p class="note">No targets, so documents are ranked by general quality. Add targets in step 1 to steer the ranking.</p>`}
139
+ ${hasT ? '<p class="hint">Keep some weight on general quality. Data tuned only to the targets overfits them.</p>' : ""}
140
+ ${slider("c-keep", "Keep the best", C.keepTop, 0.1, 1, 0.05, (v) => `${pct(v)} of each source`)}
141
+ <div class="group"><h3>Duplicates</h3><ul class="choices" style="columns:1">
142
+ <li>${check(true, "disabled", "Exact copies", "always removed")}</li>
143
+ <li>${check(C.near, 'data-c="near"', "Near copies", "MinHash, about 80% overlap")}</li></ul></div>
144
+ <div class="group"><h3>On your machines</h3>
145
+ <div class="row2"><label class="f"><span>Documents to grade with the model</span><input class="in" type="number" id="c-grade" min="100" step="100" value="${C.gradeDocs}"></label>
146
+ <label class="f"><span>Score every document by</span><select class="in" id="c-scorer"><option value="learned" ${C.scorer === "learned" ? "selected" : ""}>A scorer learned from the grades</option><option value="llm" ${C.scorer === "llm" ? "selected" : ""}>The model itself (slow, best)</option></select></label></div>
147
+ <p class="hint" style="margin-top:8px">The hosted run grades ${C.d ? 48 : "a few dozen"} documents, enough to see the rubric work but not to trust the learned scorer. In our test, 300 grades gave a rank agreement of about 0.5 with the model; more grades help.</p></div>
148
+ ${actions(true, "Continue")}`;
149
+ }
150
+ function tryHtml() {
151
+ if (C.tryErr) return `<p class="note bad">${esc(C.tryErr)}</p>`;
152
+ if (!C.tryRes) return "";
153
+ return `<div class="pairs">${C.tryRes.results.map((o) => `<div class="pcard"><div class="ph"><b>${esc(o.name)}</b>${o.ok ? "" : `<span class="flag">${esc(o.why)}</span>`}</div>
154
+ <div class="ptext">${esc(o.text || "No rewrite. The model call failed.")}</div>
155
+ ${o.fidelity && (o.fidelity.new_numbers.length || o.fidelity.new_names.length) ? `<p class="hint">Not in the source: ${esc([...o.fidelity.new_numbers, ...o.fidelity.new_names].join(", "))}</p>` : ""}</div>`).join("")}</div>`;
156
+ }
157
+ function stepCreate() {
158
+ const st = C.d ? C.d.styles : {}, picked = C.styles;
159
+ const first = C.sources.find((s) => s.preview);
160
+ return `<h2>Create</h2>
161
+ <p class="lede">Where a source is too small for its share of a mix, the best documents are rephrased into new ones: a dialogue, a tutorial, questions and answers, dense notes, a textbook section. Rewrites that add numbers or names the source does not contain are flagged and dropped.</p>
162
+ <ul class="choices" style="columns:1;margin-top:20px"><li>${check(C.rephraseOn, 'data-c="reph"', "Rephrase to fill gaps")}</li></ul>
163
+ ${C.rephraseOn ? `<div class="group"><h3>Styles <span class="hint">(none picked = chosen per kind of data)</span></h3><ul class="choices">${Object.entries(st).map(([k, n]) => `<li>${check(picked.includes(k), `data-style="${k}"`, esc(n))}</li>`).join("")}</ul></div>
164
+ <div class="group"><h3>Try it on one document</h3>
165
+ <textarea class="in" id="c-try" style="min-height:120px" placeholder="Paste a passage of 60 words or more.">${esc(C.tryText)}</textarea>
166
+ <div class="actions" style="margin-top:12px">${first ? `<button class="btn" data-c="tryfill">Use a document from ${esc(first.name)}</button>` : ""}
167
+ <button class="btn go" data-c="try" ${C.tryBusy || C.tryText.trim().split(/\s+/).length < 60 ? "disabled" : ""}>${C.tryBusy ? "Rewriting" : "Rewrite it in four styles"}</button>${C.tryBusy ? '<span class="hint">This takes about a minute.</span>' : ""}</div>
168
+ <div id="c-tryout">${tryHtml()}</div></div>` : ""}
169
+ <ul class="choices" style="columns:1;margin-top:26px"><li>${check(C.inventOn, 'data-c="inv"', "Also write new documents on a subject", "for areas your sources barely cover")}</li></ul>
170
+ ${C.inventOn ? `<div class="row2"><label class="f"><span>Subject</span><input class="in" id="c-invreq" value="${esc(C.inventReq)}" placeholder="Contract law for small businesses"></label>
171
+ <label class="f"><span>Documents on your machines</span><input class="in" type="number" id="c-invdocs" min="0" step="1000" value="${C.inventDocs}"></label></div>
172
+ <p class="hint" style="margin-top:8px">The hosted run writes four as a preview. Give invented documents a share in a mix with <code>invented_share</code> in the recipe.</p>` : ""}
173
+ ${actions(true, "Continue")}`;
174
+ }
175
+ function stepCompose() {
176
+ return `<h2>Compose</h2>
177
+ <p class="lede">Build one or more training mixes from the prepared corpus. The default share is the static mix that won our 1B-token study: 50% textbook-style, 30% web, 20% educational web, with 10% each for math and code when present. It was measured at 70M parameters, so treat it as a starting point.</p>
178
+ ${C.mixes.map((m, mi) => { const w = weightsOf(m), tot = Object.values(w).reduce((a, b) => a + b, 0) || 1;
179
+ return `<div class="group mix"><div class="mixhead"><input class="in sm" id="c-mname-${mi}" value="${esc(m.name)}" aria-label="Mix name" spellcheck="false">
180
+ <div class="seg">${BUDGETS.map(([l, v]) => `<button data-c="budget" data-m="${mi}" data-v="${v}" aria-pressed="${m.budget === v}">${l}</button>`).join("")}</div>
181
+ <label class="hint">or <input class="in sm num wide" type="number" id="c-mbud-${mi}" value="${m.budget}" min="1000000" step="1000000"> tokens</label>
182
+ ${C.mixes.length > 1 ? `<button class="linkish" data-c="rmmix" data-m="${mi}">Remove</button>` : ""}</div>
183
+ <div class="wtable">${C.sources.map((s) => `<label><span>${esc(s.name)} <small class="hint">${KIND_NAMES[s.kind] || s.kind}</small></span>
184
+ <input class="in sm num" type="number" min="0" max="100" step="1" id="c-w-${mi}-${esc(s.name)}" value="${Math.round((w[s.name] / tot) * 100)}"><span class="hint">%</span></label>`).join("")}</div></div>`; }).join("")}
185
+ <p style="margin-top:14px"><button class="linkish" data-c="addmix">Add another mix</button> <span class="hint">for a different run, from the same prepared corpus</span></p>
186
+ <div class="row2">${slider("c-epochs", "Repeat a source at most", C.maxEpochs, 1, 8, 0.5, (v) => `${v} times`)}
187
+ ${slider("c-synth", "Synthetic text at most", C.maxSynth, 0, 0.6, 0.05, (v) => `${pct(v)} of a mix`)}</div>
188
+ <p class="hint">A source short of its share is repeated first, then topped up by rephrasing its best documents. In our study a mix that leaned on synthetic-like text scored well on its own validation set but generalized worst.</p>
189
+ ${actions(true, "Continue")}`;
190
+ }
191
+
192
+ /* ------------------------------------------------------------ run + report */
193
+ const isFinal = (p) => p && ["done", "stopped", "failed"].includes(p.state);
194
+ function hist(h) {
195
+ const mx = Math.max(1, ...h);
196
+ return `<div class="hist" aria-hidden="true">${h.map((v) => `<i style="height:${Math.round((v / mx) * 100)}%"></i>`).join("")}</div>`;
197
+ }
198
+ function report(rep) {
199
+ const src = rep.sources || [], cal = (rep.stages || {}).calibrate || {}, comp = (rep.stages || {}).compose || {}, crt = (rep.stages || {}).create || {};
200
+ const agree = cal.agreement || {};
201
+ const drops = (f) => Object.entries(f.dropped).filter(([, v]) => v).sort((a, b) => b[1] - a[1]).map(([k, v]) => `${esc(k)} ${v}`).join(", ") || "none";
202
+ return `
203
+ <div class="group"><h3>What each source went through</h3><div class="tscroll"><table class="tbl"><thead><tr><th>Source</th><th>Read</th><th>Passed cleaning</th><th>Kept</th><th>Removed for test overlap</th><th>Tokens after curation</th></tr></thead><tbody>
204
+ ${src.map((f) => `<tr><td><b>${esc(f.name)}</b><br><span class="hint">${drops(f)}</span></td><td>${f.read}</td><td>${f.kept}</td><td>${f.selected}</td>
205
+ <td>${Object.keys(f.decontaminated).length ? esc(Object.entries(f.decontaminated).map(([k, v]) => `${BENCH_NAMES[k] || k} ${v}`).join(", ")) : "0"}</td>
206
+ <td>${f.available_tokens ? "about " + fmt(f.available_tokens) : '<span class="hint">unknown size</span>'}</td></tr>`).join("")}</tbody></table></div>
207
+ <p class="hint" style="margin-top:8px">Tokens after curation scale each sample's keep rate to the full dataset. They are estimates at four characters per token.</p></div>
208
+ ${src.some((f) => f.repeated_passages && f.repeated_passages.length) ? `<div class="group"><h3>Passages repeated across documents</h3>${src.filter((f) => f.repeated_passages.length).map((f) => f.repeated_passages.slice(0, 3).map((r) => `<p class="quote"><b>${esc(f.name)}</b>, in ${r.docs} documents of the sample: ${esc(r.text)}</p>`).join("")).join("")}</div>` : ""}
209
+ <div class="group"><h3>Scores</h3>
210
+ <p class="hint">The model graded ${cal.graded} documents and those keep their grades. The rest are ranked by a scorer learned from so few grades that its ranking is close to a guess${cal.graded >= 100 ? ` (rank agreement with the model on held-out documents: quality ${agree.quality ?? "n/a"}${cal.has_targets ? `, relevance ${agree.relevance ?? "n/a"}` : ""})` : ""}. The full run grades ${C.gradeDocs.toLocaleString()} documents and reports how well the scorer agrees with the model.</p>
211
+ <div class="hists">${src.map((f) => `<div><span>${esc(f.name)}</span>${hist(f.score_hist || [])}<small class="hint">low to high score</small></div>`).join("")}</div>
212
+ ${src.map((f) => `<details class="ex"><summary>Best and worst kept in ${esc(f.name)}</summary>${[...(f.examples.top || []).map((e) => ["Best", e]), ...(f.examples.bottom || []).map((e) => ["Worst", e])].map(([l, e]) => `<div class="exd"><b>${l}</b> <span class="hint">score ${e.score}, quality ${e.quality}${cal.has_targets ? `, relevance ${e.relevance}` : ""}${e.reason ? ` / ${esc(e.reason)}` : ""}</span><div class="ptext">${esc(e.text)}</div></div>`).join("")}</details>`).join("")}</div>
213
+ ${(rep.created_examples || []).length ? `<div class="group"><h3>Created</h3><p class="hint">${crt.kept} of ${crt.made} rewrites kept.${(crt.flagged || []).length ? ` ${crt.flagged.length} flagged for adding facts the source does not contain.` : ""}</p>
214
+ ${rep.created_examples.map((c) => `<details class="ex"><summary>${esc(c.style ? (C.d.styles[c.style] || c.style) : "New document: " + (c.topic || ""))}${c.kept ? "" : ` <span class="flag">${esc(c.why)}</span>`}</summary>
215
+ <div class="pairs two">${c.origin_text ? `<div class="pcard"><div class="ph"><b>Source</b></div><div class="ptext">${esc(c.origin_text)}</div></div>` : ""}<div class="pcard"><div class="ph"><b>Rewrite</b></div><div class="ptext">${esc(c.text)}</div></div></div></details>`).join("")}</div>` : ""}
216
+ <div class="group"><h3>Mixes</h3>${(comp.plans || []).map((p) => `<p style="margin-top:12px"><b>${esc(p.name)}</b>, ${esc(p.budget)} tokens</p>
217
+ <div class="tscroll"><table class="tbl"><thead><tr><th>Source</th><th>Share</th><th>Needs</th><th>Available</th><th>Repeats</th><th>Rephrased</th></tr></thead><tbody>
218
+ ${p.rows.map((r) => `<tr><td>${esc(r.name)}</td><td>${pct(r.share)}</td><td>${fmt(r.want_tokens)}</td><td>${fmt(r.available_tokens)}</td><td>${r.epochs}x</td><td>${r.synthetic_tokens ? fmt(r.synthetic_tokens) : "0"}${r.short_tokens ? ` <span class="flag">${fmt(r.short_tokens)} short</span>` : ""}</td></tr>`).join("")}</tbody></table></div>
219
+ ${p.synthetic_tokens ? `<p class="hint">About ${p.synthetic_calls.toLocaleString()} model calls to rephrase ${fmt(p.synthetic_tokens)} tokens on your endpoint.</p>` : ""}
220
+ ${p.warnings.map((w) => `<p class="note">${esc(w)}</p>`).join("")}`).join("")}</div>
221
+ ${(rep.warnings || []).map((w) => `<p class="note">${esc(w)}</p>`).join("")}`;
222
+ }
223
+ function stepRun() {
224
+ const p = C.pv, fin = isFinal(p);
225
+ const stageIdx = p ? ["setup", "clean", "calibrate", "create", "done"].indexOf(p.stage) : -1;
226
+ const out = p && p.report && p.report.output;
227
+ return `<h2>${!p ? "Run it on a sample" : !fin ? "Curating a sample" : p.state === "failed" ? "The run stopped" : "Your sample is curated"}</h2>
228
+ <p class="lede">Every step runs here on about ${C.d ? C.d.per_source : 200} documents from each source. You get the kept and created documents, a report of what each step did, and the recipe to run the same steps on all of your data.</p>
229
+ ${C.runErr ? `<p class="note bad">${esc(C.runErr)}</p>` : ""}
230
+ ${!p ? `<div class="actions"><button class="btn" data-c="back">Back</button><button class="btn go" data-c="run">Run on a sample</button></div>` : ""}
231
+ ${p && !fin ? `<ol class="stages">${["Reading", "Cleaning", "Calibrating", "Creating", "Composing"].map((s, i) => `<li class="${i < stageIdx ? "done" : i === stageIdx ? "on" : ""}">${s}</li>`).join("")}</ol>
232
+ <p class="state">${esc(p.message || "Starting")}. Elapsed ${Math.round(p.elapsed || 0)} s.</p><div class="actions"><button class="btn" data-c="stop">Stop</button></div>` : ""}
233
+ ${p && p.state === "failed" ? `<p class="note bad">${esc(p.message)}</p><div class="actions"><button class="btn" data-c="again">Change the recipe</button><button class="btn go" data-c="run">Try again</button></div>` : ""}
234
+ ${p && fin && p.report ? `<div class="stats"><div><b>${out.docs}</b><span>documents ready</span></div><div><b>${out.curated}</b><span>kept from sources</span></div><div><b>${out.created}</b><span>created</span></div><div><b>${fmt(out.tokens)}</b><span>tokens, about</span></div></div>
235
+ <div class="actions"><a class="btn go" href="/api/curate/previews/${p.id}/download" style="text-decoration:none">Download the sample (JSONL)</a><button class="btn" data-c="recipe">Download recipe</button><button class="btn" data-c="again">Change the recipe</button></div>
236
+ ${runYourself()}${report(p.report)}` : ""}`;
237
+ }
238
+ function runYourself() {
239
+ const inst = C.d ? C.d.install : "pip install datainventor";
240
+ return `<div class="group"><h3>Run it on all of your data</h3>
241
+ <p class="hint" style="max-width:62ch">On your own machines, the same recipe reads every document, grades ${C.gradeDocs.toLocaleString()} with your model, and writes the curated corpus and each mix as JSONL shards with a report. Point it at any OpenAI-compatible server (vLLM, SGLang, Ollama or a hosted API) by editing <code>endpoint</code> in the recipe. Interrupted runs resume where they stopped.</p>
242
+ <pre class="code">${esc(inst)}
243
+ export DI_API_KEY=... # key for the endpoint in the recipe
244
+ datainventor preview recipe.yaml # optional: the hosted check, locally
245
+ datainventor run recipe.yaml --out ./curated --procs 32 --workers 64</pre></div>`;
246
+ }
247
+
248
+ /* ------------------------------------------------------------ render + events */
249
+ function render() {
250
+ if (window.DI_MODE !== "curate") return;
251
+ ledger(); sheet();
252
+ const fn = [stepSources, stepClean, stepCalibrate, stepCreate, stepCompose, stepRun][C.step];
253
+ const keep = document.activeElement && document.activeElement.id;
254
+ $("#panel").innerHTML = fn();
255
+ if (keep && keep.startsWith("c-")) { const el = document.getElementById(keep); if (el) { el.focus(); if (el.setSelectionRange && el.type === "text") { const l = el.value.length; try { el.setSelectionRange(l, l); } catch (e) { /* ok */ } } } }
256
+ }
257
+ function go(i) { C.step = i; C.maxStep = Math.max(C.maxStep, i); render(); $("#panel").focus({ preventScroll: true }); window.scrollTo({ top: 0 }); }
258
+
259
+ async function addDataset(id, role, kind) {
260
+ C.addBusy = true; C.addErr = ""; render();
261
+ try {
262
+ const info = await api(`/api/curate/inspect?dataset=${encodeURIComponent(id)}`);
263
+ const list = role === "target" ? C.targets : C.sources, max = role === "target" ? (C.d ? C.d.max_targets : 2) : (C.d ? C.d.max_sources : 4);
264
+ if (list.length >= max) throw new Error(`The hosted run takes up to ${max} ${role}s. The recipe takes any number.`);
265
+ let name = info.dataset.split("/")[1]; const taken = new Set([...C.sources, ...C.targets].map((s) => s.name));
266
+ for (let k = 2; taken.has(name); k++) name = `${info.dataset.split("/")[1]}-${k}`;
267
+ list.push({ ...info, name, kind: kind || info.kind });
268
+ C.add = ""; C.pv = null;
269
+ } catch (e) { C.addErr = e.message; }
270
+ C.addBusy = false; render();
271
+ }
272
+ async function download() {
273
+ try {
274
+ const r = await fetch("/api/curate/recipe", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ recipe: recipe() }) });
275
+ if (!r.ok) { const d = await r.json().catch(() => ({})); throw new Error(d.detail || "Could not build the recipe"); }
276
+ const url = URL.createObjectURL(await r.blob()), a = document.createElement("a");
277
+ a.href = url; a.download = "recipe.yaml"; document.body.appendChild(a); a.click(); a.remove(); setTimeout(() => URL.revokeObjectURL(url), 2000);
278
+ } catch (e) { C.runErr = e.message; render(); }
279
+ }
280
+ async function run() {
281
+ C.runErr = ""; C.pv = null;
282
+ try { const r = await api("/api/curate/previews", { recipe: recipe() }); C.pid = r.id; C.pv = { id: r.id, state: "queued", stage: "setup", message: "Starting" }; render(); poll(); }
283
+ catch (e) { C.runErr = e.message; render(); }
284
+ }
285
+ async function poll() {
286
+ const id = C.pid; if (!id) return;
287
+ try {
288
+ const p = await api(`/api/curate/previews/${id}`);
289
+ if (id !== C.pid) return;
290
+ C.pv = p; if (C.step === 5) render();
291
+ if (!isFinal(p)) setTimeout(poll, 1500);
292
+ } catch (e) { C.pv = { id, state: "failed", message: e.message }; if (C.step === 5) render(); }
293
+ }
294
+ async function tryIt() {
295
+ C.tryBusy = true; C.tryErr = ""; C.tryRes = null; render();
296
+ const kind = (C.sources.find((s) => s.name === C.tryFrom) || {}).kind || "web";
297
+ try { C.tryRes = await api("/api/curate/try", { text: C.tryText, kind, styles: C.styles.slice(0, 4) }); } catch (e) { C.tryErr = e.message; }
298
+ C.tryBusy = false; render();
299
+ }
300
+
301
+ document.addEventListener("click", async (ev) => {
302
+ if (window.DI_MODE !== "curate") return;
303
+ const t = ev.target.closest("button, a"); if (!t) return;
304
+ if (t.dataset.cgo !== undefined) return go(+t.dataset.cgo);
305
+ if (t.dataset.bench) { const b = t.dataset.bench; C.decon = C.decon.includes(b) ? C.decon.filter((x) => x !== b) : [...C.decon, b]; return render(); }
306
+ if (t.dataset.style) { const s = t.dataset.style; C.styles = C.styles.includes(s) ? C.styles.filter((x) => x !== s) : [...C.styles, s]; return render(); }
307
+ const a = t.dataset.c; if (!a) return;
308
+ if (a === "next") return go(C.step + 1);
309
+ if (a === "back") return go(Math.max(0, C.step - 1));
310
+ if (a === "add") return addDataset(C.add.trim(), C.addRole);
311
+ if (a === "preset") { for (const p of PRESET) await addDataset(p.dataset, "source", p.kind); if (!C.targetsDesc) C.targetsDesc = ""; return render(); }
312
+ if (a === "rm") { (t.dataset.role === "target" ? C.targets : C.sources).splice(+t.dataset.i, 1); C.pv = null; return render(); }
313
+ if (a === "near") { C.near = !C.near; return render(); }
314
+ if (a === "reph") { C.rephraseOn = !C.rephraseOn; return render(); }
315
+ if (a === "inv") { C.inventOn = !C.inventOn; return render(); }
316
+ if (a === "tryfill") { const s = C.sources.find((x) => x.preview); if (s) { C.tryText = s.preview; C.tryFrom = s.name; } return render(); }
317
+ if (a === "try") return tryIt();
318
+ if (a === "budget") { C.mixes[+t.dataset.m].budget = +t.dataset.v; return render(); }
319
+ if (a === "addmix") { C.mixes.push({ name: `mix-${C.mixes.length + 1}`, budget: 1e9, weights: {} }); return render(); }
320
+ if (a === "rmmix") { C.mixes.splice(+t.dataset.m, 1); return render(); }
321
+ if (a === "recipe") return download();
322
+ if (a === "run") return run();
323
+ if (a === "stop") { if (C.pid) await api(`/api/curate/previews/${C.pid}/stop`, {}); return; }
324
+ if (a === "again") { C.pv = null; C.pid = null; return go(0); }
325
+ });
326
+ document.addEventListener("input", (ev) => {
327
+ if (window.DI_MODE !== "curate") return;
328
+ const el = ev.target, id = el.id || ""; if (!id.startsWith("c-")) return;
329
+ const v = el.value;
330
+ if (id === "c-add") { C.add = v; const b = $('[data-c="add"]'); if (b) b.disabled = C.addBusy || !v.trim(); return; }
331
+ if (id === "c-tdesc") { C.targetsDesc = v; return sheet(); }
332
+ if (id === "c-try") { C.tryText = v; C.tryFrom = ""; const b = $('[data-c="try"]'); if (b) b.disabled = C.tryBusy || v.trim().split(/\s+/).length < 60; return; }
333
+ if (id === "c-invreq") { C.inventReq = v; return; }
334
+ const sv = (x) => { const o = document.getElementById(id + "-v"); if (o) o.textContent = x; };
335
+ if (id === "c-focus") { C.focus = +v; sv(+v === 0 ? "general quality only" : `${pct(1 - v)} quality / ${pct(v)} targets`); return sheet(); }
336
+ if (id === "c-keep") { C.keepTop = +v; sv(`${pct(v)} of each source`); return sheet(); }
337
+ if (id === "c-epochs") { C.maxEpochs = +v; sv(`${v} times`); return; }
338
+ if (id === "c-synth") { C.maxSynth = +v; sv(`${pct(v)} of a mix`); return; }
339
+ });
340
+ document.addEventListener("change", (ev) => {
341
+ if (window.DI_MODE !== "curate") return;
342
+ const el = ev.target, id = el.id || ""; if (!id.startsWith("c-")) return;
343
+ const v = el.value; let m;
344
+ if (id === "c-addrole") C.addRole = v;
345
+ else if ((m = id.match(/^c-kind-(\d+)$/))) { C.sources[+m[1]].kind = v; C.pv = null; }
346
+ else if ((m = id.match(/^c-col-(source|target)-(\d+)$/))) { (m[1] === "target" ? C.targets : C.sources)[+m[2]].text_field = v; C.pv = null; }
347
+ else if ((m = id.match(/^c-rule-(.+)$/))) { if (v !== "") C.rules[m[1]] = +v; }
348
+ else if (id === "c-grade") C.gradeDocs = Math.max(100, +v || 2000);
349
+ else if (id === "c-scorer") C.scorer = v;
350
+ else if (id === "c-invdocs") C.inventDocs = Math.max(0, +v || 0);
351
+ else if ((m = id.match(/^c-mname-(\d+)$/))) C.mixes[+m[1]].name = v.trim().replace(/[^\w.-]+/g, "-") || `mix-${+m[1] + 1}`;
352
+ else if ((m = id.match(/^c-mbud-(\d+)$/))) C.mixes[+m[1]].budget = Math.max(1e6, Math.round(+v || 0));
353
+ else if ((m = id.match(/^c-w-(\d+)-(.+)$/))) { const mix = C.mixes[+m[1]]; mix.weights = { ...weightsOf(mix), [m[2]]: Math.max(0, +v || 0) / 100 }; }
354
+ else return;
355
+ render();
356
+ });
357
+ document.addEventListener("keydown", (ev) => { if (window.DI_MODE === "curate" && ev.key === "Enter" && ev.target.id === "c-add" && C.add.trim()) addDataset(C.add.trim(), C.addRole); });
358
+
359
+ /* ------------------------------------------------------------ mode switch */
360
+ function setMode(mode) {
361
+ window.DI_MODE = mode;
362
+ document.querySelectorAll("[data-mode]").forEach((b) => b.setAttribute("aria-pressed", String(b.dataset.mode === mode)));
363
+ $(".tag").textContent = mode === "curate" ? "Clean, score, rephrase and mix the pretraining data you already have. Preview it here on a sample, then run the recipe on your own machines." : "Describe the data you wish existed. Get training rows back. Bring any OpenAI-compatible model.";
364
+ if (mode === "curate") { if (!C.d) api("/api/curate/defaults").then((d) => { C.d = d; render(); }).catch(() => render()); render(); }
365
+ else if (window.DI_renderInvent) window.DI_renderInvent();
366
+ if ((location.hash === "#curate") !== (mode === "curate")) history.replaceState(null, "", mode === "curate" ? "#curate" : location.pathname + location.search);
367
+ }
368
+ document.addEventListener("click", (ev) => { const b = ev.target.closest("[data-mode]"); if (b) setMode(b.dataset.mode); });
369
+ window.DI_MODE = location.hash === "#curate" ? "curate" : "invent";
370
+ if (window.DI_MODE === "curate") document.addEventListener("DOMContentLoaded", () => setMode("curate"));
371
+ })();
static/index.html CHANGED
@@ -4,7 +4,7 @@
4
  <meta charset="utf-8">
5
  <meta name="viewport" content="width=device-width, initial-scale=1">
6
  <title>DataInventor</title>
7
- <meta name="description" content="Describe a dataset in plain words and get training data back. Open source, works with any OpenAI-compatible model.">
8
  <link rel="preconnect" href="https://fonts.googleapis.com">
9
  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
10
  <link href="https://fonts.googleapis.com/css2?family=Familjen+Grotesk:ital,wght@0,400..700;1,400..700&family=Literata:ital,opsz,wght@0,7..72,400..600;1,7..72,400..600&display=swap" rel="stylesheet">
@@ -17,6 +17,7 @@
17
  <span class="mark" aria-hidden="true"><i></i><i></i></span>
18
  <span class="name">DataInventor</span>
19
  </div>
 
20
  <p class="tag">Describe the data you wish existed. Get training rows back. Bring any OpenAI-compatible model.</p>
21
  </header>
22
 
@@ -29,6 +30,7 @@
29
  <footer class="foot">
30
  <span>Open source recreation of the Invent a Dataset workflow. Not affiliated with Adaption Labs.</span>
31
  </footer>
 
32
  <script src="/static/app.js"></script>
33
  </body>
34
  </html>
 
4
  <meta charset="utf-8">
5
  <meta name="viewport" content="width=device-width, initial-scale=1">
6
  <title>DataInventor</title>
7
+ <meta name="description" content="Describe a dataset in plain words and get training data back, or clean, score, rephrase and mix the pretraining data you have. Open source, works with any OpenAI-compatible model.">
8
  <link rel="preconnect" href="https://fonts.googleapis.com">
9
  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
10
  <link href="https://fonts.googleapis.com/css2?family=Familjen+Grotesk:ital,wght@0,400..700;1,400..700&family=Literata:ital,opsz,wght@0,7..72,400..600;1,7..72,400..600&display=swap" rel="stylesheet">
 
17
  <span class="mark" aria-hidden="true"><i></i><i></i></span>
18
  <span class="name">DataInventor</span>
19
  </div>
20
+ <nav class="modes" aria-label="Mode"><button data-mode="invent" aria-pressed="true">Invent rows</button><button data-mode="curate" aria-pressed="false">Curate a corpus</button></nav>
21
  <p class="tag">Describe the data you wish existed. Get training rows back. Bring any OpenAI-compatible model.</p>
22
  </header>
23
 
 
30
  <footer class="foot">
31
  <span>Open source recreation of the Invent a Dataset workflow. Not affiliated with Adaption Labs.</span>
32
  </footer>
33
+ <script src="/static/curate.js"></script>
34
  <script src="/static/app.js"></script>
35
  </body>
36
  </html>