Spaces:
Running
Running
Curate a corpus: clean, calibrate, create, compose on a sample, plus a recipe and runner for your own machines
Browse files- Dockerfile +1 -0
- README.md +29 -3
- app/curate_api.py +166 -0
- app/engine.py +2 -64
- app/main.py +8 -0
- datainventor/__init__.py +0 -0
- datainventor/calibrate.py +277 -0
- datainventor/clean.py +233 -0
- datainventor/cli.py +98 -0
- datainventor/compose.py +83 -0
- datainventor/create.py +152 -0
- datainventor/data/benchmarks.json.gz +3 -0
- datainventor/llm.py +72 -0
- datainventor/preview.py +268 -0
- datainventor/recipe.py +108 -0
- datainventor/runner.py +497 -0
- datainventor/sources.py +192 -0
- datainventor/text.py +143 -0
- make_benchmarks.py +16 -0
- pyproject.toml +24 -0
- requirements.txt +1 -0
- static/app.css +55 -0
- static/app.js +2 -0
- static/curate.js +371 -0
- static/index.html +3 -1
Dockerfile
CHANGED
|
@@ -6,6 +6,7 @@ ENV PATH="/home/user/.local/bin:$PATH"
|
|
| 6 |
WORKDIR /home/user/app
|
| 7 |
COPY --chown=user requirements.txt .
|
| 8 |
RUN pip install --no-cache-dir -r requirements.txt
|
|
|
|
| 9 |
COPY --chown=user app ./app
|
| 10 |
COPY --chown=user static ./static
|
| 11 |
WORKDIR /home/user/app/app
|
|
|
|
| 6 |
WORKDIR /home/user/app
|
| 7 |
COPY --chown=user requirements.txt .
|
| 8 |
RUN pip install --no-cache-dir -r requirements.txt
|
| 9 |
+
COPY --chown=user datainventor ./datainventor
|
| 10 |
COPY --chown=user app ./app
|
| 11 |
COPY --chown=user static ./static
|
| 12 |
WORKDIR /home/user/app/app
|
README.md
CHANGED
|
@@ -10,12 +10,15 @@ hf_oauth: true
|
|
| 10 |
hf_oauth_scopes:
|
| 11 |
- contribute-repos
|
| 12 |
hf_oauth_expiration_minutes: 120
|
| 13 |
-
short_description:
|
| 14 |
---
|
| 15 |
|
| 16 |
# DataInventor
|
| 17 |
|
| 18 |
-
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
It works with any OpenAI-compatible endpoint. The default is a free model served through Hugging Face Inference Providers.
|
| 21 |
|
|
@@ -26,7 +29,30 @@ It works with any OpenAI-compatible endpoint. The default is a free model served
|
|
| 26 |
3. **Writing.** One message per combination, then the reply (preference pairs get two replies and a ranking).
|
| 27 |
4. **Filters.** Cut-off or degenerate text, requests for many items at once, and near-duplicate questions are rejected.
|
| 28 |
|
| 29 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
```bash
|
| 32 |
pip install -r requirements.txt
|
|
|
|
| 10 |
hf_oauth_scopes:
|
| 11 |
- contribute-repos
|
| 12 |
hf_oauth_expiration_minutes: 120
|
| 13 |
+
short_description: Invent training rows, or curate your pretraining data
|
| 14 |
---
|
| 15 |
|
| 16 |
# DataInventor
|
| 17 |
|
| 18 |
+
DataInventor has two modes.
|
| 19 |
+
|
| 20 |
+
- **Invent rows.** An open-source recreation of the [Invent a Dataset](https://adaptionlabs.ai/invent-a-dataset) workflow. Describe the data you want, choose a format, domains and languages, preview a blueprint, and get an instruction or preference dataset back as CSV or JSONL.
|
| 21 |
+
- **Curate a corpus.** Clean, score, rephrase and mix the pretraining data you already have. Preview every step here on a sample, then download the recipe and run it on all of your data, on your own machines.
|
| 22 |
|
| 23 |
It works with any OpenAI-compatible endpoint. The default is a free model served through Hugging Face Inference Providers.
|
| 24 |
|
|
|
|
| 29 |
3. **Writing.** One message per combination, then the reply (preference pairs get two replies and a ranking).
|
| 30 |
4. **Filters.** Cut-off or degenerate text, requests for many items at once, and near-duplicate questions are rejected.
|
| 31 |
|
| 32 |
+
## How curation works
|
| 33 |
+
|
| 34 |
+
You register Hugging Face datasets as **sources** to curate, and optionally describe or add **targets** that show what the trained model should be good at. Then four steps run in order.
|
| 35 |
+
|
| 36 |
+
1. **Clean.** Gopher and C4 style heuristics (FineWeb thresholds) drop broken, boilerplate and repetitive documents, with looser rules for math and code. Any document that shares a 13-word run with a test set you name (GSM8K, MATH-500, HumanEval, MMLU, ARC and others, bundled with the package) or with a held-out set of your own is removed.
|
| 37 |
+
2. **Calibrate.** Exact and MinHash near-duplicates are dropped. A model grades a sample for teaching value, clarity, accuracy and density, and for relevance to your targets. A small linear scorer learns those grades and scores every document on CPU. Each source keeps its best share, ranked by a blend of quality and relevance that you set.
|
| 38 |
+
3. **Create.** Where a source is too small for its share of a mix, its best documents are rephrased into new ones (dialogue, tutorial, questions and answers, dense notes, textbook section, worked problems, code walkthrough). These are the styles from the Sutra datasets. Rewrites that add numbers or names the source lacks are dropped. You can also write new documents on a subject.
|
| 39 |
+
4. **Compose.** One prepared corpus becomes one or more mixes, each fitted to a token budget. Short sources repeat up to a cap, then get topped up with rephrased text up to a synthetic cap. The default shares come from our [1B-token mixing study](https://huggingface.co/blog/codelion/optimal-dataset-mixing): 50% textbook-style, 30% web, 20% educational web. That result was measured at 70M parameters, so treat it as a starting point.
|
| 40 |
+
|
| 41 |
+
The hosted run works on about 200 documents per source and returns the kept and created documents with a report. The runner does the same on every document:
|
| 42 |
+
|
| 43 |
+
```bash
|
| 44 |
+
pip install 'datainventor[full] @ git+https://huggingface.co/spaces/codelion/data-inventor'
|
| 45 |
+
export DI_API_KEY=... # key for the endpoint named in the recipe
|
| 46 |
+
datainventor init recipe.yaml # or download one from the app
|
| 47 |
+
datainventor preview recipe.yaml # every step on a sample
|
| 48 |
+
datainventor run recipe.yaml --out ./curated --procs 32 --workers 64
|
| 49 |
+
```
|
| 50 |
+
|
| 51 |
+
It writes `curated/<source>/`, `synthetic/<source>/` and `mixes/<mix>/` as JSONL shards, plus `REPORT.md`. Interrupted runs resume from the last finished stage or source. The endpoint is any OpenAI-compatible server: vLLM, SGLang, Ollama or a hosted API.
|
| 52 |
+
|
| 53 |
+
Two limits to know. Token counts are estimates at four characters per token. The learned scorer is a hashed n-gram model. In our test it reached a rank agreement of about 0.5 with the grading model after 300 grades, so grade more documents, or set `calibrate.scorer: llm` to grade every document with the model.
|
| 54 |
+
|
| 55 |
+
## Run the app yourself
|
| 56 |
|
| 57 |
```bash
|
| 58 |
pip install -r requirements.txt
|
app/curate_api.py
ADDED
|
@@ -0,0 +1,166 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Curate a corpus: sample previews of the clean, calibrate, create, compose workflow, plus the recipe to run it in full."""
|
| 2 |
+
from __future__ import annotations
|
| 3 |
+
|
| 4 |
+
import json
|
| 5 |
+
import os
|
| 6 |
+
import threading
|
| 7 |
+
import time
|
| 8 |
+
import uuid
|
| 9 |
+
|
| 10 |
+
from fastapi import APIRouter, HTTPException
|
| 11 |
+
from fastapi.responses import Response
|
| 12 |
+
from pydantic import BaseModel
|
| 13 |
+
|
| 14 |
+
from datainventor.clean import BENCHMARKS, DEFAULTS as CLEAN_RULES
|
| 15 |
+
from datainventor.create import DOC_TYPES, STYLES, chunks, rephrase, styles_for
|
| 16 |
+
from datainventor.llm import EndpointError
|
| 17 |
+
from datainventor.preview import Preview, run_preview
|
| 18 |
+
from datainventor.recipe import DEFAULT, dump, validate
|
| 19 |
+
from datainventor.sources import KINDS, inspect
|
| 20 |
+
|
| 21 |
+
router = APIRouter(prefix="/api/curate")
|
| 22 |
+
PER_SOURCE = int(os.environ.get("CURATE_PER_SOURCE", "200"))
|
| 23 |
+
GRADE_N = int(os.environ.get("CURATE_GRADE", "48"))
|
| 24 |
+
REPHRASE_N = int(os.environ.get("CURATE_REPHRASE", "8"))
|
| 25 |
+
MAX_SOURCES, MAX_TARGETS = 4, 2
|
| 26 |
+
INSTALL = "pip install 'datainventor[full] @ git+https://huggingface.co/spaces/codelion/data-inventor'"
|
| 27 |
+
PREVIEWS: dict[str, Preview] = {}
|
| 28 |
+
TRIES: dict[str, list] = {}
|
| 29 |
+
GUARD = threading.Lock()
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def _ep():
|
| 33 |
+
from main import make_endpoint # the host's endpoint; visitors never supply one
|
| 34 |
+
return make_endpoint()
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def _token() -> str:
|
| 38 |
+
return os.environ.get("HF_TOKEN", "")
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
@router.get("/defaults")
|
| 42 |
+
def defaults():
|
| 43 |
+
from main import MAX_ROWS
|
| 44 |
+
return {"recipe": DEFAULT, "benchmarks": list(BENCHMARKS), "kinds": list(KINDS), "clean_rules": CLEAN_RULES,
|
| 45 |
+
"styles": {k: v[0] for k, v in STYLES.items()}, "styles_by_kind": {k: styles_for(k) for k in KINDS},
|
| 46 |
+
"doc_types": list(DOC_TYPES), "per_source": PER_SOURCE, "max_sources": MAX_SOURCES, "max_targets": MAX_TARGETS,
|
| 47 |
+
"max_docs": MAX_ROWS, "install": INSTALL}
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
@router.get("/inspect")
|
| 51 |
+
def do_inspect(dataset: str):
|
| 52 |
+
ds = dataset.strip().removeprefix("https://huggingface.co/datasets/").strip("/")
|
| 53 |
+
if not ds or ds.count("/") != 1 or len(ds) > 120:
|
| 54 |
+
raise HTTPException(400, "Use a Hugging Face dataset id like HuggingFaceFW/fineweb-edu.")
|
| 55 |
+
try:
|
| 56 |
+
return inspect(ds, _token())
|
| 57 |
+
except ValueError as e:
|
| 58 |
+
raise HTTPException(404, str(e))
|
| 59 |
+
except Exception:
|
| 60 |
+
raise HTTPException(404, "The dataset viewer could not open that dataset. It must be public and have the viewer enabled.")
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
class RecipeIn(BaseModel):
|
| 64 |
+
recipe: dict
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
def _checked(rec: dict) -> dict:
|
| 68 |
+
for s in rec.get("sources", []) + rec.get("targets", {}).get("datasets", []):
|
| 69 |
+
if s.get("files"):
|
| 70 |
+
raise HTTPException(400, "Local files only work in the runner on your own machines. Use Hugging Face datasets here.")
|
| 71 |
+
if len(rec.get("sources", [])) > MAX_SOURCES or len(rec.get("targets", {}).get("datasets", [])) > MAX_TARGETS:
|
| 72 |
+
raise HTTPException(400, f"The hosted preview takes up to {MAX_SOURCES} sources and {MAX_TARGETS} target datasets.")
|
| 73 |
+
try:
|
| 74 |
+
return validate(rec)
|
| 75 |
+
except ValueError as e:
|
| 76 |
+
raise HTTPException(400, str(e))
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
@router.post("/recipe")
|
| 80 |
+
def recipe_file(body: RecipeIn):
|
| 81 |
+
"""The recipe as YAML, for the runner."""
|
| 82 |
+
r = _checked(body.recipe)
|
| 83 |
+
return Response(dump(r), media_type="application/x-yaml", headers={"Content-Disposition": 'attachment; filename="recipe.yaml"'})
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
@router.post("/previews")
|
| 87 |
+
def start(body: RecipeIn):
|
| 88 |
+
r = _checked(body.recipe)
|
| 89 |
+
ep = _ep()
|
| 90 |
+
with GUARD:
|
| 91 |
+
running = sum(1 for p in PREVIEWS.values() if p.state in ("queued", "running"))
|
| 92 |
+
if running >= 2:
|
| 93 |
+
raise HTTPException(429, "Two previews are already running. Wait for one to finish.")
|
| 94 |
+
pid = uuid.uuid4().hex[:10]
|
| 95 |
+
p = Preview(r)
|
| 96 |
+
PREVIEWS[pid] = p
|
| 97 |
+
for old in sorted(PREVIEWS, key=lambda k: PREVIEWS[k].started)[:-20]:
|
| 98 |
+
PREVIEWS.pop(old, None)
|
| 99 |
+
from main import MAX_ROWS
|
| 100 |
+
threading.Thread(target=run_preview, args=(p, ep, _token()),
|
| 101 |
+
kwargs={"per_source": PER_SOURCE, "grade_n": GRADE_N, "rephrase_n": REPHRASE_N, "invent_n": 4, "max_docs": MAX_ROWS},
|
| 102 |
+
daemon=True).start()
|
| 103 |
+
return {"id": pid}
|
| 104 |
+
|
| 105 |
+
|
| 106 |
+
def _get(pid: str) -> Preview:
|
| 107 |
+
if pid not in PREVIEWS:
|
| 108 |
+
raise HTTPException(404, "That preview is no longer here. Run it again.")
|
| 109 |
+
return PREVIEWS[pid]
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
@router.get("/previews/{pid}")
|
| 113 |
+
def status(pid: str):
|
| 114 |
+
p = _get(pid)
|
| 115 |
+
out = {"id": pid, "state": p.state, "stage": p.stage, "message": p.message, "elapsed": round((p.finished or time.time()) - p.started, 1)}
|
| 116 |
+
if p.state in ("done", "stopped"):
|
| 117 |
+
out["report"] = p.report
|
| 118 |
+
return out
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
@router.post("/previews/{pid}/stop")
|
| 122 |
+
def stop(pid: str):
|
| 123 |
+
_get(pid).stop.set()
|
| 124 |
+
return {"ok": True}
|
| 125 |
+
|
| 126 |
+
|
| 127 |
+
@router.get("/previews/{pid}/download")
|
| 128 |
+
def download(pid: str):
|
| 129 |
+
p = _get(pid)
|
| 130 |
+
if not p.docs:
|
| 131 |
+
raise HTTPException(404, "There are no documents to download yet.")
|
| 132 |
+
body = "\n".join(json.dumps(d, ensure_ascii=False) for d in p.docs) + "\n"
|
| 133 |
+
return Response(body, media_type="application/jsonl", headers={"Content-Disposition": f'attachment; filename="curated-sample-{pid}.jsonl"'})
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
class TryIn(BaseModel):
|
| 137 |
+
text: str
|
| 138 |
+
kind: str = "web"
|
| 139 |
+
styles: list[str] = []
|
| 140 |
+
|
| 141 |
+
|
| 142 |
+
@router.post("/try")
|
| 143 |
+
def try_styles(body: TryIn):
|
| 144 |
+
"""Rewrite one document in up to four styles, side by side."""
|
| 145 |
+
text = body.text.strip()
|
| 146 |
+
if len(text.split()) < 60:
|
| 147 |
+
raise HTTPException(400, "Paste or pick a document of at least 60 words.")
|
| 148 |
+
styles = [s for s in (body.styles or styles_for(body.kind)) if s in STYLES][:4]
|
| 149 |
+
if not styles:
|
| 150 |
+
raise HTTPException(400, "Choose at least one style.")
|
| 151 |
+
now = time.time()
|
| 152 |
+
with GUARD:
|
| 153 |
+
recent = [t for t in TRIES.get("all", []) if now - t < 60]
|
| 154 |
+
if len(recent) >= 6:
|
| 155 |
+
raise HTTPException(429, "Too many rewrites in the last minute. Try again shortly.")
|
| 156 |
+
TRIES["all"] = recent + [now]
|
| 157 |
+
ep = _ep()
|
| 158 |
+
piece = chunks(text)[0]
|
| 159 |
+
from concurrent.futures import ThreadPoolExecutor
|
| 160 |
+
try:
|
| 161 |
+
with ThreadPoolExecutor(len(styles)) as ex:
|
| 162 |
+
outs = list(ex.map(lambda s: rephrase(ep, piece, s), styles))
|
| 163 |
+
except EndpointError as e:
|
| 164 |
+
raise HTTPException(502, str(e))
|
| 165 |
+
return {"source": piece, "results": [{"style": o["style"], "name": STYLES[o["style"]][0], "text": o["text"], "ok": o["ok"],
|
| 166 |
+
"why": o.get("why", ""), "fidelity": o.get("fidelity")} for o in outs]}
|
app/engine.py
CHANGED
|
@@ -10,73 +10,11 @@ import random
|
|
| 10 |
import re
|
| 11 |
import threading
|
| 12 |
import time
|
| 13 |
-
import urllib.request
|
| 14 |
-
import urllib.error
|
| 15 |
from concurrent.futures import ThreadPoolExecutor
|
| 16 |
from dataclasses import dataclass, field
|
| 17 |
-
from typing import
|
| 18 |
|
| 19 |
-
HF_BASE
|
| 20 |
-
HF_MODEL = "Qwen/Qwen3-4B-Instruct-2507:nscale"
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
@dataclass
|
| 24 |
-
class Endpoint:
|
| 25 |
-
base_url: str = HF_BASE
|
| 26 |
-
api_key: str = ""
|
| 27 |
-
model: str = HF_MODEL
|
| 28 |
-
disable_thinking: bool = False
|
| 29 |
-
workers: int = 4
|
| 30 |
-
|
| 31 |
-
def chat(self, prompt: str, max_tokens: int = 1500, temperature: float = 0.8, retries: int = 5,
|
| 32 |
-
validate: Optional[Callable[[str], bool]] = None) -> str:
|
| 33 |
-
body = {"model": self.model, "max_tokens": max_tokens, "temperature": temperature,
|
| 34 |
-
"messages": [{"role": "user", "content": prompt}]}
|
| 35 |
-
if self.disable_thinking:
|
| 36 |
-
body["chat_template_kwargs"] = {"enable_thinking": False}
|
| 37 |
-
data = json.dumps(body).encode()
|
| 38 |
-
url = self.base_url.rstrip("/") + "/chat/completions"
|
| 39 |
-
last = "no attempt"
|
| 40 |
-
for attempt in range(retries):
|
| 41 |
-
try:
|
| 42 |
-
req = urllib.request.Request(url, data, {"Content-Type": "application/json", "User-Agent": "DataInventor/1.0",
|
| 43 |
-
"Authorization": "Bearer " + self.api_key})
|
| 44 |
-
resp = json.load(urllib.request.urlopen(req, timeout=180))
|
| 45 |
-
choice = (resp.get("choices") or [{}])[0] or {}
|
| 46 |
-
if choice.get("finish_reason") == "length":
|
| 47 |
-
last = "reply cut off at max_tokens"
|
| 48 |
-
time.sleep(1.5)
|
| 49 |
-
continue
|
| 50 |
-
text = ((choice.get("message") or {}).get("content") or "").strip()
|
| 51 |
-
text = re.sub(r"<think>.*?</think>", "", text, flags=re.S).strip()
|
| 52 |
-
if text and (validate is None or validate(text)):
|
| 53 |
-
return text
|
| 54 |
-
last = "reply rejected by a quality check"
|
| 55 |
-
except urllib.error.HTTPError as e:
|
| 56 |
-
detail = ""
|
| 57 |
-
try:
|
| 58 |
-
detail = e.read().decode()[:200]
|
| 59 |
-
except Exception:
|
| 60 |
-
pass
|
| 61 |
-
last = f"HTTP {e.code} {detail}"
|
| 62 |
-
if e.code in (401, 403):
|
| 63 |
-
raise EndpointError(f"The endpoint refused the key (HTTP {e.code}). Check the API key.")
|
| 64 |
-
if e.code == 404:
|
| 65 |
-
raise EndpointError("The endpoint or model was not found (HTTP 404). Check the base URL and model name.")
|
| 66 |
-
time.sleep(min(30, 2 ** attempt * 2) if e.code == 429 else 3 * (attempt + 1))
|
| 67 |
-
except Exception as e: # timeouts, resets
|
| 68 |
-
last = str(e)[:160]
|
| 69 |
-
time.sleep(3 * (attempt + 1))
|
| 70 |
-
raise EndpointError(last)
|
| 71 |
-
|
| 72 |
-
def ping(self) -> dict:
|
| 73 |
-
t = time.time()
|
| 74 |
-
text = self.chat("Reply with the single word ready.", max_tokens=16, temperature=0, retries=2)
|
| 75 |
-
return {"ok": True, "latency_ms": int((time.time() - t) * 1000), "sample": text[:40]}
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
class EndpointError(Exception):
|
| 79 |
-
pass
|
| 80 |
|
| 81 |
|
| 82 |
# ---------------------------------------------------------------- blueprint
|
|
|
|
| 10 |
import re
|
| 11 |
import threading
|
| 12 |
import time
|
|
|
|
|
|
|
| 13 |
from concurrent.futures import ThreadPoolExecutor
|
| 14 |
from dataclasses import dataclass, field
|
| 15 |
+
from typing import Optional
|
| 16 |
|
| 17 |
+
from datainventor.llm import HF_BASE, HF_MODEL, Endpoint, EndpointError # noqa: F401 (main imports them from here)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
|
| 19 |
|
| 20 |
# ---------------------------------------------------------------- blueprint
|
app/main.py
CHANGED
|
@@ -12,8 +12,11 @@ import urllib.parse
|
|
| 12 |
import urllib.request
|
| 13 |
import time
|
| 14 |
import uuid
|
|
|
|
| 15 |
from pathlib import Path
|
| 16 |
|
|
|
|
|
|
|
| 17 |
from fastapi import FastAPI, HTTPException, Request
|
| 18 |
from fastapi.responses import FileResponse, JSONResponse, RedirectResponse, Response
|
| 19 |
from fastapi.staticfiles import StaticFiles
|
|
@@ -256,6 +259,11 @@ def publish(jid: str, body: PublishIn, request: Request):
|
|
| 256 |
return {"url": url}
|
| 257 |
|
| 258 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 259 |
@app.get("/")
|
| 260 |
def index():
|
| 261 |
return FileResponse(STATIC / "index.html")
|
|
|
|
| 12 |
import urllib.request
|
| 13 |
import time
|
| 14 |
import uuid
|
| 15 |
+
import sys
|
| 16 |
from pathlib import Path
|
| 17 |
|
| 18 |
+
sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) # the datainventor package sits next to app/
|
| 19 |
+
|
| 20 |
from fastapi import FastAPI, HTTPException, Request
|
| 21 |
from fastapi.responses import FileResponse, JSONResponse, RedirectResponse, Response
|
| 22 |
from fastapi.staticfiles import StaticFiles
|
|
|
|
| 259 |
return {"url": url}
|
| 260 |
|
| 261 |
|
| 262 |
+
from curate_api import router as curate_router # noqa: E402 (imports make_endpoint from this module)
|
| 263 |
+
|
| 264 |
+
app.include_router(curate_router)
|
| 265 |
+
|
| 266 |
+
|
| 267 |
@app.get("/")
|
| 268 |
def index():
|
| 269 |
return FileResponse(STATIC / "index.html")
|
datainventor/__init__.py
ADDED
|
File without changes
|
datainventor/calibrate.py
ADDED
|
@@ -0,0 +1,277 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Calibrate: drop redundant copies, score every document, and reweight toward what you care about.
|
| 2 |
+
|
| 3 |
+
Scoring follows the FineWeb-Edu pattern. A language model grades a few thousand documents against a rubric. A cheap
|
| 4 |
+
linear model learns those grades from hashed n-grams and scores the rest of the corpus on CPU. Task relevance is
|
| 5 |
+
similarity to the target datasets you registered.
|
| 6 |
+
"""
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
import json
|
| 10 |
+
import math
|
| 11 |
+
import random
|
| 12 |
+
import re
|
| 13 |
+
from collections import Counter
|
| 14 |
+
from dataclasses import dataclass, field
|
| 15 |
+
|
| 16 |
+
from .llm import Endpoint, EndpointError
|
| 17 |
+
from .text import NearDupIndex, centroid, cosine, doc_hash, features, h64, norm
|
| 18 |
+
|
| 19 |
+
# ---------------------------------------------------------------- dedup + repetition report
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
@dataclass
|
| 23 |
+
class Deduper:
|
| 24 |
+
"""Exact (normalized hash) and near (MinHash LSH) duplicate removal. Keeps the first copy it sees and counts
|
| 25 |
+
every copy, so the report can show which passages a source repeats hundreds of times."""
|
| 26 |
+
near: bool = True
|
| 27 |
+
threshold: float = 0.8
|
| 28 |
+
exact_seen: dict = field(default_factory=dict)
|
| 29 |
+
copies: Counter = field(default_factory=Counter)
|
| 30 |
+
first_text: dict = field(default_factory=dict)
|
| 31 |
+
|
| 32 |
+
def __post_init__(self):
|
| 33 |
+
self.index = NearDupIndex(threshold=self.threshold) if self.near else None
|
| 34 |
+
|
| 35 |
+
def check(self, doc: dict) -> str:
|
| 36 |
+
h = doc_hash(doc["text"])
|
| 37 |
+
if h in self.exact_seen:
|
| 38 |
+
self.copies[h] += 1
|
| 39 |
+
return "exact duplicate"
|
| 40 |
+
self.exact_seen[h] = doc["id"]
|
| 41 |
+
self.copies[h] += 1
|
| 42 |
+
self.first_text.setdefault(h, doc["text"][:240])
|
| 43 |
+
if self.index is not None:
|
| 44 |
+
hit = self.index.check_add(h64(h), doc["text"])
|
| 45 |
+
if hit is not None:
|
| 46 |
+
return "near duplicate"
|
| 47 |
+
return ""
|
| 48 |
+
|
| 49 |
+
def report(self, top: int = 8) -> list[dict]:
|
| 50 |
+
return [{"copies": c, "text": self.first_text.get(h, "")} for h, c in self.copies.most_common(top) if c > 1]
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def repeated_passages(docs: list[dict], n: int = 50, top: int = 8) -> list[dict]:
|
| 54 |
+
"""Passages (50-word windows) that recur across documents. Whole-document dedup misses these: the same
|
| 55 |
+
worked solution pasted into many pages. Counts are within the sample, so a rate per 1,000 documents is reported."""
|
| 56 |
+
counts: Counter = Counter()
|
| 57 |
+
example: dict = {}
|
| 58 |
+
for d in docs:
|
| 59 |
+
ws = norm(d["text"]).split()
|
| 60 |
+
seen = set()
|
| 61 |
+
for i in range(0, max(0, len(ws) - n + 1), n // 2):
|
| 62 |
+
g = " ".join(ws[i:i + n])
|
| 63 |
+
k = h64(g)
|
| 64 |
+
if k not in seen:
|
| 65 |
+
seen.add(k)
|
| 66 |
+
counts[k] += 1
|
| 67 |
+
example.setdefault(k, g)
|
| 68 |
+
per = 1000 / max(1, len(docs))
|
| 69 |
+
return [{"docs": c, "per_1000": round(c * per, 1), "text": example[k][:240]} for k, c in counts.most_common(top) if c > 2]
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
# ---------------------------------------------------------------- language-model rubric
|
| 73 |
+
|
| 74 |
+
RUBRIC = """You are grading documents for a language-model pretraining corpus.
|
| 75 |
+
Score the document below on each axis from 0 to 5:
|
| 76 |
+
- "value": how useful it is for teaching a model knowledge and reasoning, at any level from school to research. 0 = spam, ads, navigation or nothing to learn; 2 = some useful facts but disorganized or shallow; 3 = coherent and informative; 4 = clearly explained, like a good textbook section or tutorial; 5 = outstanding teaching material with reasoning and examples.
|
| 77 |
+
- "clarity": how clear and well written it is.
|
| 78 |
+
- "accuracy": how likely the content is correct (5 = no visible errors).
|
| 79 |
+
- "density": information per word (0 = filler, 5 = every sentence carries content).{relevance}
|
| 80 |
+
Return ONLY JSON: {{"value": int, "clarity": int, "accuracy": int, "density": int{rel_key}, "reason": "under 20 words"}}
|
| 81 |
+
|
| 82 |
+
Document:
|
| 83 |
+
<<<
|
| 84 |
+
{doc}
|
| 85 |
+
>>>"""
|
| 86 |
+
REL_AXIS = """
|
| 87 |
+
- "relevance": how much it would help a model get better at the target capabilities: {targets}. 0 = unrelated; 3 = related background; 5 = directly practices them."""
|
| 88 |
+
AXES = ("value", "clarity", "accuracy", "density")
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
def rubric_prompt(text: str, targets: str = "") -> str:
|
| 92 |
+
t = targets.strip()
|
| 93 |
+
return RUBRIC.format(relevance=REL_AXIS.format(targets=t) if t else "", rel_key=', "relevance": int' if t else "",
|
| 94 |
+
doc=text[:6000])
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
def grade(ep: Endpoint, text: str, targets: str = "") -> dict | None:
|
| 98 |
+
"""Quality is graded without reference to the targets, so general quality and task relevance stay separate
|
| 99 |
+
signals that `focus` can trade off."""
|
| 100 |
+
axes = AXES + (("relevance",) if targets.strip() else ())
|
| 101 |
+
|
| 102 |
+
def ok(s):
|
| 103 |
+
try:
|
| 104 |
+
g = json.loads(re.search(r"\{.*\}", s, re.S).group(0))
|
| 105 |
+
return all(isinstance(g.get(k), (int, float)) for k in axes)
|
| 106 |
+
except Exception:
|
| 107 |
+
return False
|
| 108 |
+
try:
|
| 109 |
+
s = ep.chat(rubric_prompt(text, targets), max_tokens=200, temperature=0.0, retries=3, validate=ok)
|
| 110 |
+
except EndpointError:
|
| 111 |
+
return None
|
| 112 |
+
g = json.loads(re.search(r"\{.*\}", s, re.S).group(0))
|
| 113 |
+
out = {k: max(0.0, min(5.0, float(g[k]))) for k in axes}
|
| 114 |
+
out["reason"] = str(g.get("reason", ""))[:160]
|
| 115 |
+
return out
|
| 116 |
+
|
| 117 |
+
|
| 118 |
+
def rubric_score(g: dict) -> float:
|
| 119 |
+
"""General quality in [0, 1]. Value dominates; the others break ties."""
|
| 120 |
+
return round((0.55 * g["value"] + 0.15 * g["clarity"] + 0.15 * g["accuracy"] + 0.15 * g["density"]) / 5, 4)
|
| 121 |
+
|
| 122 |
+
|
| 123 |
+
# ---------------------------------------------------------------- cheap scorer trained on the grades
|
| 124 |
+
|
| 125 |
+
class LinearScorer:
|
| 126 |
+
"""Ridge regression on hashed unigram+bigram features, trained with SGD. Pure Python, so it runs anywhere;
|
| 127 |
+
it scores a few thousand documents per second per core."""
|
| 128 |
+
|
| 129 |
+
def __init__(self, l2: float = 1e-4, epochs: int = 30, lr: float = 0.5, seed: int = 0):
|
| 130 |
+
self.w: dict[int, float] = {}
|
| 131 |
+
self.b = 0.0
|
| 132 |
+
self.l2, self.epochs, self.lr, self.seed = l2, epochs, lr, seed
|
| 133 |
+
|
| 134 |
+
def fit(self, X: list[dict], y: list[float]) -> "LinearScorer":
|
| 135 |
+
self.b = sum(y) / len(y)
|
| 136 |
+
idx = list(range(len(X)))
|
| 137 |
+
rng = random.Random(self.seed)
|
| 138 |
+
for ep in range(self.epochs):
|
| 139 |
+
rng.shuffle(idx)
|
| 140 |
+
lr = self.lr / (1 + ep * 0.3)
|
| 141 |
+
for i in idx:
|
| 142 |
+
err = self.predict_vec(X[i]) - y[i]
|
| 143 |
+
for k, v in X[i].items():
|
| 144 |
+
self.w[k] = self.w.get(k, 0.0) * (1 - lr * self.l2) - lr * err * v
|
| 145 |
+
self.b -= lr * err * 0.1
|
| 146 |
+
return self
|
| 147 |
+
|
| 148 |
+
def predict_vec(self, x: dict) -> float:
|
| 149 |
+
return self.b + sum(self.w.get(k, 0.0) * v for k, v in x.items())
|
| 150 |
+
|
| 151 |
+
def predict(self, text: str) -> float:
|
| 152 |
+
return max(0.0, min(1.0, self.predict_vec(features(text))))
|
| 153 |
+
|
| 154 |
+
def to_dict(self) -> dict:
|
| 155 |
+
return {"b": self.b, "w": {str(k): round(v, 6) for k, v in self.w.items() if abs(v) > 1e-5}}
|
| 156 |
+
|
| 157 |
+
@staticmethod
|
| 158 |
+
def from_dict(d: dict) -> "LinearScorer":
|
| 159 |
+
s = LinearScorer()
|
| 160 |
+
s.b, s.w = d["b"], {int(k): v for k, v in d["w"].items()}
|
| 161 |
+
return s
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
def spearman(a: list[float], b: list[float]) -> float | None:
|
| 165 |
+
def ranks(v):
|
| 166 |
+
order = sorted(range(len(v)), key=lambda i: v[i])
|
| 167 |
+
r = [0.0] * len(v)
|
| 168 |
+
for pos, i in enumerate(order):
|
| 169 |
+
r[i] = pos
|
| 170 |
+
return r
|
| 171 |
+
if len(a) < 3:
|
| 172 |
+
return None
|
| 173 |
+
ra, rb = ranks(a), ranks(b)
|
| 174 |
+
ma, mb = sum(ra) / len(ra), sum(rb) / len(rb)
|
| 175 |
+
cov = sum((x - ma) * (y - mb) for x, y in zip(ra, rb))
|
| 176 |
+
va = math.sqrt(sum((x - ma) ** 2 for x in ra))
|
| 177 |
+
vb = math.sqrt(sum((y - mb) ** 2 for y in rb))
|
| 178 |
+
return round(cov / (va * vb), 3) if va and vb else None
|
| 179 |
+
|
| 180 |
+
|
| 181 |
+
def train_scorer(graded: list[tuple[str, float]], holdout: float = 0.25, seed: int = 0) -> tuple[LinearScorer, float | None]:
|
| 182 |
+
"""Fit on the grades and report rank agreement with the model on held-out documents. Low agreement means the
|
| 183 |
+
scorer needs more graded examples before you trust it on the full corpus."""
|
| 184 |
+
data = list(graded)
|
| 185 |
+
random.Random(seed).shuffle(data)
|
| 186 |
+
cut = max(1, int(len(data) * (1 - holdout))) if len(data) >= 8 else len(data)
|
| 187 |
+
train, test = data[:cut], data[cut:]
|
| 188 |
+
sc = LinearScorer().fit([features(t) for t, _ in train], [y for _, y in train])
|
| 189 |
+
agree = spearman([sc.predict(t) for t, _ in test], [y for _, y in test]) if len(test) >= 3 else None
|
| 190 |
+
if test: # refit on everything once agreement is measured
|
| 191 |
+
sc = LinearScorer().fit([features(t) for t, _ in data], [y for _, y in data])
|
| 192 |
+
return sc, agree
|
| 193 |
+
|
| 194 |
+
|
| 195 |
+
# ---------------------------------------------------------------- task relevance
|
| 196 |
+
|
| 197 |
+
|
| 198 |
+
class Relevance:
|
| 199 |
+
"""Cosine similarity to the centroid of the target documents, rescaled against the source sample so 1.0 means
|
| 200 |
+
'as close to the targets as the closest documents in your sources'."""
|
| 201 |
+
|
| 202 |
+
def __init__(self, target_texts: list[str]):
|
| 203 |
+
self.c = centroid([features(t) for t in target_texts]) if target_texts else {}
|
| 204 |
+
self.lo, self.hi = 0.0, 1.0
|
| 205 |
+
|
| 206 |
+
def __bool__(self):
|
| 207 |
+
return bool(self.c)
|
| 208 |
+
|
| 209 |
+
def raw(self, text: str) -> float:
|
| 210 |
+
return cosine(features(text), self.c) if self.c else 0.0
|
| 211 |
+
|
| 212 |
+
def fit_scale(self, texts: list[str]):
|
| 213 |
+
vals = sorted(self.raw(t) for t in texts)
|
| 214 |
+
if vals:
|
| 215 |
+
self.lo = vals[int(0.05 * (len(vals) - 1))]
|
| 216 |
+
self.hi = vals[int(0.95 * (len(vals) - 1))] or 1.0
|
| 217 |
+
|
| 218 |
+
def score(self, text: str) -> float:
|
| 219 |
+
if not self.c:
|
| 220 |
+
return 0.0
|
| 221 |
+
return round(max(0.0, min(1.0, (self.raw(text) - self.lo) / max(1e-6, self.hi - self.lo))), 4)
|
| 222 |
+
|
| 223 |
+
|
| 224 |
+
def combined(quality: float, relevance: float, focus: float) -> float:
|
| 225 |
+
"""focus 0 = general quality only, 1 = target relevance only."""
|
| 226 |
+
return round((1 - focus) * quality + focus * relevance, 4)
|
| 227 |
+
|
| 228 |
+
|
| 229 |
+
class Calibrator:
|
| 230 |
+
"""Turns grades into two scorers (general quality, task relevance) and scores any document with them.
|
| 231 |
+
Saved with the run, so every machine in a large run scores the same way."""
|
| 232 |
+
|
| 233 |
+
def __init__(self, focus: float = 0.3, targets: str = "", target_texts: list[str] | None = None):
|
| 234 |
+
self.targets = targets.strip()
|
| 235 |
+
self.centroid = Relevance(target_texts or [])
|
| 236 |
+
self.has_targets = bool(self.targets or self.centroid)
|
| 237 |
+
self.focus = float(focus) if self.has_targets else 0.0
|
| 238 |
+
self.q: LinearScorer | None = None
|
| 239 |
+
self.r: LinearScorer | None = None
|
| 240 |
+
self.agreement = {"quality": None, "relevance": None}
|
| 241 |
+
|
| 242 |
+
def fit(self, graded: list[tuple[str, dict]], sample_texts: list[str]) -> "Calibrator":
|
| 243 |
+
self.q, self.agreement["quality"] = train_scorer([(t, rubric_score(g)) for t, g in graded])
|
| 244 |
+
rel = [(t, g["relevance"] / 5) for t, g in graded if "relevance" in g]
|
| 245 |
+
if len(rel) >= 4:
|
| 246 |
+
self.r, self.agreement["relevance"] = train_scorer(rel)
|
| 247 |
+
if self.centroid:
|
| 248 |
+
self.centroid.fit_scale(sample_texts)
|
| 249 |
+
return self
|
| 250 |
+
|
| 251 |
+
def score(self, text: str, g: dict | None = None) -> dict:
|
| 252 |
+
quality = rubric_score(g) if g else round(self.q.predict(text), 4)
|
| 253 |
+
parts = []
|
| 254 |
+
if self.targets:
|
| 255 |
+
parts.append(g["relevance"] / 5 if g and "relevance" in g else (self.r.predict(text) if self.r else 0.0))
|
| 256 |
+
if self.centroid:
|
| 257 |
+
parts.append(self.centroid.score(text))
|
| 258 |
+
relevance = round(sum(parts) / len(parts), 4) if parts else 0.0
|
| 259 |
+
return {"quality": quality, "relevance": relevance, "score": combined(quality, relevance, self.focus)}
|
| 260 |
+
|
| 261 |
+
def to_dict(self) -> dict:
|
| 262 |
+
return {"focus": self.focus, "targets": self.targets, "q": self.q.to_dict() if self.q else None,
|
| 263 |
+
"r": self.r.to_dict() if self.r else None, "agreement": self.agreement,
|
| 264 |
+
"centroid": {"c": {str(k): v for k, v in self.centroid.c.items()}, "lo": self.centroid.lo, "hi": self.centroid.hi}}
|
| 265 |
+
|
| 266 |
+
@staticmethod
|
| 267 |
+
def from_dict(d: dict) -> "Calibrator":
|
| 268 |
+
c = Calibrator(d["focus"], d.get("targets", ""))
|
| 269 |
+
c.focus = d["focus"]
|
| 270 |
+
c.q = LinearScorer.from_dict(d["q"]) if d.get("q") else None
|
| 271 |
+
c.r = LinearScorer.from_dict(d["r"]) if d.get("r") else None
|
| 272 |
+
c.agreement = d.get("agreement", c.agreement)
|
| 273 |
+
cen = d.get("centroid") or {}
|
| 274 |
+
c.centroid.c = {int(k): v for k, v in (cen.get("c") or {}).items()}
|
| 275 |
+
c.centroid.lo, c.centroid.hi = cen.get("lo", 0.0), cen.get("hi", 1.0)
|
| 276 |
+
c.has_targets = bool(c.targets or c.centroid)
|
| 277 |
+
return c
|
datainventor/clean.py
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Clean: drop low-information documents, then remove anything that overlaps the benchmarks you will test on.
|
| 2 |
+
|
| 3 |
+
The prose rules follow the Gopher and C4 heuristics with FineWeb's thresholds. Math and code sources get their own,
|
| 4 |
+
looser rules, because LaTeX and source code fail prose checks by design.
|
| 5 |
+
"""
|
| 6 |
+
from __future__ import annotations
|
| 7 |
+
|
| 8 |
+
import re
|
| 9 |
+
from collections import Counter
|
| 10 |
+
from dataclasses import dataclass, field
|
| 11 |
+
|
| 12 |
+
from .text import ngrams, norm, words
|
| 13 |
+
|
| 14 |
+
BOILER = re.compile(r"(cookie|javascript|privacy policy|terms of (use|service)|all rights reserved|lorem ipsum|"
|
| 15 |
+
r"click here|sign up for our newsletter|accept all|log ?in to (reply|comment)|skip to (main )?content)", re.I)
|
| 16 |
+
AUTOGEN = re.compile(r"(auto-?generated|do not edit|generated by (the )?protoc|minified)", re.I)
|
| 17 |
+
|
| 18 |
+
DEFAULTS = {
|
| 19 |
+
"min_words": 50, "max_words": 100_000, "min_mean_word": 3.0, "max_mean_word": 10.0,
|
| 20 |
+
"max_symbol_ratio": 0.1, "min_alpha_words": 0.8, "max_ellipsis_lines": 0.3,
|
| 21 |
+
"max_dup_lines": 0.3, "max_top2gram": 0.2, "max_dup5gram": 0.15, "max_dup10gram": 0.1, "max_boiler_lines": 0.4,
|
| 22 |
+
"strip_boilerplate_lines": True,
|
| 23 |
+
# code
|
| 24 |
+
"code_max_line": 1000, "code_min_alpha": 0.25,
|
| 25 |
+
}
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
def dup_ngram_chars(ws: list[str], n: int) -> int:
|
| 29 |
+
"""Characters inside repeated n-grams, counting each repeat once and never the first copy (Gopher, as
|
| 30 |
+
implemented in datatrove). Overlapping windows are skipped so a repeated paragraph is not counted n times."""
|
| 31 |
+
seen, dup, i = set(), 0, 0
|
| 32 |
+
while i <= len(ws) - n:
|
| 33 |
+
g = tuple(ws[i:i + n])
|
| 34 |
+
if g in seen:
|
| 35 |
+
dup += sum(len(w) for w in g)
|
| 36 |
+
i += n
|
| 37 |
+
else:
|
| 38 |
+
seen.add(g)
|
| 39 |
+
i += 1
|
| 40 |
+
return dup
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
@dataclass
|
| 44 |
+
class Cleaner:
|
| 45 |
+
rules: dict = field(default_factory=lambda: dict(DEFAULTS))
|
| 46 |
+
|
| 47 |
+
def __post_init__(self):
|
| 48 |
+
self.rules = {**DEFAULTS, **(self.rules or {})}
|
| 49 |
+
|
| 50 |
+
def apply(self, doc: dict) -> tuple[dict | None, str]:
|
| 51 |
+
"""Return (document, "") to keep it, possibly with boilerplate lines removed, or (None, reason) to drop it."""
|
| 52 |
+
text = doc["text"]
|
| 53 |
+
kind = doc.get("kind", "web")
|
| 54 |
+
if not text.strip():
|
| 55 |
+
return None, "empty"
|
| 56 |
+
if kind == "code":
|
| 57 |
+
why = self._code(text)
|
| 58 |
+
return (None, why) if why else (doc, "")
|
| 59 |
+
if self.rules["strip_boilerplate_lines"] and kind in ("web", "educational", "multilingual"):
|
| 60 |
+
lines = text.split("\n")
|
| 61 |
+
kept = [l for l in lines if not (BOILER.search(l) and len(l.split()) < 25)]
|
| 62 |
+
if lines and len(lines) - len(kept) > self.rules["max_boiler_lines"] * len(lines):
|
| 63 |
+
return None, "mostly boilerplate"
|
| 64 |
+
if len(kept) != len(lines):
|
| 65 |
+
text = "\n".join(kept)
|
| 66 |
+
doc = {**doc, "text": text}
|
| 67 |
+
if kind == "textbook": # PDFs repeat running headers and footers on every page; drop those lines, keep the page
|
| 68 |
+
lines = text.split("\n")
|
| 69 |
+
short = Counter(l.strip() for l in lines if l.strip() and len(l.split()) <= 12)
|
| 70 |
+
running = {l for l, c in short.items() if c >= 3}
|
| 71 |
+
if running:
|
| 72 |
+
text = "\n".join(l for l in lines if l.strip() not in running)
|
| 73 |
+
doc = {**doc, "text": text}
|
| 74 |
+
why = self._prose(text, loose=kind in ("math", "textbook", "synthetic"), rep=1.6 if kind == "math" else 1.0)
|
| 75 |
+
return (None, why) if why else (doc, "")
|
| 76 |
+
|
| 77 |
+
def _prose(self, text: str, loose: bool, rep: float = 1.0) -> str:
|
| 78 |
+
"""rep scales the repetition caps: worked math repeats symbols and steps by nature."""
|
| 79 |
+
r = self.rules
|
| 80 |
+
toks = text.split()
|
| 81 |
+
n = len(toks)
|
| 82 |
+
if n < r["min_words"]:
|
| 83 |
+
return "too short"
|
| 84 |
+
if n > r["max_words"]:
|
| 85 |
+
return "too long"
|
| 86 |
+
if not loose:
|
| 87 |
+
mean = sum(len(t) for t in toks) / n
|
| 88 |
+
if not r["min_mean_word"] <= mean <= r["max_mean_word"]:
|
| 89 |
+
return "odd word lengths"
|
| 90 |
+
sym = (text.count("#") + text.count("...") + text.count("…")) / n
|
| 91 |
+
if sym > r["max_symbol_ratio"]:
|
| 92 |
+
return "too many symbols"
|
| 93 |
+
alpha = sum(1 for t in toks if any(c.isalpha() for c in t)) / n
|
| 94 |
+
if alpha < r["min_alpha_words"]:
|
| 95 |
+
return "too little real text"
|
| 96 |
+
lines = [l.strip() for l in text.split("\n") if l.strip()]
|
| 97 |
+
if lines:
|
| 98 |
+
ell = sum(1 for l in lines if l.endswith(("...", "…"))) / len(lines)
|
| 99 |
+
if ell > r["max_ellipsis_lines"]:
|
| 100 |
+
return "truncated lines"
|
| 101 |
+
if len(lines) >= 5:
|
| 102 |
+
dup = 1 - len(set(lines)) / len(lines)
|
| 103 |
+
if dup > r["max_dup_lines"] * rep:
|
| 104 |
+
return "repeated lines"
|
| 105 |
+
ws = words(text)
|
| 106 |
+
if len(ws) >= 20:
|
| 107 |
+
bi = Counter(ngrams(ws, 2))
|
| 108 |
+
top = bi.most_common(1)[0]
|
| 109 |
+
if top[1] > 2 and top[1] * len(top[0]) / max(1, sum(len(w) + 1 for w in ws)) > r["max_top2gram"] * rep:
|
| 110 |
+
return "one phrase repeated"
|
| 111 |
+
total = max(1, sum(len(w) for w in ws))
|
| 112 |
+
for n, cap in ((5, r["max_dup5gram"]), (10, r["max_dup10gram"])):
|
| 113 |
+
if dup_ngram_chars(ws, n) / total > cap * rep:
|
| 114 |
+
return "repeated passages"
|
| 115 |
+
return ""
|
| 116 |
+
|
| 117 |
+
def _code(self, text: str) -> str:
|
| 118 |
+
r = self.rules
|
| 119 |
+
lines = text.split("\n")
|
| 120 |
+
if len(text) < 100:
|
| 121 |
+
return "too short"
|
| 122 |
+
if max(len(l) for l in lines) > r["code_max_line"]:
|
| 123 |
+
return "minified or data blob"
|
| 124 |
+
if sum(c.isalpha() for c in text) / len(text) < r["code_min_alpha"]:
|
| 125 |
+
return "too little real code"
|
| 126 |
+
if AUTOGEN.search(text[:600]):
|
| 127 |
+
return "auto-generated"
|
| 128 |
+
return ""
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
# ---------------------------------------------------------------- decontamination
|
| 132 |
+
|
| 133 |
+
# Benchmarks people usually report. Each entry: (hub id, config, split, fields joined to form the test text).
|
| 134 |
+
BENCHMARKS = {
|
| 135 |
+
"gsm8k": ("openai/gsm8k", "main", "test", ["question"]),
|
| 136 |
+
"math500": ("HuggingFaceH4/MATH-500", "default", "test", ["problem"]),
|
| 137 |
+
"humaneval": ("openai/openai_humaneval", "openai_humaneval", "test", ["prompt"]),
|
| 138 |
+
"mbpp": ("google-research-datasets/mbpp", "full", "test", ["text"]),
|
| 139 |
+
"mmlu": ("cais/mmlu", "all", "test", ["question"]),
|
| 140 |
+
"arc_challenge": ("allenai/ai2_arc", "ARC-Challenge", "test", ["question"]),
|
| 141 |
+
"arc_easy": ("allenai/ai2_arc", "ARC-Easy", "test", ["question"]),
|
| 142 |
+
"hellaswag": ("Rowan/hellaswag", "default", "validation", ["ctx"]),
|
| 143 |
+
"openbookqa": ("allenai/openbookqa", "main", "test", ["question_stem"]),
|
| 144 |
+
"winogrande": ("allenai/winogrande", "winogrande_xl", "validation", ["sentence"]),
|
| 145 |
+
"truthfulqa": ("truthfulqa/truthful_qa", "generation", "validation", ["question"]),
|
| 146 |
+
"commonsenseqa": ("tau/commonsense_qa", "default", "validation", ["question"]),
|
| 147 |
+
}
|
| 148 |
+
|
| 149 |
+
|
| 150 |
+
class Decontaminator:
|
| 151 |
+
"""Drop a document when it shares any long n-gram with a held-out test item.
|
| 152 |
+
|
| 153 |
+
13-grams is the GPT-3 / OLMo convention. Short test items (under n words) are matched as a whole normalized string.
|
| 154 |
+
"""
|
| 155 |
+
|
| 156 |
+
def __init__(self, n: int = 13):
|
| 157 |
+
self.n = n
|
| 158 |
+
self.grams: dict[str, str] = {}
|
| 159 |
+
self.short: dict[str, str] = {}
|
| 160 |
+
self.counts: Counter = Counter()
|
| 161 |
+
|
| 162 |
+
def add(self, name: str, texts: list[str]):
|
| 163 |
+
for t in texts:
|
| 164 |
+
ws = norm(t).split()
|
| 165 |
+
if len(ws) < self.n:
|
| 166 |
+
if len(ws) >= 6:
|
| 167 |
+
self.short[" ".join(ws)] = name
|
| 168 |
+
continue
|
| 169 |
+
for g in ngrams(ws, self.n):
|
| 170 |
+
self.grams.setdefault(g, name)
|
| 171 |
+
|
| 172 |
+
def __len__(self):
|
| 173 |
+
return len(self.grams) + len(self.short)
|
| 174 |
+
|
| 175 |
+
def hit(self, text: str) -> str:
|
| 176 |
+
if not self:
|
| 177 |
+
return ""
|
| 178 |
+
nt = norm(text)
|
| 179 |
+
ws = nt.split()
|
| 180 |
+
for g in ngrams(ws, self.n):
|
| 181 |
+
b = self.grams.get(g)
|
| 182 |
+
if b:
|
| 183 |
+
self.counts[b] += 1
|
| 184 |
+
return b
|
| 185 |
+
if self.short:
|
| 186 |
+
for s, b in self.short.items():
|
| 187 |
+
if s in nt:
|
| 188 |
+
self.counts[b] += 1
|
| 189 |
+
return b
|
| 190 |
+
return ""
|
| 191 |
+
|
| 192 |
+
|
| 193 |
+
_BUNDLED: dict | None = None
|
| 194 |
+
|
| 195 |
+
|
| 196 |
+
def bundled_benchmarks() -> dict:
|
| 197 |
+
"""The test sets shipped with the package (data/benchmarks.json.gz, built by make_benchmarks.py)."""
|
| 198 |
+
global _BUNDLED
|
| 199 |
+
if _BUNDLED is None:
|
| 200 |
+
import gzip
|
| 201 |
+
import json
|
| 202 |
+
import os
|
| 203 |
+
path = os.path.join(os.path.dirname(__file__), "data", "benchmarks.json.gz")
|
| 204 |
+
try:
|
| 205 |
+
with gzip.open(path, "rt", encoding="utf-8") as f:
|
| 206 |
+
_BUNDLED = json.load(f)
|
| 207 |
+
except OSError:
|
| 208 |
+
_BUNDLED = {}
|
| 209 |
+
return _BUNDLED
|
| 210 |
+
|
| 211 |
+
|
| 212 |
+
def benchmark_texts(key: str, token: str = "") -> list[str]:
|
| 213 |
+
"""Bundled copy first; the dataset viewer only when a set is missing from the bundle."""
|
| 214 |
+
return bundled_benchmarks().get(key) or load_benchmark_texts(key, token)
|
| 215 |
+
|
| 216 |
+
|
| 217 |
+
def load_benchmark_texts(key: str, token: str = "", limit: int = 20000) -> list[str]:
|
| 218 |
+
"""Test items through the dataset viewer API (no download). Test sets are small, so this is enough."""
|
| 219 |
+
from .sources import _get
|
| 220 |
+
ds, cfg, split, fields = BENCHMARKS[key]
|
| 221 |
+
out, off = [], 0
|
| 222 |
+
while off < limit:
|
| 223 |
+
got = _get("rows", {"dataset": ds, "config": cfg, "split": split, "offset": off, "length": 100}, token)
|
| 224 |
+
rows = got.get("rows", [])
|
| 225 |
+
for r in rows:
|
| 226 |
+
row = r.get("row") or {}
|
| 227 |
+
t = " ".join(str(row.get(f, "")) for f in fields).strip()
|
| 228 |
+
if t:
|
| 229 |
+
out.append(t)
|
| 230 |
+
if len(rows) < 100:
|
| 231 |
+
break
|
| 232 |
+
off += 100
|
| 233 |
+
return out
|
datainventor/cli.py
ADDED
|
@@ -0,0 +1,98 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""datainventor: curate a pretraining corpus on your own machines.
|
| 2 |
+
|
| 3 |
+
datainventor init recipe.yaml write a starter recipe to edit
|
| 4 |
+
datainventor preview recipe.yaml run every stage on a small sample and print what each did
|
| 5 |
+
datainventor run recipe.yaml --out ./curated run it in full (resumable)
|
| 6 |
+
|
| 7 |
+
The model endpoint is any OpenAI-compatible server: vLLM, SGLang, Ollama, a hosted API. Set its key in the
|
| 8 |
+
environment variable named by endpoint.api_key_env (DI_API_KEY by default).
|
| 9 |
+
"""
|
| 10 |
+
from __future__ import annotations
|
| 11 |
+
|
| 12 |
+
import argparse
|
| 13 |
+
import json
|
| 14 |
+
import os
|
| 15 |
+
import sys
|
| 16 |
+
|
| 17 |
+
from .compose import fmt
|
| 18 |
+
from .recipe import DEFAULT, dump, endpoint_of, load
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
def main(argv=None):
|
| 22 |
+
ap = argparse.ArgumentParser(prog="datainventor", description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
| 23 |
+
sub = ap.add_subparsers(dest="cmd", required=True)
|
| 24 |
+
a = sub.add_parser("init", help="write a starter recipe")
|
| 25 |
+
a.add_argument("path")
|
| 26 |
+
a = sub.add_parser("preview", help="run every stage on a sample")
|
| 27 |
+
a.add_argument("recipe")
|
| 28 |
+
a.add_argument("--per-source", type=int, default=300)
|
| 29 |
+
a.add_argument("--grade", type=int, default=64)
|
| 30 |
+
a.add_argument("--rephrase", type=int, default=6)
|
| 31 |
+
a.add_argument("--workers", type=int, default=None)
|
| 32 |
+
a.add_argument("--json", help="also write the full report here")
|
| 33 |
+
a = sub.add_parser("run", help="run the recipe in full")
|
| 34 |
+
a.add_argument("recipe")
|
| 35 |
+
a.add_argument("--out", default=None, help="output folder (default: output.dir in the recipe)")
|
| 36 |
+
a.add_argument("--procs", type=int, default=0, help="CPU processes for cleaning and scoring (default: all but one)")
|
| 37 |
+
a.add_argument("--workers", type=int, default=None, help="parallel model calls (default: endpoint.workers)")
|
| 38 |
+
a.add_argument("--stages", default="grade,curate,create,compose")
|
| 39 |
+
a.add_argument("--limit", type=int, default=0, help="read at most this many documents per source (for a trial run)")
|
| 40 |
+
args = ap.parse_args(argv)
|
| 41 |
+
|
| 42 |
+
if args.cmd == "init":
|
| 43 |
+
if os.path.exists(args.path):
|
| 44 |
+
sys.exit(f"{args.path} exists; not overwriting it.")
|
| 45 |
+
r = json.loads(json.dumps(DEFAULT))
|
| 46 |
+
r["sources"] = [
|
| 47 |
+
{"name": "finepdfs", "dataset": "codelion/finepdfs-1B", "kind": "textbook"},
|
| 48 |
+
{"name": "dclm", "dataset": "codelion/dclm-baseline-1B", "kind": "web"},
|
| 49 |
+
{"name": "fineweb-edu", "dataset": "codelion/fineweb-edu-1B", "kind": "educational"},
|
| 50 |
+
]
|
| 51 |
+
r["targets"]["description"] = "grade-school to university math and science reasoning"
|
| 52 |
+
r["compose"]["mixes"][0]["budget_tokens"] = 2_000_000_000
|
| 53 |
+
with open(args.path, "w") as f:
|
| 54 |
+
f.write(dump(r))
|
| 55 |
+
print(f"Wrote {args.path}. Edit the sources, targets and budget, then: datainventor preview {args.path}")
|
| 56 |
+
return
|
| 57 |
+
|
| 58 |
+
r = load(args.recipe)
|
| 59 |
+
if args.cmd == "preview":
|
| 60 |
+
from .preview import Preview, run_preview
|
| 61 |
+
from .sources import inspect
|
| 62 |
+
token = os.environ.get("HF_TOKEN", "")
|
| 63 |
+
for s in r["sources"]: # row counts make the token estimates possible
|
| 64 |
+
if s.get("dataset") and not s.get("num_rows"):
|
| 65 |
+
try:
|
| 66 |
+
s["num_rows"] = inspect(s["dataset"], token)["num_rows"] if (s.get("config", "default") == "default") else 0
|
| 67 |
+
except Exception:
|
| 68 |
+
pass
|
| 69 |
+
p = Preview(r)
|
| 70 |
+
ep = endpoint_of(r, args.workers)
|
| 71 |
+
run_preview(p, ep, token, per_source=args.per_source, grade_n=args.grade, rephrase_n=args.rephrase, invent_n=2)
|
| 72 |
+
if p.state != "done":
|
| 73 |
+
sys.exit(f"Preview {p.state}: {p.message}")
|
| 74 |
+
rep = p.report
|
| 75 |
+
for f in rep["sources"]:
|
| 76 |
+
drops = ", ".join(f"{k} {v}" for k, v in sorted(f["dropped"].items(), key=lambda x: -x[1]))
|
| 77 |
+
print(f"{f['name']}: read {f['read']}, selected {f['selected']}; dropped: {drops}; decontaminated: {f['decontaminated'] or 'none'}")
|
| 78 |
+
if f.get("available_tokens"):
|
| 79 |
+
print(f" about {fmt(f['available_tokens'])} tokens available after curation")
|
| 80 |
+
c = rep["stages"]["calibrate"]
|
| 81 |
+
print(f"graded {c['graded']}; scorer agreement {c['agreement']}")
|
| 82 |
+
for plan in rep["stages"]["compose"]["plans"]:
|
| 83 |
+
print(f"mix {plan['name']} ({plan['budget']}): " + "; ".join(f"{x['name']} {int(x['share'] * 100)}% x{x['epochs']}" for x in plan["rows"]))
|
| 84 |
+
for w in plan["warnings"]:
|
| 85 |
+
print(" - " + w)
|
| 86 |
+
if args.json:
|
| 87 |
+
with open(args.json, "w") as fh:
|
| 88 |
+
json.dump({"report": rep, "docs": p.docs}, fh, indent=1, ensure_ascii=False)
|
| 89 |
+
return
|
| 90 |
+
|
| 91 |
+
from .runner import Run
|
| 92 |
+
out = args.out or r["output"]["dir"]
|
| 93 |
+
Run(r, out, procs=args.procs, workers=args.workers, limit=args.limit).go(tuple(args.stages.split(",")))
|
| 94 |
+
print(f"Done. See {os.path.join(out, 'REPORT.md')}")
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
if __name__ == "__main__":
|
| 98 |
+
main()
|
datainventor/compose.py
ADDED
|
@@ -0,0 +1,83 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Compose: turn one prepared corpus into one or more training mixes that fit a token budget.
|
| 2 |
+
|
| 3 |
+
Defaults come from our own 1B-token study (huggingface.co/blog/codelion/optimal-dataset-mixing): a static mix of
|
| 4 |
+
50% textbook-style text, 30% filtered web and 20% educational web beat every curriculum we tried at 70M parameters
|
| 5 |
+
and 1B tokens. That is one model size and one budget, so treat it as a starting point and fork it.
|
| 6 |
+
|
| 7 |
+
When a source is too small for its share, it is repeated up to `max_epochs` (about four epochs of repetition cost
|
| 8 |
+
little in Muennighoff et al., 2023). Past that, the gap is filled by rephrasing that source's best documents,
|
| 9 |
+
up to `max_synthetic_share` of the mix, since a mix that leans on synthetic text generalized worst in our study.
|
| 10 |
+
"""
|
| 11 |
+
from __future__ import annotations
|
| 12 |
+
|
| 13 |
+
GROUP_OF_KIND = {"textbook": "textbook", "synthetic": "textbook", "web": "web", "multilingual": "web",
|
| 14 |
+
"educational": "educational", "math": "math", "code": "code", "other": "web"}
|
| 15 |
+
BASE_SHARES = {"textbook": 0.50, "web": 0.30, "educational": 0.20}
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
def default_weights(sources: list[dict]) -> dict[str, float]:
|
| 19 |
+
"""Share per source name. Math and code, when present, take 10% each off the top; the rest follows 50/30/20
|
| 20 |
+
over the groups that exist, split evenly inside each group."""
|
| 21 |
+
groups: dict[str, list[str]] = {}
|
| 22 |
+
for s in sources:
|
| 23 |
+
groups.setdefault(GROUP_OF_KIND.get(s.get("kind", "web"), "web"), []).append(s["name"])
|
| 24 |
+
shares: dict[str, float] = {}
|
| 25 |
+
fixed = {g: 0.10 for g in ("math", "code") if g in groups}
|
| 26 |
+
rest = 1.0 - sum(fixed.values())
|
| 27 |
+
base = {g: v for g, v in BASE_SHARES.items() if g in groups}
|
| 28 |
+
if not base:
|
| 29 |
+
# only math and/or code sources: split evenly
|
| 30 |
+
return {s["name"]: round(1 / len(sources), 4) for s in sources}
|
| 31 |
+
tot = sum(base.values())
|
| 32 |
+
for g, v in base.items():
|
| 33 |
+
shares[g] = rest * v / tot
|
| 34 |
+
shares.update(fixed)
|
| 35 |
+
out = {}
|
| 36 |
+
for g, names in groups.items():
|
| 37 |
+
for n in names:
|
| 38 |
+
out[n] = round(shares[g] / len(names), 4)
|
| 39 |
+
return out
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def plan_mix(budget_tokens: int, sources: list[dict], weights: dict[str, float], max_epochs: float = 4.0,
|
| 43 |
+
max_synthetic_share: float = 0.25, rephrase_yield: float = 4.0, avg_out_tokens: int = 700) -> dict:
|
| 44 |
+
"""sources: [{name, kind, available_tokens (after cleaning and dedup), high_tokens (top tier)}].
|
| 45 |
+
Returns per-source rows and totals, including how much synthetic text to generate and roughly how many calls."""
|
| 46 |
+
tot_w = sum(weights.get(s["name"], 0) for s in sources) or 1.0
|
| 47 |
+
rows, synth_total, short_total = [], 0, 0
|
| 48 |
+
for s in sources:
|
| 49 |
+
share = weights.get(s["name"], 0) / tot_w
|
| 50 |
+
want = int(budget_tokens * share)
|
| 51 |
+
avail = max(0, int(s.get("available_tokens", 0)))
|
| 52 |
+
fresh = min(want, avail)
|
| 53 |
+
repeat_cap = int(avail * max_epochs)
|
| 54 |
+
from_repeat = min(want, repeat_cap)
|
| 55 |
+
gap = want - from_repeat
|
| 56 |
+
synth = min(gap, int(s.get("high_tokens", avail) * rephrase_yield)) if gap > 0 else 0
|
| 57 |
+
short = gap - synth
|
| 58 |
+
epochs = round(from_repeat / avail, 2) if avail else 0.0
|
| 59 |
+
rows.append({"name": s["name"], "kind": s.get("kind", "web"), "share": round(share, 4), "want_tokens": want,
|
| 60 |
+
"available_tokens": avail, "fresh_tokens": fresh, "epochs": epochs, "synthetic_tokens": synth,
|
| 61 |
+
"short_tokens": short})
|
| 62 |
+
synth_total += synth
|
| 63 |
+
short_total += short
|
| 64 |
+
cap = int(budget_tokens * max_synthetic_share)
|
| 65 |
+
warnings = []
|
| 66 |
+
if synth_total > cap:
|
| 67 |
+
warnings.append(f"The plan needs {fmt(synth_total)} synthetic tokens, over the {int(max_synthetic_share * 100)}% cap "
|
| 68 |
+
f"({fmt(cap)}). Add sources, lower the budget, or raise the cap knowingly.")
|
| 69 |
+
if short_total:
|
| 70 |
+
warnings.append(f"{fmt(short_total)} tokens cannot be filled even with repetition and rephrasing.")
|
| 71 |
+
for r in rows:
|
| 72 |
+
if r["epochs"] > 1.0:
|
| 73 |
+
warnings.append(f"{r['name']} repeats {r['epochs']}x to reach its share.")
|
| 74 |
+
return {"budget_tokens": budget_tokens, "rows": rows, "synthetic_tokens": synth_total,
|
| 75 |
+
"synthetic_calls": synth_total // max(1, avg_out_tokens), "short_tokens": short_total, "warnings": warnings}
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
def fmt(n: float) -> str:
|
| 79 |
+
n = float(n)
|
| 80 |
+
for unit, div in (("T", 1e12), ("B", 1e9), ("M", 1e6), ("K", 1e3)):
|
| 81 |
+
if abs(n) >= div:
|
| 82 |
+
return f"{n / div:.3g}{unit}"
|
| 83 |
+
return f"{int(n)}"
|
datainventor/create.py
ADDED
|
@@ -0,0 +1,152 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Create: rephrase good source documents into new training documents, or write new ones where sources are thin.
|
| 2 |
+
|
| 3 |
+
Rephrase styles are the five from Sutra (weights 30/25/20/15/10, after BeyondWeb), plus math and code styles.
|
| 4 |
+
A fidelity check flags rephrasings that introduce numbers or names the source does not contain, which matters when the
|
| 5 |
+
training data must respect a knowledge cutoff.
|
| 6 |
+
"""
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
import random
|
| 10 |
+
import re
|
| 11 |
+
|
| 12 |
+
from .llm import Endpoint, EndpointError
|
| 13 |
+
|
| 14 |
+
NO_PREAMBLE = ("Output ONLY the rewritten document. No preamble, no meta-commentary, no phrases like \"Here is\", "
|
| 15 |
+
"\"Let me\", \"Sure\" or \"Okay\". Do not add facts, numbers, names or dates that are not in the original.")
|
| 16 |
+
|
| 17 |
+
STYLES = {
|
| 18 |
+
"conversational": ("Teacher-student dialogue", 30, """Rewrite the content as a natural conversation between a teacher and a student.
|
| 19 |
+
The student asks curious, natural questions; the teacher explains clearly with concrete examples taken from the content.
|
| 20 |
+
Keep all key information and the same technical accuracy. Keep a similar length (within 20%).
|
| 21 |
+
Use the labels **Teacher**: and **Student**: and start directly with "**Teacher**:"."""),
|
| 22 |
+
"tutorial": ("Step-by-step tutorial", 25, """Rewrite the content as a clear step-by-step tutorial.
|
| 23 |
+
Start with "## Tutorial: <topic>", then an overview, prerequisites, numbered steps with what to do and how to check it,
|
| 24 |
+
a short practice exercise and key takeaways. Keep all key information accurate. Keep a similar length (within 20%)."""),
|
| 25 |
+
"qa": ("Questions and answers", 20, """Rewrite the content as 4 to 7 questions and thorough answers that together cover every key point,
|
| 26 |
+
moving from basic (what) to deeper (why, how) to applied. Start with "## Understanding <topic>: Questions and Answers".
|
| 27 |
+
Use **Q:** and **A:** labels. Keep a similar total length (within 20%)."""),
|
| 28 |
+
"dense_reference": ("Dense reference notes", 15, """Rewrite the content as compact reference notes at 70 to 85 percent of the original length.
|
| 29 |
+
Remove filler and repetition. Use headings and bullet points: core definition, key properties, relationships,
|
| 30 |
+
essential examples, formulas or procedures if any, pitfalls."""),
|
| 31 |
+
"pedagogical": ("Textbook section", 10, """Rewrite the content as a well-structured textbook section: learning objectives, the connection to what
|
| 32 |
+
the reader already knows, the core explanation, one worked example with step-by-step reasoning, a common question
|
| 33 |
+
with its answer, and a quick self-check question. Keep a similar length."""),
|
| 34 |
+
"worked_examples": ("Worked problems (math)", 0, """Rewrite the content as 2 to 4 worked problems that use exactly the ideas in it. For each: state the problem,
|
| 35 |
+
solve it step by step with the reasoning for each step, and check the answer. Use LaTeX for mathematics."""),
|
| 36 |
+
"code_walkthrough": ("Code walkthrough (code)", 0, """Rewrite the content as an explained code walkthrough: what the code does, then the code in a fenced block
|
| 37 |
+
(keep it correct and runnable; you may tidy names and add comments), then an explanation of the key lines and one
|
| 38 |
+
example of using it."""),
|
| 39 |
+
}
|
| 40 |
+
|
| 41 |
+
KIND_STYLES = {"math": ["worked_examples", "pedagogical", "qa", "tutorial"],
|
| 42 |
+
"code": ["code_walkthrough", "tutorial", "qa", "dense_reference"]}
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
def styles_for(kind: str) -> list[str]:
|
| 46 |
+
return KIND_STYLES.get(kind, ["conversational", "tutorial", "qa", "dense_reference", "pedagogical"])
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def pick_style(kind: str, rng: random.Random, allowed: list[str] | None = None) -> str:
|
| 50 |
+
opts = [s for s in (allowed or styles_for(kind)) if s in STYLES]
|
| 51 |
+
weights = [max(STYLES[s][1], 10) for s in opts]
|
| 52 |
+
return rng.choices(opts, weights=weights, k=1)[0]
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
def chunks(text: str, max_chars: int = 6000) -> list[str]:
|
| 56 |
+
"""Split a long document at paragraph breaks so each rewrite sees a piece it can keep at a similar length."""
|
| 57 |
+
if len(text) <= max_chars:
|
| 58 |
+
return [text]
|
| 59 |
+
out, cur = [], ""
|
| 60 |
+
for para in re.split(r"\n\s*\n", text):
|
| 61 |
+
if cur and len(cur) + len(para) + 2 > max_chars:
|
| 62 |
+
out.append(cur)
|
| 63 |
+
cur = ""
|
| 64 |
+
cur = (cur + "\n\n" + para) if cur else para
|
| 65 |
+
while len(cur) > max_chars: # one giant paragraph
|
| 66 |
+
out.append(cur[:max_chars])
|
| 67 |
+
cur = cur[max_chars:]
|
| 68 |
+
if cur.strip():
|
| 69 |
+
out.append(cur)
|
| 70 |
+
return [c for c in out if len(c.split()) >= 80] or [text[:max_chars]]
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
def rephrase_prompt(text: str, style: str, language: str = "") -> str:
|
| 74 |
+
lang = f"\nWrite it in {language}." if language and language.lower() != "same" else ""
|
| 75 |
+
return f"""{STYLES[style][2]}{lang}
|
| 76 |
+
{NO_PREAMBLE}
|
| 77 |
+
|
| 78 |
+
Original content:
|
| 79 |
+
<<<
|
| 80 |
+
{text[:8000]}
|
| 81 |
+
>>>"""
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
LEAD = re.compile(r"^(here('s| is)|sure|okay|certainly|let me|i('ll| will))\b", re.I)
|
| 85 |
+
NUM = re.compile(r"\b\d[\d,.]*\b")
|
| 86 |
+
# a capitalized word that follows a lowercase word mid-sentence: names, places, products. Words at the start of a
|
| 87 |
+
# sentence, a line or a markdown label are skipped, so "Teacher:" or "Great question" never count.
|
| 88 |
+
HEADINGS = re.compile(r"^#+ .*$|\*\*[^*\n]{1,80}\*\*", re.M)
|
| 89 |
+
NAME = re.compile(r"\b[a-z][a-z'-]*[,;]? ([A-Z][a-zA-Z]{2,}(?: [A-Z][a-zA-Z]{2,})*)")
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
def fidelity(src: str, out: str, template: str = "") -> dict:
|
| 93 |
+
"""Share of numbers and mid-sentence names in the rewrite that never appear in the source. A rough proxy for
|
| 94 |
+
invented facts; the threshold is a choice, not a law."""
|
| 95 |
+
s_low = (src + " " + template).lower()
|
| 96 |
+
body = HEADINGS.sub(" ", out) # headings and bold labels the style asks for are not new facts
|
| 97 |
+
nums = {n.rstrip(".,") for n in NUM.findall(out)}
|
| 98 |
+
names = set(NAME.findall(body))
|
| 99 |
+
new_nums = [n for n in nums if n and n not in src]
|
| 100 |
+
new_names = [n for n in names if n.lower() not in s_low]
|
| 101 |
+
total = len(nums) + len(names)
|
| 102 |
+
share = (len(new_nums) + len(new_names)) / total if total else 0.0
|
| 103 |
+
return {"new_numbers": sorted(new_nums)[:8], "new_names": sorted(new_names)[:8], "new_share": round(share, 3),
|
| 104 |
+
"checked": total}
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def rephrase(ep: Endpoint, text: str, style: str, language: str = "", max_new_share: float = 0.35) -> dict:
|
| 108 |
+
"""One rewrite. Returns {"text", "style", "fidelity", "ok", "why"}."""
|
| 109 |
+
def ok(s):
|
| 110 |
+
return len(s.split()) >= 60 and not LEAD.search(s.strip()[:40])
|
| 111 |
+
try:
|
| 112 |
+
out = ep.chat(rephrase_prompt(text, style, language), max_tokens=3000, temperature=0.7, retries=3, validate=ok)
|
| 113 |
+
except EndpointError as e:
|
| 114 |
+
return {"text": "", "style": style, "ok": False, "why": f"model call failed: {e}"[:200]}
|
| 115 |
+
fid = fidelity(text, out, STYLES[style][2])
|
| 116 |
+
new = len(fid["new_numbers"]) + len(fid["new_names"])
|
| 117 |
+
# invented numbers are the clearest sign of made-up content; one stray number or name is not a pattern
|
| 118 |
+
good = len(fid["new_numbers"]) < 3 and (new < 2 or fid["new_share"] <= max_new_share)
|
| 119 |
+
return {"text": out, "style": style, "fidelity": fid, "ok": good, "why": "" if good else "adds facts the source does not contain"}
|
| 120 |
+
|
| 121 |
+
|
| 122 |
+
# ---------------------------------------------------------------- new documents where sources are thin
|
| 123 |
+
|
| 124 |
+
DOC_TYPES = { # a subset of Sutra's content types that reads well as pretraining text
|
| 125 |
+
"concept_introduction": "an introduction to the concept: definition, intuition, why it matters, how it works, two examples, a common misconception, and how it connects to related ideas",
|
| 126 |
+
"reasoning_demonstration": "a worked demonstration of reasoning about a concrete problem in this area, thinking step by step and checking the result",
|
| 127 |
+
"worked_examples": "two or three worked examples of increasing difficulty, each solved step by step",
|
| 128 |
+
"common_misconceptions": "an explanation of the most common misconceptions about the concept, why people hold them, and the correct understanding",
|
| 129 |
+
"problem_set": "a short problem set of four problems with full solutions",
|
| 130 |
+
"case_study": "a realistic case study that applies the concept, with the decisions and their reasons",
|
| 131 |
+
"technical_documentation": "clear technical documentation of the concept or tool, with a usage example",
|
| 132 |
+
"historical_context": "the history of how the idea developed, who contributed what, and why it mattered",
|
| 133 |
+
}
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
def invent_prompt(topic: str, doc_type: str, level: int, request: str, language: str = "English") -> str:
|
| 137 |
+
return f"""You are writing high-quality pretraining text for a language model. Collection: "{request}".
|
| 138 |
+
Write {DOC_TYPES[doc_type]}.
|
| 139 |
+
Concept: {topic}
|
| 140 |
+
Difficulty: {level} on a 1 (introductory) to 10 (expert) scale. Language: {language}.
|
| 141 |
+
Write 350 to 700 words of dense, accurate prose with markdown headings where useful. Every sentence should teach
|
| 142 |
+
something. Do not invent statistics, quotes or citations. {NO_PREAMBLE.split('. Do not add')[0]}."""
|
| 143 |
+
|
| 144 |
+
|
| 145 |
+
def invent(ep: Endpoint, topic: str, doc_type: str, level: int, request: str, language: str = "English") -> dict:
|
| 146 |
+
def ok(s):
|
| 147 |
+
return len(s.split()) >= 150 and not LEAD.search(s.strip()[:40])
|
| 148 |
+
try:
|
| 149 |
+
out = ep.chat(invent_prompt(topic, doc_type, level, request, language), max_tokens=2500, temperature=0.8, retries=3, validate=ok)
|
| 150 |
+
except EndpointError as e:
|
| 151 |
+
return {"text": "", "ok": False, "why": f"model call failed: {e}"[:200]}
|
| 152 |
+
return {"text": out, "ok": True, "doc_type": doc_type, "topic": topic, "level": level}
|
datainventor/data/benchmarks.json.gz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e1b3b8b79353bdece54384094cc308a52e577dab21e195a9fdd8ba7a66e862de
|
| 3 |
+
size 2370882
|
datainventor/llm.py
ADDED
|
@@ -0,0 +1,72 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""One OpenAI-compatible chat endpoint: HF Inference Providers, vLLM, SGLang, Ollama, OptiQ serve or a hosted API."""
|
| 2 |
+
from __future__ import annotations
|
| 3 |
+
|
| 4 |
+
import json
|
| 5 |
+
import re
|
| 6 |
+
import time
|
| 7 |
+
import urllib.error
|
| 8 |
+
import urllib.request
|
| 9 |
+
from dataclasses import dataclass
|
| 10 |
+
from typing import Callable, Optional
|
| 11 |
+
|
| 12 |
+
HF_BASE = "https://router.huggingface.co/v1"
|
| 13 |
+
HF_MODEL = "Qwen/Qwen3-4B-Instruct-2507:nscale"
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
@dataclass
|
| 17 |
+
class Endpoint:
|
| 18 |
+
base_url: str = HF_BASE
|
| 19 |
+
api_key: str = ""
|
| 20 |
+
model: str = HF_MODEL
|
| 21 |
+
disable_thinking: bool = False
|
| 22 |
+
workers: int = 4
|
| 23 |
+
|
| 24 |
+
def chat(self, prompt: str, max_tokens: int = 1500, temperature: float = 0.8, retries: int = 5,
|
| 25 |
+
validate: Optional[Callable[[str], bool]] = None) -> str:
|
| 26 |
+
body = {"model": self.model, "max_tokens": max_tokens, "temperature": temperature,
|
| 27 |
+
"messages": [{"role": "user", "content": prompt}]}
|
| 28 |
+
if self.disable_thinking:
|
| 29 |
+
body["chat_template_kwargs"] = {"enable_thinking": False}
|
| 30 |
+
data = json.dumps(body).encode()
|
| 31 |
+
url = self.base_url.rstrip("/") + "/chat/completions"
|
| 32 |
+
last = "no attempt"
|
| 33 |
+
for attempt in range(retries):
|
| 34 |
+
try:
|
| 35 |
+
req = urllib.request.Request(url, data, {"Content-Type": "application/json", "User-Agent": "DataInventor/1.0",
|
| 36 |
+
"Authorization": "Bearer " + self.api_key})
|
| 37 |
+
resp = json.load(urllib.request.urlopen(req, timeout=180))
|
| 38 |
+
choice = (resp.get("choices") or [{}])[0] or {}
|
| 39 |
+
if choice.get("finish_reason") == "length":
|
| 40 |
+
last = "reply cut off at max_tokens"
|
| 41 |
+
time.sleep(1.5)
|
| 42 |
+
continue
|
| 43 |
+
text = ((choice.get("message") or {}).get("content") or "").strip()
|
| 44 |
+
text = re.sub(r"<think>.*?</think>", "", text, flags=re.S).strip()
|
| 45 |
+
if text and (validate is None or validate(text)):
|
| 46 |
+
return text
|
| 47 |
+
last = "reply rejected by a quality check"
|
| 48 |
+
except urllib.error.HTTPError as e:
|
| 49 |
+
detail = ""
|
| 50 |
+
try:
|
| 51 |
+
detail = e.read().decode()[:200]
|
| 52 |
+
except Exception:
|
| 53 |
+
pass
|
| 54 |
+
last = f"HTTP {e.code} {detail}"
|
| 55 |
+
if e.code in (401, 403):
|
| 56 |
+
raise EndpointError(f"The endpoint refused the key (HTTP {e.code}). Check the API key.")
|
| 57 |
+
if e.code == 404:
|
| 58 |
+
raise EndpointError("The endpoint or model was not found (HTTP 404). Check the base URL and model name.")
|
| 59 |
+
time.sleep(min(30, 2 ** attempt * 2) if e.code == 429 else 3 * (attempt + 1))
|
| 60 |
+
except Exception as e: # timeouts, resets
|
| 61 |
+
last = str(e)[:160]
|
| 62 |
+
time.sleep(3 * (attempt + 1))
|
| 63 |
+
raise EndpointError(last)
|
| 64 |
+
|
| 65 |
+
def ping(self) -> dict:
|
| 66 |
+
t = time.time()
|
| 67 |
+
text = self.chat("Reply with the single word ready.", max_tokens=16, temperature=0, retries=2)
|
| 68 |
+
return {"ok": True, "latency_ms": int((time.time() - t) * 1000), "sample": text[:40]}
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
class EndpointError(Exception):
|
| 72 |
+
pass
|
datainventor/preview.py
ADDED
|
@@ -0,0 +1,268 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Run a recipe on a sample: every stage, small enough for a hosted app.
|
| 2 |
+
|
| 3 |
+
It reads a spread sample of each source through the dataset viewer, applies clean, calibrate, create and compose,
|
| 4 |
+
and reports what each stage did, with estimates scaled up to the full sources. The kept and created documents
|
| 5 |
+
come back too, so a sample run is also a small curated dataset.
|
| 6 |
+
"""
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
import random
|
| 10 |
+
import threading
|
| 11 |
+
import time
|
| 12 |
+
from concurrent.futures import ThreadPoolExecutor
|
| 13 |
+
from dataclasses import dataclass, field
|
| 14 |
+
from typing import Callable
|
| 15 |
+
|
| 16 |
+
from . import calibrate as cal
|
| 17 |
+
from .clean import Cleaner, Decontaminator, benchmark_texts
|
| 18 |
+
from .compose import default_weights, fmt, plan_mix
|
| 19 |
+
from .create import chunks, invent, pick_style, rephrase, styles_for
|
| 20 |
+
from .llm import Endpoint, EndpointError
|
| 21 |
+
from .recipe import held_out_of, sources_of, targets_of
|
| 22 |
+
from .sources import sample
|
| 23 |
+
from .text import est_tokens
|
| 24 |
+
|
| 25 |
+
_BENCH_CACHE: dict[str, list[str]] = {}
|
| 26 |
+
_BENCH_LOCK = threading.Lock()
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def bench_texts(key: str, token: str = "") -> list[str]:
|
| 30 |
+
with _BENCH_LOCK:
|
| 31 |
+
if key not in _BENCH_CACHE:
|
| 32 |
+
_BENCH_CACHE[key] = benchmark_texts(key, token)
|
| 33 |
+
return _BENCH_CACHE[key]
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
@dataclass
|
| 37 |
+
class Preview:
|
| 38 |
+
recipe: dict
|
| 39 |
+
state: str = "queued" # queued | running | done | failed | stopped
|
| 40 |
+
message: str = ""
|
| 41 |
+
stage: str = ""
|
| 42 |
+
report: dict = field(default_factory=dict)
|
| 43 |
+
docs: list = field(default_factory=list)
|
| 44 |
+
started: float = field(default_factory=time.time)
|
| 45 |
+
finished: float = 0.0
|
| 46 |
+
stop: threading.Event = field(default_factory=threading.Event)
|
| 47 |
+
|
| 48 |
+
def say(self, stage: str, msg: str):
|
| 49 |
+
self.stage, self.message = stage, msg
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
def _pmap(fn: Callable, items: list, workers: int, stop: threading.Event, tick: Callable[[int], None] | None = None) -> list:
|
| 53 |
+
done = [0]
|
| 54 |
+
lock = threading.Lock()
|
| 55 |
+
|
| 56 |
+
def guarded(x):
|
| 57 |
+
if stop.is_set():
|
| 58 |
+
return None
|
| 59 |
+
out = fn(x)
|
| 60 |
+
if tick:
|
| 61 |
+
with lock:
|
| 62 |
+
done[0] += 1
|
| 63 |
+
tick(done[0])
|
| 64 |
+
return out
|
| 65 |
+
with ThreadPoolExecutor(max_workers=max(1, workers)) as pool:
|
| 66 |
+
return list(pool.map(guarded, items))
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
def run_preview(p: Preview, ep: Endpoint, token: str = "", per_source: int = 200, grade_n: int = 32,
|
| 70 |
+
rephrase_n: int = 8, invent_n: int = 4, max_docs: int = 1000, seed: int = 7):
|
| 71 |
+
r = p.recipe
|
| 72 |
+
rng = random.Random(seed)
|
| 73 |
+
try:
|
| 74 |
+
p.state = "running"
|
| 75 |
+
rep: dict = {"sources": [], "stages": {}}
|
| 76 |
+
|
| 77 |
+
# ---- targets
|
| 78 |
+
targets_text: list[str] = []
|
| 79 |
+
for t in targets_of(r):
|
| 80 |
+
p.say("setup", f"Reading target {t.name}")
|
| 81 |
+
targets_text += [d["text"] for d in sample(t, min(100, per_source), token)]
|
| 82 |
+
calib = cal.Calibrator(float(r["calibrate"]["focus"]), r["targets"]["description"], targets_text)
|
| 83 |
+
|
| 84 |
+
# ---- clean
|
| 85 |
+
cleaner = Cleaner(r["clean"]["rules"])
|
| 86 |
+
dec = Decontaminator(int(r["clean"].get("ngram", 13)))
|
| 87 |
+
for b in r["clean"]["decontaminate"]:
|
| 88 |
+
p.say("clean", f"Loading the {b} test set for decontamination")
|
| 89 |
+
try:
|
| 90 |
+
dec.add(b, bench_texts(b, token))
|
| 91 |
+
except Exception as e:
|
| 92 |
+
rep.setdefault("warnings", []).append(f"Could not load {b} for decontamination: {e}")
|
| 93 |
+
for h in held_out_of(r):
|
| 94 |
+
p.say("clean", f"Loading held-out set {h.name}")
|
| 95 |
+
try:
|
| 96 |
+
dec.add(h.name, [d["text"] for d in sample(h, 2000, token)])
|
| 97 |
+
except Exception as e:
|
| 98 |
+
rep.setdefault("warnings", []).append(f"Could not load held-out set {h.name}: {e}")
|
| 99 |
+
dedup = cal.Deduper(near=r["calibrate"]["dedup"]["near"], threshold=float(r["calibrate"]["dedup"]["threshold"]))
|
| 100 |
+
survivors: list[dict] = []
|
| 101 |
+
for s in sources_of(r):
|
| 102 |
+
if p.stop.is_set():
|
| 103 |
+
break
|
| 104 |
+
p.say("clean", f"Sampling {s.name}")
|
| 105 |
+
raw = sample(s, per_source, token)
|
| 106 |
+
f = {"name": s.name, "dataset": s.dataset or s.files, "kind": s.kind, "num_rows": s.num_rows, "read": len(raw),
|
| 107 |
+
"dropped": {}, "decontaminated": {}, "kept": 0}
|
| 108 |
+
f["repeated_passages"] = cal.repeated_passages(raw)
|
| 109 |
+
f["raw_tokens"] = sum(est_tokens(d["text"]) for d in raw)
|
| 110 |
+
for d in raw:
|
| 111 |
+
d2, why = cleaner.apply(d)
|
| 112 |
+
if not d2:
|
| 113 |
+
f["dropped"][why] = f["dropped"].get(why, 0) + 1
|
| 114 |
+
continue
|
| 115 |
+
b = dec.hit(d2["text"])
|
| 116 |
+
if b:
|
| 117 |
+
f["decontaminated"][b] = f["decontaminated"].get(b, 0) + 1
|
| 118 |
+
continue
|
| 119 |
+
why = dedup.check(d2)
|
| 120 |
+
if why:
|
| 121 |
+
f["dropped"][why] = f["dropped"].get(why, 0) + 1
|
| 122 |
+
continue
|
| 123 |
+
survivors.append(d2)
|
| 124 |
+
f["kept"] += 1
|
| 125 |
+
rep["sources"].append(f)
|
| 126 |
+
rep["stages"]["clean"] = {"benchmarks": r["clean"]["decontaminate"], "test_items": len(dec)}
|
| 127 |
+
if not survivors:
|
| 128 |
+
raise ValueError("Nothing survived cleaning. Loosen the rules, or check the text column of each source.")
|
| 129 |
+
|
| 130 |
+
# ---- calibrate: grade, learn, score
|
| 131 |
+
by_src: dict[str, list[dict]] = {}
|
| 132 |
+
for d in survivors:
|
| 133 |
+
by_src.setdefault(d["source"], []).append(d)
|
| 134 |
+
to_grade = []
|
| 135 |
+
per = max(2, grade_n // max(1, len(by_src)))
|
| 136 |
+
for docs in by_src.values():
|
| 137 |
+
to_grade += rng.sample(docs, min(per, len(docs)))
|
| 138 |
+
p.say("calibrate", f"Grading {len(to_grade)} documents with the model")
|
| 139 |
+
desc = r["targets"]["description"]
|
| 140 |
+
grades = _pmap(lambda d: cal.grade(ep, d["text"], desc), to_grade, ep.workers, p.stop,
|
| 141 |
+
lambda n: p.say("calibrate", f"Graded {n} of {len(to_grade)} documents with the model"))
|
| 142 |
+
graded = [(d, g) for d, g in zip(to_grade, grades) if g]
|
| 143 |
+
if len(graded) < 4:
|
| 144 |
+
raise EndpointError("The model returned too few usable grades. Check the endpoint, or try again.")
|
| 145 |
+
calib.fit([(d["text"], g) for d, g in graded], [d["text"] for d in survivors])
|
| 146 |
+
gmap = {d["id"]: g for d, g in graded}
|
| 147 |
+
for d in survivors:
|
| 148 |
+
g = gmap.get(d["id"])
|
| 149 |
+
if g:
|
| 150 |
+
d["grade"] = g
|
| 151 |
+
d.update(calib.score(d["text"], g))
|
| 152 |
+
d["tokens"] = est_tokens(d["text"])
|
| 153 |
+
keep_top = float(r["calibrate"]["keep_top"])
|
| 154 |
+
for f in rep["sources"]:
|
| 155 |
+
docs = sorted(by_src.get(f["name"], []), key=lambda d: -d["score"])
|
| 156 |
+
k = max(1, round(len(docs) * keep_top)) if docs else 0
|
| 157 |
+
hi = max(1, round(len(docs) * 0.2)) if docs else 0
|
| 158 |
+
for i, d in enumerate(docs):
|
| 159 |
+
d["kept"] = i < k
|
| 160 |
+
d["tier"] = "high" if i < hi else "medium" if i < k else "low"
|
| 161 |
+
kept = [d for d in docs if d["kept"]]
|
| 162 |
+
f["selected"] = len(kept)
|
| 163 |
+
f["dropped"]["below score cut"] = len(docs) - len(kept)
|
| 164 |
+
rate = len(kept) / max(1, f["read"])
|
| 165 |
+
mean_tok = sum(d["tokens"] for d in kept) / max(1, len(kept))
|
| 166 |
+
f["mean_tokens"] = round(mean_tok)
|
| 167 |
+
f["keep_rate"] = round(rate, 3)
|
| 168 |
+
if f["num_rows"]:
|
| 169 |
+
f["available_tokens"] = int(f["num_rows"] * rate * mean_tok)
|
| 170 |
+
f["high_tokens"] = int(f["num_rows"] * (hi / max(1, f["read"])) * mean_tok)
|
| 171 |
+
f["score_hist"] = _hist([d["score"] for d in docs])
|
| 172 |
+
f["examples"] = {"top": [_ex(d) for d in docs[:2]], "bottom": [_ex(d) for d in docs[-2:]]}
|
| 173 |
+
rep["stages"]["calibrate"] = {"graded": len(graded), "agreement": calib.agreement, "focus": calib.focus, "keep_top": keep_top,
|
| 174 |
+
"has_targets": calib.has_targets, "grades": [{"id": d["id"], **g} for d, g in graded][:12]}
|
| 175 |
+
|
| 176 |
+
# ---- create
|
| 177 |
+
created: list[dict] = []
|
| 178 |
+
rcfg, icfg = r["create"]["rephrase"], r["create"]["invent"]
|
| 179 |
+
if rcfg["enabled"] and rephrase_n and not p.stop.is_set():
|
| 180 |
+
queues = [sorted([d for d in docs if d.get("tier") == "high"], key=lambda d: -d["score"]) for docs in by_src.values()]
|
| 181 |
+
picks = []
|
| 182 |
+
while len(picks) < rephrase_n and any(queues): # round-robin so every source is represented
|
| 183 |
+
for q in queues:
|
| 184 |
+
if q and len(picks) < rephrase_n:
|
| 185 |
+
picks.append(q.pop(0))
|
| 186 |
+
jobs = [({**d, "text": chunks(d["text"])[0]}, pick_style(d["kind"], rng, rcfg["styles"] or styles_for(d["kind"]))) for d in picks]
|
| 187 |
+
p.say("create", f"Rephrasing {len(jobs)} of the best documents")
|
| 188 |
+
outs = _pmap(lambda j: rephrase(ep, j[0]["text"], j[1], rcfg.get("language", ""), float(rcfg["max_new_share"])), jobs, ep.workers, p.stop,
|
| 189 |
+
lambda n: p.say("create", f"Rephrased {n} of {len(jobs)} of the best documents"))
|
| 190 |
+
for (d, style), o in zip(jobs, outs):
|
| 191 |
+
if o and o["text"]:
|
| 192 |
+
created.append({"id": f"{d['id']}:{style}", "text": o["text"], "source": d["source"], "kind": "synthetic",
|
| 193 |
+
"origin": d["id"], "origin_text": d["text"], "style": style, "fidelity": o.get("fidelity"), "kept": o["ok"],
|
| 194 |
+
"why": o.get("why", ""), "tokens": est_tokens(o["text"])})
|
| 195 |
+
if icfg["enabled"] and invent_n and icfg["request"].strip() and not p.stop.is_set():
|
| 196 |
+
p.say("create", "Writing new documents")
|
| 197 |
+
topics = plan_topics(ep, icfg["request"], invent_n)
|
| 198 |
+
jobs = [(t, rng.choice(icfg["doc_types"]), rng.choice(icfg["levels"])) for t in topics[:invent_n]]
|
| 199 |
+
outs = _pmap(lambda j: invent(ep, j[0], j[1], j[2], icfg["request"], icfg.get("language", "English")), jobs, ep.workers, p.stop)
|
| 200 |
+
for (t, dt, lv), o in zip(jobs, outs):
|
| 201 |
+
if o and o["text"]:
|
| 202 |
+
created.append({"id": f"invent:{len(created)}", "text": o["text"], "source": "invented", "kind": "synthetic",
|
| 203 |
+
"topic": t, "doc_type": dt, "level": lv, "kept": True, "tokens": est_tokens(o["text"])})
|
| 204 |
+
rep["stages"]["create"] = {"made": len(created), "kept": sum(1 for c in created if c["kept"]),
|
| 205 |
+
"flagged": [{"id": c["id"], "why": c["why"], "fidelity": c.get("fidelity")} for c in created if not c["kept"]]}
|
| 206 |
+
|
| 207 |
+
# ---- compose
|
| 208 |
+
src_rows = [{"name": f["name"], "kind": f["kind"], "available_tokens": f.get("available_tokens", 0),
|
| 209 |
+
"high_tokens": f.get("high_tokens", 0)} for f in rep["sources"]]
|
| 210 |
+
defaults = default_weights(src_rows)
|
| 211 |
+
plans = []
|
| 212 |
+
for m in r["compose"]["mixes"]:
|
| 213 |
+
w = {**defaults, **{k: float(v) for k, v in (m.get("weights") or {}).items()}}
|
| 214 |
+
plan = plan_mix(int(m["budget_tokens"]), src_rows, w, float(r["compose"]["max_epochs"]), float(r["compose"]["max_synthetic_share"]))
|
| 215 |
+
plan["name"] = m["name"]
|
| 216 |
+
plan["budget"] = fmt(m["budget_tokens"])
|
| 217 |
+
plans.append(plan)
|
| 218 |
+
if any(not f["num_rows"] for f in rep["sources"]):
|
| 219 |
+
rep.setdefault("warnings", []).append("Some sources report no row count, so their available tokens are unknown and plans treat them as empty.")
|
| 220 |
+
rep["stages"]["compose"] = {"default_weights": defaults, "plans": plans}
|
| 221 |
+
|
| 222 |
+
# ---- output sample
|
| 223 |
+
kept_docs = [d for d in survivors if d.get("kept")]
|
| 224 |
+
kept_docs.sort(key=lambda d: -d["score"])
|
| 225 |
+
out = [c for c in created if c["kept"]] + kept_docs
|
| 226 |
+
p.docs = [_out(d) for d in out[:max_docs]]
|
| 227 |
+
n_created = min(sum(1 for c in created if c["kept"]), max_docs)
|
| 228 |
+
rep["output"] = {"docs": len(p.docs), "curated": len(p.docs) - n_created, "created": n_created,
|
| 229 |
+
"tokens": sum(d["tokens"] for d in out[:max_docs])}
|
| 230 |
+
rep["created_examples"] = [{k: c.get(k) for k in ("id", "style", "doc_type", "topic", "kept", "why", "fidelity")} | {"text": c["text"][:1600], "origin_text": c.get("origin_text", "")[:1600]} for c in created[:6]]
|
| 231 |
+
p.report = rep
|
| 232 |
+
p.state = "stopped" if p.stop.is_set() else "done"
|
| 233 |
+
p.say("done", f"{len(p.docs)} documents ready" if p.state == "done" else "Stopped")
|
| 234 |
+
except EndpointError as e:
|
| 235 |
+
p.state, p.message = "failed", str(e)
|
| 236 |
+
except Exception as e:
|
| 237 |
+
p.state, p.message = "failed", str(e) or e.__class__.__name__
|
| 238 |
+
finally:
|
| 239 |
+
p.finished = time.time()
|
| 240 |
+
|
| 241 |
+
|
| 242 |
+
def plan_topics(ep: Endpoint, request: str, n: int) -> list[str]:
|
| 243 |
+
import json
|
| 244 |
+
import re
|
| 245 |
+
s = ep.chat(f"""List {n} specific concepts to write teaching documents about, for this collection: "{request}".
|
| 246 |
+
Spread them across the whole subject and across difficulty. Return ONLY a JSON list of strings, each 2 to 6 words.""",
|
| 247 |
+
max_tokens=600, temperature=0.8, validate=lambda t: bool(re.search(r"\[.*\]", t, re.S)))
|
| 248 |
+
try:
|
| 249 |
+
return [str(x) for x in json.loads(re.search(r"\[.*\]", s, re.S).group(0))][:n]
|
| 250 |
+
except Exception:
|
| 251 |
+
return [request] * n
|
| 252 |
+
|
| 253 |
+
|
| 254 |
+
def _hist(vals: list[float], bins: int = 10) -> list[int]:
|
| 255 |
+
h = [0] * bins
|
| 256 |
+
for v in vals:
|
| 257 |
+
h[min(bins - 1, int(v * bins))] += 1
|
| 258 |
+
return h
|
| 259 |
+
|
| 260 |
+
|
| 261 |
+
def _ex(d: dict) -> dict:
|
| 262 |
+
return {"id": d["id"], "score": d["score"], "quality": d["quality"], "relevance": d["relevance"],
|
| 263 |
+
"reason": (d.get("grade") or {}).get("reason", ""), "text": d["text"][:700]}
|
| 264 |
+
|
| 265 |
+
|
| 266 |
+
def _out(d: dict) -> dict:
|
| 267 |
+
keep = ("id", "text", "source", "kind", "quality", "relevance", "score", "tier", "style", "origin", "doc_type", "topic", "level", "tokens")
|
| 268 |
+
return {k: d[k] for k in keep if k in d}
|
datainventor/recipe.py
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""A recipe is the whole curation plan in one file. The hosted app builds and previews it on a sample; the runner
|
| 2 |
+
executes it in full on your own machines. Fork the defaults and change any stage."""
|
| 3 |
+
from __future__ import annotations
|
| 4 |
+
|
| 5 |
+
import copy
|
| 6 |
+
import json
|
| 7 |
+
import os
|
| 8 |
+
|
| 9 |
+
from .clean import BENCHMARKS, DEFAULTS as CLEAN_DEFAULTS
|
| 10 |
+
from .create import DOC_TYPES, STYLES
|
| 11 |
+
from .llm import HF_BASE, HF_MODEL, Endpoint
|
| 12 |
+
from .sources import KINDS, Source
|
| 13 |
+
|
| 14 |
+
DEFAULT = {
|
| 15 |
+
"version": 1,
|
| 16 |
+
"name": "my-corpus",
|
| 17 |
+
"endpoint": {"base_url": HF_BASE, "model": HF_MODEL, "api_key_env": "DI_API_KEY", "disable_thinking": False, "workers": 16},
|
| 18 |
+
"sources": [],
|
| 19 |
+
"targets": {"description": "", "datasets": [], "max_docs": 2000},
|
| 20 |
+
"clean": {"rules": dict(CLEAN_DEFAULTS), "decontaminate": ["gsm8k", "math500", "humaneval", "mmlu", "arc_challenge", "hellaswag"],
|
| 21 |
+
"held_out": [], "ngram": 13},
|
| 22 |
+
"calibrate": {"dedup": {"near": True, "threshold": 0.8}, "grade_docs": 2000, "scorer": "learned", "focus": 0.3, "keep_top": 0.6},
|
| 23 |
+
"create": {"rephrase": {"enabled": True, "styles": [], "max_new_share": 0.35, "language": ""},
|
| 24 |
+
"invent": {"enabled": False, "request": "", "docs": 0, "doc_types": list(DOC_TYPES), "levels": [3, 5, 7], "language": "English"}},
|
| 25 |
+
"compose": {"mixes": [{"name": "main", "budget_tokens": 10_000_000_000, "weights": {}}],
|
| 26 |
+
"max_epochs": 4.0, "max_synthetic_share": 0.25},
|
| 27 |
+
"output": {"dir": "./curated", "shard_docs": 50_000},
|
| 28 |
+
}
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
def merge(base: dict, over: dict) -> dict:
|
| 32 |
+
out = copy.deepcopy(base)
|
| 33 |
+
for k, v in (over or {}).items():
|
| 34 |
+
if isinstance(v, dict) and isinstance(out.get(k), dict):
|
| 35 |
+
out[k] = merge(out[k], v)
|
| 36 |
+
else:
|
| 37 |
+
out[k] = copy.deepcopy(v)
|
| 38 |
+
return out
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
def validate(r: dict) -> dict:
|
| 42 |
+
"""Fill defaults and check the parts a run cannot recover from. Returns the full recipe."""
|
| 43 |
+
r = merge(DEFAULT, r)
|
| 44 |
+
srcs = r["sources"]
|
| 45 |
+
if not srcs:
|
| 46 |
+
raise ValueError("Add at least one source dataset.")
|
| 47 |
+
names = set()
|
| 48 |
+
for s in srcs + r["targets"]["datasets"] + r["clean"]["held_out"]:
|
| 49 |
+
if not (s.get("dataset") or s.get("files")):
|
| 50 |
+
raise ValueError("Every dataset needs a Hugging Face id or a files glob.")
|
| 51 |
+
s.setdefault("name", (s.get("dataset") or s.get("files")).split("/")[-1])
|
| 52 |
+
if s["name"] in names:
|
| 53 |
+
raise ValueError(f"Two datasets share the name {s['name']}. Give each a unique name.")
|
| 54 |
+
names.add(s["name"])
|
| 55 |
+
if s.get("kind", "web") not in KINDS:
|
| 56 |
+
raise ValueError(f"Kind must be one of {', '.join(KINDS)}.")
|
| 57 |
+
unknown = [b for b in r["clean"]["decontaminate"] if b not in BENCHMARKS]
|
| 58 |
+
if unknown:
|
| 59 |
+
raise ValueError(f"Unknown benchmarks: {', '.join(unknown)}. Known: {', '.join(BENCHMARKS)}.")
|
| 60 |
+
bad = [s for s in r["create"]["rephrase"]["styles"] if s not in STYLES]
|
| 61 |
+
if bad:
|
| 62 |
+
raise ValueError(f"Unknown rephrase styles: {', '.join(bad)}.")
|
| 63 |
+
c = r["calibrate"]
|
| 64 |
+
if c["scorer"] not in ("learned", "llm"):
|
| 65 |
+
raise ValueError('calibrate.scorer must be "learned" (grade a sample, learn the rest) or "llm" (grade everything).')
|
| 66 |
+
if not 0 <= c["focus"] <= 1 or not 0 < c["keep_top"] <= 1:
|
| 67 |
+
raise ValueError("focus must be in [0, 1] and keep_top in (0, 1].")
|
| 68 |
+
for m in r["compose"]["mixes"]:
|
| 69 |
+
if int(m.get("budget_tokens", 0)) <= 0:
|
| 70 |
+
raise ValueError("Each mix needs a positive token budget.")
|
| 71 |
+
return r
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
def sources_of(r: dict) -> list[Source]:
|
| 75 |
+
return [Source.from_dict({**s, "role": "source"}) for s in r["sources"]]
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
def held_out_of(r: dict) -> list[Source]:
|
| 79 |
+
"""Your own test sets for decontamination, read like any other dataset."""
|
| 80 |
+
return [Source.from_dict({**s, "role": "held_out"}) for s in r["clean"]["held_out"]]
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def targets_of(r: dict) -> list[Source]:
|
| 84 |
+
return [Source.from_dict({**s, "role": "target"}) for s in r["targets"]["datasets"]]
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def endpoint_of(r: dict, workers: int | None = None) -> Endpoint:
|
| 88 |
+
e = r["endpoint"]
|
| 89 |
+
key = os.environ.get(e.get("api_key_env") or "DI_API_KEY") or os.environ.get("HF_TOKEN", "")
|
| 90 |
+
return Endpoint(base_url=e["base_url"], api_key=key, model=e["model"], disable_thinking=bool(e.get("disable_thinking")),
|
| 91 |
+
workers=int(workers or e.get("workers", 16)))
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
def dump(r: dict) -> str:
|
| 95 |
+
"""YAML when PyYAML is installed (easier to edit), JSON otherwise. Both load with load()."""
|
| 96 |
+
try:
|
| 97 |
+
import yaml
|
| 98 |
+
return yaml.safe_dump(r, sort_keys=False, allow_unicode=True, width=110)
|
| 99 |
+
except ImportError:
|
| 100 |
+
return json.dumps(r, indent=2, ensure_ascii=False)
|
| 101 |
+
|
| 102 |
+
|
| 103 |
+
def load(path: str) -> dict:
|
| 104 |
+
text = open(path, encoding="utf-8").read()
|
| 105 |
+
if path.endswith((".yaml", ".yml")):
|
| 106 |
+
import yaml
|
| 107 |
+
return validate(yaml.safe_load(text))
|
| 108 |
+
return validate(json.loads(text))
|
datainventor/runner.py
ADDED
|
@@ -0,0 +1,497 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Run a recipe in full on your own machines.
|
| 2 |
+
|
| 3 |
+
datainventor run recipe.yaml --out ./curated --procs 16 --workers 32
|
| 4 |
+
|
| 5 |
+
Stages, each resumable (re-run the same command after an interruption):
|
| 6 |
+
grade grade a sample of every source with your model, fit the scorers -> calibrator.json, grades.jsonl
|
| 7 |
+
curate stream every document: clean, decontaminate, dedup, score -> curated/<source>/part-*.jsonl
|
| 8 |
+
create rephrase the best documents where a mix needs more tokens, invent -> synthetic/<source>/part-*.jsonl
|
| 9 |
+
compose build each mix to its token budget -> mixes/<mix>/part-*.jsonl
|
| 10 |
+
Then REPORT.md and report.json summarize what every stage did.
|
| 11 |
+
|
| 12 |
+
Token counts are estimates (four characters per token). Calibrate them against your tokenizer if the budget is tight.
|
| 13 |
+
"""
|
| 14 |
+
from __future__ import annotations
|
| 15 |
+
|
| 16 |
+
import itertools
|
| 17 |
+
import json
|
| 18 |
+
import os
|
| 19 |
+
import random
|
| 20 |
+
import shutil
|
| 21 |
+
import time
|
| 22 |
+
from concurrent.futures import ThreadPoolExecutor
|
| 23 |
+
from multiprocessing import get_context
|
| 24 |
+
|
| 25 |
+
from . import calibrate as cal
|
| 26 |
+
from .clean import Cleaner, Decontaminator, benchmark_texts
|
| 27 |
+
from .compose import default_weights, fmt, plan_mix
|
| 28 |
+
from .create import chunks, invent, rephrase, styles_for
|
| 29 |
+
from .preview import plan_topics
|
| 30 |
+
from .recipe import dump, endpoint_of, held_out_of, sources_of, targets_of
|
| 31 |
+
from .sources import Source, stream
|
| 32 |
+
from .text import MinHasher, NearDupIndex, doc_hash, est_tokens, h64
|
| 33 |
+
|
| 34 |
+
STAGES = ("grade", "curate", "create", "compose")
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
class Shards:
|
| 38 |
+
"""Append-only JSONL shards of a fixed number of documents."""
|
| 39 |
+
|
| 40 |
+
def __init__(self, folder: str, per: int):
|
| 41 |
+
os.makedirs(folder, exist_ok=True)
|
| 42 |
+
self.folder, self.per, self.n, self.f, self.count = folder, per, 0, None, 0
|
| 43 |
+
|
| 44 |
+
def write(self, doc: dict):
|
| 45 |
+
if self.f is None or self.count >= self.per:
|
| 46 |
+
if self.f:
|
| 47 |
+
self.f.close()
|
| 48 |
+
self.f = open(os.path.join(self.folder, f"part-{self.n:05d}.jsonl"), "w", encoding="utf-8")
|
| 49 |
+
self.n += 1
|
| 50 |
+
self.count = 0
|
| 51 |
+
self.f.write(json.dumps(doc, ensure_ascii=False) + "\n")
|
| 52 |
+
self.count += 1
|
| 53 |
+
|
| 54 |
+
def close(self):
|
| 55 |
+
if self.f:
|
| 56 |
+
self.f.close()
|
| 57 |
+
self.f = None
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
def read_dir(folder: str):
|
| 61 |
+
if not os.path.isdir(folder):
|
| 62 |
+
return
|
| 63 |
+
for name in sorted(os.listdir(folder)):
|
| 64 |
+
if name.endswith(".jsonl"):
|
| 65 |
+
with open(os.path.join(folder, name), encoding="utf-8") as f:
|
| 66 |
+
for line in f:
|
| 67 |
+
if line.strip():
|
| 68 |
+
yield json.loads(line)
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
class Run:
|
| 72 |
+
def __init__(self, recipe: dict, out: str, procs: int = 0, workers: int | None = None, limit: int = 0, log=print):
|
| 73 |
+
self.r, self.out, self.limit, self.log = recipe, os.path.abspath(out), limit, log
|
| 74 |
+
self.procs = procs or max(1, (os.cpu_count() or 2) - 1)
|
| 75 |
+
self.ep = endpoint_of(recipe, workers)
|
| 76 |
+
os.makedirs(self.out, exist_ok=True)
|
| 77 |
+
self.state_path = os.path.join(self.out, "state.json")
|
| 78 |
+
self.state = json.load(open(self.state_path)) if os.path.exists(self.state_path) else {"done": [], "sources": {}}
|
| 79 |
+
with open(os.path.join(self.out, "recipe.yaml"), "w", encoding="utf-8") as f:
|
| 80 |
+
f.write(dump(recipe))
|
| 81 |
+
|
| 82 |
+
def save(self):
|
| 83 |
+
tmp = self.state_path + ".tmp"
|
| 84 |
+
with open(tmp, "w") as f:
|
| 85 |
+
json.dump(self.state, f, indent=1)
|
| 86 |
+
os.replace(tmp, self.state_path)
|
| 87 |
+
|
| 88 |
+
def path(self, *p) -> str:
|
| 89 |
+
return os.path.join(self.out, *p)
|
| 90 |
+
|
| 91 |
+
def go(self, stages=STAGES):
|
| 92 |
+
if not self.ep.api_key and self.r["endpoint"]["base_url"].startswith("https://router.huggingface.co"):
|
| 93 |
+
raise SystemExit("Set DI_API_KEY (or HF_TOKEN) for the model endpoint.")
|
| 94 |
+
for st in STAGES:
|
| 95 |
+
if st not in stages:
|
| 96 |
+
continue
|
| 97 |
+
if st in self.state["done"]:
|
| 98 |
+
self.log(f"[{st}] already done, skipping (delete {self.state_path} to start over)")
|
| 99 |
+
continue
|
| 100 |
+
t = time.time()
|
| 101 |
+
getattr(self, "stage_" + st)()
|
| 102 |
+
self.state["done"].append(st)
|
| 103 |
+
self.save()
|
| 104 |
+
self.log(f"[{st}] done in {time.time() - t:.0f}s")
|
| 105 |
+
self.write_report()
|
| 106 |
+
|
| 107 |
+
# ------------------------------------------------------------ grade
|
| 108 |
+
def stage_grade(self):
|
| 109 |
+
r = self.r
|
| 110 |
+
n_total = int(r["calibrate"]["grade_docs"])
|
| 111 |
+
srcs = sources_of(r)
|
| 112 |
+
per = max(4, n_total // len(srcs))
|
| 113 |
+
cleaner = Cleaner(r["clean"]["rules"])
|
| 114 |
+
pool: list[dict] = []
|
| 115 |
+
for s in srcs:
|
| 116 |
+
got = []
|
| 117 |
+
for d in stream(s):
|
| 118 |
+
d2, why = cleaner.apply(d)
|
| 119 |
+
if d2:
|
| 120 |
+
got.append(d2)
|
| 121 |
+
if len(got) >= per * 3:
|
| 122 |
+
break
|
| 123 |
+
pool += random.Random(1).sample(got, min(per, len(got)))
|
| 124 |
+
self.log(f"[grade] {s.name}: sampled {min(per, len(got))} documents")
|
| 125 |
+
gpath = self.path("grades.jsonl")
|
| 126 |
+
done = {}
|
| 127 |
+
if os.path.exists(gpath):
|
| 128 |
+
for line in open(gpath, encoding="utf-8"):
|
| 129 |
+
g = json.loads(line)
|
| 130 |
+
done[g["id"]] = g
|
| 131 |
+
todo = [d for d in pool if d["id"] not in done]
|
| 132 |
+
desc = r["targets"]["description"]
|
| 133 |
+
self.log(f"[grade] grading {len(todo)} documents ({len(done)} already graded) with {self.ep.model}")
|
| 134 |
+
with open(gpath, "a", encoding="utf-8") as f, ThreadPoolExecutor(self.ep.workers) as ex:
|
| 135 |
+
for i, (d, g) in enumerate(zip(todo, ex.map(lambda d: cal.grade(self.ep, d["text"], desc), todo))):
|
| 136 |
+
if g:
|
| 137 |
+
rec = {"id": d["id"], "source": d["source"], "text": d["text"][:12000], **g}
|
| 138 |
+
f.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
| 139 |
+
done[d["id"]] = rec
|
| 140 |
+
if (i + 1) % 200 == 0:
|
| 141 |
+
f.flush()
|
| 142 |
+
self.log(f"[grade] {i + 1}/{len(todo)}")
|
| 143 |
+
graded = [(g["text"], g) for g in done.values()]
|
| 144 |
+
target_texts = []
|
| 145 |
+
for t in targets_of(r):
|
| 146 |
+
for i, d in enumerate(stream(t)):
|
| 147 |
+
if i >= int(r["targets"]["max_docs"]):
|
| 148 |
+
break
|
| 149 |
+
target_texts.append(d["text"])
|
| 150 |
+
calib = cal.Calibrator(float(r["calibrate"]["focus"]), desc, target_texts).fit(graded, [d["text"] for d in pool])
|
| 151 |
+
with open(self.path("calibrator.json"), "w") as f:
|
| 152 |
+
json.dump(calib.to_dict(), f)
|
| 153 |
+
self.state["grade"] = {"graded": len(graded), "agreement": calib.agreement}
|
| 154 |
+
self.log(f"[grade] scorer agreement with the model on held-out documents: {calib.agreement}")
|
| 155 |
+
|
| 156 |
+
# ------------------------------------------------------------ curate
|
| 157 |
+
def stage_curate(self):
|
| 158 |
+
r = self.r
|
| 159 |
+
dec_keys = r["clean"]["decontaminate"]
|
| 160 |
+
index = NearDupIndex(MinHasher(), threshold=float(r["calibrate"]["dedup"]["threshold"])) if r["calibrate"]["dedup"]["near"] else None
|
| 161 |
+
exact: set = set()
|
| 162 |
+
ctx = get_context("fork") if hasattr(os, "fork") else get_context("spawn")
|
| 163 |
+
held = {}
|
| 164 |
+
for h in held_out_of(r):
|
| 165 |
+
held[h.name] = [d["text"] for _, d in zip(range(200_000), stream(h))]
|
| 166 |
+
self.log(f"[curate] held-out set {h.name}: {len(held[h.name]):,} items")
|
| 167 |
+
init = (r["clean"]["rules"], dec_keys, held, int(r["clean"].get("ngram", 13)), self.path("calibrator.json"), bool(index))
|
| 168 |
+
with ctx.Pool(self.procs, initializer=_worker_init, initargs=init) as pool:
|
| 169 |
+
for s in sources_of(r):
|
| 170 |
+
st = self.state["sources"].get(s.name, {})
|
| 171 |
+
folder = self.path("curated", s.name)
|
| 172 |
+
if st.get("done"):
|
| 173 |
+
self.log(f"[curate] {s.name}: done earlier, loading it into the dedup index")
|
| 174 |
+
for d in read_dir(folder):
|
| 175 |
+
exact.add(d["h"])
|
| 176 |
+
if index is not None:
|
| 177 |
+
for sig, d in zip(pool.imap(_sig_only, (d["text"] for d in read_dir(folder)), chunksize=64), read_dir(folder)):
|
| 178 |
+
index.check_add_sig(h64(d["h"]), sig)
|
| 179 |
+
continue
|
| 180 |
+
if os.path.isdir(folder):
|
| 181 |
+
shutil.rmtree(folder)
|
| 182 |
+
sh = Shards(folder, int(r["output"]["shard_docs"]))
|
| 183 |
+
c = {"read": 0, "kept": 0, "dropped": {}, "decontaminated": {}, "tokens": 0}
|
| 184 |
+
scores: list[float] = []
|
| 185 |
+
t0 = time.time()
|
| 186 |
+
llm_all = r["calibrate"].get("scorer") == "llm"
|
| 187 |
+
calib = cal.Calibrator.from_dict(json.load(open(self.path("calibrator.json")))) if llm_all else None
|
| 188 |
+
ex = ThreadPoolExecutor(self.ep.workers) if llm_all else None
|
| 189 |
+
desc = r["targets"]["description"]
|
| 190 |
+
for batch in pool.imap(_worker_batch, _batches(stream(s), 256, self.limit), chunksize=1):
|
| 191 |
+
if llm_all: # grade every surviving document with the model instead of the learned scorer
|
| 192 |
+
live = [d for d, why, _ in batch if d]
|
| 193 |
+
for d, g in zip(live, ex.map(lambda d: cal.grade(self.ep, d["text"], desc), live)):
|
| 194 |
+
if g:
|
| 195 |
+
d.update(calib.score(d["text"], g))
|
| 196 |
+
for d, why, sig in batch:
|
| 197 |
+
c["read"] += 1
|
| 198 |
+
if why:
|
| 199 |
+
key = "decontaminated" if why.startswith("bench:") else "dropped"
|
| 200 |
+
name = why[6:] if key == "decontaminated" else why
|
| 201 |
+
c[key][name] = c[key].get(name, 0) + 1
|
| 202 |
+
continue
|
| 203 |
+
if d["h"] in exact:
|
| 204 |
+
c["dropped"]["exact duplicate"] = c["dropped"].get("exact duplicate", 0) + 1
|
| 205 |
+
continue
|
| 206 |
+
exact.add(d["h"])
|
| 207 |
+
if index is not None and index.check_add_sig(h64(d["h"]), sig) is not None:
|
| 208 |
+
c["dropped"]["near duplicate"] = c["dropped"].get("near duplicate", 0) + 1
|
| 209 |
+
continue
|
| 210 |
+
sh.write(d)
|
| 211 |
+
c["kept"] += 1
|
| 212 |
+
c["tokens"] += d["tokens"]
|
| 213 |
+
scores.append(d["score"])
|
| 214 |
+
if c["read"] % 50_000 < 256:
|
| 215 |
+
rate = c["read"] / max(1e-6, time.time() - t0)
|
| 216 |
+
self.log(f"[curate] {s.name}: {c['read']:,} read, {c['kept']:,} kept, {rate:,.0f} docs/s")
|
| 217 |
+
sh.close()
|
| 218 |
+
if ex:
|
| 219 |
+
ex.shutdown()
|
| 220 |
+
scores.sort(reverse=True)
|
| 221 |
+
keep_top, n = float(r["calibrate"]["keep_top"]), len(scores)
|
| 222 |
+
c["cut"] = scores[max(0, int(n * keep_top) - 1)] if n else 1.0
|
| 223 |
+
c["high_cut"] = scores[max(0, int(n * 0.2) - 1)] if n else 1.0
|
| 224 |
+
c["selected_tokens"], c["high_tokens"] = 0, 0
|
| 225 |
+
for d in read_dir(folder):
|
| 226 |
+
if d["score"] >= c["cut"]:
|
| 227 |
+
c["selected_tokens"] += d["tokens"]
|
| 228 |
+
if d["score"] >= c["high_cut"]:
|
| 229 |
+
c["high_tokens"] += d["tokens"]
|
| 230 |
+
c["done"] = True
|
| 231 |
+
self.state["sources"][s.name] = c
|
| 232 |
+
self.save()
|
| 233 |
+
self.log(f"[curate] {s.name}: kept {c['kept']:,} of {c['read']:,}; {fmt(c['selected_tokens'])} tokens above the score cut")
|
| 234 |
+
|
| 235 |
+
# ------------------------------------------------------------ create
|
| 236 |
+
def plans(self, synthetic: dict | None = None) -> list[dict]:
|
| 237 |
+
rows = [{"name": s.name, "kind": s.kind, "available_tokens": self.state["sources"].get(s.name, {}).get("selected_tokens", 0),
|
| 238 |
+
"high_tokens": self.state["sources"].get(s.name, {}).get("high_tokens", 0)} for s in sources_of(self.r)]
|
| 239 |
+
defaults = default_weights(rows)
|
| 240 |
+
out = []
|
| 241 |
+
for m in self.r["compose"]["mixes"]:
|
| 242 |
+
w = {**defaults, **{k: float(v) for k, v in (m.get("weights") or {}).items()}}
|
| 243 |
+
inv = float(m.get("invented_share", 0))
|
| 244 |
+
p = plan_mix(int(int(m["budget_tokens"]) * (1 - inv)), rows, w, float(self.r["compose"]["max_epochs"]), float(self.r["compose"]["max_synthetic_share"]))
|
| 245 |
+
p["name"], p["weights"], p["invented_share"], p["budget_tokens"] = m["name"], w, inv, int(m["budget_tokens"])
|
| 246 |
+
out.append(p)
|
| 247 |
+
return out
|
| 248 |
+
|
| 249 |
+
def stage_create(self):
|
| 250 |
+
r = self.r
|
| 251 |
+
rc, ic = r["create"]["rephrase"], r["create"]["invent"]
|
| 252 |
+
need: dict[str, int] = {}
|
| 253 |
+
for p in self.plans():
|
| 254 |
+
for row in p["rows"]:
|
| 255 |
+
need[row["name"]] = max(need.get(row["name"], 0), row["synthetic_tokens"])
|
| 256 |
+
made = self.state.setdefault("synthetic", {})
|
| 257 |
+
rng = random.Random(3)
|
| 258 |
+
if rc["enabled"]:
|
| 259 |
+
for s in sources_of(r):
|
| 260 |
+
want = need.get(s.name, 0)
|
| 261 |
+
have = made.get(s.name, {}).get("tokens", 0)
|
| 262 |
+
if want <= have:
|
| 263 |
+
continue
|
| 264 |
+
self.log(f"[create] {s.name}: rephrasing to add {fmt(want - have)} tokens")
|
| 265 |
+
self._rephrase_source(s, want, rc, rng)
|
| 266 |
+
if ic["enabled"] and int(ic["docs"]) > 0 and ic["request"].strip():
|
| 267 |
+
self._invent(ic, rng)
|
| 268 |
+
|
| 269 |
+
def _rephrase_source(self, s: Source, want: int, rc: dict, rng: random.Random):
|
| 270 |
+
folder = self.path("synthetic", s.name)
|
| 271 |
+
st = self.state["synthetic"].setdefault(s.name, {"tokens": 0, "docs": 0, "flagged": 0, "seen": 0})
|
| 272 |
+
cut = self.state["sources"][s.name]["high_cut"]
|
| 273 |
+
styles = rc["styles"] or styles_for(s.kind)
|
| 274 |
+
|
| 275 |
+
def gen():
|
| 276 |
+
# each pass over the best documents uses a different style, so later passes add variety, not copies
|
| 277 |
+
k = 0
|
| 278 |
+
for pass_ in range(len(styles)):
|
| 279 |
+
for d in read_dir(self.path("curated", s.name)):
|
| 280 |
+
if d["score"] >= cut:
|
| 281 |
+
for i, ch in enumerate(chunks(d["text"])):
|
| 282 |
+
yield d["id"], i, ch, styles[(pass_ + k) % len(styles)]
|
| 283 |
+
k += 1
|
| 284 |
+
jobs = itertools.islice(gen(), st["seen"], None)
|
| 285 |
+
sh = Shards(folder, int(self.r["output"]["shard_docs"]))
|
| 286 |
+
sh.n = len([f for f in os.listdir(folder)]) if os.path.isdir(folder) else 0
|
| 287 |
+
step = self.ep.workers * 4
|
| 288 |
+
with ThreadPoolExecutor(self.ep.workers) as ex:
|
| 289 |
+
while st["tokens"] < want:
|
| 290 |
+
part = list(itertools.islice(jobs, step))
|
| 291 |
+
if not part:
|
| 292 |
+
break
|
| 293 |
+
for (oid, i, ch, style), o in zip(part, ex.map(lambda j: rephrase(self.ep, j[2], j[3], rc.get("language", ""), float(rc["max_new_share"])), part)):
|
| 294 |
+
st["seen"] += 1
|
| 295 |
+
if not o["text"]:
|
| 296 |
+
continue
|
| 297 |
+
if not o["ok"]:
|
| 298 |
+
st["flagged"] += 1
|
| 299 |
+
continue
|
| 300 |
+
t = est_tokens(o["text"])
|
| 301 |
+
sh.write({"id": f"{oid}:{i}:{style}", "text": o["text"], "source": s.name, "kind": "synthetic", "origin": oid,
|
| 302 |
+
"style": style, "tokens": t, "fidelity": o["fidelity"]})
|
| 303 |
+
st["tokens"] += t
|
| 304 |
+
st["docs"] += 1
|
| 305 |
+
sh.close()
|
| 306 |
+
self.save()
|
| 307 |
+
self.log(f"[create] {s.name}: {fmt(st['tokens'])} of {fmt(want)} tokens, {st['flagged']} flagged")
|
| 308 |
+
if st["tokens"] < want:
|
| 309 |
+
self.log(f"[create] {s.name}: ran out of documents to rephrase at {fmt(st['tokens'])} tokens")
|
| 310 |
+
|
| 311 |
+
def _invent(self, ic: dict, rng: random.Random):
|
| 312 |
+
st = self.state["synthetic"].setdefault("invented", {"tokens": 0, "docs": 0, "flagged": 0, "seen": 0})
|
| 313 |
+
n = int(ic["docs"]) - st["docs"]
|
| 314 |
+
if n <= 0:
|
| 315 |
+
return
|
| 316 |
+
topics = plan_topics(self.ep, ic["request"], min(200, max(10, n // 5)))
|
| 317 |
+
jobs = [(rng.choice(topics), rng.choice(ic["doc_types"]), rng.choice(ic["levels"])) for _ in range(n)]
|
| 318 |
+
sh = Shards(self.path("synthetic", "invented"), int(self.r["output"]["shard_docs"]))
|
| 319 |
+
sh.n = len(os.listdir(self.path("synthetic", "invented")))
|
| 320 |
+
self.log(f"[create] inventing {n} documents over {len(topics)} topics")
|
| 321 |
+
with ThreadPoolExecutor(self.ep.workers) as ex:
|
| 322 |
+
for (t, dt, lv), o in zip(jobs, ex.map(lambda j: invent(self.ep, j[0], j[1], j[2], ic["request"], ic.get("language", "English")), jobs)):
|
| 323 |
+
if o.get("text"):
|
| 324 |
+
tk = est_tokens(o["text"])
|
| 325 |
+
sh.write({"id": f"invented:{st['docs']}", "text": o["text"], "source": "invented", "kind": "synthetic",
|
| 326 |
+
"topic": t, "doc_type": dt, "level": lv, "tokens": tk})
|
| 327 |
+
st["docs"] += 1
|
| 328 |
+
st["tokens"] += tk
|
| 329 |
+
sh.close()
|
| 330 |
+
|
| 331 |
+
# ------------------------------------------------------------ compose
|
| 332 |
+
def stage_compose(self):
|
| 333 |
+
r = self.r
|
| 334 |
+
synth = self.state.get("synthetic", {})
|
| 335 |
+
results = []
|
| 336 |
+
for p in self.plans():
|
| 337 |
+
name = p["name"]
|
| 338 |
+
folder = self.path("mixes", name)
|
| 339 |
+
if os.path.isdir(folder):
|
| 340 |
+
shutil.rmtree(folder)
|
| 341 |
+
sh = Shards(folder, int(r["output"]["shard_docs"]))
|
| 342 |
+
buckets = []
|
| 343 |
+
for row in p["rows"]:
|
| 344 |
+
s = row["name"]
|
| 345 |
+
cut = self.state["sources"].get(s, {}).get("cut", 1.0)
|
| 346 |
+
real_target = row["want_tokens"] - min(row["synthetic_tokens"], synth.get(s, {}).get("tokens", 0))
|
| 347 |
+
buckets.append(_Bucket(s, self.path("curated", s), cut, real_target, float(r["compose"]["max_epochs"]),
|
| 348 |
+
self.state["sources"].get(s, {}).get("selected_tokens", 0)))
|
| 349 |
+
if row["synthetic_tokens"] and synth.get(s, {}).get("tokens"):
|
| 350 |
+
buckets.append(_Bucket(s + ":synthetic", self.path("synthetic", s), -1, min(row["synthetic_tokens"], synth[s]["tokens"]), 1.0,
|
| 351 |
+
synth[s]["tokens"]))
|
| 352 |
+
share = float(p.get("invented_share", 0))
|
| 353 |
+
if share and synth.get("invented", {}).get("tokens"):
|
| 354 |
+
buckets.append(_Bucket("invented", self.path("synthetic", "invented"), -1, int(p["budget_tokens"] * share), 1.0,
|
| 355 |
+
synth["invented"]["tokens"]))
|
| 356 |
+
rng = random.Random(11)
|
| 357 |
+
buf: list[dict] = []
|
| 358 |
+
live = [b for b in buckets if b.target > 0]
|
| 359 |
+
while live:
|
| 360 |
+
# draw the next bucket in proportion to the tokens it still owes, which interleaves sources evenly
|
| 361 |
+
b = rng.choices(live, weights=[x.target - x.taken for x in live], k=1)[0]
|
| 362 |
+
d = b.next()
|
| 363 |
+
if d is None:
|
| 364 |
+
live.remove(b)
|
| 365 |
+
continue
|
| 366 |
+
buf.append({"id": d["id"], "text": d["text"], "source": d["source"], "kind": d.get("kind", "")})
|
| 367 |
+
if len(buf) >= 20_000:
|
| 368 |
+
rng.shuffle(buf)
|
| 369 |
+
for x in buf[:10_000]:
|
| 370 |
+
sh.write(x)
|
| 371 |
+
buf = buf[10_000:]
|
| 372 |
+
if b.taken >= b.target:
|
| 373 |
+
live.remove(b)
|
| 374 |
+
rng.shuffle(buf)
|
| 375 |
+
for x in buf:
|
| 376 |
+
sh.write(x)
|
| 377 |
+
sh.close()
|
| 378 |
+
manifest = {"name": name, "budget_tokens": p["budget_tokens"], "weights": p["weights"],
|
| 379 |
+
"buckets": [{"name": b.name, "target_tokens": b.target, "tokens": b.taken, "docs": b.docs, "epochs": b.epochs_used()} for b in buckets],
|
| 380 |
+
"tokens": sum(b.taken for b in buckets), "warnings": p["warnings"]}
|
| 381 |
+
with open(os.path.join(folder, "manifest.json"), "w") as f:
|
| 382 |
+
json.dump(manifest, f, indent=1)
|
| 383 |
+
results.append(manifest)
|
| 384 |
+
self.log(f"[compose] {name}: {fmt(manifest['tokens'])} tokens of {fmt(p['budget_tokens'])}")
|
| 385 |
+
self.state["mixes"] = results
|
| 386 |
+
|
| 387 |
+
# ------------------------------------------------------------ report
|
| 388 |
+
def write_report(self):
|
| 389 |
+
st = self.state
|
| 390 |
+
lines = [f"# Curation report: {self.r['name']}", "", f"Model: `{self.ep.model}`. Stages done: {', '.join(st['done']) or 'none'}.", ""]
|
| 391 |
+
if st.get("grade"):
|
| 392 |
+
a = st["grade"]["agreement"]
|
| 393 |
+
lines += [f"Graded {st['grade']['graded']:,} documents. Scorer rank agreement with the model on held-out documents: "
|
| 394 |
+
f"quality {a.get('quality')}, relevance {a.get('relevance')}.", ""]
|
| 395 |
+
if st["sources"]:
|
| 396 |
+
lines += ["## Sources", "", "| Source | Read | Kept | Decontaminated | Top drop reasons | Tokens above cut |", "|---|---|---|---|---|---|"]
|
| 397 |
+
for n, c in st["sources"].items():
|
| 398 |
+
top = ", ".join(f"{k} {v:,}" for k, v in sorted(c["dropped"].items(), key=lambda x: -x[1])[:3])
|
| 399 |
+
dec = sum(c["decontaminated"].values())
|
| 400 |
+
lines.append(f"| {n} | {c['read']:,} | {c['kept']:,} | {dec:,} | {top} | {fmt(c.get('selected_tokens', 0))} |")
|
| 401 |
+
lines.append("")
|
| 402 |
+
if st.get("synthetic"):
|
| 403 |
+
lines += ["## Created", "", "| From | Documents | Tokens | Flagged |", "|---|---|---|---|"]
|
| 404 |
+
for n, c in st["synthetic"].items():
|
| 405 |
+
lines.append(f"| {n} | {c['docs']:,} | {fmt(c['tokens'])} | {c['flagged']:,} |")
|
| 406 |
+
lines.append("")
|
| 407 |
+
for m in st.get("mixes", []):
|
| 408 |
+
lines += [f"## Mix `{m['name']}`: {fmt(m['tokens'])} of {fmt(m['budget_tokens'])} tokens", "",
|
| 409 |
+
"| Bucket | Target | Written | Epochs |", "|---|---|---|---|"]
|
| 410 |
+
for b in m["buckets"]:
|
| 411 |
+
lines.append(f"| {b['name']} | {fmt(b['target_tokens'])} | {fmt(b['tokens'])} | {b['epochs']} |")
|
| 412 |
+
lines += [""] + [f"- {w}" for w in m["warnings"]] + [""]
|
| 413 |
+
with open(self.path("REPORT.md"), "w") as f:
|
| 414 |
+
f.write("\n".join(lines))
|
| 415 |
+
with open(self.path("report.json"), "w") as f:
|
| 416 |
+
json.dump(st, f, indent=1)
|
| 417 |
+
|
| 418 |
+
|
| 419 |
+
class _Bucket:
|
| 420 |
+
"""Documents of one source above its score cut, cycled up to max_epochs times."""
|
| 421 |
+
|
| 422 |
+
def __init__(self, name: str, folder: str, cut: float, target: int, max_epochs: float, pass_tokens: int = 0):
|
| 423 |
+
self.name, self.folder, self.cut, self.target, self.max_epochs = name, folder, cut, int(target), max_epochs
|
| 424 |
+
self.pass_tokens = pass_tokens
|
| 425 |
+
self.taken, self.docs, self.epoch, self.it, self.epoch_tokens = 0, 0, 0, None, 0
|
| 426 |
+
|
| 427 |
+
def next(self):
|
| 428 |
+
while True:
|
| 429 |
+
if self.it is None:
|
| 430 |
+
if self.epoch >= max(1, int(self.max_epochs + 0.999)):
|
| 431 |
+
return None
|
| 432 |
+
self.it = iter(read_dir(self.folder))
|
| 433 |
+
self.epoch += 1
|
| 434 |
+
for d in self.it:
|
| 435 |
+
if d.get("score", 1.0) >= self.cut:
|
| 436 |
+
self.taken += d.get("tokens") or est_tokens(d["text"])
|
| 437 |
+
self.docs += 1
|
| 438 |
+
return d
|
| 439 |
+
self.it = None
|
| 440 |
+
if self.taken == 0:
|
| 441 |
+
return None
|
| 442 |
+
|
| 443 |
+
def epochs_used(self) -> float:
|
| 444 |
+
return round(self.taken / self.pass_tokens, 2) if self.pass_tokens else float(self.epoch)
|
| 445 |
+
|
| 446 |
+
|
| 447 |
+
# ---------------------------------------------------------------- worker processes
|
| 448 |
+
|
| 449 |
+
_W: dict = {}
|
| 450 |
+
|
| 451 |
+
|
| 452 |
+
def _worker_init(rules, dec_keys, held, ngram, calib_path, near):
|
| 453 |
+
_W["clean"] = Cleaner(rules)
|
| 454 |
+
dec = Decontaminator(ngram)
|
| 455 |
+
for k in dec_keys:
|
| 456 |
+
dec.add(k, benchmark_texts(k))
|
| 457 |
+
for k, texts in held.items():
|
| 458 |
+
dec.add(k, texts)
|
| 459 |
+
_W["dec"] = dec
|
| 460 |
+
_W["calib"] = cal.Calibrator.from_dict(json.load(open(calib_path)))
|
| 461 |
+
_W["mh"] = MinHasher() if near else None
|
| 462 |
+
|
| 463 |
+
|
| 464 |
+
def _worker_batch(batch: list[dict]) -> list[tuple]:
|
| 465 |
+
out = []
|
| 466 |
+
for d in batch:
|
| 467 |
+
d2, why = _W["clean"].apply(d)
|
| 468 |
+
if not d2:
|
| 469 |
+
out.append((None, why, None))
|
| 470 |
+
continue
|
| 471 |
+
b = _W["dec"].hit(d2["text"])
|
| 472 |
+
if b:
|
| 473 |
+
out.append((None, "bench:" + b, None))
|
| 474 |
+
continue
|
| 475 |
+
sc = _W["calib"].score(d2["text"])
|
| 476 |
+
rec = {**d2, **sc, "tokens": est_tokens(d2["text"]), "h": doc_hash(d2["text"])}
|
| 477 |
+
out.append((rec, "", _W["mh"].signature(d2["text"]) if _W["mh"] else None))
|
| 478 |
+
return out
|
| 479 |
+
|
| 480 |
+
|
| 481 |
+
def _sig_only(text: str):
|
| 482 |
+
return _W["mh"].signature(text) if _W.get("mh") else None
|
| 483 |
+
|
| 484 |
+
|
| 485 |
+
def _batches(it, n: int, limit: int = 0):
|
| 486 |
+
buf, seen = [], 0
|
| 487 |
+
for d in it:
|
| 488 |
+
buf.append(d)
|
| 489 |
+
seen += 1
|
| 490 |
+
if len(buf) >= n:
|
| 491 |
+
yield buf
|
| 492 |
+
buf = []
|
| 493 |
+
if limit and seen >= limit:
|
| 494 |
+
break
|
| 495 |
+
if buf:
|
| 496 |
+
yield buf
|
| 497 |
+
|
datainventor/sources.py
ADDED
|
@@ -0,0 +1,192 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Where documents come from.
|
| 2 |
+
|
| 3 |
+
Two readers share one Source spec:
|
| 4 |
+
- sample(): a spread sample through the Hugging Face dataset viewer API. No downloads, used by the hosted preview.
|
| 5 |
+
- stream(): every document, through `datasets` streaming or local JSONL/Parquet files. Used by the runner on your own machines.
|
| 6 |
+
"""
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
import glob
|
| 10 |
+
import json
|
| 11 |
+
import os
|
| 12 |
+
import time
|
| 13 |
+
import urllib.error
|
| 14 |
+
import urllib.parse
|
| 15 |
+
import urllib.request
|
| 16 |
+
from dataclasses import asdict, dataclass, field
|
| 17 |
+
from typing import Iterator
|
| 18 |
+
|
| 19 |
+
VIEWER = "https://datasets-server.huggingface.co"
|
| 20 |
+
KINDS = ("web", "educational", "textbook", "math", "code", "multilingual", "synthetic", "other")
|
| 21 |
+
TEXT_FIELDS = ("text", "content", "markdown", "body", "document", "raw_content", "code", "problem", "question", "prompt")
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
@dataclass
|
| 25 |
+
class Source:
|
| 26 |
+
"""One dataset, registered as a source to curate or as a target that says what good looks like."""
|
| 27 |
+
name: str
|
| 28 |
+
dataset: str = "" # Hugging Face id, e.g. HuggingFaceFW/fineweb-edu
|
| 29 |
+
config: str = "default"
|
| 30 |
+
split: str = "train"
|
| 31 |
+
text_field: str = "text"
|
| 32 |
+
files: str = "" # local glob instead of a hub id: /data/*.jsonl or /data/*.parquet
|
| 33 |
+
kind: str = "web"
|
| 34 |
+
role: str = "source" # source | target
|
| 35 |
+
num_rows: int = 0 # filled by inspect(); 0 means unknown
|
| 36 |
+
bytes: int = 0
|
| 37 |
+
extra: dict = field(default_factory=dict)
|
| 38 |
+
|
| 39 |
+
def to_dict(self) -> dict:
|
| 40 |
+
return {k: v for k, v in asdict(self).items() if v not in ("", 0, {}, None)}
|
| 41 |
+
|
| 42 |
+
@staticmethod
|
| 43 |
+
def from_dict(d: dict) -> "Source":
|
| 44 |
+
keys = Source.__dataclass_fields__.keys()
|
| 45 |
+
return Source(**{k: v for k, v in d.items() if k in keys})
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def _get(path: str, params: dict, token: str = "", timeout: int = 60, retries: int = 7) -> dict:
|
| 49 |
+
url = f"{VIEWER}/{path}?{urllib.parse.urlencode(params)}"
|
| 50 |
+
headers = {"User-Agent": "DataInventor/1.0"}
|
| 51 |
+
if token:
|
| 52 |
+
headers["Authorization"] = "Bearer " + token
|
| 53 |
+
for attempt in range(retries):
|
| 54 |
+
try:
|
| 55 |
+
return json.load(urllib.request.urlopen(urllib.request.Request(url, headers=headers), timeout=timeout))
|
| 56 |
+
except urllib.error.HTTPError as e:
|
| 57 |
+
if e.code not in (429, 500, 502, 503, 504) or attempt == retries - 1:
|
| 58 |
+
raise
|
| 59 |
+
time.sleep(min(30, 2 * 2 ** attempt))
|
| 60 |
+
except (TimeoutError, urllib.error.URLError):
|
| 61 |
+
if attempt == retries - 1:
|
| 62 |
+
raise
|
| 63 |
+
time.sleep(2)
|
| 64 |
+
raise RuntimeError("unreachable")
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
def inspect(dataset: str, token: str = "") -> dict:
|
| 68 |
+
"""Configs, splits, string columns and sizes for a hub dataset, so the user can pick what to read."""
|
| 69 |
+
splits = _get("splits", {"dataset": dataset}, token).get("splits", [])
|
| 70 |
+
if not splits:
|
| 71 |
+
raise ValueError("The dataset viewer has no splits for that dataset. Check the id, or that it is public.")
|
| 72 |
+
first = splits[0]
|
| 73 |
+
cfg, spl = first["config"], first["split"]
|
| 74 |
+
for s in splits: # prefer a train split of the default config
|
| 75 |
+
if s["split"] == "train" and s["config"] in ("default", cfg):
|
| 76 |
+
cfg, spl = s["config"], s["split"]
|
| 77 |
+
break
|
| 78 |
+
cols, sample = [], ""
|
| 79 |
+
try:
|
| 80 |
+
fr = _get("first-rows", {"dataset": dataset, "config": cfg, "split": spl}, token)
|
| 81 |
+
cols = [f["name"] for f in fr.get("features", []) if (f.get("type") or {}).get("dtype") in ("string", "large_string")]
|
| 82 |
+
if fr.get("rows"):
|
| 83 |
+
sample = fr["rows"][0]["row"]
|
| 84 |
+
except Exception:
|
| 85 |
+
pass
|
| 86 |
+
field_ = next((c for c in TEXT_FIELDS if c in cols), cols[0] if cols else "text")
|
| 87 |
+
sizes = {}
|
| 88 |
+
try:
|
| 89 |
+
for c in _get("size", {"dataset": dataset}, token).get("size", {}).get("splits", []):
|
| 90 |
+
sizes[(c["config"], c["split"])] = (c.get("num_rows", 0), c.get("num_bytes_parquet_files", 0))
|
| 91 |
+
except Exception:
|
| 92 |
+
pass
|
| 93 |
+
rows, nbytes = sizes.get((cfg, spl), (0, 0))
|
| 94 |
+
preview = str((sample or {}).get(field_, ""))[:4000] if isinstance(sample, dict) else ""
|
| 95 |
+
return {"dataset": dataset, "config": cfg, "split": spl, "splits": [{"config": s["config"], "split": s["split"]} for s in splits][:40],
|
| 96 |
+
"columns": cols, "text_field": field_, "num_rows": rows, "bytes": nbytes, "preview": preview,
|
| 97 |
+
"kind": guess_kind(dataset)}
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
def guess_kind(dataset: str) -> str:
|
| 101 |
+
d = dataset.lower().split("/")[-1] # the owner name says nothing about the content
|
| 102 |
+
if any(k in d for k in ("math", "gsm", "proof", "numina")):
|
| 103 |
+
return "math"
|
| 104 |
+
if any(k in d for k in ("stack", "code", "github")):
|
| 105 |
+
return "code"
|
| 106 |
+
if any(k in d for k in ("pdf", "textbook", "sutra", "cosmopedia", "wiki")):
|
| 107 |
+
return "textbook"
|
| 108 |
+
if "edu" in d:
|
| 109 |
+
return "educational"
|
| 110 |
+
if any(k in d for k in ("fineweb-2", "multiling", "culturax", "madlad", "hplt")):
|
| 111 |
+
return "multilingual"
|
| 112 |
+
return "web"
|
| 113 |
+
|
| 114 |
+
|
| 115 |
+
def sample(src: Source, n: int, token: str = "", chunks: int = 4) -> list[dict]:
|
| 116 |
+
"""Up to n documents read from `chunks` evenly spaced offsets, so a sample is not just the first shard."""
|
| 117 |
+
if src.files:
|
| 118 |
+
out = []
|
| 119 |
+
for d in stream(src):
|
| 120 |
+
out.append(d)
|
| 121 |
+
if len(out) >= n:
|
| 122 |
+
break
|
| 123 |
+
return out
|
| 124 |
+
total = src.num_rows or 0
|
| 125 |
+
per = max(1, min(100, -(-n // chunks)))
|
| 126 |
+
offsets = [0] if total <= n else [int(total * (i + 0.5) / chunks) for i in range(chunks)]
|
| 127 |
+
out, seen = [], set()
|
| 128 |
+
for off in offsets:
|
| 129 |
+
try:
|
| 130 |
+
got = _get("rows", {"dataset": src.dataset, "config": src.config, "split": src.split, "offset": off, "length": per}, token)
|
| 131 |
+
except Exception as e:
|
| 132 |
+
if not out and off == offsets[-1]:
|
| 133 |
+
raise ValueError(f"Could not read rows from {src.dataset}: {e}")
|
| 134 |
+
continue
|
| 135 |
+
for r in got.get("rows", []):
|
| 136 |
+
idx = r.get("row_idx", off)
|
| 137 |
+
text = (r.get("row") or {}).get(src.text_field)
|
| 138 |
+
if isinstance(text, str) and text.strip() and idx not in seen:
|
| 139 |
+
seen.add(idx)
|
| 140 |
+
out.append({"id": f"{src.name}:{idx}", "text": text, "source": src.name, "kind": src.kind})
|
| 141 |
+
if len(out) >= n:
|
| 142 |
+
break
|
| 143 |
+
return out[:n]
|
| 144 |
+
|
| 145 |
+
|
| 146 |
+
def stream(src: Source, skip: int = 0) -> Iterator[dict]:
|
| 147 |
+
"""Every document in the source. Local files are read directly; hub datasets stream through `datasets`."""
|
| 148 |
+
if src.files:
|
| 149 |
+
paths = sorted(glob.glob(os.path.expanduser(src.files)))
|
| 150 |
+
if not paths:
|
| 151 |
+
raise ValueError(f"No files match {src.files}")
|
| 152 |
+
i = 0
|
| 153 |
+
for p in paths:
|
| 154 |
+
for row in _read_file(p):
|
| 155 |
+
text = row.get(src.text_field)
|
| 156 |
+
if isinstance(text, str) and text.strip():
|
| 157 |
+
if i >= skip:
|
| 158 |
+
yield {"id": f"{src.name}:{i}", "text": text, "source": src.name, "kind": src.kind}
|
| 159 |
+
i += 1
|
| 160 |
+
return
|
| 161 |
+
try:
|
| 162 |
+
from datasets import load_dataset
|
| 163 |
+
except ImportError as e: # pragma: no cover
|
| 164 |
+
raise SystemExit("Reading a hub dataset in full needs `pip install datasets`.") from e
|
| 165 |
+
ds = load_dataset(src.dataset, None if src.config == "default" else src.config, split=src.split, streaming=True)
|
| 166 |
+
if skip:
|
| 167 |
+
ds = ds.skip(skip)
|
| 168 |
+
for i, row in enumerate(ds, start=skip):
|
| 169 |
+
text = row.get(src.text_field)
|
| 170 |
+
if isinstance(text, str) and text.strip():
|
| 171 |
+
yield {"id": f"{src.name}:{i}", "text": text, "source": src.name, "kind": src.kind}
|
| 172 |
+
|
| 173 |
+
|
| 174 |
+
def _read_file(path: str) -> Iterator[dict]:
|
| 175 |
+
if path.endswith((".jsonl", ".json")):
|
| 176 |
+
with open(path, encoding="utf-8") as f:
|
| 177 |
+
for line in f:
|
| 178 |
+
line = line.strip()
|
| 179 |
+
if line:
|
| 180 |
+
try:
|
| 181 |
+
yield json.loads(line)
|
| 182 |
+
except json.JSONDecodeError:
|
| 183 |
+
continue
|
| 184 |
+
elif path.endswith(".parquet"):
|
| 185 |
+
import pyarrow.parquet as pq
|
| 186 |
+
for batch in pq.ParquetFile(path).iter_batches(batch_size=2048):
|
| 187 |
+
yield from batch.to_pylist()
|
| 188 |
+
elif path.endswith(".txt"):
|
| 189 |
+
with open(path, encoding="utf-8") as f:
|
| 190 |
+
yield {"text": f.read()}
|
| 191 |
+
else:
|
| 192 |
+
raise ValueError(f"Unsupported file type: {path} (use .jsonl, .parquet or .txt)")
|
datainventor/text.py
ADDED
|
@@ -0,0 +1,143 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Text primitives shared by every stage: normalization, token estimates, n-grams, MinHash, hashed features.
|
| 2 |
+
|
| 3 |
+
Pure standard library so the hosted preview needs no heavy installs. numpy is used when present to speed up MinHash.
|
| 4 |
+
"""
|
| 5 |
+
from __future__ import annotations
|
| 6 |
+
|
| 7 |
+
import hashlib
|
| 8 |
+
import math
|
| 9 |
+
import re
|
| 10 |
+
import struct
|
| 11 |
+
import unicodedata
|
| 12 |
+
|
| 13 |
+
try: # optional speed-up for MinHash on large runs
|
| 14 |
+
import numpy as _np
|
| 15 |
+
except Exception: # pragma: no cover
|
| 16 |
+
_np = None
|
| 17 |
+
|
| 18 |
+
WORD = re.compile(r"\w+", re.U)
|
| 19 |
+
_MERSENNE = (1 << 61) - 1
|
| 20 |
+
_MAXH = (1 << 32) - 1
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def norm(text: str) -> str:
|
| 24 |
+
"""Lowercase, strip accents and punctuation, squeeze whitespace. Used for exact dedup and n-gram matching."""
|
| 25 |
+
t = unicodedata.normalize("NFKD", text.lower())
|
| 26 |
+
t = "".join(c for c in t if not unicodedata.combining(c))
|
| 27 |
+
return " ".join(WORD.findall(t))
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
def words(text: str) -> list[str]:
|
| 31 |
+
return WORD.findall(text.lower())
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def est_tokens(text: str) -> int:
|
| 35 |
+
"""About four characters per token for English prose. Good enough to plan a budget, not to bill one."""
|
| 36 |
+
return max(1, len(text) // 4)
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def h64(s: str) -> int:
|
| 40 |
+
return int.from_bytes(hashlib.blake2b(s.encode("utf-8", "ignore"), digest_size=8).digest(), "little")
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def doc_hash(text: str) -> str:
|
| 44 |
+
return hashlib.blake2b(norm(text).encode(), digest_size=12).hexdigest()
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def ngrams(ws: list[str], n: int) -> list[str]:
|
| 48 |
+
return [" ".join(ws[i:i + n]) for i in range(len(ws) - n + 1)] if len(ws) >= n else ([" ".join(ws)] if ws else [])
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
# ---------------------------------------------------------------- MinHash + LSH (near-duplicate detection)
|
| 52 |
+
|
| 53 |
+
class MinHasher:
|
| 54 |
+
"""MinHash over word 5-gram shingles. 112 permutations in 14 bands of 8 rows find pairs at roughly 0.8 Jaccard,
|
| 55 |
+
the setting FineWeb used for near-duplicate removal."""
|
| 56 |
+
|
| 57 |
+
def __init__(self, num_perm: int = 112, bands: int = 14, shingle: int = 5, seed: int = 1):
|
| 58 |
+
assert num_perm % bands == 0
|
| 59 |
+
self.num_perm, self.bands, self.rows, self.shingle = num_perm, bands, num_perm // bands, shingle
|
| 60 |
+
import random
|
| 61 |
+
rng = random.Random(seed)
|
| 62 |
+
# a, b < 2**32 keep a*x + b inside 64 bits, so the numpy and pure-Python paths agree exactly
|
| 63 |
+
self.a = [rng.randrange(1, _MAXH) for _ in range(num_perm)]
|
| 64 |
+
self.b = [rng.randrange(0, _MAXH) for _ in range(num_perm)]
|
| 65 |
+
if _np is not None:
|
| 66 |
+
self._a = _np.array(self.a, dtype=_np.uint64)
|
| 67 |
+
self._b = _np.array(self.b, dtype=_np.uint64)
|
| 68 |
+
|
| 69 |
+
def signature(self, text: str) -> list[int]:
|
| 70 |
+
sh = {h64(g) & _MAXH for g in ngrams(words(text), self.shingle)} or {0}
|
| 71 |
+
if _np is not None:
|
| 72 |
+
x = _np.fromiter(sh, dtype=_np.uint64)[:, None]
|
| 73 |
+
v = ((x * self._a + self._b) % _np.uint64(_MERSENNE)) & _np.uint64(_MAXH)
|
| 74 |
+
return v.min(axis=0).tolist()
|
| 75 |
+
return [min(((a * x + b) % _MERSENNE) & _MAXH for x in sh) for a, b in zip(self.a, self.b)]
|
| 76 |
+
|
| 77 |
+
def band_keys(self, sig: list[int]) -> list[bytes]:
|
| 78 |
+
r = self.rows
|
| 79 |
+
return [bytes([i]) + struct.pack(f"<{r}I", *sig[i * r:(i + 1) * r]) for i in range(self.bands)]
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def jaccard_sig(a: list[int], b: list[int]) -> float:
|
| 83 |
+
return sum(1 for x, y in zip(a, b) if x == y) / len(a)
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
class NearDupIndex:
|
| 87 |
+
"""In-memory LSH index. Holds one band table per band; fine for tens of millions of documents on a large box.
|
| 88 |
+
Past that, shard by a hash prefix and run one index per shard."""
|
| 89 |
+
|
| 90 |
+
def __init__(self, hasher: MinHasher | None = None, threshold: float = 0.8):
|
| 91 |
+
self.h = hasher or MinHasher()
|
| 92 |
+
self.threshold = threshold
|
| 93 |
+
self.tables: list[dict] = [dict() for _ in range(self.h.bands)]
|
| 94 |
+
self.sigs: dict[int, list[int]] = {}
|
| 95 |
+
|
| 96 |
+
def check_add(self, key: int, text: str) -> int | None:
|
| 97 |
+
"""Return the key of an earlier near-duplicate, or None after adding this document."""
|
| 98 |
+
return self.check_add_sig(key, self.h.signature(text))
|
| 99 |
+
|
| 100 |
+
def check_add_sig(self, key: int, sig: list[int]) -> int | None:
|
| 101 |
+
"""Same, with a signature computed elsewhere (in a worker process)."""
|
| 102 |
+
cands = set()
|
| 103 |
+
for i, k in enumerate(self.h.band_keys(sig)):
|
| 104 |
+
hit = self.tables[i].get(k)
|
| 105 |
+
if hit is not None:
|
| 106 |
+
cands.add(hit)
|
| 107 |
+
for c in cands:
|
| 108 |
+
if jaccard_sig(sig, self.sigs[c]) >= self.threshold:
|
| 109 |
+
return c
|
| 110 |
+
for i, k in enumerate(self.h.band_keys(sig)):
|
| 111 |
+
self.tables[i].setdefault(k, key)
|
| 112 |
+
self.sigs[key] = sig
|
| 113 |
+
return None
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
# ---------------------------------------------------------------- hashed features (cheap classifier + relevance)
|
| 117 |
+
|
| 118 |
+
def features(text: str, dim: int = 1 << 18) -> dict[int, float]:
|
| 119 |
+
"""Hashed unigrams and bigrams, log-scaled and L2-normalized. Works for any language with word characters."""
|
| 120 |
+
ws = words(text[:20000])
|
| 121 |
+
counts: dict[int, float] = {}
|
| 122 |
+
for g in ws + ngrams(ws, 2):
|
| 123 |
+
k = h64(g) % dim
|
| 124 |
+
counts[k] = counts.get(k, 0.0) + 1.0
|
| 125 |
+
for k in counts:
|
| 126 |
+
counts[k] = 1.0 + math.log(counts[k])
|
| 127 |
+
n = math.sqrt(sum(v * v for v in counts.values())) or 1.0
|
| 128 |
+
return {k: v / n for k, v in counts.items()}
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
def cosine(a: dict[int, float], b: dict[int, float]) -> float:
|
| 132 |
+
if len(a) > len(b):
|
| 133 |
+
a, b = b, a
|
| 134 |
+
return sum(v * b.get(k, 0.0) for k, v in a.items())
|
| 135 |
+
|
| 136 |
+
|
| 137 |
+
def centroid(vecs: list[dict[int, float]]) -> dict[int, float]:
|
| 138 |
+
c: dict[int, float] = {}
|
| 139 |
+
for v in vecs:
|
| 140 |
+
for k, x in v.items():
|
| 141 |
+
c[k] = c.get(k, 0.0) + x
|
| 142 |
+
n = math.sqrt(sum(x * x for x in c.values())) or 1.0
|
| 143 |
+
return {k: x / n for k, x in c.items()}
|
make_benchmarks.py
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Bundle the decontamination test sets with the package, so runs never depend on the dataset viewer.
|
| 2 |
+
Needs `datasets`. Run again to refresh: python make_benchmarks.py"""
|
| 3 |
+
import gzip
|
| 4 |
+
import json
|
| 5 |
+
|
| 6 |
+
from datasets import load_dataset
|
| 7 |
+
|
| 8 |
+
from datainventor.clean import BENCHMARKS
|
| 9 |
+
|
| 10 |
+
out = {}
|
| 11 |
+
for key, (ds, cfg, split, fields) in BENCHMARKS.items():
|
| 12 |
+
rows = load_dataset(ds, None if cfg == "default" else cfg, split=split)
|
| 13 |
+
out[key] = [t for t in (" ".join(str(r.get(f, "")) for f in fields).strip() for r in rows) if t]
|
| 14 |
+
print(key, len(out[key]), flush=True)
|
| 15 |
+
with gzip.open("datainventor/data/benchmarks.json.gz", "wt", encoding="utf-8") as f:
|
| 16 |
+
json.dump(out, f, ensure_ascii=False)
|
pyproject.toml
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[build-system]
|
| 2 |
+
requires = ["setuptools>=68"]
|
| 3 |
+
build-backend = "setuptools.build_meta"
|
| 4 |
+
|
| 5 |
+
[project]
|
| 6 |
+
name = "datainventor"
|
| 7 |
+
version = "0.2.0"
|
| 8 |
+
description = "Curate a pretraining corpus on your own machines: clean, calibrate, create, compose."
|
| 9 |
+
readme = "README.md"
|
| 10 |
+
requires-python = ">=3.10"
|
| 11 |
+
license = { text = "Apache-2.0" }
|
| 12 |
+
dependencies = ["pyyaml>=6"]
|
| 13 |
+
|
| 14 |
+
[project.optional-dependencies]
|
| 15 |
+
full = ["datasets>=2.18", "pyarrow>=14", "numpy>=1.24"]
|
| 16 |
+
|
| 17 |
+
[project.scripts]
|
| 18 |
+
datainventor = "datainventor.cli:main"
|
| 19 |
+
|
| 20 |
+
[tool.setuptools]
|
| 21 |
+
packages = ["datainventor"]
|
| 22 |
+
|
| 23 |
+
[tool.setuptools.package-data]
|
| 24 |
+
datainventor = ["data/*.json.gz"]
|
requirements.txt
CHANGED
|
@@ -2,3 +2,4 @@ fastapi==0.115.*
|
|
| 2 |
uvicorn[standard]==0.32.*
|
| 3 |
pydantic>=2.7
|
| 4 |
huggingface_hub>=0.30
|
|
|
|
|
|
| 2 |
uvicorn[standard]==0.32.*
|
| 3 |
pydantic>=2.7
|
| 4 |
huggingface_hub>=0.30
|
| 5 |
+
pyyaml>=6
|
static/app.css
CHANGED
|
@@ -188,3 +188,58 @@ textarea.in { min-height: 118px; resize: vertical; font-family: var(--doc); font
|
|
| 188 |
.stats { grid-template-columns: repeat(2, minmax(0, 1fr)); }
|
| 189 |
.press { height: 250px; }
|
| 190 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 188 |
.stats { grid-template-columns: repeat(2, minmax(0, 1fr)); }
|
| 189 |
.press { height: 250px; }
|
| 190 |
}
|
| 191 |
+
|
| 192 |
+
/* mode switch */
|
| 193 |
+
.modes { display: inline-flex; border: 1.5px solid var(--ink); border-radius: var(--r); overflow: hidden; flex: none; }
|
| 194 |
+
.modes button { background: var(--paper-2); border: 0; padding: 7px 14px; font-weight: 600; cursor: pointer; }
|
| 195 |
+
.modes button + button { border-left: 1.5px solid var(--ink); }
|
| 196 |
+
.modes button[aria-pressed="true"] { background: var(--ink); color: #fff; }
|
| 197 |
+
|
| 198 |
+
/* curate */
|
| 199 |
+
.addrow { display: grid; grid-template-columns: minmax(0, 1fr) auto auto; gap: 10px; margin-top: 8px; }
|
| 200 |
+
.in.sm { width: auto; padding: 7px 10px; font-size: 14.5px; }
|
| 201 |
+
.in.num { width: 7.5em; font-variant-numeric: tabular-nums; }
|
| 202 |
+
.srow { display: grid; grid-template-columns: minmax(0, 1fr) auto; gap: 14px; padding: 12px 0; border-bottom: 1px solid var(--rule); align-items: start; }
|
| 203 |
+
.srow:first-of-type { border-top: 1px solid var(--ink); margin-top: 10px; }
|
| 204 |
+
.sname b { display: block; overflow-wrap: anywhere; }
|
| 205 |
+
.sname .hint { display: block; }
|
| 206 |
+
.sctl { display: flex; gap: 10px; align-items: center; flex-wrap: wrap; justify-content: flex-end; }
|
| 207 |
+
.peek, .ptext { font-family: var(--doc); white-space: pre-wrap; font-size: 14.5px; line-height: 1.5; max-height: 260px; overflow: auto; margin-top: 6px; padding: 10px 12px; background: var(--paper-2); border: 1px solid var(--rule); border-radius: 2px; }
|
| 208 |
+
details summary { cursor: pointer; color: var(--blue); font-weight: 600; font-size: 14px; margin-top: 4px; }
|
| 209 |
+
code { font-size: .92em; background: var(--paper-2); padding: 0 4px; border-radius: 2px; }
|
| 210 |
+
.range { width: 100%; accent-color: var(--blue); }
|
| 211 |
+
.sv { font-weight: 600; color: var(--blue); margin-left: 6px; }
|
| 212 |
+
.pairs { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 14px; margin-top: 16px; }
|
| 213 |
+
.pcard .ph { display: flex; gap: 10px; align-items: baseline; flex-wrap: wrap; }
|
| 214 |
+
.flag { color: var(--bad); font-size: 13px; font-weight: 600; }
|
| 215 |
+
.mix { padding-top: 14px; border-top: 1px solid var(--rule); }
|
| 216 |
+
.mixhead { display: flex; flex-wrap: wrap; gap: 12px; align-items: center; }
|
| 217 |
+
.mixhead .seg { margin-top: 0; }
|
| 218 |
+
.wtable { display: grid; grid-template-columns: repeat(auto-fill, minmax(230px, 1fr)); gap: 10px 22px; margin-top: 14px; }
|
| 219 |
+
.wtable label { display: grid; grid-template-columns: minmax(0, 1fr) auto auto; gap: 6px; align-items: center; }
|
| 220 |
+
.wtable label > span:first-child { overflow-wrap: anywhere; }
|
| 221 |
+
.stages { list-style: none; display: flex; flex-wrap: wrap; gap: 6px 22px; padding: 0; margin: 26px 0 0; }
|
| 222 |
+
.stages li { color: var(--slate); }
|
| 223 |
+
.stages li.done { color: var(--ink); }
|
| 224 |
+
.stages li.on { color: var(--blue); font-weight: 600; }
|
| 225 |
+
.stages li.on::after { content: " …"; }
|
| 226 |
+
.tscroll { overflow-x: auto; margin-top: 8px; }
|
| 227 |
+
.tbl { width: 100%; border-collapse: collapse; font-size: 14.5px; font-variant-numeric: tabular-nums; }
|
| 228 |
+
.tbl th { text-align: left; font-weight: 600; border-bottom: 1px solid var(--ink); padding: 8px 12px 8px 0; white-space: nowrap; }
|
| 229 |
+
.tbl td { border-bottom: 1px solid var(--rule); padding: 9px 12px 9px 0; vertical-align: top; }
|
| 230 |
+
.hists { display: grid; grid-template-columns: repeat(auto-fill, minmax(180px, 1fr)); gap: 18px; margin-top: 12px; }
|
| 231 |
+
.hists span { font-weight: 600; font-size: 14px; }
|
| 232 |
+
.hist { display: flex; align-items: flex-end; gap: 3px; height: 56px; margin-top: 6px; border-bottom: 1px solid var(--ink); }
|
| 233 |
+
.hist i { flex: 1; background: var(--blue); min-height: 1px; }
|
| 234 |
+
.ex { margin-top: 10px; }
|
| 235 |
+
.exd { margin-top: 10px; }
|
| 236 |
+
.quote { font-family: var(--doc); font-size: 14.5px; margin-top: 8px; }
|
| 237 |
+
.code { background: var(--ink); color: #f1f2f4; padding: 14px 16px; border-radius: var(--r); overflow-x: auto; font-size: 13.5px; line-height: 1.6; margin-top: 12px; white-space: pre; }
|
| 238 |
+
@media (max-width: 640px) {
|
| 239 |
+
.pairs { grid-template-columns: 1fr; }
|
| 240 |
+
.addrow { grid-template-columns: 1fr auto; }
|
| 241 |
+
.addrow .btn { grid-column: 1 / -1; }
|
| 242 |
+
.srow { grid-template-columns: 1fr; }
|
| 243 |
+
.sctl { justify-content: flex-start; }
|
| 244 |
+
}
|
| 245 |
+
.in.num.wide { width: 11em; }
|
static/app.js
CHANGED
|
@@ -209,6 +209,7 @@ function updateRun() {
|
|
| 209 |
|
| 210 |
/* ------------------------------------------------------------ render + events */
|
| 211 |
function render() {
|
|
|
|
| 212 |
renderLedger(); renderSheet();
|
| 213 |
const fn = [stepFormat, stepDomains, stepSubs, stepLangs, stepSummary, stepReview, stepRun][S.step];
|
| 214 |
const keep = document.activeElement && document.activeElement.id;
|
|
@@ -262,6 +263,7 @@ async function poll() {
|
|
| 262 |
if (j.stats && j.stats.rows) S.maxStep = 6;
|
| 263 |
} catch (e) { console.error(e.stack); S.jobState = { state: "failed", message: e.message, count: S.jobRows.length, target: S.rows, elapsed: 0, id }; if (S.step === 6) render(); }
|
| 264 |
}
|
|
|
|
| 265 |
/* "Start writing" on the review step starts the run */
|
| 266 |
const _go = go;
|
| 267 |
go = function (i) { const wasReview = S.step === 5; _go(i); if (i === 6 && wasReview && !S.jobState) startRun(); return true; };
|
|
|
|
| 209 |
|
| 210 |
/* ------------------------------------------------------------ render + events */
|
| 211 |
function render() {
|
| 212 |
+
if (window.DI_MODE === "curate") return;
|
| 213 |
renderLedger(); renderSheet();
|
| 214 |
const fn = [stepFormat, stepDomains, stepSubs, stepLangs, stepSummary, stepReview, stepRun][S.step];
|
| 215 |
const keep = document.activeElement && document.activeElement.id;
|
|
|
|
| 263 |
if (j.stats && j.stats.rows) S.maxStep = 6;
|
| 264 |
} catch (e) { console.error(e.stack); S.jobState = { state: "failed", message: e.message, count: S.jobRows.length, target: S.rows, elapsed: 0, id }; if (S.step === 6) render(); }
|
| 265 |
}
|
| 266 |
+
window.DI_renderInvent = () => render();
|
| 267 |
/* "Start writing" on the review step starts the run */
|
| 268 |
const _go = go;
|
| 269 |
go = function (i) { const wasReview = S.step === 5; _go(i); if (i === 6 && wasReview && !S.jobState) startRun(); return true; };
|
static/curate.js
ADDED
|
@@ -0,0 +1,371 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"use strict";
|
| 2 |
+
/* Curate a corpus: register sources and targets, then clean, calibrate, create, compose.
|
| 3 |
+
The hosted run works on a sample; the recipe runs in full on your own machines. */
|
| 4 |
+
(function () {
|
| 5 |
+
const CSTEPS = ["Sources", "Clean", "Calibrate", "Create", "Compose", "Run"];
|
| 6 |
+
const BUDGETS = [["1B", 1e9], ["10B", 1e10], ["100B", 1e11], ["1T", 1e12]];
|
| 7 |
+
const BENCH_NAMES = { gsm8k: "GSM8K", math500: "MATH-500", humaneval: "HumanEval", mbpp: "MBPP", mmlu: "MMLU", arc_challenge: "ARC-Challenge",
|
| 8 |
+
arc_easy: "ARC-Easy", hellaswag: "HellaSwag", openbookqa: "OpenBookQA", winogrande: "WinoGrande", truthfulqa: "TruthfulQA", commonsenseqa: "CommonsenseQA" };
|
| 9 |
+
const KIND_NAMES = { web: "Web", educational: "Educational web", textbook: "Textbook-style", math: "Math", code: "Code", multilingual: "Multilingual", synthetic: "Synthetic", other: "Other" };
|
| 10 |
+
const RULES = [["min_words", "Fewest words in a document", 1], ["max_dup_lines", "Most repeated lines (share)", 0.01], ["max_dup10gram", "Most repeated passages (share)", 0.01], ["min_alpha_words", "Fewest words with letters (share)", 0.01]];
|
| 11 |
+
const PRESET = [
|
| 12 |
+
{ dataset: "codelion/finepdfs-1B", kind: "textbook" }, { dataset: "codelion/dclm-baseline-1B", kind: "web" }, { dataset: "codelion/fineweb-edu-1B", kind: "educational" }];
|
| 13 |
+
|
| 14 |
+
const C = {
|
| 15 |
+
step: 0, maxStep: 0, d: null, sources: [], targets: [], targetsDesc: "", add: "", addRole: "source", addBusy: false, addErr: "",
|
| 16 |
+
decon: ["gsm8k", "math500", "humaneval", "mmlu", "arc_challenge", "hellaswag"], rules: {},
|
| 17 |
+
focus: 0.3, keepTop: 0.6, near: true, threshold: 0.8, gradeDocs: 2000, scorer: "learned",
|
| 18 |
+
rephraseOn: true, styles: [], maxNew: 0.35, inventOn: false, inventReq: "", inventDocs: 0,
|
| 19 |
+
mixes: [{ name: "main", budget: 1e10, weights: {} }], maxEpochs: 4, maxSynth: 0.25,
|
| 20 |
+
tryText: "", tryBusy: false, tryErr: "", tryRes: null, tryFrom: "",
|
| 21 |
+
pid: null, pv: null, runErr: "",
|
| 22 |
+
};
|
| 23 |
+
const $ = (s, el = document) => el.querySelector(s);
|
| 24 |
+
const esc = (t) => String(t ?? "").replace(/[&<>"']/g, (c) => ({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" }[c]));
|
| 25 |
+
const fmt = (n) => { n = +n || 0; for (const [u, d] of [["T", 1e12], ["B", 1e9], ["M", 1e6], ["K", 1e3]]) if (Math.abs(n) >= d) return `${+(n / d).toPrecision(3)}${u}`; return `${Math.round(n)}`; };
|
| 26 |
+
const pct = (x) => `${Math.round((+x || 0) * 100)}%`;
|
| 27 |
+
async function api(path, body, method) {
|
| 28 |
+
const r = await fetch(path, { method: method || (body ? "POST" : "GET"), headers: body ? { "Content-Type": "application/json" } : {}, body: body ? JSON.stringify(body) : undefined });
|
| 29 |
+
let data = null; try { data = await r.json(); } catch (e) { /* not json */ }
|
| 30 |
+
if (!r.ok) throw new Error((data && (data.detail || data.message)) || `Request failed (${r.status})`);
|
| 31 |
+
return data;
|
| 32 |
+
}
|
| 33 |
+
|
| 34 |
+
/* ------------------------------------------------------------ recipe */
|
| 35 |
+
function srcOut(s) { return { name: s.name, dataset: s.dataset, config: s.config, split: s.split, text_field: s.text_field, kind: s.kind, num_rows: s.num_rows || 0 }; }
|
| 36 |
+
function recipe() {
|
| 37 |
+
return {
|
| 38 |
+
name: "my-corpus", sources: C.sources.map(srcOut),
|
| 39 |
+
targets: { description: C.targetsDesc.trim(), datasets: C.targets.map(srcOut) },
|
| 40 |
+
clean: { decontaminate: C.decon, rules: C.rules },
|
| 41 |
+
calibrate: { dedup: { near: C.near, threshold: C.threshold }, grade_docs: C.gradeDocs, scorer: C.scorer, focus: C.focus, keep_top: C.keepTop },
|
| 42 |
+
create: { rephrase: { enabled: C.rephraseOn, styles: C.styles, max_new_share: C.maxNew },
|
| 43 |
+
invent: { enabled: C.inventOn && !!C.inventReq.trim(), request: C.inventReq.trim(), docs: C.inventDocs } },
|
| 44 |
+
compose: { mixes: C.mixes.map((m) => ({ name: m.name, budget_tokens: m.budget, weights: m.weights })), max_epochs: C.maxEpochs, max_synthetic_share: C.maxSynth },
|
| 45 |
+
};
|
| 46 |
+
}
|
| 47 |
+
/* mirrors datainventor.compose.default_weights */
|
| 48 |
+
function defaultWeights() {
|
| 49 |
+
const G = { textbook: "textbook", synthetic: "textbook", web: "web", multilingual: "web", other: "web", educational: "educational", math: "math", code: "code" };
|
| 50 |
+
const BASE = { textbook: 0.5, web: 0.3, educational: 0.2 };
|
| 51 |
+
const groups = {}; C.sources.forEach((s) => (groups[G[s.kind] || "web"] = groups[G[s.kind] || "web"] || []).push(s.name));
|
| 52 |
+
const fixed = {}; ["math", "code"].forEach((g) => { if (groups[g]) fixed[g] = 0.1; });
|
| 53 |
+
const base = Object.fromEntries(Object.entries(BASE).filter(([g]) => groups[g]));
|
| 54 |
+
const out = {};
|
| 55 |
+
if (!Object.keys(base).length) { C.sources.forEach((s) => (out[s.name] = 1 / C.sources.length)); return out; }
|
| 56 |
+
const rest = 1 - Object.values(fixed).reduce((a, b) => a + b, 0), tot = Object.values(base).reduce((a, b) => a + b, 0);
|
| 57 |
+
const shares = { ...Object.fromEntries(Object.entries(base).map(([g, v]) => [g, (rest * v) / tot])), ...fixed };
|
| 58 |
+
Object.entries(groups).forEach(([g, names]) => names.forEach((n) => (out[n] = shares[g] / names.length)));
|
| 59 |
+
return out;
|
| 60 |
+
}
|
| 61 |
+
const weightsOf = (m) => ({ ...defaultWeights(), ...m.weights });
|
| 62 |
+
|
| 63 |
+
/* ------------------------------------------------------------ chrome */
|
| 64 |
+
function ledger() {
|
| 65 |
+
$("#steps").innerHTML = CSTEPS.map((n, i) => {
|
| 66 |
+
const cls = i === C.step ? "on" : i < C.step ? "done" : "";
|
| 67 |
+
return `<li class="${cls}"><button data-cgo="${i}" ${i > C.maxStep ? "disabled" : ""} ${i === C.step ? 'aria-current="step"' : ""}><span class="n">${i + 1}</span><span>${n}</span></button></li>`;
|
| 68 |
+
}).join("");
|
| 69 |
+
}
|
| 70 |
+
function sheet() {
|
| 71 |
+
const line = (k, v) => `<div class="line"><dt>${k}</dt><dd>${v}</dd></div>`;
|
| 72 |
+
const m = C.mixes[0];
|
| 73 |
+
$("#sheet").innerHTML = `<h2>Your recipe</h2><dl>
|
| 74 |
+
${line("Sources", C.sources.length ? esc(C.sources.map((s) => s.name).join(", ")) : '<span class="hint">None yet</span>')}
|
| 75 |
+
${line("Targets", C.targetsDesc.trim() || C.targets.length ? esc([C.targetsDesc.trim().slice(0, 80), ...C.targets.map((t) => t.name)].filter(Boolean).join("; ")) : '<span class="hint">General quality only</span>')}
|
| 76 |
+
${line("Decontam", C.decon.length ? esc(C.decon.map((b) => BENCH_NAMES[b] || b).join(", ")) : "Off")}
|
| 77 |
+
${line("Focus", C.focus === 0 || !(C.targetsDesc.trim() || C.targets.length) ? "General quality" : `${pct(1 - C.focus)} quality, ${pct(C.focus)} targets`)}
|
| 78 |
+
${line("Keep", `Top ${pct(C.keepTop)} by score`)}
|
| 79 |
+
${line("Create", C.rephraseOn ? "Rephrase to fill gaps" + (C.inventOn ? ", invent" : "") : C.inventOn ? "Invent" : "Off")}
|
| 80 |
+
${line("Mixes", C.mixes.map((x) => `${esc(x.name)} ${fmt(x.budget)}`).join(", "))}
|
| 81 |
+
</dl>
|
| 82 |
+
<p class="est">The hosted run works on about <b>${C.d ? C.d.per_source : 200}</b> documents per source. The recipe runs the same steps on every document, on your machines.</p>
|
| 83 |
+
<div class="actions" style="margin-top:14px"><button class="btn" data-c="recipe" ${C.sources.length ? "" : "disabled"}>Download recipe</button></div>`;
|
| 84 |
+
}
|
| 85 |
+
function actions(back, next, ok = true, extra = "") {
|
| 86 |
+
return `<div class="actions">${back ? `<button class="btn" data-c="back">Back</button>` : ""}${next ? `<button class="btn go" data-c="next" ${ok ? "" : "disabled"}>${next}</button>` : ""}${extra}</div>`;
|
| 87 |
+
}
|
| 88 |
+
function check(on, attr, label, hint = "") {
|
| 89 |
+
return `<button class="choice" ${attr} aria-pressed="${on}"><span class="box"></span><span>${label}${hint ? ` <span class="hint">${hint}</span>` : ""}</span></button>`;
|
| 90 |
+
}
|
| 91 |
+
function slider(id, label, val, min, max, step, show) {
|
| 92 |
+
return `<label class="f"><span>${label} <b class="sv" id="${id}-v">${show(val)}</b></span><input type="range" class="range" id="${id}" min="${min}" max="${max}" step="${step}" value="${val}"></label>`;
|
| 93 |
+
}
|
| 94 |
+
|
| 95 |
+
/* ------------------------------------------------------------ steps */
|
| 96 |
+
function srcRow(s, i, role) {
|
| 97 |
+
const kinds = C.d ? C.d.kinds : Object.keys(KIND_NAMES);
|
| 98 |
+
return `<div class="srow">
|
| 99 |
+
<div class="sname"><b>${esc(s.dataset)}</b><span class="hint">${esc(s.config)} / ${esc(s.split)} / column <code>${esc(s.text_field)}</code>${s.num_rows ? ` / ${fmt(s.num_rows)} rows` : ""}</span>
|
| 100 |
+
${s.preview ? `<details><summary>Peek at a document</summary><div class="peek">${esc(s.preview)}</div></details>` : ""}</div>
|
| 101 |
+
<div class="sctl">${role === "source" ? `<select class="in sm" id="c-kind-${i}" aria-label="Kind of data">${kinds.map((k) => `<option value="${k}" ${s.kind === k ? "selected" : ""}>${KIND_NAMES[k] || k}</option>`).join("")}</select>` : ""}
|
| 102 |
+
${s.columns && s.columns.length > 1 ? `<select class="in sm" id="c-col-${role}-${i}" aria-label="Text column">${s.columns.map((c) => `<option ${c === s.text_field ? "selected" : ""}>${esc(c)}</option>`).join("")}</select>` : ""}
|
| 103 |
+
<button class="linkish" data-c="rm" data-role="${role}" data-i="${i}">Remove</button></div></div>`;
|
| 104 |
+
}
|
| 105 |
+
function stepSources() {
|
| 106 |
+
return `<h1>Bring the data you have</h1>
|
| 107 |
+
<p class="lede">Register each Hugging Face dataset as a source to curate, or as a target that shows what good looks like. Targets steer every later step toward your tasks.</p>
|
| 108 |
+
<div class="group"><h3>Add a dataset</h3>
|
| 109 |
+
<div class="addrow"><input class="in" id="c-add" placeholder="HuggingFaceFW/fineweb-edu" value="${esc(C.add)}" spellcheck="false" autocomplete="off">
|
| 110 |
+
<select class="in sm" id="c-addrole" aria-label="Register as"><option value="source" ${C.addRole === "source" ? "selected" : ""}>Source</option><option value="target" ${C.addRole === "target" ? "selected" : ""}>Target</option></select>
|
| 111 |
+
<button class="btn" data-c="add" ${C.addBusy || !C.add.trim() ? "disabled" : ""}>${C.addBusy ? "Reading" : "Add"}</button></div>
|
| 112 |
+
${C.addErr ? `<p class="note bad">${esc(C.addErr)}</p>` : ""}
|
| 113 |
+
${C.sources.length ? "" : `<p class="hint" style="margin-top:10px">Or start from <button class="linkish" data-c="preset">the 1B-token mix</button>: textbook-style PDFs, DCLM and FineWeb-Edu.</p>`}</div>
|
| 114 |
+
<div class="group"><h3>Sources <span class="hint">(up to ${C.d ? C.d.max_sources : 4} here)</span></h3>
|
| 115 |
+
${C.sources.length ? C.sources.map((s, i) => srcRow(s, i, "source")).join("") : '<p class="hint">None yet.</p>'}</div>
|
| 116 |
+
<div class="group"><h3>Targets</h3>
|
| 117 |
+
<label class="f" style="margin-top:6px"><span>What should the trained model be good at? <span class="hint">(optional)</span></span>
|
| 118 |
+
<textarea class="in" id="c-tdesc" style="min-height:80px" placeholder="Reading and drafting commercial contracts. Grade-school math word problems.">${esc(C.targetsDesc)}</textarea></label>
|
| 119 |
+
${C.targets.map((s, i) => srcRow(s, i, "target")).join("")}
|
| 120 |
+
<p class="hint" style="margin-top:8px">Target datasets are examples of the work you care about, such as a benchmark's training split or your own documents.</p></div>
|
| 121 |
+
${actions(false, "Continue", C.sources.length > 0)}`;
|
| 122 |
+
}
|
| 123 |
+
function stepClean() {
|
| 124 |
+
const b = C.d ? C.d.benchmarks : Object.keys(BENCH_NAMES), r = { ...(C.d ? C.d.clean_rules : {}), ...C.rules };
|
| 125 |
+
return `<h2>Clean</h2>
|
| 126 |
+
<p class="lede">Heuristic filters drop broken text, boilerplate, empty and repetitive documents. Math and code get their own looser rules. Then any document that shares a 13-word run with a test set you name is removed.</p>
|
| 127 |
+
<div class="group"><h3>Decontaminate against</h3><ul class="choices">${b.map((k) => `<li>${check(C.decon.includes(k), `data-bench="${k}"`, BENCH_NAMES[k] || k)}</li>`).join("")}</ul>
|
| 128 |
+
<p class="hint" style="margin-top:6px">To check against test sets of your own, list them under <code>clean.held_out</code> in the recipe, the same way as sources.</p></div>
|
| 129 |
+
<div class="group"><h3>Filters</h3><div class="row2">${RULES.map(([k, label, step]) => `<label class="f"><span>${label}</span><input class="in" type="number" id="c-rule-${k}" step="${step}" value="${r[k]}"></label>`).join("")}</div>
|
| 130 |
+
<p class="hint" style="margin-top:8px">The defaults follow the Gopher and C4 rules with FineWeb's thresholds. Every rule is in the recipe.</p></div>
|
| 131 |
+
${actions(true, "Continue")}`;
|
| 132 |
+
}
|
| 133 |
+
function stepCalibrate() {
|
| 134 |
+
const hasT = !!(C.targetsDesc.trim() || C.targets.length);
|
| 135 |
+
return `<h2>Calibrate</h2>
|
| 136 |
+
<p class="lede">A model grades a sample of each source for teaching value, clarity, accuracy and density${hasT ? ", and for relevance to your targets" : ""}. A small scorer learns those grades and scores every document. Redundant copies are dropped first.</p>
|
| 137 |
+
${hasT ? slider("c-focus", "Balance between general quality and your targets", C.focus, 0, 1, 0.05, (v) => (+v === 0 ? "general quality only" : `${pct(1 - v)} quality / ${pct(v)} targets`))
|
| 138 |
+
: `<p class="note">No targets, so documents are ranked by general quality. Add targets in step 1 to steer the ranking.</p>`}
|
| 139 |
+
${hasT ? '<p class="hint">Keep some weight on general quality. Data tuned only to the targets overfits them.</p>' : ""}
|
| 140 |
+
${slider("c-keep", "Keep the best", C.keepTop, 0.1, 1, 0.05, (v) => `${pct(v)} of each source`)}
|
| 141 |
+
<div class="group"><h3>Duplicates</h3><ul class="choices" style="columns:1">
|
| 142 |
+
<li>${check(true, "disabled", "Exact copies", "always removed")}</li>
|
| 143 |
+
<li>${check(C.near, 'data-c="near"', "Near copies", "MinHash, about 80% overlap")}</li></ul></div>
|
| 144 |
+
<div class="group"><h3>On your machines</h3>
|
| 145 |
+
<div class="row2"><label class="f"><span>Documents to grade with the model</span><input class="in" type="number" id="c-grade" min="100" step="100" value="${C.gradeDocs}"></label>
|
| 146 |
+
<label class="f"><span>Score every document by</span><select class="in" id="c-scorer"><option value="learned" ${C.scorer === "learned" ? "selected" : ""}>A scorer learned from the grades</option><option value="llm" ${C.scorer === "llm" ? "selected" : ""}>The model itself (slow, best)</option></select></label></div>
|
| 147 |
+
<p class="hint" style="margin-top:8px">The hosted run grades ${C.d ? 48 : "a few dozen"} documents, enough to see the rubric work but not to trust the learned scorer. In our test, 300 grades gave a rank agreement of about 0.5 with the model; more grades help.</p></div>
|
| 148 |
+
${actions(true, "Continue")}`;
|
| 149 |
+
}
|
| 150 |
+
function tryHtml() {
|
| 151 |
+
if (C.tryErr) return `<p class="note bad">${esc(C.tryErr)}</p>`;
|
| 152 |
+
if (!C.tryRes) return "";
|
| 153 |
+
return `<div class="pairs">${C.tryRes.results.map((o) => `<div class="pcard"><div class="ph"><b>${esc(o.name)}</b>${o.ok ? "" : `<span class="flag">${esc(o.why)}</span>`}</div>
|
| 154 |
+
<div class="ptext">${esc(o.text || "No rewrite. The model call failed.")}</div>
|
| 155 |
+
${o.fidelity && (o.fidelity.new_numbers.length || o.fidelity.new_names.length) ? `<p class="hint">Not in the source: ${esc([...o.fidelity.new_numbers, ...o.fidelity.new_names].join(", "))}</p>` : ""}</div>`).join("")}</div>`;
|
| 156 |
+
}
|
| 157 |
+
function stepCreate() {
|
| 158 |
+
const st = C.d ? C.d.styles : {}, picked = C.styles;
|
| 159 |
+
const first = C.sources.find((s) => s.preview);
|
| 160 |
+
return `<h2>Create</h2>
|
| 161 |
+
<p class="lede">Where a source is too small for its share of a mix, the best documents are rephrased into new ones: a dialogue, a tutorial, questions and answers, dense notes, a textbook section. Rewrites that add numbers or names the source does not contain are flagged and dropped.</p>
|
| 162 |
+
<ul class="choices" style="columns:1;margin-top:20px"><li>${check(C.rephraseOn, 'data-c="reph"', "Rephrase to fill gaps")}</li></ul>
|
| 163 |
+
${C.rephraseOn ? `<div class="group"><h3>Styles <span class="hint">(none picked = chosen per kind of data)</span></h3><ul class="choices">${Object.entries(st).map(([k, n]) => `<li>${check(picked.includes(k), `data-style="${k}"`, esc(n))}</li>`).join("")}</ul></div>
|
| 164 |
+
<div class="group"><h3>Try it on one document</h3>
|
| 165 |
+
<textarea class="in" id="c-try" style="min-height:120px" placeholder="Paste a passage of 60 words or more.">${esc(C.tryText)}</textarea>
|
| 166 |
+
<div class="actions" style="margin-top:12px">${first ? `<button class="btn" data-c="tryfill">Use a document from ${esc(first.name)}</button>` : ""}
|
| 167 |
+
<button class="btn go" data-c="try" ${C.tryBusy || C.tryText.trim().split(/\s+/).length < 60 ? "disabled" : ""}>${C.tryBusy ? "Rewriting" : "Rewrite it in four styles"}</button>${C.tryBusy ? '<span class="hint">This takes about a minute.</span>' : ""}</div>
|
| 168 |
+
<div id="c-tryout">${tryHtml()}</div></div>` : ""}
|
| 169 |
+
<ul class="choices" style="columns:1;margin-top:26px"><li>${check(C.inventOn, 'data-c="inv"', "Also write new documents on a subject", "for areas your sources barely cover")}</li></ul>
|
| 170 |
+
${C.inventOn ? `<div class="row2"><label class="f"><span>Subject</span><input class="in" id="c-invreq" value="${esc(C.inventReq)}" placeholder="Contract law for small businesses"></label>
|
| 171 |
+
<label class="f"><span>Documents on your machines</span><input class="in" type="number" id="c-invdocs" min="0" step="1000" value="${C.inventDocs}"></label></div>
|
| 172 |
+
<p class="hint" style="margin-top:8px">The hosted run writes four as a preview. Give invented documents a share in a mix with <code>invented_share</code> in the recipe.</p>` : ""}
|
| 173 |
+
${actions(true, "Continue")}`;
|
| 174 |
+
}
|
| 175 |
+
function stepCompose() {
|
| 176 |
+
return `<h2>Compose</h2>
|
| 177 |
+
<p class="lede">Build one or more training mixes from the prepared corpus. The default share is the static mix that won our 1B-token study: 50% textbook-style, 30% web, 20% educational web, with 10% each for math and code when present. It was measured at 70M parameters, so treat it as a starting point.</p>
|
| 178 |
+
${C.mixes.map((m, mi) => { const w = weightsOf(m), tot = Object.values(w).reduce((a, b) => a + b, 0) || 1;
|
| 179 |
+
return `<div class="group mix"><div class="mixhead"><input class="in sm" id="c-mname-${mi}" value="${esc(m.name)}" aria-label="Mix name" spellcheck="false">
|
| 180 |
+
<div class="seg">${BUDGETS.map(([l, v]) => `<button data-c="budget" data-m="${mi}" data-v="${v}" aria-pressed="${m.budget === v}">${l}</button>`).join("")}</div>
|
| 181 |
+
<label class="hint">or <input class="in sm num wide" type="number" id="c-mbud-${mi}" value="${m.budget}" min="1000000" step="1000000"> tokens</label>
|
| 182 |
+
${C.mixes.length > 1 ? `<button class="linkish" data-c="rmmix" data-m="${mi}">Remove</button>` : ""}</div>
|
| 183 |
+
<div class="wtable">${C.sources.map((s) => `<label><span>${esc(s.name)} <small class="hint">${KIND_NAMES[s.kind] || s.kind}</small></span>
|
| 184 |
+
<input class="in sm num" type="number" min="0" max="100" step="1" id="c-w-${mi}-${esc(s.name)}" value="${Math.round((w[s.name] / tot) * 100)}"><span class="hint">%</span></label>`).join("")}</div></div>`; }).join("")}
|
| 185 |
+
<p style="margin-top:14px"><button class="linkish" data-c="addmix">Add another mix</button> <span class="hint">for a different run, from the same prepared corpus</span></p>
|
| 186 |
+
<div class="row2">${slider("c-epochs", "Repeat a source at most", C.maxEpochs, 1, 8, 0.5, (v) => `${v} times`)}
|
| 187 |
+
${slider("c-synth", "Synthetic text at most", C.maxSynth, 0, 0.6, 0.05, (v) => `${pct(v)} of a mix`)}</div>
|
| 188 |
+
<p class="hint">A source short of its share is repeated first, then topped up by rephrasing its best documents. In our study a mix that leaned on synthetic-like text scored well on its own validation set but generalized worst.</p>
|
| 189 |
+
${actions(true, "Continue")}`;
|
| 190 |
+
}
|
| 191 |
+
|
| 192 |
+
/* ------------------------------------------------------------ run + report */
|
| 193 |
+
const isFinal = (p) => p && ["done", "stopped", "failed"].includes(p.state);
|
| 194 |
+
function hist(h) {
|
| 195 |
+
const mx = Math.max(1, ...h);
|
| 196 |
+
return `<div class="hist" aria-hidden="true">${h.map((v) => `<i style="height:${Math.round((v / mx) * 100)}%"></i>`).join("")}</div>`;
|
| 197 |
+
}
|
| 198 |
+
function report(rep) {
|
| 199 |
+
const src = rep.sources || [], cal = (rep.stages || {}).calibrate || {}, comp = (rep.stages || {}).compose || {}, crt = (rep.stages || {}).create || {};
|
| 200 |
+
const agree = cal.agreement || {};
|
| 201 |
+
const drops = (f) => Object.entries(f.dropped).filter(([, v]) => v).sort((a, b) => b[1] - a[1]).map(([k, v]) => `${esc(k)} ${v}`).join(", ") || "none";
|
| 202 |
+
return `
|
| 203 |
+
<div class="group"><h3>What each source went through</h3><div class="tscroll"><table class="tbl"><thead><tr><th>Source</th><th>Read</th><th>Passed cleaning</th><th>Kept</th><th>Removed for test overlap</th><th>Tokens after curation</th></tr></thead><tbody>
|
| 204 |
+
${src.map((f) => `<tr><td><b>${esc(f.name)}</b><br><span class="hint">${drops(f)}</span></td><td>${f.read}</td><td>${f.kept}</td><td>${f.selected}</td>
|
| 205 |
+
<td>${Object.keys(f.decontaminated).length ? esc(Object.entries(f.decontaminated).map(([k, v]) => `${BENCH_NAMES[k] || k} ${v}`).join(", ")) : "0"}</td>
|
| 206 |
+
<td>${f.available_tokens ? "about " + fmt(f.available_tokens) : '<span class="hint">unknown size</span>'}</td></tr>`).join("")}</tbody></table></div>
|
| 207 |
+
<p class="hint" style="margin-top:8px">Tokens after curation scale each sample's keep rate to the full dataset. They are estimates at four characters per token.</p></div>
|
| 208 |
+
${src.some((f) => f.repeated_passages && f.repeated_passages.length) ? `<div class="group"><h3>Passages repeated across documents</h3>${src.filter((f) => f.repeated_passages.length).map((f) => f.repeated_passages.slice(0, 3).map((r) => `<p class="quote"><b>${esc(f.name)}</b>, in ${r.docs} documents of the sample: ${esc(r.text)}</p>`).join("")).join("")}</div>` : ""}
|
| 209 |
+
<div class="group"><h3>Scores</h3>
|
| 210 |
+
<p class="hint">The model graded ${cal.graded} documents and those keep their grades. The rest are ranked by a scorer learned from so few grades that its ranking is close to a guess${cal.graded >= 100 ? ` (rank agreement with the model on held-out documents: quality ${agree.quality ?? "n/a"}${cal.has_targets ? `, relevance ${agree.relevance ?? "n/a"}` : ""})` : ""}. The full run grades ${C.gradeDocs.toLocaleString()} documents and reports how well the scorer agrees with the model.</p>
|
| 211 |
+
<div class="hists">${src.map((f) => `<div><span>${esc(f.name)}</span>${hist(f.score_hist || [])}<small class="hint">low to high score</small></div>`).join("")}</div>
|
| 212 |
+
${src.map((f) => `<details class="ex"><summary>Best and worst kept in ${esc(f.name)}</summary>${[...(f.examples.top || []).map((e) => ["Best", e]), ...(f.examples.bottom || []).map((e) => ["Worst", e])].map(([l, e]) => `<div class="exd"><b>${l}</b> <span class="hint">score ${e.score}, quality ${e.quality}${cal.has_targets ? `, relevance ${e.relevance}` : ""}${e.reason ? ` / ${esc(e.reason)}` : ""}</span><div class="ptext">${esc(e.text)}</div></div>`).join("")}</details>`).join("")}</div>
|
| 213 |
+
${(rep.created_examples || []).length ? `<div class="group"><h3>Created</h3><p class="hint">${crt.kept} of ${crt.made} rewrites kept.${(crt.flagged || []).length ? ` ${crt.flagged.length} flagged for adding facts the source does not contain.` : ""}</p>
|
| 214 |
+
${rep.created_examples.map((c) => `<details class="ex"><summary>${esc(c.style ? (C.d.styles[c.style] || c.style) : "New document: " + (c.topic || ""))}${c.kept ? "" : ` <span class="flag">${esc(c.why)}</span>`}</summary>
|
| 215 |
+
<div class="pairs two">${c.origin_text ? `<div class="pcard"><div class="ph"><b>Source</b></div><div class="ptext">${esc(c.origin_text)}</div></div>` : ""}<div class="pcard"><div class="ph"><b>Rewrite</b></div><div class="ptext">${esc(c.text)}</div></div></div></details>`).join("")}</div>` : ""}
|
| 216 |
+
<div class="group"><h3>Mixes</h3>${(comp.plans || []).map((p) => `<p style="margin-top:12px"><b>${esc(p.name)}</b>, ${esc(p.budget)} tokens</p>
|
| 217 |
+
<div class="tscroll"><table class="tbl"><thead><tr><th>Source</th><th>Share</th><th>Needs</th><th>Available</th><th>Repeats</th><th>Rephrased</th></tr></thead><tbody>
|
| 218 |
+
${p.rows.map((r) => `<tr><td>${esc(r.name)}</td><td>${pct(r.share)}</td><td>${fmt(r.want_tokens)}</td><td>${fmt(r.available_tokens)}</td><td>${r.epochs}x</td><td>${r.synthetic_tokens ? fmt(r.synthetic_tokens) : "0"}${r.short_tokens ? ` <span class="flag">${fmt(r.short_tokens)} short</span>` : ""}</td></tr>`).join("")}</tbody></table></div>
|
| 219 |
+
${p.synthetic_tokens ? `<p class="hint">About ${p.synthetic_calls.toLocaleString()} model calls to rephrase ${fmt(p.synthetic_tokens)} tokens on your endpoint.</p>` : ""}
|
| 220 |
+
${p.warnings.map((w) => `<p class="note">${esc(w)}</p>`).join("")}`).join("")}</div>
|
| 221 |
+
${(rep.warnings || []).map((w) => `<p class="note">${esc(w)}</p>`).join("")}`;
|
| 222 |
+
}
|
| 223 |
+
function stepRun() {
|
| 224 |
+
const p = C.pv, fin = isFinal(p);
|
| 225 |
+
const stageIdx = p ? ["setup", "clean", "calibrate", "create", "done"].indexOf(p.stage) : -1;
|
| 226 |
+
const out = p && p.report && p.report.output;
|
| 227 |
+
return `<h2>${!p ? "Run it on a sample" : !fin ? "Curating a sample" : p.state === "failed" ? "The run stopped" : "Your sample is curated"}</h2>
|
| 228 |
+
<p class="lede">Every step runs here on about ${C.d ? C.d.per_source : 200} documents from each source. You get the kept and created documents, a report of what each step did, and the recipe to run the same steps on all of your data.</p>
|
| 229 |
+
${C.runErr ? `<p class="note bad">${esc(C.runErr)}</p>` : ""}
|
| 230 |
+
${!p ? `<div class="actions"><button class="btn" data-c="back">Back</button><button class="btn go" data-c="run">Run on a sample</button></div>` : ""}
|
| 231 |
+
${p && !fin ? `<ol class="stages">${["Reading", "Cleaning", "Calibrating", "Creating", "Composing"].map((s, i) => `<li class="${i < stageIdx ? "done" : i === stageIdx ? "on" : ""}">${s}</li>`).join("")}</ol>
|
| 232 |
+
<p class="state">${esc(p.message || "Starting")}. Elapsed ${Math.round(p.elapsed || 0)} s.</p><div class="actions"><button class="btn" data-c="stop">Stop</button></div>` : ""}
|
| 233 |
+
${p && p.state === "failed" ? `<p class="note bad">${esc(p.message)}</p><div class="actions"><button class="btn" data-c="again">Change the recipe</button><button class="btn go" data-c="run">Try again</button></div>` : ""}
|
| 234 |
+
${p && fin && p.report ? `<div class="stats"><div><b>${out.docs}</b><span>documents ready</span></div><div><b>${out.curated}</b><span>kept from sources</span></div><div><b>${out.created}</b><span>created</span></div><div><b>${fmt(out.tokens)}</b><span>tokens, about</span></div></div>
|
| 235 |
+
<div class="actions"><a class="btn go" href="/api/curate/previews/${p.id}/download" style="text-decoration:none">Download the sample (JSONL)</a><button class="btn" data-c="recipe">Download recipe</button><button class="btn" data-c="again">Change the recipe</button></div>
|
| 236 |
+
${runYourself()}${report(p.report)}` : ""}`;
|
| 237 |
+
}
|
| 238 |
+
function runYourself() {
|
| 239 |
+
const inst = C.d ? C.d.install : "pip install datainventor";
|
| 240 |
+
return `<div class="group"><h3>Run it on all of your data</h3>
|
| 241 |
+
<p class="hint" style="max-width:62ch">On your own machines, the same recipe reads every document, grades ${C.gradeDocs.toLocaleString()} with your model, and writes the curated corpus and each mix as JSONL shards with a report. Point it at any OpenAI-compatible server (vLLM, SGLang, Ollama or a hosted API) by editing <code>endpoint</code> in the recipe. Interrupted runs resume where they stopped.</p>
|
| 242 |
+
<pre class="code">${esc(inst)}
|
| 243 |
+
export DI_API_KEY=... # key for the endpoint in the recipe
|
| 244 |
+
datainventor preview recipe.yaml # optional: the hosted check, locally
|
| 245 |
+
datainventor run recipe.yaml --out ./curated --procs 32 --workers 64</pre></div>`;
|
| 246 |
+
}
|
| 247 |
+
|
| 248 |
+
/* ------------------------------------------------------------ render + events */
|
| 249 |
+
function render() {
|
| 250 |
+
if (window.DI_MODE !== "curate") return;
|
| 251 |
+
ledger(); sheet();
|
| 252 |
+
const fn = [stepSources, stepClean, stepCalibrate, stepCreate, stepCompose, stepRun][C.step];
|
| 253 |
+
const keep = document.activeElement && document.activeElement.id;
|
| 254 |
+
$("#panel").innerHTML = fn();
|
| 255 |
+
if (keep && keep.startsWith("c-")) { const el = document.getElementById(keep); if (el) { el.focus(); if (el.setSelectionRange && el.type === "text") { const l = el.value.length; try { el.setSelectionRange(l, l); } catch (e) { /* ok */ } } } }
|
| 256 |
+
}
|
| 257 |
+
function go(i) { C.step = i; C.maxStep = Math.max(C.maxStep, i); render(); $("#panel").focus({ preventScroll: true }); window.scrollTo({ top: 0 }); }
|
| 258 |
+
|
| 259 |
+
async function addDataset(id, role, kind) {
|
| 260 |
+
C.addBusy = true; C.addErr = ""; render();
|
| 261 |
+
try {
|
| 262 |
+
const info = await api(`/api/curate/inspect?dataset=${encodeURIComponent(id)}`);
|
| 263 |
+
const list = role === "target" ? C.targets : C.sources, max = role === "target" ? (C.d ? C.d.max_targets : 2) : (C.d ? C.d.max_sources : 4);
|
| 264 |
+
if (list.length >= max) throw new Error(`The hosted run takes up to ${max} ${role}s. The recipe takes any number.`);
|
| 265 |
+
let name = info.dataset.split("/")[1]; const taken = new Set([...C.sources, ...C.targets].map((s) => s.name));
|
| 266 |
+
for (let k = 2; taken.has(name); k++) name = `${info.dataset.split("/")[1]}-${k}`;
|
| 267 |
+
list.push({ ...info, name, kind: kind || info.kind });
|
| 268 |
+
C.add = ""; C.pv = null;
|
| 269 |
+
} catch (e) { C.addErr = e.message; }
|
| 270 |
+
C.addBusy = false; render();
|
| 271 |
+
}
|
| 272 |
+
async function download() {
|
| 273 |
+
try {
|
| 274 |
+
const r = await fetch("/api/curate/recipe", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ recipe: recipe() }) });
|
| 275 |
+
if (!r.ok) { const d = await r.json().catch(() => ({})); throw new Error(d.detail || "Could not build the recipe"); }
|
| 276 |
+
const url = URL.createObjectURL(await r.blob()), a = document.createElement("a");
|
| 277 |
+
a.href = url; a.download = "recipe.yaml"; document.body.appendChild(a); a.click(); a.remove(); setTimeout(() => URL.revokeObjectURL(url), 2000);
|
| 278 |
+
} catch (e) { C.runErr = e.message; render(); }
|
| 279 |
+
}
|
| 280 |
+
async function run() {
|
| 281 |
+
C.runErr = ""; C.pv = null;
|
| 282 |
+
try { const r = await api("/api/curate/previews", { recipe: recipe() }); C.pid = r.id; C.pv = { id: r.id, state: "queued", stage: "setup", message: "Starting" }; render(); poll(); }
|
| 283 |
+
catch (e) { C.runErr = e.message; render(); }
|
| 284 |
+
}
|
| 285 |
+
async function poll() {
|
| 286 |
+
const id = C.pid; if (!id) return;
|
| 287 |
+
try {
|
| 288 |
+
const p = await api(`/api/curate/previews/${id}`);
|
| 289 |
+
if (id !== C.pid) return;
|
| 290 |
+
C.pv = p; if (C.step === 5) render();
|
| 291 |
+
if (!isFinal(p)) setTimeout(poll, 1500);
|
| 292 |
+
} catch (e) { C.pv = { id, state: "failed", message: e.message }; if (C.step === 5) render(); }
|
| 293 |
+
}
|
| 294 |
+
async function tryIt() {
|
| 295 |
+
C.tryBusy = true; C.tryErr = ""; C.tryRes = null; render();
|
| 296 |
+
const kind = (C.sources.find((s) => s.name === C.tryFrom) || {}).kind || "web";
|
| 297 |
+
try { C.tryRes = await api("/api/curate/try", { text: C.tryText, kind, styles: C.styles.slice(0, 4) }); } catch (e) { C.tryErr = e.message; }
|
| 298 |
+
C.tryBusy = false; render();
|
| 299 |
+
}
|
| 300 |
+
|
| 301 |
+
document.addEventListener("click", async (ev) => {
|
| 302 |
+
if (window.DI_MODE !== "curate") return;
|
| 303 |
+
const t = ev.target.closest("button, a"); if (!t) return;
|
| 304 |
+
if (t.dataset.cgo !== undefined) return go(+t.dataset.cgo);
|
| 305 |
+
if (t.dataset.bench) { const b = t.dataset.bench; C.decon = C.decon.includes(b) ? C.decon.filter((x) => x !== b) : [...C.decon, b]; return render(); }
|
| 306 |
+
if (t.dataset.style) { const s = t.dataset.style; C.styles = C.styles.includes(s) ? C.styles.filter((x) => x !== s) : [...C.styles, s]; return render(); }
|
| 307 |
+
const a = t.dataset.c; if (!a) return;
|
| 308 |
+
if (a === "next") return go(C.step + 1);
|
| 309 |
+
if (a === "back") return go(Math.max(0, C.step - 1));
|
| 310 |
+
if (a === "add") return addDataset(C.add.trim(), C.addRole);
|
| 311 |
+
if (a === "preset") { for (const p of PRESET) await addDataset(p.dataset, "source", p.kind); if (!C.targetsDesc) C.targetsDesc = ""; return render(); }
|
| 312 |
+
if (a === "rm") { (t.dataset.role === "target" ? C.targets : C.sources).splice(+t.dataset.i, 1); C.pv = null; return render(); }
|
| 313 |
+
if (a === "near") { C.near = !C.near; return render(); }
|
| 314 |
+
if (a === "reph") { C.rephraseOn = !C.rephraseOn; return render(); }
|
| 315 |
+
if (a === "inv") { C.inventOn = !C.inventOn; return render(); }
|
| 316 |
+
if (a === "tryfill") { const s = C.sources.find((x) => x.preview); if (s) { C.tryText = s.preview; C.tryFrom = s.name; } return render(); }
|
| 317 |
+
if (a === "try") return tryIt();
|
| 318 |
+
if (a === "budget") { C.mixes[+t.dataset.m].budget = +t.dataset.v; return render(); }
|
| 319 |
+
if (a === "addmix") { C.mixes.push({ name: `mix-${C.mixes.length + 1}`, budget: 1e9, weights: {} }); return render(); }
|
| 320 |
+
if (a === "rmmix") { C.mixes.splice(+t.dataset.m, 1); return render(); }
|
| 321 |
+
if (a === "recipe") return download();
|
| 322 |
+
if (a === "run") return run();
|
| 323 |
+
if (a === "stop") { if (C.pid) await api(`/api/curate/previews/${C.pid}/stop`, {}); return; }
|
| 324 |
+
if (a === "again") { C.pv = null; C.pid = null; return go(0); }
|
| 325 |
+
});
|
| 326 |
+
document.addEventListener("input", (ev) => {
|
| 327 |
+
if (window.DI_MODE !== "curate") return;
|
| 328 |
+
const el = ev.target, id = el.id || ""; if (!id.startsWith("c-")) return;
|
| 329 |
+
const v = el.value;
|
| 330 |
+
if (id === "c-add") { C.add = v; const b = $('[data-c="add"]'); if (b) b.disabled = C.addBusy || !v.trim(); return; }
|
| 331 |
+
if (id === "c-tdesc") { C.targetsDesc = v; return sheet(); }
|
| 332 |
+
if (id === "c-try") { C.tryText = v; C.tryFrom = ""; const b = $('[data-c="try"]'); if (b) b.disabled = C.tryBusy || v.trim().split(/\s+/).length < 60; return; }
|
| 333 |
+
if (id === "c-invreq") { C.inventReq = v; return; }
|
| 334 |
+
const sv = (x) => { const o = document.getElementById(id + "-v"); if (o) o.textContent = x; };
|
| 335 |
+
if (id === "c-focus") { C.focus = +v; sv(+v === 0 ? "general quality only" : `${pct(1 - v)} quality / ${pct(v)} targets`); return sheet(); }
|
| 336 |
+
if (id === "c-keep") { C.keepTop = +v; sv(`${pct(v)} of each source`); return sheet(); }
|
| 337 |
+
if (id === "c-epochs") { C.maxEpochs = +v; sv(`${v} times`); return; }
|
| 338 |
+
if (id === "c-synth") { C.maxSynth = +v; sv(`${pct(v)} of a mix`); return; }
|
| 339 |
+
});
|
| 340 |
+
document.addEventListener("change", (ev) => {
|
| 341 |
+
if (window.DI_MODE !== "curate") return;
|
| 342 |
+
const el = ev.target, id = el.id || ""; if (!id.startsWith("c-")) return;
|
| 343 |
+
const v = el.value; let m;
|
| 344 |
+
if (id === "c-addrole") C.addRole = v;
|
| 345 |
+
else if ((m = id.match(/^c-kind-(\d+)$/))) { C.sources[+m[1]].kind = v; C.pv = null; }
|
| 346 |
+
else if ((m = id.match(/^c-col-(source|target)-(\d+)$/))) { (m[1] === "target" ? C.targets : C.sources)[+m[2]].text_field = v; C.pv = null; }
|
| 347 |
+
else if ((m = id.match(/^c-rule-(.+)$/))) { if (v !== "") C.rules[m[1]] = +v; }
|
| 348 |
+
else if (id === "c-grade") C.gradeDocs = Math.max(100, +v || 2000);
|
| 349 |
+
else if (id === "c-scorer") C.scorer = v;
|
| 350 |
+
else if (id === "c-invdocs") C.inventDocs = Math.max(0, +v || 0);
|
| 351 |
+
else if ((m = id.match(/^c-mname-(\d+)$/))) C.mixes[+m[1]].name = v.trim().replace(/[^\w.-]+/g, "-") || `mix-${+m[1] + 1}`;
|
| 352 |
+
else if ((m = id.match(/^c-mbud-(\d+)$/))) C.mixes[+m[1]].budget = Math.max(1e6, Math.round(+v || 0));
|
| 353 |
+
else if ((m = id.match(/^c-w-(\d+)-(.+)$/))) { const mix = C.mixes[+m[1]]; mix.weights = { ...weightsOf(mix), [m[2]]: Math.max(0, +v || 0) / 100 }; }
|
| 354 |
+
else return;
|
| 355 |
+
render();
|
| 356 |
+
});
|
| 357 |
+
document.addEventListener("keydown", (ev) => { if (window.DI_MODE === "curate" && ev.key === "Enter" && ev.target.id === "c-add" && C.add.trim()) addDataset(C.add.trim(), C.addRole); });
|
| 358 |
+
|
| 359 |
+
/* ------------------------------------------------------------ mode switch */
|
| 360 |
+
function setMode(mode) {
|
| 361 |
+
window.DI_MODE = mode;
|
| 362 |
+
document.querySelectorAll("[data-mode]").forEach((b) => b.setAttribute("aria-pressed", String(b.dataset.mode === mode)));
|
| 363 |
+
$(".tag").textContent = mode === "curate" ? "Clean, score, rephrase and mix the pretraining data you already have. Preview it here on a sample, then run the recipe on your own machines." : "Describe the data you wish existed. Get training rows back. Bring any OpenAI-compatible model.";
|
| 364 |
+
if (mode === "curate") { if (!C.d) api("/api/curate/defaults").then((d) => { C.d = d; render(); }).catch(() => render()); render(); }
|
| 365 |
+
else if (window.DI_renderInvent) window.DI_renderInvent();
|
| 366 |
+
if ((location.hash === "#curate") !== (mode === "curate")) history.replaceState(null, "", mode === "curate" ? "#curate" : location.pathname + location.search);
|
| 367 |
+
}
|
| 368 |
+
document.addEventListener("click", (ev) => { const b = ev.target.closest("[data-mode]"); if (b) setMode(b.dataset.mode); });
|
| 369 |
+
window.DI_MODE = location.hash === "#curate" ? "curate" : "invent";
|
| 370 |
+
if (window.DI_MODE === "curate") document.addEventListener("DOMContentLoaded", () => setMode("curate"));
|
| 371 |
+
})();
|
static/index.html
CHANGED
|
@@ -4,7 +4,7 @@
|
|
| 4 |
<meta charset="utf-8">
|
| 5 |
<meta name="viewport" content="width=device-width, initial-scale=1">
|
| 6 |
<title>DataInventor</title>
|
| 7 |
-
<meta name="description" content="Describe a dataset in plain words and get training data back. Open source, works with any OpenAI-compatible model.">
|
| 8 |
<link rel="preconnect" href="https://fonts.googleapis.com">
|
| 9 |
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
| 10 |
<link href="https://fonts.googleapis.com/css2?family=Familjen+Grotesk:ital,wght@0,400..700;1,400..700&family=Literata:ital,opsz,wght@0,7..72,400..600;1,7..72,400..600&display=swap" rel="stylesheet">
|
|
@@ -17,6 +17,7 @@
|
|
| 17 |
<span class="mark" aria-hidden="true"><i></i><i></i></span>
|
| 18 |
<span class="name">DataInventor</span>
|
| 19 |
</div>
|
|
|
|
| 20 |
<p class="tag">Describe the data you wish existed. Get training rows back. Bring any OpenAI-compatible model.</p>
|
| 21 |
</header>
|
| 22 |
|
|
@@ -29,6 +30,7 @@
|
|
| 29 |
<footer class="foot">
|
| 30 |
<span>Open source recreation of the Invent a Dataset workflow. Not affiliated with Adaption Labs.</span>
|
| 31 |
</footer>
|
|
|
|
| 32 |
<script src="/static/app.js"></script>
|
| 33 |
</body>
|
| 34 |
</html>
|
|
|
|
| 4 |
<meta charset="utf-8">
|
| 5 |
<meta name="viewport" content="width=device-width, initial-scale=1">
|
| 6 |
<title>DataInventor</title>
|
| 7 |
+
<meta name="description" content="Describe a dataset in plain words and get training data back, or clean, score, rephrase and mix the pretraining data you have. Open source, works with any OpenAI-compatible model.">
|
| 8 |
<link rel="preconnect" href="https://fonts.googleapis.com">
|
| 9 |
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
| 10 |
<link href="https://fonts.googleapis.com/css2?family=Familjen+Grotesk:ital,wght@0,400..700;1,400..700&family=Literata:ital,opsz,wght@0,7..72,400..600;1,7..72,400..600&display=swap" rel="stylesheet">
|
|
|
|
| 17 |
<span class="mark" aria-hidden="true"><i></i><i></i></span>
|
| 18 |
<span class="name">DataInventor</span>
|
| 19 |
</div>
|
| 20 |
+
<nav class="modes" aria-label="Mode"><button data-mode="invent" aria-pressed="true">Invent rows</button><button data-mode="curate" aria-pressed="false">Curate a corpus</button></nav>
|
| 21 |
<p class="tag">Describe the data you wish existed. Get training rows back. Bring any OpenAI-compatible model.</p>
|
| 22 |
</header>
|
| 23 |
|
|
|
|
| 30 |
<footer class="foot">
|
| 31 |
<span>Open source recreation of the Invent a Dataset workflow. Not affiliated with Adaption Labs.</span>
|
| 32 |
</footer>
|
| 33 |
+
<script src="/static/curate.js"></script>
|
| 34 |
<script src="/static/app.js"></script>
|
| 35 |
</body>
|
| 36 |
</html>
|