curator-teacher2-v2: the curation classifier (E030)

curator-teacher2-v2/README.md

@@ -0,0 +1,52 @@
+---
+license: mit
+datasets:
+- ks46/usernames
+tags:
+- username
+- byte-level
+- classifier
+- curation
+---
+
+# curator-teacher2-v2
+
+The curation classifier behind the `ks46/usernames` corpus: for a username it
+predicts a signup-form quality score `q` (1–5, regression) and an offensiveness
+level `o` (0–4, softmax). `prep` keeps or drops every name of the raw corpus on
+these two numbers; the published corpus is what survived.
+
+Files:
+
+- `curator.pt`: the byte GPT `teacher2-16x1024` (16 layers × 1024, 205.9M
+  parameters) turned into an encoder with two small heads, fine-tuned end to
+  end for 3 epochs, batch 512, lr 1e-4, on 1,676,379 judge labels.
+- `ngram.pt`: the same heads on a hashed byte n-gram bag, trained as the
+  baseline (10 epochs).
+- `report.json`: the training setup and validation metrics below.
+
+Labels are `training.judge --label` output from an LLM judge (deepseek-flash,
+batch 20 with anchors): 998,963 names sampled by hash stride from the
+uncurated corpus plus two targeted draws for offensiveness (word-list hits and
+substrings), 1,863,434 rows in all; they are `labels/judge-*.parquet` in
+`ks46/usernames`. The validation split is `xxh3_64(name) % 10 == 0`, 187,055
+names.
+
+| model | q MAE | q exact | q within 1 | Spearman | o acc | o≥2 P/R | o≥3 P/R |
+|---|---|---|---|---|---|---|---|
+| `curator.pt` | 0.355 | 0.726 | 0.996 | 0.820 | 0.911 | 0.84/0.84 | 0.75/0.76 |
+| `ngram.pt` | 0.473 | 0.606 | 0.981 | 0.685 | 0.872 | 0.72/0.76 | 0.56/0.67 |
+
+The judge's own self-agreement on relabelled names is at this level, so the
+curator reproduces the judge rather than smoothing it. Scoring the whole
+uncurated corpus (197.1M names) took 371 s on four RTX 5090s; the keep rule
+(a ramp on `q` with `o` capped) kept 149,142,110 names, 75.7 %.
+
+Usage, from the `name-dreamer` training package:
+
+```
+uv run python -m training.curate score --curator curator.pt --names names.parquet --out scores/
+uv run python -m training.prep ... --curated scores/ --min-q 3.5 --max-o 1.0
+```
+
+Not a generator: sampling needs one of the sibling folders in this repo.

curator-teacher2-v2/curator.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:241ec7b35fb65da3623dc5ffc39c2e3a6f39c97d27a8a90085f54acce3bdaaf2
+size 825681485

curator-teacher2-v2/ngram.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05313861c914d140ef594b770e4858b30795a542bb395b33b9961a7c7d89a78b
+size 67185015

curator-teacher2-v2/report.json

@@ -0,0 +1,39 @@
+{
+ "labels": "/root/data/name-dreamer/labels-all.csv",
+ "n_train": 1676379,
+ "n_val": 187055,
+ "epochs": 3,
+ "ngram_epochs": 10,
+ "batch": 512,
+ "ngram": {
+  "n": 187055,
+  "q_mae": 0.47344785928726196,
+  "q_exact": 0.6056186683061132,
+  "q_within1": 0.9805351367244928,
+  "q_spearman": 0.684556585799076,
+  "o_acc": 0.8720643660955334,
+  "o2_precision": 0.7186619350798455,
+  "o2_recall": 0.7624716239410886,
+  "o2_support": 18061,
+  "o3_precision": 0.5552031230938148,
+  "o3_recall": 0.6746219982211681,
+  "o3_support": 6746,
+  "epoch": 10
+ },
+ "gpt": {
+  "n": 187055,
+  "q_mae": 0.3546030819416046,
+  "q_exact": 0.7256528828419448,
+  "q_within1": 0.9962257090160648,
+  "q_spearman": 0.8201309928486379,
+  "o_acc": 0.9109566704979819,
+  "o2_precision": 0.8402582353914915,
+  "o2_recall": 0.8431426831293948,
+  "o2_support": 18061,
+  "o3_precision": 0.7457205559619605,
+  "o3_recall": 0.7555588496887045,
+  "o3_support": 6746,
+  "epoch": 3,
+  "ckpt": "/root/models/name-dreamer/teacher2-16x1024/ckpt.pt"
+ }
+}