Datasets — sources, splits, licenses
Every state string in this repo is verbatim dataset text. The only edits are
truncation (long fields capped) and label remapping to opaque anchors.
MANIFEST.json carries per-file SHA-256 and per-domain row counts; the
preparation recipe (and the three failure-mode fixes it bakes in) is in
MANIFEST.json -> recipe.
Included corpora (16 domains)
| domain | source | train | cal | hidden | notes |
|---|---|---|---|---|---|
| tickets_en | repo synthetic support corpus (data/big, SHA-256 manifest) | 8,400 | 1,800 | 1,800 | choice, 3 departments |
| tickets_bn/hi/ru | repo synthetic multilingual support corpus | 2,520-2,522 | 539-540 | 539-540 | bengali/devanagari/cyrillic substrates |
| game | repo synthetic game-command corpus | 5,600 | 1,200 | 1,200 | choice, 3 actions |
| guard | repo synthetic guardrail corpus | 5,600 | 1,200 | 1,200 | noul (irreversible, off_task) + choice (scope); original schema kept verbatim |
| bank77 | PolyAI banking77 official splits | 9,000 | 1,003 | 3,080 | 77 intents, official test = hidden |
| sms | UCI SMS Spam Collection | 3,900 | 836 | 838 | free-for-research corpus |
| clinc | clinc/clinc_oos (plus config) | 10,000 | 3,000 | 4,500 | 151 intents incl. out-of-scope |
| enron | SetFit/enron_spam | 6,000 | 758 | 1,113 | cal/hidden carved from official test, disjoint |
| emotion | dair-ai/emotion | 6,000 | 700 | 2,000 | 6 emotions |
| hate | tdavidson/hate_speech_offensive | 4,078 | 400 | 1,500 | official train only -> stratified carve (seed 77); offensive class capped at 2,500 in train, natural distribution in eval |
| agnews | fancyzhx/ag_news | 6,000 | 400 | 2,000 | 4 topics; cal carved from train |
| polarity | fancyzhx/amazon_polarity | 6,000 | 400 | 2,000 | 2 classes; cal carved from train |
| dbpedia | fancyzhx/dbpedia_14 | 6,000 | 400 | 2,000 | 14 entity classes |
| boolq | google/boolq | 6,000 | 500 | 1,500 | question+passage, boolean choice; cal+hidden carved from official validation |
Total: 76,142 train rows (152,284 lessons per epoch; the shipped model was trained ONE interleaved pass — see docs/EVALUATION.md §Honest reading for the consolidation analysis).
Split discipline
- Hidden files carry
_hiddenin their names; the M1 firewall refuses them forlearnandcalibrate(hidden rows never teach and never set calibration scalars). - cal and hidden are disjoint per domain (enforced in
tools/giant_prepare.py); cal feeds per-script temperature/Platt fitting only. - Model/flag selection stays on cal; hidden is scored once for the report.
File format (eval + cal)
The shipped eval/*_hidden.jsonl and cal_*.jsonl rows are SCHEMA-FREE:
{"state", "domain", "labels"} (guard/noul rows also carry their
questions inline, since noul instructions are the probe carrier). The
question objects (router + class criteria) are rebuilt from
schemas.json — see build_questions() in the main repo's
tools/giant_bench.py. This keeps the repo small; the full-schema corpus
regenerates via tools/giant_prepare.py in the main SyFox repo.
Licenses
- SyFox engine: MIT (see LICENSE).
- bank77: released by PolyAI for research (check the repository for the current license text).
- UCI SMS Spam Collection: free for research with attribution.
- HuggingFace datasets (clinc_oos, enron_spam, emotion, hate_speech_offensive, ag_news, amazon_polarity, dbpedia_14, boolq): each retains its original publisher license — consult the dataset page before redistributing or commercial use.
- The repo synthetic corpora are original generated data (MIT, same as the engine).