5ak3t/scm-mid-4b-v1

SCM-mid 4B v1 (experimental research release)

SCM ("Sovereign Classification Model") makes typed decisions in one forward pass: choice (pick one of K), noul (yes/no) and score (ordinal level) questions about a text. This repo holds a LoRA adapter only for Qwen/Qwen3-4B-Base at revision 906bfd4b4dc7f14ee4320094d8b41684abff8539 (training run s88-v1mid-H100-20261015-113727; lr 1e-4 (fp16, 1 GPU), LoRA r=16 / alpha=32 on all attention and MLP projections, 16 questions per step, one epoch, prompts left-truncated at 3,072 tokens). It is not a production model.

How it scores (joint-option format)

All options of one question appear in one prompt as labelled lines; the score of an option is the next-token logit of its label. Labels are A to T, then U to Z, then single-token two-letter labels (AA, AB, ...), so questions with up to 255 options work (v0 handled at most 20). Four option orders are scored and averaged. l87_joint_prompt.py in this repo is the exact prompt and label rule. No calibration or temperature is applied.

Results

S1MB English v1 (137 benchmarks, all complete, 0 failed decisions), baseline-adjusted scores, own measurement with the evaluator's SCM adapter (renderer scm-joint-letters-v2), submitted as a results pull request: Task Avg 30.15 (noul 45.92, choice 38.22, score 6.30); Borda 46.06; #94 of 165 in the complete published cohort at the time of measurement (our own computation from the saved results; the shared leaderboard may order differently). no S1MB measurement exists for the v0 4B model.

Important: this is not a zero-shot result. v1 was trained on the train splits of many datasets that S1MB benchmarks are built from (79 same_source_family disclosures in the results submission, rebuilt from the final retained training inventory). Training items whose input matched an S1MB test input exactly or by 8-gram overlap were removed (4,524 dropped), but a record-level audit of the final pool still finds 304 of 15,609 English test cases with an approximate match (mostly templated generator families: snake, spartqa, ir-control, typed-decisions, browser-control, silent-failure-control, sponsor-segment). These are approximate-match candidates, not confirmed use of evaluation inputs. Compare with models that did not train on these source families with care.

Release gate (fixed before any v1 score; open data, not S1MB): held-out accuracy 0.6945 (v0) to 0.6990 (v1), difference +0.0045, 95% CI [-0.0091, +0.0174] (not distinguishable); NLL 0.707 to 0.774 (worse). These held-out groups come from workflow families v1 trained on (other groups), so the gate favours v1 and only guards against a broken run; it does not measure generalisation to unseen workflows.

Training data

Pool v6 lite, 50,464 records / 56,280 questions (choice 40,596, noul 11,870, score 3,814; 10.3% of questions have 21 to 160 options): a seeded sample of 26,991 records from the v0 pool (v5), 23,473 new records: the train splits of 65 families of the Bekko System One dataset (revision 9c673a09, capped per family) plus three extra wide-label sets (trec_fine, ledgar, dbpedia level 2). The full recipe, per-source counts, licences as recorded by the builders and decontamination counts are in pool-v6-lite-manifest.json. Sources labelled by an LLM (gretel_pii, laya typed-decisions, LLM-jury aegis2 labels) were excluded from the new data. The sampled v0 part still contains typed-decisions and wanli records (as v0 did), and the v0 pool's 780 support tickets were written by an LLM from deterministic specs. No label was generated by an LLM for this training run. No raw training records are in this repo. Licences are recorded, not gated: some sources are non-commercial or share-alike; check them before commercial use.

Limitations

• Accuracy on workflows unlike the training data is far lower than on in-distribution suites; probabilities are the model's softmax with no calibration or coverage guarantee, and Score-type answers are weak (see the Score task above).
• Yes/no and choice errors are workflow-specific (a model can over-answer "yes" in one workflow and the opposite in another); test on your own data first.
• The measured revision of this repo is the first upload; the model card was written afterwards (card-only change, the adapter weights are identical).
• Adapter only: the base weights follow the Qwen3 licence (Apache-2.0).

Source and documents: https://github.com/5ak3t/scm