Bring the card to the preprint, and ship the scoring head as safetensors

The wording bench now reports both candidate-set sizes and marks which rows are
controls. The Russian word and the English key are the wordings both releases
trained on, so those rows measure recall of a trained form and their near-zero
gap is what a control should show; the Russian phrase is a training style for
v1.3 only, so its gap is expected by construction. Only the held-out rows speak
to generalisation, and the reading they support is narrower than before:
training a label in several surface forms helps on unfamiliar wordings made of
words, and does nothing for abbreviations or opaque codes, where the label
string carries little of its meaning.

Confidence no longer follows accuracy in the summary either. With eighteen
candidates the count of confident errors rises on jargon and on slang, because
the augmented release is more confident overall, so a deployment that invents
its own label vocabulary is pointed at fit_temperature.py instead.

The calibration figures on the comparable rows are recomputed with a recorded
split seed. The previous implementation shuffled its index list in place without
resetting it between splits, so the numbers depended on the order of rows in the
file. The corrected figures are 2.0054 against 1.5593 for the fitted temperature
and 0.0190 against 0.0191 for the ECE that remains, favouring v1.3 in 14 of 25
splits rather than 12. Nothing turns over.

Latency is measured per decision at two, five and eighteen candidates against an
open comparison model on the same machine, replacing three figures that did not
say how many candidates they covered.

head.safetensors is added in this same commit because the usage snippet now
reads it. head.pt stays for older code.

The demo link is relabelled: its probabilities are precomputed, and a live CPU
space would need a paid plan.

README.md

@@ -31,7 +31,7 @@ options, and ordinal scale. It is a compact encoder (mmBERT-base, 307M), so it
 runs on CPU and can be deployed inside a closed perimeter.
 
 - Project page: https://ru-jev.besk.tech
-- Interactive demo: https://huggingface.co/spaces/besktech/ru-jev-demo
+- Demo with precomputed examples: https://huggingface.co/spaces/besktech/ru-jev-demo
 - Training and evaluation code: https://github.com/beskvladimir-create/ru-jev
 
 Earlier releases are kept on branches: `v1.1` and `v1`. Load one with
@@ -65,30 +65,79 @@ so the model has to read an option instead of recognising it. For the routing
 tasks that includes the English key written in Cyrillic and a clipped form of
 it, generated by rule.
 
-| Option wording | v1.1 accuracy | v1.1 mean confidence | v1.3 accuracy | v1.3 mean confidence |
-| --- | --- | --- | --- | --- |
-| Russian word, `погода` | 0.956 | 0.965 | 0.960 | 0.973 |
-| English key, `weather` | 0.957 | 0.968 | 0.962 | 0.974 |
-| Descriptive sentence | 0.939 | 0.930 | 0.958 | 0.961 |
-| Russian phrase, `прогноз погоды` | 0.881 | 0.896 | 0.957 | 0.972 |
-| Department name, `метеослужба` | 0.835 | 0.865 | 0.869 | 0.922 |
-| Jargon, `вэзер` | 0.633 | 0.754 | 0.855 | 0.885 |
-
-A thousand routing questions per row, the same eighteen categories every time,
-only the wording differs. Every row in this table covers all eighteen
-categories, for both versions.
-
-One caveat on the jargon row, because it is the headline number. An intent key
-begins with its scenario name, so three of the eighteen jargon labels coincide
-with what the transliteration rule produces and reach training as a first word.
-Restricted to the fifteen that do not coincide, which is the honest measure of
-generalisation to wording the model has never seen, the figures are 0.624
-for v1.1 and 0.838 for v1.3. That strict pair is the one this project
-quotes for the jargon result. The all-eighteen pair, 0.633 against
-0.855, is the full test and is what the table above shows. The two pairs
-measure different label sets, so a figure from one never belongs beside a figure
-from the other. The rules were not tuned against this test
-vocabulary; `scripts/robustness.py` in the code repository reports both figures.
+The same 1,000 routing questions were then asked over and over with the label
+wording varied and nothing else. Two candidate-set sizes are reported, because
+the gap depends on the size in different ways for different vocabularies and a
+probe at one size alone can misstate the effect: five candidates drawn from the
+eighteen categories, which is what earlier versions of this card reported, and
+all eighteen candidates at once. Chance is 0.20 and 0.056.
+
+| Option wording, example for `weather` | Status | v1.1, 5 | v1.3, 5 | v1.1, 18 | v1.3, 18 |
+| --- | --- | --- | --- | --- | --- |
+| Russian word, `погода` | training style, both releases | 0.956 | 0.960 | 0.883 | 0.892 |
+| English key, `weather` | training style, both releases | 0.957 | 0.962 | 0.894 | 0.900 |
+| Russian phrase, `прогноз погоды` | training style, v1.3 only | 0.881 | 0.957 | 0.703 | 0.885 |
+| Descriptive sentence | held out | 0.939 | 0.958 | 0.848 | 0.878 |
+| Department name, `метеослужба` | held out | 0.835 | 0.869 | 0.632 | 0.733 |
+| Anglicism jargon, `вэзер`, all 18 labels | held out | 0.633 | 0.855 | 0.431 | 0.640 |
+| Anglicism jargon, 15 clean labels | held out | 0.624 | 0.838 | 0.427 | 0.595 |
+| Anglicism jargon, 11 strict labels | held out | 0.550 | 0.786 | 0.328 | 0.470 |
+| Office slang, `синоптик`, all 18 | held out | 0.648 | 0.780 | 0.428 | 0.476 |
+| Office slang, 14 strict labels | held out | 0.595 | 0.761 | 0.369 | 0.419 |
+| Abbreviation, `ПОГ`, 17 clean labels | held out | 0.420 | 0.360 | not scored | |
+| Opaque code, `ПГД`, 9 strict labels | held out | 0.135 | 0.168 | 0.018 | 0.061 |
+| Code identifier, `WEATHER_SCN` | contains the trained key | 0.856 | 0.889 | not scored | |
+
+**The first two rows are controls, not results.** The Russian word and the
+English key are the wordings both releases trained on: the bench vocabulary for
+them is the training style itself, taken from the same table in the data
+builder. Those two rows measure recall of a trained form, and the gap near zero
+is what a control should show. The Russian phrase is a training style for v1.3
+and not for v1.1, so its gap is expected by construction and is not evidence of
+robustness either. Only the rows marked held out speak to generalisation.
+
+Two caveats on the held-out rows. An intent key begins with its scenario name,
+so three of the eighteen jargon labels coincide with what the transliteration
+rule produces and reach training as a first word; the clean slice drops those
+three and the strict slice drops every label with any exact or prefix relation
+to a training option. The rules were not tuned against this test vocabulary, and
+the vocabulary is the older of the two: it was committed six hours and
+thirty-four minutes before the rules. Figures from different slices measure
+different label sets, so one never belongs beside another.
+
+**What the augmentation helps with.** Training a label in several surface forms
+helps on held-out wordings made of words, ordinary or borrowed, including kinds
+of words that were never in training at all: jargon gains +0.235 with five
+candidates and +0.142 with eighteen on the strict slice, office slang +0.166 and
++0.050, department names +0.034 and +0.101, descriptive sentences +0.019 and
++0.030. The direction is the same in all fourteen word-vocabulary cells.
+
+**What it does not help with.** On three-letter abbreviations v1.3 is not better:
+-0.024 on all labels and -0.060 on the clean slice. Opaque codes are unreadable
+for both releases, with v1.1 below chance and v1.3 at chance on the strict slice
+with eighteen candidates. Code identifiers such as `WEATHER_SCN` contain the
+trained key and are easy for both, so they are not treated as a test of
+robustness. Where the label string carries little of its meaning, several
+surface forms in training buy nothing.
+
+**Confidence does not follow accuracy here.** With five candidates the count of
+confident errors falls on jargon, from 110 to 62 per 1,000 on the strict slice.
+With eighteen it rises, on jargon from 79 to 109 on all labels and on slang from
+151 to 191 on the strict slice, because the augmented release is more confident
+overall and more of its errors cross the 0.8 threshold. On abbreviations it
+rises from 77 to 118. The augmentation does not by itself make confidence safer
+on unfamiliar wording. A deployment that invents its own label vocabulary should
+refit the temperature on its own decisions; `scripts/fit_temperature.py` in the
+code repository does that on a few hundred rows and reports whether there were
+enough of them.
+
+Significance is exact McNemar on the paired questions with a Holm correction
+over the 23 held-out cells, and the interval resamples labels before questions.
+The five-candidate columns for slang and opaque codes are obtained by
+renormalising the eighteen-candidate scores over the same five candidates the
+main bench uses, which is exact for Ru-Jev because every candidate is scored in
+its own pass. Raw counts, per-label tables and the analysis script are in the
+code repository under `results/analysis/`.
 
 ## Results
 
@@ -153,12 +202,14 @@ task can no longer move the split in another.
 | --- | --- | --- |
 | Correct answers | 8,120 | 8,120 |
 | Accuracy (micro) | 0.7994 | 0.7994 |
-| Fitted temperature on these rows | 1.995 | 1.541 |
-| ECE after each model's own temperature | 0.0185 | 0.0188 |
+| Fitted temperature on these rows | 2.0054 | 1.5593 |
+| ECE after each model's own temperature | 0.0190 | 0.0191 |
 
 Accuracy does not depend on the temperature, so it is reported on all 10,157 decisions rather than on a split half. The two releases land on the same count: 8,120 correct out of 10,157 each. They do not get the same rows right. They disagree on 1,680 of them, 840 where only v1.1 is right and 840 where only v1.3 is. One standard error on the difference is 0.0056, and the difference is +0.0000.
 
-The temperature and ECE rows are the mean over 25 random half-splits, fitting on one half and scoring the other. v1.3 needs a lower temperature in 25 of 25 splits, with the two ranges not overlapping (1.92 to 2.10 against 1.47 to 1.60), so its raw scores are genuinely less overconfident. The ECE that remains after each model gets its own temperature is the same for both: the difference averages +0.0004 and runs from -0.0146 to +0.0096, favouring v1.3 in 12 of 25 splits. An earlier version of this card reported 0.0172 against 0.0160 from a single fixed split and called v1.3 better calibrated here. On a different split the sign flips, so that claim is withdrawn.
+The temperature and ECE rows are the mean over 25 random half-splits, fitting on one half and scoring the other. v1.3 needs a lower temperature in 25 of 25 splits, with the two ranges not overlapping (1.942 to 2.098 against 1.500 to 1.604), so its raw scores are genuinely less overconfident. The ECE that remains after each model gets its own temperature is the same for both: the difference averages +0.0001 and runs from -0.0063 to +0.0109, favouring v1.3 in 14 of 25 splits. An earlier version of this card reported 0.0172 against 0.0160 from a single fixed split and called v1.3 better calibrated here. On a different split the sign flips, so that claim is withdrawn.
+
+The split seed is recorded as `SPLIT_SEED = 20261105` in `scripts/calib.py`. An earlier implementation shuffled its index list in place without resetting it between splits, so each split was applied to the permutation left by the previous one and the result depended on the order of rows in the file. These figures come from the corrected code; the shift it caused is in the third decimal and turns over nothing.
 
 | Task | v1.1 | v1.3 | n |
 | --- | --- | --- | --- |
@@ -287,19 +338,29 @@ disturb the order, so nothing here should be assumed to carry over to it.
 
 ## Speed on CPU
 
-Apple M4, 4 threads, float32, median of 25 runs:
+Apple M4, float32, median of 25 runs after 5 warm-up runs, nothing else running
+on the machine. Latency is the whole decision: tokenisation, the forward passes
+and the softmax.
 
-| Decision | Latency |
-| --- | --- |
-| Short yes/no | 25 ms |
-| Choice from three options | 31 ms |
-| Full 256-token context | 131 ms |
+| Candidates | Ru-Jev v1.3 | Laya | Ratio |
+| --- | --- | --- | --- |
+| 2 | 18.5 ms | 15.5 ms | 1.19 |
+| 5 | 32.8 ms | 17.8 ms | 1.84 |
+| 18 | 73.8 ms | 30.5 ms | 2.42 |
+
+Ru-Jev scores each (context, option) pair in its own forward pass, so the cost
+of a decision grows linearly with the number of candidates, by roughly 4 ms per
+candidate on this machine. That is also why option order is not an input to it.
+Laya reads all options in one pass, so its cost is almost flat in the number of
+candidates and order is an input. Ru-Jev is slower in both settings and no
+speed advantage is claimed here.
 
 ## Usage
 
 ```python
 import json, torch, torch.nn as nn
 from huggingface_hub import hf_hub_download
+from safetensors.torch import load_file
 from transformers import AutoModel, AutoTokenizer
 
 REPO = "besktech/ru-jev"
@@ -311,7 +372,7 @@ class Scorer(nn.Module):
         self.backbone = AutoModel.from_pretrained(repo)
         self.head = nn.Linear(self.backbone.config.hidden_size, 1)
         self.head.load_state_dict(
-            torch.load(hf_hub_download(repo, "head.pt"), map_location="cpu"))
+            load_file(hf_hub_download(repo, "head.safetensors")))
 
     def forward(self, ids, mask):
         cls = self.backbone(input_ids=ids, attention_mask=mask).last_hidden_state[:, 0]
@@ -353,7 +414,9 @@ That check exists because an earlier run silently trained on eleven.
 
 ## License
 
-MIT, weights and code.
+MIT covers the code and the weights. The training data keeps the licences of
+its sources, which are listed task by task in `DATASETS.md` in the code
+repository.
 
 Vladimir Beskorovainyi, Enterprise AI Architect & CTO, author of the Amazon
 bestseller "Enterprise AI: The KATA Way", founder of Besk Tech (besk.tech)

head.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:724740455841c6c4c14ff53489b8c9e3956952d2cbf32ab1884b1f0855508bcf
+size 3244