ru-jev / README.md
add the preprint link and a citation block
a86ed31 verified about 24 hours ago
25.3 kB
---
license: mit
language:
  - ru
  - en
  - multilingual
pipeline_tag: text-classification
tags:
  - text-classification
  - zero-shot-classification
  - typed-decision
  - russian
  - routing
  - intent-classification
  - calibrated
  - calibration
  - on-prem
  - cpu
  - mmbert
library_name: transformers
base_model: jhu-clsp/mmBERT-base
inference: false
---
# Ру-Жев (Ru-Jev) v1.3

**Do not use the standard zero-shot pipeline.** The call
`pipeline("zero-shot-classification", model="besktech/ru-jev")` neither fails nor
works: it returns numbers drawn from a randomly initialized head, and they change
on every load. Load the model with the snippet below.

Open, Russian-first typed-decision model. It does not generate text. Given a
context and a set of candidate options, it returns a calibrated probability over
the options and picks one. It covers three decision shapes: yes/no, choice from
options, and ordinal scale. It is a compact encoder (mmBERT-base, 307M), so it
runs on CPU and can be deployed inside a closed perimeter.

- Project page: https://ru-jev.besk.tech
- Demo with precomputed examples: https://huggingface.co/spaces/besktech/ru-jev-demo
- Training and evaluation code: https://github.com/beskvladimir-create/ru-jev

Earlier releases are kept on branches: `v1.1` and `v1`. Load one with
`revision="v1.1"` if you depend on its exact behaviour.

The inference widget is switched off on purpose, and the stock pipelines do not
work either. This is a backbone plus a separate scoring head, not a standard
sequence-classification checkpoint. Every stock pipeline loads the checkpoint
through `AutoModelForSequenceClassification`, which quietly initializes a fresh
head instead of loading `head.safetensors`, and then scores with it. On the
example below the standard zero-shot pipeline put at most 0.44 on the right
option where this model puts 0.95, and over three loads of the same code it
named a different top option twice. Use the snippet below.

**Not affiliated with TypeSafe AI. Not trained on Jev outputs.** Ru-Jev is a
Jev-class typed-decision model built independently: the scoring head is trained
from random initialisation and the mmBERT-base backbone is fine-tuned along with
it, on open, human-labelled datasets only. No closed decision API was called,
distilled or used as a label source at any point. The name is a play on the
category, not a claim of compatibility or endorsement.

## What changed since v1.1

v1.1 read the option text, which is the point of the design, but it read it
narrowly. It had learned the exact wordings its training data used. Rename a
category and accuracy fell off a cliff. A scenario called `погода` was routed
correctly in 0.956 of cases, and the same scenario called `вэзер` in
0.633. Worse, the
confidence did not fall with it: on the jargon wording v1.1 still reported
0.754 mean confidence against 0.633 accuracy, a gap of 0.121,
against 0.030 for v1.3. The probability stopped carrying information exactly
where a caller needed it most.

v1.3 ships every fixed option set in several wordings and draws one per example,
so the model has to read an option instead of recognising it. For the routing
tasks that includes the English key written in Cyrillic and a clipped form of
it, generated by rule.

The same 1,000 routing questions were then asked over and over with the label
wording varied and nothing else. Two candidate-set sizes are reported, because
the gap depends on the size in different ways for different vocabularies and a
probe at one size alone can misstate the effect: five candidates drawn from the
eighteen categories, which is what earlier versions of this card reported, and
all eighteen candidates at once. Chance is 0.20 and 0.056.

| Option wording, example for `weather` | Status | v1.1, 5 | v1.3, 5 | v1.1, 18 | v1.3, 18 |
| --- | --- | --- | --- | --- | --- |
| Russian word, `погода` | training style, both releases | 0.956 | 0.960 | 0.883 | 0.892 |
| English key, `weather` | training style, both releases | 0.957 | 0.962 | 0.894 | 0.900 |
| Russian phrase, `прогноз погоды` | training style, v1.3 only | 0.881 | 0.957 | 0.703 | 0.885 |
| Descriptive sentence | held out | 0.939 | 0.958 | 0.848 | 0.878 |
| Department name, `метеослужба` | held out | 0.835 | 0.869 | 0.632 | 0.733 |
| Anglicism jargon, `вэзер`, all 18 labels | held out | 0.633 | 0.855 | 0.431 | 0.640 |
| Anglicism jargon, 15 clean labels | held out | 0.624 | 0.838 | 0.427 | 0.595 |
| Anglicism jargon, 11 strict labels | held out | 0.550 | 0.786 | 0.328 | 0.470 |
| Office slang, `синоптик`, all 18 | held out | 0.648 | 0.780 | 0.428 | 0.476 |
| Office slang, 14 strict labels | held out | 0.595 | 0.761 | 0.369 | 0.419 |
| Abbreviation, `ПОГ`, 17 clean labels | held out | 0.420 | 0.360 | not scored | |
| Opaque code, `ПГД`, 9 strict labels | held out | 0.135 | 0.168 | 0.018 | 0.061 |
| Code identifier, `WEATHER_SCN` | contains the trained key | 0.856 | 0.889 | not scored | |

**The first two rows are controls, not results.** The Russian word and the
English key are the wordings both releases trained on: the bench vocabulary for
them is the training style itself, taken from the same table in the data
builder. Those two rows measure recall of a trained form, and the gap near zero
is what a control should show. The Russian phrase is a training style for v1.3
and not for v1.1, so its gap is expected by construction and is not evidence of
robustness either. Only the rows marked held out speak to generalisation.

Two caveats on the held-out rows. An intent key begins with its scenario name,
so three of the eighteen jargon labels coincide with what the transliteration
rule produces and reach training as a first word; the clean slice drops those
three and the strict slice drops every label with any exact or prefix relation
to a training option. The rules were not tuned against this test vocabulary, and
the vocabulary is the older of the two: it was committed six hours and
thirty-four minutes before the rules. Figures from different slices measure
different label sets, so one never belongs beside another.

**What the augmentation helps with.** Training a label in several surface forms
helps on held-out wordings made of words, ordinary or borrowed, including kinds
of words that were never in training at all: jargon gains +0.235 with five
candidates and +0.142 with eighteen on the strict slice, office slang +0.166 and
+0.050, department names +0.034 and +0.101, descriptive sentences +0.019 and
+0.030. The direction is the same in all fourteen word-vocabulary cells.

**What it does not help with.** On three-letter abbreviations v1.3 is not better:
-0.024 on all labels and -0.060 on the clean slice. Opaque codes are unreadable
for both releases, with v1.1 below chance and v1.3 at chance on the strict slice
with eighteen candidates. Code identifiers such as `WEATHER_SCN` contain the
trained key and are easy for both, so they are not treated as a test of
robustness. Where the label string carries little of its meaning, several
surface forms in training buy nothing.

**Confidence does not follow accuracy here.** With five candidates the count of
confident errors falls on jargon, from 110 to 62 per 1,000 on the strict slice.
With eighteen it rises, on jargon from 79 to 109 on all labels and on slang from
151 to 191 on the strict slice, because the augmented release is more confident
overall and more of its errors cross the 0.8 threshold. On abbreviations it
rises from 77 to 118. The augmentation does not by itself make confidence safer
on unfamiliar wording. A deployment that invents its own label vocabulary should
refit the temperature on its own decisions; `scripts/fit_temperature.py` in the
code repository does that on a few hundred rows and reports whether there were
enough of them.

Thresholding tells a kinder story than the raw count does. In most cells where
accuracy improves, the area under the risk-coverage curve falls, so a caller who
answers only above a confidence threshold is better served: on strict jargon it
goes from 0.235 to 0.078 with five candidates and from 0.452 to 0.418 with
eighteen, and on strict slang from 0.223 to 0.121 with five candidates. The
exception is slang on all labels with eighteen candidates, where it rises from
0.354 to 0.395. So the release is easier to threshold on unfamiliar wording while
being worse at the fixed 0.8 cut, and which of those matters depends on whether
a threshold is tuned.

**These are two trained models, not an ablation.** The two releases differ in
more than the label augmentation, and each was trained once. The results above
are consistent with the augmentation being the cause, and a later attempt to
isolate it by retraining the recipe on other seeds was void because the mixture
shipped to that run predated the augmentation commit. Nothing here is a
controlled ablation.

Significance is exact McNemar on the paired questions with a Holm correction
over the 23 held-out cells, and the interval resamples labels before questions.
The five-candidate columns for slang and opaque codes are obtained by
renormalising the eighteen-candidate scores over the same five candidates the
main bench uses, which is exact for Ru-Jev because every candidate is scored in
its own pass. Raw counts, per-label tables and the analysis script are in the
code repository under `results/analysis/`.

## Results

v1.3 on its own validation split, 13,373 decisions, with the temperature
fitted on the other half:

| Metric | Value |
| --- | --- |
| Accuracy (micro) | 0.823 |
| ECE, temperature fitted on a disjoint half | 0.0185 |
| ECE, temperature fitted on this same data | 0.0159 |
| Shipped temperature | 1.8640 |

Two ECE figures, because the number depends on where the temperature came from.
Fitting and scoring on the same rows is in-sample and optimistic. The first
figure is what to expect on new data of the same kind, and it is the one to
quote.

### Why most of the comparison with v1.1 had to be thrown away

Two things stood between the two releases, and both were found by checking
rather than by reasoning.

The data builder carved each task's validation slice by shuffling that task's
pool with a module-level random generator, seeded once and consumed by every
other random call in the file. Adding two label styles to one task changed how
many bits those draws consumed, the stream diverged from that point, and every
task below it in the build order received a different slice. Measured: of
v1.3's validation rows, 90.3% of rucola and 89.8% of mera_qa had been in v1.1's
training set, along with 47.2% of georeview and 21.7% of toxicity.
The second is the point of this release working against its own measurement.
Where v1.3 ships an option set in several wordings, its validation rows carry
wordings v1.1 never trained on. Scoring v1.1 there measures how well it handles
renamed options, which is a real question but not the question a per-task
accuracy table is asking. Comparing by text alone hides this, so the rows were
compared by text and option wordings together.
| Task | Same rows | Same rows and same option wordings |
| --- | --- | --- |
| terra, danetqa, russe, rwsd, parus, rcb | 100% | 100% |
| rureviews | 100% | 19.6% |
| massive_scenario, massive_intent | 63% | 0% |
| mera_qa | 10.6% | 10.6% |
| rucola | 9.7% | 9.7% |
| georeview | 5.3% | 1.5% |
| toxicity | 2.4% | 0.4% |

What this cost in published claims is worth stating plainly. An early draft of
this card read rucola 0.909 against 0.647 and mera_qa 0.939 against 0.596, both
of which were v1.1 being scored on its own training data. It also read toxicity
0.722 against 0.954 as a gain in capability, when v1.1 scores 0.961 on toxicity
against the wordings it trained on; 0.722 is that same model with the labels
renamed. None of those three numbers belonged in a per-task accuracy table.
The builder now seeds each carve from the task name, so a wording change in one
task can no longer move the split in another.
### Six tasks measured on identical rows with identical option wordings
| Metric, six comparable tasks, all 10,157 decisions | v1.1 | v1.3 |
| --- | --- | --- |
| Correct answers | 8,120 | 8,120 |
| Accuracy (micro) | 0.7994 | 0.7994 |
| Fitted temperature on these rows | 2.0054 | 1.5593 |
| ECE after each model's own temperature | 0.0190 | 0.0191 |
Accuracy does not depend on the temperature, so it is reported on all 10,157 decisions rather than on a split half. The two releases land on the same count: 8,120 correct out of 10,157 each. They do not get the same rows right. They disagree on 1,680 of them, 840 where only v1.1 is right and 840 where only v1.3 is. One standard error on the difference is 0.0056, and the difference is +0.0000.
The temperature and ECE rows are the mean over 25 random half-splits, fitting on one half and scoring the other. v1.3 needs a lower temperature in 25 of 25 splits, with the two ranges not overlapping (1.942 to 2.098 against 1.500 to 1.604), so its raw scores are genuinely less overconfident. The ECE that remains after each model gets its own temperature is the same for both: the difference averages +0.0001 and runs from -0.0063 to +0.0109, favouring v1.3 in 14 of 25 splits. An earlier version of this card reported 0.0172 against 0.0160 from a single fixed split and called v1.3 better calibrated here. On a different split the sign flips, so that claim is withdrawn.
The split seed is recorded as `SPLIT_SEED = 20261105` in `scripts/calib.py`. An earlier implementation shuffled its index list in place without resetting it between splits, so each split was applied to the permutation left by the previous one and the result depended on the order of rows in the file. These figures come from the corrected code; the shift it caused is in the third decimal and turns over nothing.

| Task | v1.1 | v1.3 | n |
| --- | --- | --- | --- |
| russe (word sense) | 0.822 | 0.824 | 8,505 |
| danetqa (yes/no) | 0.742 | 0.753 | 821 |
| terra (entailment) | 0.775 | 0.733 | 307 |
| parus (choice) | 0.680 | 0.590 | 100 |
| rcb (3-way NLI) | 0.450 | 0.509 | 220 |
| rwsd (coreference) | 0.564 | 0.480 | 204 |
| **all six** | **0.7994** | **0.7994** | **10,157** |

On the only ground where the two releases can be compared directly, they are
the same model for accuracy. v1.3 is not more accurate than v1.1 and this
release does not claim to be. What it is, is far less dependent on how the
options are worded, and less overconfident before calibration, needing a
noticeably lower temperature on identical rows. After each model is given its
own temperature, the calibration error that remains is the same for both.

One caveat on this table. `russe` supplies 8,505 of the 10,157
decisions, 84% of the total, so the micro average is mostly a
statement about one word-sense task. The five smaller tasks move in both
directions and none of them moves far enough to matter against its own standard
error.

### The remaining seven tasks, each model on its own held-out slice

No shared rows exist for these, so each column is that model's own validation
slice scored with its own option wordings. That is an honest measurement of
each model separately and a weak comparison between them.

| Task | v1.1, own slice | v1.3, own slice | n |
| --- | --- | --- | --- |
| massive_intent (60 labels) | not measured | 0.960 | 3,000 |
| massive_scenario (18 labels) | not measured | 0.955 | 3,000 |
| toxicity | 0.961 | 0.954 | 3,000 |
| rureviews (sentiment) | not measured | 0.774 | 3,000 |
| rucola (acceptability) | 0.650 | 0.647 | 1,344 |
| georeview (5 levels) | 0.629 | 0.631 | 3,000 |
| mera_qa (reasoning) | 0.522 | 0.596 | 245 |
## Four epochs were trained and the third was shipped
The run went four epochs and each one was kept. The fourth overfits: training
loss fell to 0.085 while validation loss rose from 0.483 to 0.820, and a
temperature fitted on it comes out at 3.92, which is what overconfidence looks
like as a number.
| Epoch | Training loss | Validation loss | Validation accuracy |
| --- | --- | --- | --- |
| 1 | 0.6029 | 0.5054 | 0.7536 |
| 2 | 0.4336 | 0.4576 | 0.7920 |
| 3 | 0.2714 | 0.4831 | 0.8181 |
| 4 | 0.0849 | 0.8204 | 0.8139 |
The shipped weights are the third epoch. Had only the last checkpoint been kept,
the released model would have been the worse one and nothing would have shown it.
## Name your options carefully
The model reads the option text, so wording still matters, just much less than
it did.
| Option wording | v1.1 accuracy | v1.1 confident errors | v1.3 accuracy | v1.3 confident errors |
| --- | --- | --- | --- | --- |
| Russian word, `погода` | 0.956 | 20 | 0.960 | 19 |
| English key, `weather` | 0.957 | 16 | 0.962 | 26 |
| Descriptive sentence | 0.939 | 14 | 0.958 | 16 |
| Russian phrase, `прогноз погоды` | 0.881 | 34 | 0.957 | 14 |
| Department name, `метеослужба` | 0.835 | 54 | 0.869 | 52 |
| Jargon, `вэзер` | 0.633 | 85 | 0.855 | 44 |
The confident-error columns count answers that were wrong while above 0.8
confidence, out of a thousand questions per row. They are measured with five
candidates; with all eighteen at once the count rises for v1.3 on jargon and on
slang, because it is more confident overall.
Three rules follow from the wording bench above.
**Write options as ordinary words that describe the category.** That is where
several surface forms in training actually pay: on held-out wordings made of
words, including borrowed words and office slang the model never saw.
**Do not use abbreviations or opaque codes as option text.** `ПОГ` and `ПГД` for
the weather category are not better read by v1.3 than by v1.1, and opaque codes
sit at chance for both. A code that contains a readable key, such as
`WEATHER_SCN`, works for the same reason the key does, so it is the key that is
doing the work, not the code.

**Measure your own vocabulary and refit the temperature on it.**
`scripts/robustness.py` scores a label vocabulary of your own, and
`scripts/fit_temperature.py` fits a temperature on a few hundred of your own
decisions and says whether there were enough of them. Accuracy does not depend
on the temperature, so refitting changes only the number next to the answer, but
on unfamiliar wording that number is where this release is least trustworthy.

## Withholding an answer

A calibrated probability is worth something only if the caller acts on it. Set a
confidence threshold, answer above it and send the rest to a human. The serving
API takes a `threshold` and returns the withheld decisions in a review list.

| Share answered | Confidence threshold | Accuracy on those | Errors per 1000 |
| --- | --- | --- | --- |
| 50% | 0.85 | 0.967 | 33 |
| 60% | 0.77 | 0.947 | 53 |
| 70% | 0.69 | 0.921 | 79 |
| 80% | 0.61 | 0.892 | 108 |
| 90% | 0.54 | 0.859 | 141 |
| 100% | 0.23 | 0.823 | 177 |

## Limitations

Hard reasoning tasks with small training sets remain weak and are not fixed by
anything in this release: `rwsd` coreference, `rcb` three-way inference and
`parus` causal choice all train on fewer than a thousand rows each. For a 307M
encoder that is a size limit, not a data-wording limit. A larger backbone was
tried once, xlm-roberta-large at 560M, and lost to v1.

Those same tasks are where the two releases differ most in the comparable
table, and in both directions: v1.3 is 0.059 above v1.1 on `rcb` and 0.084 and
0.090 below it on `rwsd` and `parus`. Their validation slices hold 220, 204 and
100 rows, so one standard error is roughly 0.03 to 0.05 and none of those three
gaps is far outside it. Read them as scatter, not as a trend, and do not plan
around either direction.

The model answers only within the option set it is given. It cannot say that
none of the options fit, so include an explicit "none of these" option when that
outcome is possible.

Probabilities are calibrated in aggregate, not per task. A single temperature
serves all thirteen tasks, so a task far from the mix average can be
systematically overconfident or underconfident.

Every training recipe here was run once. The wording-robustness gap is large
enough that seed noise is unlikely to account for it, but it has not been
measured across seeds, and the small per-task differences in the comparable
table certainly should not be read as reproducible.

Robustness to wording was measured systematically on the choice-from-options
shape only, using the eighteen-way routing task. For yes/no there is a single
renaming test, on toxicity, reported above. The ordinal-scale shape has no
wording test at all, and it is the shape where renaming a level could also
disturb the order, so nothing here should be assumed to carry over to it.

## Speed on CPU

Apple M4, float32, median of 25 runs after 5 warm-up runs, nothing else running
on the machine. Latency is the whole decision: tokenisation, the forward passes
and the softmax.

| Candidates | Ru-Jev v1.3 | Laya | Ratio |
| --- | --- | --- | --- |
| 2 | 18.5 ms | 15.5 ms | 1.19 |
| 5 | 32.8 ms | 17.8 ms | 1.84 |
| 18 | 73.8 ms | 30.5 ms | 2.42 |

Ru-Jev scores each (context, option) pair in its own forward pass, so the cost
of a decision grows linearly with the number of candidates, by roughly 4 ms per
candidate on this machine. That is also why option order is not an input to it.
Laya reads all options in one pass, so its cost is almost flat in the number of
candidates and order is an input. Ru-Jev is slower in both settings and no
speed advantage is claimed here.

## Usage

```python
import json, torch, torch.nn as nn
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModel, AutoTokenizer
REPO = "besktech/ru-jev"
class Scorer(nn.Module):
    def __init__(self, repo):
        super().__init__()
        self.backbone = AutoModel.from_pretrained(repo)
        self.head = nn.Linear(self.backbone.config.hidden_size, 1)
        self.head.load_state_dict(
            load_file(hf_hub_download(repo, "head.safetensors")))
    def forward(self, ids, mask):
        cls = self.backbone(input_ids=ids, attention_mask=mask).last_hidden_state[:, 0]
        return self.head(cls).squeeze(-1)
calib = json.load(open(hf_hub_download(REPO, "calibration.json")))
T = calib["temperature"]
tok = AutoTokenizer.from_pretrained(REPO)
model = Scorer(REPO).eval()
context = "Приложение падает при входе после обновления."
options = ["техническая проблема", "оплата, счета и возвраты", "продажи"]
with torch.no_grad():
    enc = tok([context] * len(options), options, truncation=True,
              max_length=calib["max_len"], padding=True, return_tensors="pt")
    probs = torch.softmax(model(enc["input_ids"], enc["attention_mask"]) / T, dim=0)
for option, p in zip(options, probs.tolist()):
    print(f"{p:.3f}  {option}")
```

Divide by the shipped temperature before the softmax. Without it the
probabilities are overconfident and the threshold logic above does not hold.

## Training data

Thirteen tasks from open Russian datasets, 143,991 training and 26,746
validation decisions. Russian SuperGLUE (`terra`, `danetqa`, `russe`, `rwsd`,
`parus`, `rcb`), RuCoLA, MERA (`ruopenbookqa`, `ruworldtree`), MTEB Russian
(`RuReviewsClassification`, `GeoreviewClassification`, `amazon_massive_intent`,
`amazon_massive_scenario`) and `rudetoxifier_data`. Each source keeps its own
licence; see `DATASETS.md` in the code repository.

Every fixed option set ships in several wordings. The builder verifies that all
thirteen tasks are present and refuses to write the data if any is short.
That check exists because an earlier run silently trained on eleven.

## Citation

The two releases are compared in the preprint
[arXiv:2611.08439](https://arxiv.org/abs/2611.08439). The code, the per-row
scores and the evaluation scripts behind every figure in it are archived at
Zenodo under [10.5281/zenodo.23986846](https://doi.org/10.5281/zenodo.23986846),
which always resolves to the latest version. Cite 10.5281/zenodo.23986847 for
the exact version the paper was written against.

```bibtex
@misc{beskorovainyi2026rujev,
  title         = {Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Words, Not on Abbreviations or Codes},
  author        = {Beskorovainyi, Vladimir},
  year          = {2026},
  eprint        = {2611.08439},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2611.08439}
}
@software{beskorovainyi2026rujevcode,
  title     = {Ru-Jev: open typed-decision model (v1.3) — code, per-row results and evaluation scripts},
  author    = {Beskorovainyi, Vladimir},
  year      = {2026},
  version   = {1.3.0},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23986846},
  url       = {https://doi.org/10.5281/zenodo.23986846}
}
```

## License

MIT covers the code and the weights. The training data keeps the licences of
its sources, which are listed task by task in `DATASETS.md` in the code
repository.

Vladimir Beskorovainyi, Enterprise AI Architect & CTO, author of the Amazon
bestseller "Enterprise AI: The KATA Way", founder of Besk Tech (besk.tech)