Ру-Жев (Ru-Jev) v1.3

Do not use the standard zero-shot pipeline. The call pipeline("zero-shot-classification", model="besktech/ru-jev") neither fails nor works: it returns numbers drawn from a randomly initialized head, and they change on every load. Load the model with the snippet below.

Open, Russian-first typed-decision model. It does not generate text. Given a context and a set of candidate options, it returns a calibrated probability over the options and picks one. It covers three decision shapes: yes/no, choice from options, and ordinal scale. It is a compact encoder (mmBERT-base, 307M), so it runs on CPU and can be deployed inside a closed perimeter.

• Project page: https://ru-jev.besk.tech
• Demo with precomputed examples: https://huggingface.co/spaces/besktech/ru-jev-demo
• Training and evaluation code: https://github.com/beskvladimir-create/ru-jev

Earlier releases are kept on branches: v1.1 and v1. Load one with revision="v1.1" if you depend on its exact behaviour.

The inference widget is switched off on purpose, and the stock pipelines do not work either. This is a backbone plus a separate scoring head, not a standard sequence-classification checkpoint. Every stock pipeline loads the checkpoint through AutoModelForSequenceClassification, which quietly initializes a fresh head instead of loading head.safetensors, and then scores with it. On the example below the standard zero-shot pipeline put at most 0.44 on the right option where this model puts 0.95, and over three loads of the same code it named a different top option twice. Use the snippet below.

Not affiliated with TypeSafe AI. Not trained on Jev outputs. Ru-Jev is a Jev-class typed-decision model built independently: the scoring head is trained from random initialisation and the mmBERT-base backbone is fine-tuned along with it, on open, human-labelled datasets only. No closed decision API was called, distilled or used as a label source at any point. The name is a play on the category, not a claim of compatibility or endorsement.

What changed since v1.1

v1.1 read the option text, which is the point of the design, but it read it narrowly. It had learned the exact wordings its training data used. Rename a category and accuracy fell off a cliff. A scenario called погода was routed correctly in 0.956 of cases, and the same scenario called вэзер in 0.633. Worse, the confidence did not fall with it: on the jargon wording v1.1 still reported 0.754 mean confidence against 0.633 accuracy, a gap of 0.121, against 0.030 for v1.3. The probability stopped carrying information exactly where a caller needed it most.

v1.3 ships every fixed option set in several wordings and draws one per example, so the model has to read an option instead of recognising it. For the routing tasks that includes the English key written in Cyrillic and a clipped form of it, generated by rule.

The same 1,000 routing questions were then asked over and over with the label wording varied and nothing else. Two candidate-set sizes are reported, because the gap depends on the size in different ways for different vocabularies and a probe at one size alone can misstate the effect: five candidates drawn from the eighteen categories, which is what earlier versions of this card reported, and all eighteen candidates at once. Chance is 0.20 and 0.056.

Option wording, example for weather	Status	v1.1, 5	v1.3, 5	v1.1, 18	v1.3, 18
Russian word, погода	training style, both releases	0.956	0.960	0.883	0.892
English key, weather	training style, both releases	0.957	0.962	0.894	0.900
Russian phrase, прогноз погоды	training style, v1.3 only	0.881	0.957	0.703	0.885
Descriptive sentence	held out	0.939	0.958	0.848	0.878
Department name, метеослужба	held out	0.835	0.869	0.632	0.733
Anglicism jargon, вэзер, all 18 labels	held out	0.633	0.855	0.431	0.640
Anglicism jargon, 15 clean labels	held out	0.624	0.838	0.427	0.595
Anglicism jargon, 11 strict labels	held out	0.550	0.786	0.328	0.470
Office slang, синоптик, all 18	held out	0.648	0.780	0.428	0.476
Office slang, 14 strict labels	held out	0.595	0.761	0.369	0.419
Abbreviation, ПОГ, 17 clean labels	held out	0.420	0.360	not scored
Opaque code, ПГД, 9 strict labels	held out	0.135	0.168	0.018	0.061
Code identifier, WEATHER_SCN	contains the trained key	0.856	0.889	not scored

The first two rows are controls, not results. The Russian word and the English key are the wordings both releases trained on: the bench vocabulary for them is the training style itself, taken from the same table in the data builder. Those two rows measure recall of a trained form, and the gap near zero is what a control should show. The Russian phrase is a training style for v1.3 and not for v1.1, so its gap is expected by construction and is not evidence of robustness either. Only the rows marked held out speak to generalisation.

Two caveats on the held-out rows. An intent key begins with its scenario name, so three of the eighteen jargon labels coincide with what the transliteration rule produces and reach training as a first word; the clean slice drops those three and the strict slice drops every label with any exact or prefix relation to a training option. The rules were not tuned against this test vocabulary, and the vocabulary is the older of the two: it was committed six hours and thirty-four minutes before the rules. Figures from different slices measure different label sets, so one never belongs beside another.

What the augmentation helps with. Training a label in several surface forms helps on held-out wordings made of words, ordinary or borrowed, including kinds of words that were never in training at all: jargon gains +0.235 with five candidates and +0.142 with eighteen on the strict slice, office slang +0.166 and +0.050, department names +0.034 and +0.101, descriptive sentences +0.019 and +0.030. The direction is the same in all fourteen word-vocabulary cells.

What it does not help with. On three-letter abbreviations v1.3 is not better: -0.024 on all labels and -0.060 on the clean slice. Opaque codes are unreadable for both releases, with v1.1 below chance and v1.3 at chance on the strict slice with eighteen candidates. Code identifiers such as WEATHER_SCN contain the trained key and are easy for both, so they are not treated as a test of robustness. Where the label string carries little of its meaning, several surface forms in training buy nothing.

Confidence does not follow accuracy here. With five candidates the count of confident errors falls on jargon, from 110 to 62 per 1,000 on the strict slice. With eighteen it rises, on jargon from 79 to 109 on all labels and on slang from 151 to 191 on the strict slice, because the augmented release is more confident overall and more of its errors cross the 0.8 threshold. On abbreviations it rises from 77 to 118. The augmentation does not by itself make confidence safer on unfamiliar wording. A deployment that invents its own label vocabulary should refit the temperature on its own decisions; scripts/fit_temperature.py in the code repository does that on a few hundred rows and reports whether there were enough of them.

Thresholding tells a kinder story than the raw count does. In most cells where accuracy improves, the area under the risk-coverage curve falls, so a caller who answers only above a confidence threshold is better served: on strict jargon it goes from 0.235 to 0.078 with five candidates and from 0.452 to 0.418 with eighteen, and on strict slang from 0.223 to 0.121 with five candidates. The exception is slang on all labels with eighteen candidates, where it rises from 0.354 to 0.395. So the release is easier to threshold on unfamiliar wording while being worse at the fixed 0.8 cut, and which of those matters depends on whether a threshold is tuned.

These are two trained models, not an ablation. The two releases differ in more than the label augmentation, and each was trained once. The results above are consistent with the augmentation being the cause, and a later attempt to isolate it by retraining the recipe on other seeds was void because the mixture shipped to that run predated the augmentation commit. Nothing here is a controlled ablation.

Significance is exact McNemar on the paired questions with a Holm correction over the 23 held-out cells, and the interval resamples labels before questions. The five-candidate columns for slang and opaque codes are obtained by renormalising the eighteen-candidate scores over the same five candidates the main bench uses, which is exact for Ru-Jev because every candidate is scored in its own pass. Raw counts, per-label tables and the analysis script are in the code repository under results/analysis/.

Results

v1.3 on its own validation split, 13,373 decisions, with the temperature fitted on the other half:

Metric	Value
Accuracy (micro)	0.823
ECE, temperature fitted on a disjoint half	0.0185
ECE, temperature fitted on this same data	0.0159
Shipped temperature	1.8640

Two ECE figures, because the number depends on where the temperature came from. Fitting and scoring on the same rows is in-sample and optimistic. The first figure is what to expect on new data of the same kind, and it is the one to quote.

Why most of the comparison with v1.1 had to be thrown away

Two things stood between the two releases, and both were found by checking rather than by reasoning.

The data builder carved each task's validation slice by shuffling that task's pool with a module-level random generator, seeded once and consumed by every other random call in the file. Adding two label styles to one task changed how many bits those draws consumed, the stream diverged from that point, and every task below it in the build order received a different slice. Measured: of v1.3's validation rows, 90.3% of rucola and 89.8% of mera_qa had been in v1.1's training set, along with 47.2% of georeview and 21.7% of toxicity.

The second is the point of this release working against its own measurement. Where v1.3 ships an option set in several wordings, its validation rows carry wordings v1.1 never trained on. Scoring v1.1 there measures how well it handles renamed options, which is a real question but not the question a per-task accuracy table is asking. Comparing by text alone hides this, so the rows were compared by text and option wordings together.

Task	Same rows	Same rows and same option wordings
terra, danetqa, russe, rwsd, parus, rcb	100%	100%
rureviews	100%	19.6%
massive_scenario, massive_intent	63%	0%
mera_qa	10.6%	10.6%
rucola	9.7%	9.7%
georeview	5.3%	1.5%
toxicity	2.4%	0.4%

What this cost in published claims is worth stating plainly. An early draft of this card read rucola 0.909 against 0.647 and mera_qa 0.939 against 0.596, both of which were v1.1 being scored on its own training data. It also read toxicity 0.722 against 0.954 as a gain in capability, when v1.1 scores 0.961 on toxicity against the wordings it trained on; 0.722 is that same model with the labels renamed. None of those three numbers belonged in a per-task accuracy table.

The builder now seeds each carve from the task name, so a wording change in one task can no longer move the split in another.

Six tasks measured on identical rows with identical option wordings

Metric, six comparable tasks, all 10,157 decisions	v1.1	v1.3
Correct answers	8,120	8,120
Accuracy (micro)	0.7994	0.7994
Fitted temperature on these rows	2.0054	1.5593
ECE after each model's own temperature	0.0190	0.0191

Accuracy does not depend on the temperature, so it is reported on all 10,157 decisions rather than on a split half. The two releases land on the same count: 8,120 correct out of 10,157 each. They do not get the same rows right. They disagree on 1,680 of them, 840 where only v1.1 is right and 840 where only v1.3 is. One standard error on the difference is 0.0056, and the difference is +0.0000.

The temperature and ECE rows are the mean over 25 random half-splits, fitting on one half and scoring the other. v1.3 needs a lower temperature in 25 of 25 splits, with the two ranges not overlapping (1.942 to 2.098 against 1.500 to 1.604), so its raw scores are genuinely less overconfident. The ECE that remains after each model gets its own temperature is the same for both: the difference averages +0.0001 and runs from -0.0063 to +0.0109, favouring v1.3 in 14 of 25 splits. An earlier version of this card reported 0.0172 against 0.0160 from a single fixed split and called v1.3 better calibrated here. On a different split the sign flips, so that claim is withdrawn.

The split seed is recorded as SPLIT_SEED = 20261105 in scripts/calib.py. An earlier implementation shuffled its index list in place without resetting it between splits, so each split was applied to the permutation left by the previous one and the result depended on the order of rows in the file. These figures come from the corrected code; the shift it caused is in the third decimal and turns over nothing.

Task	v1.1	v1.3	n
russe (word sense)	0.822	0.824	8,505
danetqa (yes/no)	0.742	0.753	821
terra (entailment)	0.775	0.733	307
parus (choice)	0.680	0.590	100
rcb (3-way NLI)	0.450	0.509	220
rwsd (coreference)	0.564	0.480	204
all six	0.7994	0.7994	10,157

On the only ground where the two releases can be compared directly, they are the same model for accuracy. v1.3 is not more accurate than v1.1 and this release does not claim to be. What it is, is far less dependent on how the options are worded, and less overconfident before calibration, needing a noticeably lower temperature on identical rows. After each model is given its own temperature, the calibration error that remains is the same for both.

One caveat on this table. russe supplies 8,505 of the 10,157 decisions, 84% of the total, so the micro average is mostly a statement about one word-sense task. The five smaller tasks move in both directions and none of them moves far enough to matter against its own standard error.

The remaining seven tasks, each model on its own held-out slice

No shared rows exist for these, so each column is that model's own validation slice scored with its own option wordings. That is an honest measurement of each model separately and a weak comparison between them.

Task	v1.1, own slice	v1.3, own slice	n
massive_intent (60 labels)	not measured	0.960	3,000
massive_scenario (18 labels)	not measured	0.955	3,000
toxicity	0.961	0.954	3,000
rureviews (sentiment)	not measured	0.774	3,000
rucola (acceptability)	0.650	0.647	1,344
georeview (5 levels)	0.629	0.631	3,000
mera_qa (reasoning)	0.522	0.596	245

Four epochs were trained and the third was shipped

The run went four epochs and each one was kept. The fourth overfits: training loss fell to 0.085 while validation loss rose from 0.483 to 0.820, and a temperature fitted on it comes out at 3.92, which is what overconfidence looks like as a number.

Epoch	Training loss	Validation loss	Validation accuracy
1	0.6029	0.5054	0.7536
2	0.4336	0.4576	0.7920
3	0.2714	0.4831	0.8181
4	0.0849	0.8204	0.8139

The shipped weights are the third epoch. Had only the last checkpoint been kept, the released model would have been the worse one and nothing would have shown it.

Name your options carefully

The model reads the option text, so wording still matters, just much less than it did.

Option wording	v1.1 accuracy	v1.1 confident errors	v1.3 accuracy	v1.3 confident errors
Russian word, погода	0.956	20	0.960	19
English key, weather	0.957	16	0.962	26
Descriptive sentence	0.939	14	0.958	16
Russian phrase, прогноз погоды	0.881	34	0.957	14
Department name, метеослужба	0.835	54	0.869	52
Jargon, вэзер	0.633	85	0.855	44

The confident-error columns count answers that were wrong while above 0.8 confidence, out of a thousand questions per row. They are measured with five candidates; with all eighteen at once the count rises for v1.3 on jargon and on slang, because it is more confident overall.

Three rules follow from the wording bench above.

Write options as ordinary words that describe the category. That is where several surface forms in training actually pay: on held-out wordings made of words, including borrowed words and office slang the model never saw.

Do not use abbreviations or opaque codes as option text. ПОГ and ПГД for the weather category are not better read by v1.3 than by v1.1, and opaque codes sit at chance for both. A code that contains a readable key, such as WEATHER_SCN, works for the same reason the key does, so it is the key that is doing the work, not the code.

Measure your own vocabulary and refit the temperature on it. scripts/robustness.py scores a label vocabulary of your own, and scripts/fit_temperature.py fits a temperature on a few hundred of your own decisions and says whether there were enough of them. Accuracy does not depend on the temperature, so refitting changes only the number next to the answer, but on unfamiliar wording that number is where this release is least trustworthy.

Withholding an answer

A calibrated probability is worth something only if the caller acts on it. Set a confidence threshold, answer above it and send the rest to a human. The serving API takes a threshold and returns the withheld decisions in a review list.

Share answered	Confidence threshold	Accuracy on those	Errors per 1000
50%	0.85	0.967	33
60%	0.77	0.947	53
70%	0.69	0.921	79
80%	0.61	0.892	108
90%	0.54	0.859	141
100%	0.23	0.823	177

Limitations

Hard reasoning tasks with small training sets remain weak and are not fixed by anything in this release: rwsd coreference, rcb three-way inference and parus causal choice all train on fewer than a thousand rows each. For a 307M encoder that is a size limit, not a data-wording limit. A larger backbone was tried once, xlm-roberta-large at 560M, and lost to v1.

Those same tasks are where the two releases differ most in the comparable table, and in both directions: v1.3 is 0.059 above v1.1 on rcb and 0.084 and 0.090 below it on rwsd and parus. Their validation slices hold 220, 204 and 100 rows, so one standard error is roughly 0.03 to 0.05 and none of those three gaps is far outside it. Read them as scatter, not as a trend, and do not plan around either direction.

The model answers only within the option set it is given. It cannot say that none of the options fit, so include an explicit "none of these" option when that outcome is possible.

Probabilities are calibrated in aggregate, not per task. A single temperature serves all thirteen tasks, so a task far from the mix average can be systematically overconfident or underconfident.

Every training recipe here was run once. The wording-robustness gap is large enough that seed noise is unlikely to account for it, but it has not been measured across seeds, and the small per-task differences in the comparable table certainly should not be read as reproducible.

Robustness to wording was measured systematically on the choice-from-options shape only, using the eighteen-way routing task. For yes/no there is a single renaming test, on toxicity, reported above. The ordinal-scale shape has no wording test at all, and it is the shape where renaming a level could also disturb the order, so nothing here should be assumed to carry over to it.

Speed on CPU

Apple M4, float32, median of 25 runs after 5 warm-up runs, nothing else running on the machine. Latency is the whole decision: tokenisation, the forward passes and the softmax.

Candidates	Ru-Jev v1.3	Laya	Ratio
2	18.5 ms	15.5 ms	1.19
5	32.8 ms	17.8 ms	1.84
18	73.8 ms	30.5 ms	2.42

Ru-Jev scores each (context, option) pair in its own forward pass, so the cost of a decision grows linearly with the number of candidates, by roughly 4 ms per candidate on this machine. That is also why option order is not an input to it. Laya reads all options in one pass, so its cost is almost flat in the number of candidates and order is an input. Ru-Jev is slower in both settings and no speed advantage is claimed here.

Usage

import json, torch, torch.nn as nn
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModel, AutoTokenizer

REPO = "besktech/ru-jev"


class Scorer(nn.Module):
    def __init__(self, repo):
        super().__init__()
        self.backbone = AutoModel.from_pretrained(repo)
        self.head = nn.Linear(self.backbone.config.hidden_size, 1)
        self.head.load_state_dict(
            load_file(hf_hub_download(repo, "head.safetensors")))

    def forward(self, ids, mask):
        cls = self.backbone(input_ids=ids, attention_mask=mask).last_hidden_state[:, 0]
        return self.head(cls).squeeze(-1)


calib = json.load(open(hf_hub_download(REPO, "calibration.json")))
T = calib["temperature"]
tok = AutoTokenizer.from_pretrained(REPO)
model = Scorer(REPO).eval()

context = "Приложение падает при входе после обновления."
options = ["техническая проблема", "оплата, счета и возвраты", "продажи"]

with torch.no_grad():
    enc = tok([context] * len(options), options, truncation=True,
              max_length=calib["max_len"], padding=True, return_tensors="pt")
    probs = torch.softmax(model(enc["input_ids"], enc["attention_mask"]) / T, dim=0)

for option, p in zip(options, probs.tolist()):
    print(f"{p:.3f}  {option}")

Divide by the shipped temperature before the softmax. Without it the probabilities are overconfident and the threshold logic above does not hold.

Training data

Thirteen tasks from open Russian datasets, 143,991 training and 26,746 validation decisions. Russian SuperGLUE (terra, danetqa, russe, rwsd, parus, rcb), RuCoLA, MERA (ruopenbookqa, ruworldtree), MTEB Russian (RuReviewsClassification, GeoreviewClassification, amazon_massive_intent, amazon_massive_scenario) and rudetoxifier_data. Each source keeps its own licence; see DATASETS.md in the code repository.

Every fixed option set ships in several wordings. The builder verifies that all thirteen tasks are present and refuses to write the data if any is short. That check exists because an earlier run silently trained on eleven.

Citation

The two releases are compared in the preprint arXiv:2611.08439. The code, the per-row scores and the evaluation scripts behind every figure in it are archived at Zenodo under 10.5281/zenodo.23986846, which always resolves to the latest version. Cite 10.5281/zenodo.23986847 for the exact version the paper was written against.

@misc{beskorovainyi2026rujev,
  title         = {Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Words, Not on Abbreviations or Codes},
  author        = {Beskorovainyi, Vladimir},
  year          = {2026},
  eprint        = {2611.08439},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2611.08439}
}

@software{beskorovainyi2026rujevcode,
  title     = {Ru-Jev: open typed-decision model (v1.3) — code, per-row results and evaluation scripts},
  author    = {Beskorovainyi, Vladimir},
  year      = {2026},
  version   = {1.3.0},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23986846},
  url       = {https://doi.org/10.5281/zenodo.23986846}
}

License

MIT covers the code and the weights. The training data keeps the licences of its sources, which are listed task by task in DATASETS.md in the code repository.

Vladimir Beskorovainyi, Enterprise AI Architect & CTO, author of the Amazon bestseller "Enterprise AI: The KATA Way", founder of Besk Tech (besk.tech)