Post
17
Ru-Jev v1.3: label augmentation helps with unfamiliar words, not with abbreviations or codes.
Ru-Jev is an open typed-decision model, a pointwise cross-encoder on mmBERT-base (307M, MIT). You pass a context and candidate options, it returns a probability per option. It runs on CPU and is trained and tested on Russian-language data.
v1.3 trains each label in several surface forms. I renamed 18 routing labels with 10 vocabularies, scored each with 5 candidates per question and eight of them also with 18, and checked every label against the training data.
On held-out word vocabularies the new version is better in both settings. Jargon labels with no exact or prefix match to training: 0.550 to 0.786 with 5 candidates, 0.328 to 0.470 with 18. Department names: 0.632 to 0.733 with 18. Office slang: 0.595 to 0.761 with 5, and a small gain with 18 that does not survive Holm correction. On abbreviations it is not better, and opaque codes defeat both versions. With 18 candidates confident errors rise on jargon and slang, so refit the temperature on your own labels.
Per-row predictions and all evaluation scripts are in the repo. The write-up also covers four evaluation bugs and the numbers they forced me to withdraw.
Preprint: arxiv.org/abs/2611.08439 Model: besktech/ru-jev Code: github.com/beskvladimir-create/ru-jev
Ru-Jev is an open typed-decision model, a pointwise cross-encoder on mmBERT-base (307M, MIT). You pass a context and candidate options, it returns a probability per option. It runs on CPU and is trained and tested on Russian-language data.
v1.3 trains each label in several surface forms. I renamed 18 routing labels with 10 vocabularies, scored each with 5 candidates per question and eight of them also with 18, and checked every label against the training data.
On held-out word vocabularies the new version is better in both settings. Jargon labels with no exact or prefix match to training: 0.550 to 0.786 with 5 candidates, 0.328 to 0.470 with 18. Department names: 0.632 to 0.733 with 18. Office slang: 0.595 to 0.761 with 5, and a small gain with 18 that does not survive Holm correction. On abbreviations it is not better, and opaque codes defeat both versions. With 18 candidates confident errors rise on jargon and slang, so refit the temperature on your own labels.
Per-row predictions and all evaluation scripts are in the repo. The write-up also covers four evaluation bugs and the numbers they forced me to withdraw.
Preprint: arxiv.org/abs/2611.08439 Model: besktech/ru-jev Code: github.com/beskvladimir-create/ru-jev