Instructions to use ITLL/krasivo-jev-bf16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use ITLL/krasivo-jev-bf16 with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
jev pointer, all data (Qwen3.5-2B, run 20261018-160511)
A "system one" model. It reads a state (a context), a question and a list of options, and in one forward pass returns a probability for every option. It is meant for classification, routing, extraction checks, NLI, guardrails and reading comprehension, where the answer follows from the text it is given. It is not meant to answer from world knowledge.
This is the final model, trained on all data, benchmarks included. It was trained with
train_on_benchmark, so the train splits of the 20 benchmark datasets were part of its training mix. Its benchmark scores therefore measure fit to tasks it has seen, not transfer to new ones. For transfer numbers, see the held-out run20261017-192041(run files, model), which is the same recipe without the benchmarks.
- Base model: Qwen/Qwen3.5-2B-Base. LoRA rank 16 on every
attention, linear-attention and MLP projection, plus a 256-dim pointer head
(
src/jev_model/pointer/). - Languages: English and Russian.
- Questions:
noul(yes/no) andchoice(any number of options; there is no fixed slot count). - Training: the
cestage only, 34,315 optimizer steps (2 epochs over 100 datasets). - Run:
runs/pointer/20261018-160511in the training repo: config, training history and test scores. - Code:
krasivo-jev@v0.1.0, the release that includes this run. - Headline: on the test splits of the 80 core datasets, macro accuracy is 0.877, NLL 0.359 and ECE 0.043. On the test splits of the 20 benchmark datasets (whose train splits it trained on), macro accuracy is 0.791, NLL 0.549 and ECE 0.064.
How it works
The sample is laid out as one causal sequence. The delimiters reuse rarely used Qwen special tokens, so no vocabulary rows are added:
<state> state <q> question <opt> option 1 </opt> <opt> option 2 </opt> ... <decide>
The pointer head projects the hidden state at <decide> and the hidden state at each </opt> to 256 dimensions.
An option's logit is their scaled dot product. A softmax over the options, at the fitted temperature (0.626 for
this run), gives the probabilities. Because the logit comes from each option's own text, any number of options
works and the option labels can be arbitrary. Sequences are cut to 8,192 tokens: a longer state loses its middle.
Attention is plain causal, so an option can see the options before it, and the output can depend somewhat on option order. Training shuffles the options on every draw so that the model can't learn a position prior.
The design is adapted from kev. There is one question per sequence and no option isolation, which suits the linear-attention (Gated DeltaNet) layers of Qwen3.5.
Training
| setting | value |
|---|---|
| objective | focal loss (γ = 2) over the option distribution, which supports soft labels |
| optimizer steps | 34,315 (2 passes over the training set, sized sampling) |
| batch | up to 8,192 padded tokens × 2 accumulation steps |
| learning rate | 2e-4, 500 warmup steps, weight decay 0.01 |
| precision | bf16 base weights and autocast; LoRA and head in float32 |
| augmentation | options reshuffled on every draw. With p = 0.2, some wrong options are dropped (so the option count doesn't reveal the dataset). With p = 0.1, the true option's text is replaced by "None of the above" |
| max length | 8,192 tokens (the held-out run used 4,096) |
| model selection | every 10,000 steps, on up to 1,000 validation rows per dataset; the weights with the best macro NLL are kept, then one temperature is fitted on the whole validation partition |
With sized sampling, each pass shuffles the whole training set and packs it into token-budget batches. Every
sample is seen once per pass, so the datasets are mixed in proportion to their size.
To train it from scratch, you need at least 4 A100 GPUs; this run takes about 12–13 hours on 4 of them. Build the data, then train with DDP from this run's config:
uv run jev prepare all --max-samples 25000 -j 8
uv run accelerate launch --multi_gpu --num_processes 4 -m jev_model.cli train pointer \
-c runs/pointer/20261018-160511/config.json -s data_dir=data # the saved data_dir is the training machine's path
Revisions
The pinned versions behind this model.
| what | revision |
|---|---|
| base model | Qwen/Qwen3.5-2B-Base @ b1485b2fa6dfa1287294f269f5fb618e03d52d7c, the snapshot merged into these weights at export |
| code | krasivo-jev @ v0.1.0, the release that includes this run. Trained at ee22754, the last commit before the run started (2026-10-18 16:05 +0300; the run doesn't record its commit), and exported at 0b3be4c |
| libraries | torch 2.14.0, transformers 5.17.0, peft 0.21.1, accelerate 1.15.0, datasets 5.0.1, huggingface-hub 1.33.0 (uv.lock) |
| serving | vllm/vllm-openai:v0.17.1 (docker/vllm.Dockerfile) |
| training data | built by jev prepare from the source datasets listed in the metadata, at their Hub main as of 2026-10-18. The source revisions weren't recorded |
To load the base model at the same revision, pass revision="b1485b2fa6dfa1287294f269f5fb618e03d52d7c" to
from_pretrained/snapshot_download.
Data
All data is converted by the krasivo-jev pipeline (src/jev_datasets/) into one format:
state -> question -> options -> label. The label is a probability distribution over the options: one-hot, or
annotator vote shares for sets such as ChaosNLI. Sources, subsets and rows that need outside knowledge were
dropped. Examples: SciQ keeps only rows that have a support passage, and BBH leaves out its knowledge-heavy tasks.
MMLU, ARC, OpenBookQA, CommonsenseQA, HellaSwag and WinoGrande are excluded for this reason. Each dataset uses
about 5 paraphrases of its question and of its instruction, picked deterministically per sample.
The model trained on the train splits of all 100 datasets: the 80 core datasets and the 20 benchmarks. They cover 14 topics: bugs, business, classification, guardrails, knowledge, logic, nlu, reasoning, retrieval, sentiment, spatial, structured, toxicity and truthfulness. The full list, with sources, is in the tables below. The splits are the same as in the held-out run, so test rows are identical across the two runs.
Split method
Each dataset is saved with exactly train, validation and test splits, built as follows:
- Canonical names.
dev/valbecomevalidation; unknown split names becometrain. - Disjoint contexts. Rows are grouped by source document (an explicit
groupkey when one document yields several passages or turns), or otherwise by a hash of their context (state, orquestionwhen there is no state). Each group moves whole into the first split it appears in, checked in the order test, validation, train. So no context appears in two splits, and a test passage is never trained on through another question about it. Exact duplicate rows are dropped, and the held-out copy is kept. - Missing splits. A missing validation or test split is carved from train by a hash of the group, 5% each. A source with no train split at all is re-split 80/10/10.
- Caps. Each split is capped with a seeded random sample. This build used 25,000 rows per split.
- Sources that share data. Many logic and spatial collections bundle each other (for example, SpaRP includes StepGame, and RuleCollection bundles ProofWriter and LogicNLI). Each underlying set is taken from one source only, so a benchmark's rows can't leak in through another dataset's train split.
Every step is deterministic, seeded from the sample content.
Benchmarks
20 datasets are marked as benchmarks. In the normal recipe they are held out completely. This model trained
on their train splits (train_on_benchmark); only their validation and test splits stay unseen. They
are OpenDecisions (split into noul and choice), plus one converter per topic chosen by a seeded draw:
ar_lsat, github_issue_type, hwu64, jailbreak_classification, logiqa, the six nomiracl_* sets, ropes,
rublimp, russian_toxicity, stepgame, tweet_sentiment, vitaminc and webnlg_en. For this model their test
scores are in-distribution, just like the core datasets. The training-time validation that picked the checkpoint
used only the 80 core datasets.
Evaluation
All scores are on the test split. Each row is scored by the option with the highest probability, at the fitted temperature:
- accuracy: the argmax option is a correct one.
- NLL: the cross-entropy of the predicted distribution against the label distribution.
- ECE: expected calibration error of the top-option confidence, with 15 equal-width bins.
- macro: the mean over datasets, so every dataset counts the same however large it is.
The comparison model is TypeSafe's hosted typesafe/jev-1.13, called through its system_one API and scored on
the same test rows. jev-1.13's training data is not known, so for that model neither the held-in sets nor
the benchmarks are guaranteed to be held out.
In the tables and plots, "held-in" means the 80 core datasets. "Benchmarks" are the 20 benchmark datasets, whose train splits this model also trained on.
| datasets | rows | accuracy | NLL | ECE | jev-1.13 accuracy | jev-1.13 NLL | jev-1.13 ECE | |
|---|---|---|---|---|---|---|---|---|
| held-in test | 80 | 294,408 | 0.877 | 0.359 | 0.043 | 0.829 | 0.882 | 0.080 |
held-in, noul |
14 | 0.935 | 0.165 | 0.030 | 0.858 | 0.326 | 0.052 | |
held-in, choice |
66 | 0.864 | 0.400 | 0.046 | 0.823 | 1.000 | 0.085 | |
| benchmarks | 20 | 102,175 | 0.791 | 0.549 | 0.064 | 0.825 | 0.748 | 0.067 |
benchmarks, noul |
6 | 0.856 | 0.304 | 0.064 | 0.896 | 0.254 | 0.051 | |
benchmarks, choice |
14 | 0.763 | 0.654 | 0.065 | 0.795 | 0.960 | 0.074 |
All values are macro means over datasets; the better value in each pair is in bold.
Against the held-out run (20261017-192041)
The held-out run is the same recipe without the benchmark train splits, and with a 4,096-token limit.
| macro on test | this run | held-out run |
|---|---|---|
| held-in accuracy / NLL / ECE | 0.877 / 0.359 / 0.043 | 0.879 / 0.351 / 0.041 |
| benchmark accuracy / NLL / ECE | 0.791 / 0.549 / 0.064 | 0.640 / 0.990 / 0.123 |
Adding the benchmarks costs nothing on the core datasets: every metric is within 0.01. The benchmarks improve on 19
of 20 sets, most on the ones the held-out run failed to transfer to: rublimp +0.47, stepgame +0.44,
open_decisions_noul +0.24, russian_toxicity +0.23. logiqa (0.490) and ar_lsat (0.247) stay near where they
were, so training on those formats did not help.
Held-in datasets, against jev-1.13
This model beats jev-1.13 on 53 of 80 core datasets, and its macro NLL is less than half of jev-1.13's. The
pattern matches the held-out run. It gains on spatial QA (spartqa_mchoice +0.41, spartqa_yn +0.33,
sparp +0.29), emotion labels (emotion +0.34, go_emotions +0.27), aegis_category (+0.33) and
halueval_summarization (+0.29). It falls behind on multi-choice reading comprehension and argument reasoning
(reclor −0.29, race −0.14, quail −0.13, dream −0.12), on tweet_hate (−0.19), on rcb (−0.12) and on
folio (−0.10).
Benchmarks, against jev-1.13
On the benchmarks this model wins 10 of 20 datasets, with webnlg_en tied at 0.999. jev-1.13 has the higher macro
accuracy (0.825 vs 0.791), but this model has the lower NLL and ECE. This model leads on stepgame (+0.25), hwu64
(+0.13), tweet_sentiment, vitaminc and github_issue_type. It trails on logiqa (−0.29), open_decisions_choice
(−0.26), ar_lsat (−0.22), open_decisions_noul (−0.19) and the nomiracl choice sets.
Usage
The repo holds merged bf16 weights that vLLM can serve, plus the LoRA adapter/ and head.pt for in-process
use. Both paths need the krasivo-jev package (jev_model and the jev_vllm plugins). Install it at the
v0.1.0 release, which matches these weights:
git clone https://github.com/SergeyBazhmin/krasivo-jev && cd krasivo-jev && git checkout v0.1.0
uv sync
from jev_model.system_one import Choice, JevHolder, Noul
holder = JevHolder.from_pretrained("bizhmans/krasivo-jev-bf16", revision="v0.1.0")
answers = holder.system_one(
state="I was charged twice for my order. Please help.",
questions={
"billing": Noul(instructions="Is this about billing?"),
"tone": Choice(instructions="What is the tone?", criteria={"angry": None, "calm": "polite"}),
},
)
answers["billing"].noul # P(yes)
answers["tone"].choice # the likeliest key; also .confidence and .probabilities
uv run serve bizhmans/krasivo-jev-bf16 -- --revision v0.1.0 --tokenizer-revision v0.1.0
# vLLM 0.17.1 + jev_vllm plugins in Docker, http://localhost:8080/pooling
curl localhost:8080/pooling -H 'Content-Type: application/json' -d '{"model": "pointer", "data": {
"state": "I was charged twice. Please help.",
"questions": {"billing": {"type": "noul", "instructions": "Is this about billing?"}}}}'
Code written against typesafe-sdk can call the served model through
jev_vllm.client.JevClient, which takes the same question types and returns the SDK's
SystemOneResponse. Swap the client to switch between them:
from typesafe_sdk import Choice, Noul
from jev_vllm.client import JevClient
with JevClient() as client: # http://localhost:8080, model "pointer"
response = client.system_one(
state="I was charged twice for my order. Please help.",
questions={
"billing": Noul(instructions="Is this about billing?"),
"tone": Choice(instructions="What is the tone?", criteria={"angry": None, "calm": "polite"}),
},
)
response.nouls["billing"].noul, response.choices["tone"].choice
AsyncJevClient has the same system_one, awaited. vLLM batches concurrent requests itself, so asyncio.gather over
many calls keeps the GPU busy.
Limitations
- The model is not a knowledge base. Questions whose answer is not in the state get confident-looking guesses.
- Option order can shift the probabilities slightly, because options attend causally to the ones before them.
- "None of the above" was trained only as an augmentation; the scores here don't measure it.
- Every reported score is in-distribution: no dataset here is unseen. Expect lower accuracy and worse calibration on new tasks; the held-out run's benchmark numbers (macro accuracy 0.640, ECE 0.123) are the better estimate.
- Multi-step logic (
ar_lsat,logiqa) stays weak even after training on it. - The training mix is weighted by dataset size, so large English sets dominate it.
Per-dataset scores
n is the number of test rows. type is noul (yes/no) or choice.
Held-in (test)
| dataset | topic | type | source | n | accuracy | NLL | ECE | jev-1.13 acc | jev-1.13 NLL | jev-1.13 ECE |
|---|---|---|---|---|---|---|---|---|---|---|
| vscode_triage | bugs | choice | Tilakoid/vscode-bug-feature-triage | 200 | 0.890 | 0.319 | 0.050 | 0.890 | 0.986 | 0.099 |
| financial_news_topic | business | choice | zeroshot/twitter-financial-news-topic | 854 | 0.899 | 0.332 | 0.020 | 0.657 | 2.161 | 0.171 |
| ag_news | classification | choice | fancyzhx/ag_news | 7,600 | 0.925 | 0.220 | 0.026 | 0.889 | 0.894 | 0.067 |
| atis | classification | choice | tuetschek/atis | 848 | 0.988 | 0.058 | 0.006 | 0.929 | 0.256 | 0.022 |
| banking77 | classification | choice | legacy-datasets/banking77 | 3,080 | 0.929 | 0.280 | 0.011 | 0.799 | 1.705 | 0.087 |
| bitext_customer_support | classification | choice | bitext/Bitext-customer-support-llm-chatbot-training-dataset | 1,217 | 1.000 | 0.002 | 0.002 | 0.921 | 0.285 | 0.024 |
| bitext_insurance | classification | choice | bitext/Bitext-insurance-llm-chatbot-training-dataset | 1,934 | 1.000 | 0.002 | 0.001 | 0.896 | 0.391 | 0.020 |
| bitext_retail_banking | classification | choice | bitext/Bitext-retail-banking-llm-chatbot-training-dataset | 1,246 | 0.998 | 0.006 | 0.001 | 0.954 | 0.374 | 0.030 |
| bitext_telco | classification | choice | bitext/Bitext-telco-llm-chatbot-training-dataset | 1,284 | 1.000 | 0.002 | 0.002 | 0.921 | 0.292 | 0.038 |
| clinc_oos | classification | choice | clinc/clinc_oos plus |
5,502 | 0.921 | 0.314 | 0.033 | 0.885 | 0.726 | 0.024 |
| dbpedia | classification | choice | fancyzhx/dbpedia_14 | 25,000 | 0.991 | 0.038 | 0.003 | 0.987 | 0.083 | 0.004 |
| massive_intent | classification | choice | mteb/amazon_massive_intent en |
2,972 | 0.911 | 0.373 | 0.014 | 0.744 | 1.761 | 0.111 |
| massive_intent_ru | classification | choice | mteb/MassiveIntentClassification ru |
2,925 | 0.883 | 0.503 | 0.031 | 0.777 | 1.739 | 0.108 |
| rus_med_dialogues | classification | choice | Mykes/rus_med_dialogues | 329 | 0.875 | 0.407 | 0.045 | 0.802 | 1.209 | 0.120 |
| rus_med_dialogues_noul | classification | noul | Mykes/rus_med_dialogues | 1,004 | 0.977 | 0.086 | 0.025 | 0.950 | 0.159 | 0.049 |
| ticket_routing | classification | choice | Tobi-Bueck/customer-support-tickets | 4,093 | 0.436 | 1.578 | 0.064 | 0.278 | 7.707 | 0.441 |
| ticket_subject | classification | choice | Tobi-Bueck/customer-support-tickets | 3,509 | 0.899 | 0.280 | 0.035 | 0.856 | 0.551 | 0.053 |
| ticket_type | classification | choice | Tobi-Bueck/customer-support-tickets | 4,093 | 0.806 | 0.395 | 0.048 | 0.677 | 1.861 | 0.226 |
| aegis_category | guardrails | choice | nvidia/Aegis-AI-Content-Safety-Dataset-2.0 | 672 | 0.726 | 0.862 | 0.056 | 0.396 | 3.819 | 0.346 |
| safeguard_prompt_injection | guardrails | noul | xTRam1/safe-guard-prompt-injection | 2,049 | 0.998 | 0.015 | 0.007 | 0.917 | 0.198 | 0.035 |
| spml_prompt_injection | guardrails | noul | reshabhs/SPML_Chatbot_Prompt_Injection | 852 | 0.996 | 0.016 | 0.003 | 0.897 | 0.286 | 0.025 |
| qasc | knowledge | choice | allenai/qasc | 419 | 0.990 | 0.030 | 0.013 | 0.964 | 0.095 | 0.021 |
| quartz | knowledge | choice | allenai/quartz | 784 | 0.902 | 0.269 | 0.030 | 0.952 | 0.615 | 0.037 |
| sciq | knowledge | choice | allenai/sciq | 886 | 0.995 | 0.017 | 0.008 | 0.999 | 0.007 | 0.002 |
| clutrr | logic | choice | tasksource/clutrr | 1,045 | 0.434 | 4.283 | 0.440 | 0.414 | 2.174 | 0.187 |
| folio | logic | choice | yfxiao/folio-refined | 39 | 0.667 | 0.717 | 0.129 | 0.769 | 1.183 | 0.195 |
| logicnli | logic | choice | tasksource/LogicNLI | 1,999 | 0.733 | 0.616 | 0.024 | 0.606 | 2.330 | 0.205 |
| proofwriter | logic | choice | tasksource/proofwriter | 25,000 | 0.931 | 0.165 | 0.021 | 0.923 | 0.206 | 0.005 |
| proverqa | logic | choice | opendatalab/ProverQA | 313 | 0.805 | 0.403 | 0.036 | 0.719 | 0.641 | 0.101 |
| anli | nlu | choice | facebook/anli | 8,204 | 0.848 | 0.403 | 0.029 | 0.865 | 0.453 | 0.039 |
| babi_nli | nlu | noul | tasksource/babi_nli | 11,934 | 0.968 | 0.095 | 0.026 | 0.924 | 0.210 | 0.011 |
| boolq | nlu | noul | google/boolq | 392 | 0.862 | 0.380 | 0.068 | 0.926 | 0.215 | 0.035 |
| chaos_nli | nlu | choice | tasksource/chaos-mnli-ambiguity | 79 | 0.671 | 0.923 | 0.089 | 0.658 | 2.649 | 0.190 |
| danetqa | nlu | choice | RussianNLP/russian_super_glue DaNetQA |
67 | 0.910 | 0.261 | 0.084 | 0.896 | 0.207 | 0.066 |
| dream | nlu | choice | dataset-org/dream | 2,041 | 0.852 | 0.392 | 0.017 | 0.971 | 0.174 | 0.010 |
| hotpot_qa | nlu | choice | hotpotqa/hotpot_qa distractor |
325 | 0.938 | 0.167 | 0.036 | 0.951 | 0.181 | 0.027 |
| hotpot_qa_noul | nlu | noul | hotpotqa/hotpot_qa distractor |
264 | 0.917 | 0.213 | 0.042 | 0.943 | 0.184 | 0.034 |
| lidirus | nlu | choice | RussianNLP/russian_super_glue LiDiRus |
105 | 0.810 | 0.435 | 0.086 | 0.867 | 0.305 | 0.061 |
| mera_multiq | nlu | choice | ai-forever/MERA multiq |
46 | 0.913 | 0.204 | 0.096 | 0.739 | 1.159 | 0.172 |
| multi_nli | nlu | choice | nyu-mll/multi_nli | 20,287 | 0.867 | 0.366 | 0.034 | 0.850 | 0.546 | 0.049 |
| multi_nli_genre | nlu | choice | nyu-mll/multi_nli | 6,623 | 0.930 | 0.200 | 0.013 | 0.841 | 0.697 | 0.060 |
| muserc | nlu | choice | RussianNLP/russian_super_glue MuSeRC |
536 | 0.873 | 0.302 | 0.036 | 0.937 | 0.464 | 0.042 |
| paws | nlu | noul | google-research-datasets/paws labeled_final |
7,987 | 0.942 | 0.171 | 0.030 | 0.860 | 0.322 | 0.022 |
| quail | nlu | choice | textmachinelab/quail | 496 | 0.728 | 0.724 | 0.064 | 0.857 | 0.595 | 0.060 |
| race | nlu | choice | ehovy/race all |
4,934 | 0.795 | 0.574 | 0.029 | 0.929 | 0.369 | 0.022 |
| rcb | nlu | choice | RussianNLP/russian_super_glue RCB |
26 | 0.538 | 0.869 | 0.101 | 0.654 | 3.922 | 0.340 |
| ru_paraphraser | nlu | choice | merionum/ru_paraphraser | 1,906 | 0.698 | 0.646 | 0.023 | 0.716 | 0.714 | 0.102 |
| rucola | nlu | choice | RussianNLP/rucola | 357 | 0.784 | 0.540 | 0.120 | 0.796 | 0.835 | 0.126 |
| russe | nlu | choice | RussianNLP/russian_super_glue RUSSE |
550 | 0.660 | 0.627 | 0.037 | 0.715 | 0.648 | 0.095 |
| sberquad | nlu | choice | kuznetsoffandrey/sberquad | 1,209 | 0.967 | 0.112 | 0.015 | 0.984 | 0.073 | 0.006 |
| terra | nlu | choice | RussianNLP/russian_super_glue TERRa |
106 | 0.858 | 0.405 | 0.110 | 0.915 | 0.200 | 0.045 |
| reclor | reasoning | choice | tasksource/reclor | 244 | 0.664 | 0.934 | 0.108 | 0.951 | 0.253 | 0.018 |
| wikiqa | retrieval | choice | microsoft/wiki_qa | 615 | 0.738 | 0.779 | 0.099 | 0.727 | 1.706 | 0.148 |
| wikiqa_noul | retrieval | noul | microsoft/wiki_qa | 1,587 | 0.899 | 0.252 | 0.040 | 0.895 | 0.250 | 0.016 |
| cedr | sentiment | choice | sagteam/cedr_v1 main |
9,378 | 0.955 | 0.131 | 0.029 | 0.918 | 0.316 | 0.030 |
| emotion | sentiment | choice | dair-ai/emotion split |
2,014 | 0.929 | 0.134 | 0.015 | 0.591 | 3.516 | 0.258 |
| go_emotions | sentiment | choice | google-research-datasets/go_emotions simplified |
5,439 | 0.600 | 1.421 | 0.061 | 0.330 | 6.248 | 0.340 |
| go_emotions_noul | sentiment | noul | google-research-datasets/go_emotions simplified |
15,413 | 0.903 | 0.241 | 0.037 | 0.821 | 0.400 | 0.031 |
| imdb | sentiment | choice | stanfordnlp/imdb | 24,801 | 0.961 | 0.127 | 0.038 | 0.964 | 0.194 | 0.018 |
| sst2 | sentiment | choice | stanfordnlp/sst2 | 3,338 | 0.941 | 0.170 | 0.037 | 0.931 | 0.222 | 0.016 |
| sparp | spatial | choice | UKPLab/sparp SpaRP-PS1 (SpaRTUN) |
1,557 | 0.972 | 0.090 | 0.014 | 0.681 | 1.323 | 0.051 |
| spartqa_mchoice | spatial | choice | tasksource/spartqa-mchoice | 3,537 | 0.962 | 0.117 | 0.013 | 0.550 | 1.151 | 0.119 |
| spartqa_yn | spatial | choice | tasksource/spartqa-yn | 3,894 | 0.889 | 0.316 | 0.056 | 0.559 | 0.928 | 0.073 |
| atis_slots | structured | choice | tuetschek/atis | 1,816 | 0.998 | 0.010 | 0.006 | 0.997 | 0.012 | 0.008 |
| cuad_spans | structured | choice | The-Atticus-Project/cuad | 259 | 1.000 | 0.003 | 0.002 | 1.000 | 0.000 | 0.000 |
| massive_slots | structured | choice | AmazonScience/massive en-US |
1,937 | 0.997 | 0.016 | 0.003 | 0.989 | 0.059 | 0.006 |
| massive_slots_ru | structured | choice | AmazonScience/massive ru-RU |
1,927 | 0.988 | 0.031 | 0.003 | 0.966 | 0.124 | 0.014 |
| multiwoz_state | structured | choice | budzianowski/multiwoz multiwoz |
3,856 | 0.996 | 0.018 | 0.007 | 0.990 | 0.069 | 0.004 |
| nerel_relations | structured | choice | nerel-ds/NEREL | 307 | 0.928 | 0.219 | 0.032 | 0.925 | 0.317 | 0.025 |
| sgd_state | structured | choice | google-research-datasets/dstc8-schema-guided-dialogue sgd |
9,234 | 0.993 | 0.023 | 0.006 | 0.988 | 0.048 | 0.011 |
| skillspan | structured | choice | kris927b/SkillSpan | 921 | 0.999 | 0.007 | 0.002 | 0.999 | 0.006 | 0.003 |
| totto_cells | structured | choice | GEM/totto | 4,882 | 0.996 | 0.013 | 0.002 | 0.999 | 0.002 | 0.001 |
| totto_statements | structured | choice | GEM/totto | 1,267 | 0.995 | 0.009 | 0.003 | 0.998 | 0.008 | 0.003 |
| beavertails | toxicity | noul | PKU-Alignment/BeaverTails | 14,163 | 0.747 | 0.510 | 0.059 | 0.714 | 0.685 | 0.143 |
| toxic_chat | toxicity | noul | lmsys/toxic-chat toxicchat0124 |
4,976 | 0.970 | 0.088 | 0.024 | 0.974 | 0.080 | 0.017 |
| tweet_hate | toxicity | choice | cardiffnlp/tweet_eval hate |
3,056 | 0.511 | 0.803 | 0.224 | 0.699 | 0.733 | 0.146 |
| tweet_offensive | toxicity | choice | cardiffnlp/tweet_eval offensive |
850 | 0.854 | 0.376 | 0.099 | 0.815 | 0.529 | 0.082 |
| halueval_dialogue | truthfulness | noul | pminervini/HaluEval dialogue |
994 | 0.921 | 0.209 | 0.048 | 0.665 | 0.633 | 0.166 |
| halueval_qa | truthfulness | noul | pminervini/HaluEval qa |
954 | 0.995 | 0.028 | 0.011 | 0.824 | 0.360 | 0.053 |
| halueval_summarization | truthfulness | noul | pminervini/HaluEval summarization |
970 | 0.999 | 0.009 | 0.002 | 0.708 | 0.579 | 0.093 |
Benchmarks (test)
| dataset | topic | type | source | n | accuracy | NLL | ECE | jev-1.13 acc | jev-1.13 NLL | jev-1.13 ECE |
|---|---|---|---|---|---|---|---|---|---|---|
| github_issue_type | bugs | choice | khanmu2003/issue-classification | 21,501 | 0.859 | 0.397 | 0.033 | 0.809 | 1.407 | 0.111 |
| open_decisions_choice | business | choice | jgeuter/OpenDecisions | 125 | 0.640 | 0.822 | 0.079 | 0.896 | 0.286 | 0.059 |
| open_decisions_noul | business | noul | jgeuter/OpenDecisions | 68 | 0.721 | 0.444 | 0.172 | 0.912 | 0.217 | 0.097 |
| hwu64 | classification | choice | DeepPavlov/hwu64 | 1,076 | 0.944 | 0.188 | 0.013 | 0.818 | 1.157 | 0.055 |
| jailbreak_classification | guardrails | noul | jackhhao/jailbreak-classification | 262 | 0.985 | 0.048 | 0.023 | 0.962 | 0.125 | 0.026 |
| ropes | knowledge | choice | allenai/ropes plain_text |
504 | 0.899 | 0.240 | 0.035 | 0.948 | 0.294 | 0.022 |
| ar_lsat | logic | choice | RuleReasoner/RuleCollection-32K | 1,513 | 0.247 | 1.666 | 0.089 | 0.463 | 1.316 | 0.063 |
| rublimp | nlu | choice | RussianNLP/RuBLiMP | 4,570 | 0.972 | 0.074 | 0.013 | 0.969 | 0.097 | 0.033 |
| logiqa | reasoning | choice | lucasmccabe/logiqa | 651 | 0.490 | 1.208 | 0.099 | 0.779 | 1.116 | 0.076 |
| nomiracl_choice_en | retrieval | choice | miracl/nomiracl english |
312 | 0.644 | 1.456 | 0.179 | 0.718 | 1.922 | 0.143 |
| nomiracl_choice_ru | retrieval | choice | miracl/nomiracl russian |
231 | 0.628 | 1.419 | 0.226 | 0.736 | 1.183 | 0.099 |
| nomiracl_en | retrieval | noul | miracl/nomiracl english |
313 | 0.885 | 0.272 | 0.042 | 0.875 | 0.313 | 0.053 |
| nomiracl_passage_en | retrieval | noul | miracl/nomiracl english |
3,160 | 0.872 | 0.308 | 0.026 | 0.858 | 0.343 | 0.050 |
| nomiracl_passage_ru | retrieval | noul | miracl/nomiracl russian |
2,358 | 0.869 | 0.308 | 0.019 | 0.891 | 0.260 | 0.020 |
| nomiracl_ru | retrieval | noul | miracl/nomiracl russian |
231 | 0.805 | 0.447 | 0.105 | 0.879 | 0.267 | 0.059 |
| tweet_sentiment | sentiment | choice | cardiffnlp/tweet_eval sentiment |
12,284 | 0.724 | 0.617 | 0.042 | 0.668 | 1.935 | 0.201 |
| stepgame | spatial | choice | tasksource/stepgame | 25,000 | 0.793 | 0.605 | 0.025 | 0.548 | 1.785 | 0.082 |
| webnlg_en | structured | choice | GEM/web_nlg en |
2,773 | 0.999 | 0.005 | 0.001 | 0.999 | 0.002 | 0.002 |
| russian_toxicity | toxicity | choice | textdetox/multilingual_toxicity_dataset | 243 | 0.979 | 0.080 | 0.042 | 0.963 | 0.084 | 0.015 |
| vitaminc | truthfulness | choice | tals/vitaminc | 25,000 | 0.865 | 0.374 | 0.029 | 0.815 | 0.860 | 0.072 |
- Downloads last month
- -
Model tree for ITLL/krasivo-jev-bf16
Base model
Qwen/Qwen3.5-2B-Base



