Nefelim584/jev-lite-stage-a

jev-lite (Stage A)

jev-lite is a typed decision model. It takes a state (free text, a JSON object, or a list of texts) and one or more typed questions:

Type	Question	Output
bool	a statement about the state	P(true) + confidence
choice	a prompt + 1–32 options	distribution over the options + argmax + confidence
score	a prompt + 1–32 ordered levels	distribution over the levels + argmax + expected level + confidence

It does not generate text. The answer is always one of the caller's own candidates, so the output always fits the schema. The model is meant to be a fast, cheap replacement for an LLM-as-a-judge on checks such as faithfulness, answerability and rubric scoring.

This checkpoint is Stage A: run stage_a-full-s42, trained only on public data. Not affiliated with TypeSafe AI.

Code: github.com/Nefelim584/JEV_Judge

Model details

• Base: answerdotai/ModernBERT-large (395M), full fine-tuning.
• Architecture: packed encoding. Each question is one sequence (state + question + candidates), with one [MASK] marker per candidate. The marker hidden states go through a shared 1024 → 256 trunk and three heads (Choice / Score / Bool).
• Max length: 512 tokens. When the state is longer, it is truncated head + tail (50/50), and the response sets meta.truncated.
• Confidence: v1 heuristic 1 − H(p) / log K. No temperature scaling and no learned confidence head yet: the probabilities are raw.
• Language: English only.

Usage

The heads are custom, so the checkpoint is loaded with the jevlite package from the code repository.

from jevlite.checkpoint import load_final
from jevlite.encoding import text_encoder_from_config
from jevlite.inference import Predictor
from jevlite.schema import Request

model, tokenizer, cfg = load_final("path/to/final", device="cuda")
predictor = Predictor(model, text_encoder_from_config(cfg, tokenizer))

request = Request(
    state="The Eiffel Tower was completed in 1889 for the World's Fair in Paris.",
    questions=[
        {"id": "faithful", "type": "bool",
         "prompt": "The text states that the Eiffel Tower was finished in 1889."},
        {"id": "topic", "type": "choice", "prompt": "What is the text about?",
         "options": ["architecture", "sports", "finance"]},
        {"id": "clarity", "type": "score", "prompt": "How clear is the text?",
         "levels": ["0", "1", "2", "3", "4"]},
    ],
)
response = predictor.predict(request)   # all questions in one forward pass
print(response.answers["faithful"].probability)

Training data

The stage_a mix contains 25 public English datasets, with 300k records per epoch (Bool 50% / Choice 20% / Score 30%) over 2 epochs. Choice options are shuffled every epoch. No dataset under a non-commercial license was used. Licenses were checked on 2026-10-02 against the HF Hub cards and the original sources.

Dataset	Use	License
BoolQ	Bool	CC BY-SA 3.0
MultiNLI	Bool / Choice	OANC; CC BY-SA 3.0 / CC BY 3.0 / PD (fiction)
SNLI	Bool / Choice	CC BY-SA 4.0
WANLI	Bool / Choice	CC BY 4.0 (pairs generated by GPT-3, labeled by humans)
QNLI	Bool	CC BY-SA 4.0 (from SQuAD 1.1)
PAWS	Bool	free use (repo LICENSE)
FEVER	Bool / Choice	CC BY-SA 3.0
VitaminC	Bool / Choice	CC BY-SA 3.0
HoVer	Bool	CC BY-SA 4.0
TabFact	Bool	CC BY 4.0
Natural Questions	Bool	CC BY-SA 3.0
SQuAD 2.0	Bool	CC BY-SA 4.0
ClapNQ	Bool	Apache-2.0 (NQ text CC BY-SA 3.0)
ARC	Choice	CC BY-SA 4.0
OpenBookQA	Choice	Apache-2.0
CommonsenseQA	Choice	MIT
Cosmos QA	Choice	CC BY 4.0
Social IQa	Choice	CC BY 4.0
CLINC150	Choice	CC BY 3.0
MASSIVE (en-US)	Choice	CC BY 4.0
DBpedia-14	Choice	CC BY-SA 3.0
GoEmotions	Choice	Apache-2.0
summarize_from_feedback	Choice	modified MIT (labels); CC BY 4.0 (posts)
HelpSteer2	Score	CC BY 4.0
HelpSteer3	Score / Choice	CC BY 4.0

Evaluation only (never trained on): ContractNLI (CC BY 4.0 from the official source), BANKING77 (CC BY 4.0), MultiNLI mismatched genres, MMLU (MIT), FaithEval (research use only).

License note. The weights are released under Apache-2.0. Some training datasets are licensed under CC BY-SA; they were used as natural-language training text with human labels (Wikipedia passages, image captions, exam questions). The model outputs only probabilities over caller-supplied candidates and does not generate text. All datasets are attributed in the table above.

Training procedure

Setting	Value
Hardware	2× T4 (Kaggle), DDP, fp16 AMP, SDPA, gradient checkpointing
Optimizer	8-bit AdamW, lr 3e-5, warmup 6%, linear decay, weight decay 0.01, grad clip 1.0
Batch	16 per GPU, effective 32, length-grouped
Steps	18,750 (2 epochs), ~9.5 h
Loss	CE (Choice), CE + 0.5·EMD² (Score), BCE + 0.1·Brier (Bool)
Seed	42

Evaluation

Baselines are zero-shot: an NLI cross-encoder and Laya. Test sets are cleaned: records whose passage appears in the training data are removed, and so are records used for mid-run eval. Significance uses paired bootstrap + McNemar with Holm correction.

Accuracy (ECE): NLI / Laya / jev-lite

	test_in (19,272)	test_ood_template (17,302)	test_ood_source (11,883)
Bool	0.763 / 0.723 / 0.882 (ECE 0.031)	0.569 / 0.770 / 0.902 (0.026)	0.542 / 0.519 / 0.603 (0.295)
Choice	0.595 / 0.596 / 0.803 (0.040)	0.617 / 0.622 / 0.794 (0.052)	0.558 / 0.585 / 0.635 (0.114)
Score	0.242 / 0.309 / 0.620	0.234 / 0.315 / 0.591	—

• test_in: trained sources and templates. test_ood_template: trained sources, held-out question templates. test_ood_source: sources never seen in training.
• Across 61 source × split cells: vs NLI 43 wins / 16 ties / 2 losses; vs Laya 52 / 9 / 0.
• Faithfulness: HoVer (multi-hop) 0.866 vs 0.707 (NLI). Accuracy holds across 2–4 hops, and "not supported" is caught 84% of the time.
• Answerability (excluding ClapNQ): 0.907 vs 0.746 (Laya).
• Score: Spearman 0.67, MAE 0.59 levels (Laya: 0.22 / 1.03).
• Selective prediction: answering only the 80% most confident Bool questions gives 0.945 accuracy (AURC 0.030 vs 0.147 for Laya).
• Robustness: the argmax changes under option permutation in 2.8% of cases (Laya 12.4%). With an empty state, the model answers "no" for every source, so there is no claim-only shortcut.
• Latency: median 73 ms per request (p90 152 ms) on an Apple M1 Pro GPU.

Limitations

• Weak transfer to new domains. On unseen sources the lead over the baselines falls to 5–6 points, and there are two significant losses to NLI: ContractNLI Bool (−4.8) and FaithEval counterfactual (−6.1).
• The model is too strict on new domains. On ContractNLI it ranks well (AUC 0.931 vs 0.884 for NLI), but its probabilities are shifted down, so it calls half of the supported claims unsupported. Temperature scaling cannot fix this; a bias or threshold calibrated on in-domain labels is needed.
• Calibration breaks out of domain. Bool ECE is 0.031 in-domain but 0.295 on new sources.
• Refusal detection does not transfer. On FaithEval-unanswerable (answer removed from the context) accuracy is 0.49.
• Table shortcut. On TabFact (0.71) the model often says "yes" even for a mismatched table (54%). Part of its TabFact score is not real table reading.
• ClapNQ is a data artifact. It scores 1.000 because the two classes come from different passage pools; it is excluded from the headline numbers.
• Weak on knowledge-heavy multiple-choice QA (MMLU 0.38, ARC 0.63). This is expected from a 395M encoder.
• Context limit of 512 tokens; longer states are truncated.
• Single run, single seed. No temperatures and no learned confidence yet.
• Public proxies only. Quality as a judge on a real production task has not been measured yet.

Intended use

• Fast, schema-constrained checks over text: faithfulness of a claim to a context, answerability, classification into a fixed label set, and rubric scores.
• A first stage in an LLM-judge cascade: answer when confident, escalate otherwise.

Not intended for: final decisions on new domains without in-domain calibration, factual-knowledge QA, long documents (> 512 tokens), or languages other than English.

Citation

TODO