RubriX-8B

RubriX is a non-autoregressive rubric judge. Given a criterion, the rubric's level descriptions and a response (optionally with the conversation it answers), it returns a probability for every level in a single pass, with no text generation. Level descriptions are embedded once per rubric and cached, so the cost of each judgment does not grow with the number or length of the levels.

Read more in the blog post: Introducing RubriX.

How it works

One encoder, Qwen3-8B with this repository's LoRA adapter, embeds the input and each level description separately. Two projection heads map them into a shared space, and

• s_k = τ · cos(f(input), g(level_k)), with τ = 100
• p = softmax(s_1 … s_K)

The model was trained with cross-entropy over each rubric's own levels plus an adaptively weighted ranked probability score, so that near-miss levels cost less than distant ones.

Usage

# pip install torch transformers peft safetensors huggingface_hub
from transformers.dynamic_module_utils import get_class_from_dynamic_module

RubriX = get_class_from_dynamic_module("rubrix.RubriX", "CobuildX-AI/RubriX-8B")
judge = RubriX.from_pretrained("CobuildX-AI/RubriX-8B")    # GPU with ~20 GB recommended (bf16)

result = judge.score(
    criterion="Does the response answer the question correctly and completely?",
    levels=["Wrong or unrelated.", "Partly correct.", "Correct but incomplete.", "Correct and complete."],
    response="Paris is the capital of France.",
    conversation="What is the capital of France?",
)
print(result)
# {'probabilities': [...], 'level': 3, 'expected_level': ...}   levels are 0-based, lowest first

• score(criterion, levels, response, conversation=None, system_prompt=None, reference_answer=None) scores one response.
• score_batch([{...}, ...]) scores many; level vectors are cached across calls (clear_cache() resets them).
• levels must be ordered from lowest to highest. Inputs are truncated at 32,768 tokens (Qwen3-8B's native context); RubriX was trained on inputs of up to 8,192 tokens, so longer ones are outside what it was trained on. Pass max_tokens= to from_pretrained to change the cap. Memory grows with input length, so lower batch_size for very long inputs.

The base model is loaded from Qwen/Qwen3-8B at the pinned revision it was trained on (b968826d9c46dd6066d109eabc6255188de91218). The adapter is merged at load time by default (merge=False keeps it separate). This module reproduces the evaluated model: on 60 benchmark judgments, its probabilities differ from the recorded evaluation by at most 0.02 per level (bf16 numerics), and the predicted level matches in all 60.

Files

File	Contents
adapter/	LoRA adapter (rank 16, α 32, all attention and MLP projections) for Qwen3Model
heads.safetensors	the two projection heads (4096 → 1536 → 1536 → 512, GELU, LayerNorm) and the temperature
rubrix_config.json	base model, revision, head configuration and pooling
rubrix.py	loading and scoring code

No training data is included.

Results

Exact accuracy (%) against human labels; every judge scored the same judgments.

Set	RubriX	Jev	Qwen3-8B judge
Held-out rubrics (6,749 judgments, criteria not in training)	75.4	57.1	49.8
HealthBench	68.7	73.5	64.5
BiGGen	37.8	43.0	40.7
ELLIPSE	30.8	32.0	47.8
SummEval	32.8	44.4	12.5
JudgmentBench, binary	66.6	80.7	76.8
JudgmentBench, multi-level	46.2	37.5	86.3
QAGS-CNN	80.4	82.1	78.7
QAGS-XSUM	75.7	78.7	78.2

Ranking correlation: summary-level Spearman ρ within groups of responses to the same input, averaged per category.

Benchmark	RubriX	Jev	Qwen3-8B judge
SummEval	0.483	0.541	0.428
QAGS	0.620	0.684	0.616
ELLIPSE	0.515	0.601	0.485
HealthBench	0.320	0.441	0.273
BiGGen	0.434	0.526	0.527
JudgmentBench	0.045	0.068	0.002

Latency, one request at a time on an H100 with cached level embeddings: 27.5–28.3 ms per judgment from 2 to 100 levels of 1,500 tokens and from 10 to 3,000 tokens per level (5 levels). At 3,000 tokens per level that is 7.7× faster than Jev (approximate network time subtracted) and 18.8× faster than a Qwen3-8B judge.

Training

34,261 examples covering 2,284 criteria, at most 25 per criterion: public rubric datasets (Feedback Collection, HelpSteer, HelpSteer2, HelpSteer3, UltraFeedback, ASAP 2.0) and a proprietary dataset. One epoch on one H100 (2.8 hours); AdamW, learning rates 1e-4 (adapter) and 5e-4 (heads), effective batch 32, 5% warm-up then linear decay.

Limitations

• The input and the level descriptions are encoded separately, so verdicts that depend on checking a specific fact against a level are harder than for a judge that reads both together.
• Strongest on rubrics like those it was trained on; on unfamiliar benchmark formats it trails Jev on most sets. Fine-tuning on in-domain rubrics is the intended way to adapt it.
• 8B encoder with small heads; a second training epoch did not improve validation loss.