README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-8B
library_name: peft
pipeline_tag: text-classification
tags:
- rubric-scoring
- llm-as-a-judge
- evaluation
- lora
- contrastive
---

# RubriX-8B

RubriX is a **non-autoregressive rubric judge**. Given a criterion, the rubric's level descriptions and a response
(optionally with the conversation it answers), it returns a probability for every level in a single pass, with no
text generation. Level descriptions are embedded once per rubric and cached, so the cost of each judgment does not
grow with the number or length of the levels.

Read more in the blog post: [Introducing RubriX](https://cobuildx.ai/blog/rubrix-rubric-judge).

## How it works

One encoder, Qwen3-8B with this repository's LoRA adapter, embeds the input and each level description separately.
Two projection heads map them into a shared space, and

- `s_k = τ · cos(f(input), g(level_k))`, with τ = 100
- `p = softmax(s_1 … s_K)`

The model was trained with cross-entropy over each rubric's own levels plus an adaptively weighted ranked probability
score, so that near-miss levels cost less than distant ones.

## Usage

```python
# pip install torch transformers peft safetensors huggingface_hub
from transformers.dynamic_module_utils import get_class_from_dynamic_module

RubriX = get_class_from_dynamic_module("rubrix.RubriX", "CobuildX-AI/RubriX-8B")
judge = RubriX.from_pretrained("CobuildX-AI/RubriX-8B")    # GPU with ~20 GB recommended (bf16)

result = judge.score(
    criterion="Does the response answer the question correctly and completely?",
    levels=["Wrong or unrelated.", "Partly correct.", "Correct but incomplete.", "Correct and complete."],
    response="Paris is the capital of France.",
    conversation="What is the capital of France?",
)
print(result)
# {'probabilities': [...], 'level': 3, 'expected_level': ...}   levels are 0-based, lowest first
```

- `score(criterion, levels, response, conversation=None, system_prompt=None, reference_answer=None)` scores one response.
- `score_batch([{...}, ...])` scores many; level vectors are cached across calls (`clear_cache()` resets them).
- `levels` must be ordered from lowest to highest. Inputs are truncated at 32,768 tokens (Qwen3-8B's native context); RubriX was trained on inputs of up to 8,192 tokens, so longer ones are outside what it was trained on. Pass `max_tokens=` to `from_pretrained` to change the cap. Memory grows with input length, so lower `batch_size` for very long inputs.

The base model is loaded from `Qwen/Qwen3-8B` at the pinned revision it was trained on
(`b968826d9c46dd6066d109eabc6255188de91218`). The adapter is merged at load time by default (`merge=False` keeps
it separate). This module reproduces the evaluated model: on 60 benchmark judgments, its probabilities differ from
the recorded evaluation by at most 0.02 per level (bf16 numerics), and the predicted level matches in all 60.

## Files

| File | Contents |
|---|---|
| `adapter/` | LoRA adapter (rank 16, α 32, all attention and MLP projections) for `Qwen3Model` |
| `heads.safetensors` | the two projection heads (4096 → 1536 → 1536 → 512, GELU, LayerNorm) and the temperature |
| `rubrix_config.json` | base model, revision, head configuration and pooling |
| `rubrix.py` | loading and scoring code |

No training data is included.

## Results

Exact accuracy (%) against human labels; every judge scored the same judgments.

| Set | RubriX | Jev | Qwen3-8B judge |
|---|---|---|---|
| Held-out rubrics (6,749 judgments, criteria not in training) | **75.4** | 57.1 | 49.8 |
| HealthBench | 68.7 | 73.5 | 64.5 |
| BiGGen | 37.8 | 43.0 | 40.7 |
| ELLIPSE | 30.8 | 32.0 | 47.8 |
| SummEval | 32.8 | 44.4 | 12.5 |
| JudgmentBench, binary | 66.6 | 80.7 | 76.8 |
| JudgmentBench, multi-level | 46.2 | 37.5 | 86.3 |
| QAGS-CNN | 80.4 | 82.1 | 78.7 |
| QAGS-XSUM | 75.7 | 78.7 | 78.2 |

Ranking correlation: summary-level Spearman ρ within groups of responses to the same input, averaged per category.

| Benchmark | RubriX | Jev | Qwen3-8B judge |
|---|---|---|---|
| SummEval | 0.483 | 0.541 | 0.428 |
| QAGS | 0.620 | 0.684 | 0.616 |
| ELLIPSE | 0.515 | 0.601 | 0.485 |
| HealthBench | 0.320 | 0.441 | 0.273 |
| BiGGen | 0.434 | 0.526 | 0.527 |
| JudgmentBench | 0.045 | 0.068 | 0.002 |

Latency, one request at a time on an H100 with cached level embeddings: 27.5–28.3 ms per
judgment from 2 to 100 levels of 1,500 tokens and from 10 to 3,000 tokens per level (5 levels). At 3,000 tokens per
level that is 7.7× faster than Jev (approximate network time
subtracted) and 18.8× faster than a Qwen3-8B judge.

## Training

34,261 examples covering 2,284 criteria, at most 25 per criterion: public
rubric datasets (Feedback Collection, HelpSteer, HelpSteer2, HelpSteer3, UltraFeedback, ASAP 2.0) and a proprietary
dataset. One epoch on one H100 (2.8 hours); AdamW, learning rates 1e-4 (adapter)
and 5e-4 (heads), effective batch 32, 5% warm-up then linear decay.

## Limitations

- The input and the level descriptions are encoded separately, so verdicts that depend on checking a specific fact
  against a level are harder than for a judge that reads both together.
- Strongest on rubrics like those it was trained on; on unfamiliar benchmark formats it trails Jev on most sets.
  Fine-tuning on in-domain rubrics is the intended way to adapt it.
- 8B encoder with small heads; a second training epoch did not improve validation loss.