README.md

---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-classification
base_model: Qwen/Qwen3-Reranker-8B
base_model_relation: finetune
datasets:
- oklenAI/MathPinpoint
tags:
- math
- retrieval
- reranker
- relevance
- llm-as-a-judge
- data-labeling
- distillation
---

# MathPinpoint-Judge-8B

A relevance judge for math retrieval. Give it a math question and a candidate document; it returns a score from 0 to 2 for **whether the document answers that exact problem, correctly**.

Use it to label (question, document) pairs at a scale where an LLM judge is too slow or too expensive:

- **Training data for a math retriever.** Retrieve candidates for your questions with any retriever, score them, keep high scores as positives and low scores as hard negatives.
- **Reranking.** Order one question's candidates by score.
- **Triage.** Accept the confident scores and send only the uncertain middle to an LLM or a human.

It is GPT-5.6-Sol's judgment distilled into Qwen3-Reranker-8B. On retrieval candidates, a 2 from this model is a full answer for the LLM 83.5% of the time, and a 0 hides a full answer 0.5% of the time.

## What "relevant" means here

The labels follow one strict rubric ([`judge_prompt.md`](https://huggingface.co/datasets/oklenAI/MathPinpoint/blob/main/prompts/judge_prompt.md)), which scores usefulness toward a *correct* answer to *the same* problem:

- **2, highly relevant:** directly and substantially answers the same mathematical need, with correct central reasoning and conclusion.
- **1, partially relevant:** substantive, mathematically correct material that advances the same problem but leaves important work, for example a correct lemma, method or solved subcase. Topical overlap alone does not qualify.
- **0, not relevant or unusable:** a different problem or changed conditions, keyword overlap only, no answer, a major mathematical error, a correct-looking conclusion with no reliable basis, or mathematics that cannot be verified.

**This is problem-level relevance, not topical relevance.** A page that solves the same kind of problem with different numbers gets a 0. If you want documents on the same topic, or that use the same theorem or technique, this is the wrong model.

## Quick start

```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

name = "oklenAI/MathPinpoint-Judge-8B"
device = "cuda" if torch.cuda.is_available() else "cpu"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name, dtype=torch.bfloat16).to(device).eval()


@torch.no_grad()
def score(queries, docs, batch_size=8):
    """Raw scores on the 0-2 scale, one per (query, document) pair."""
    scores = []
    for i in range(0, len(queries), batch_size):
        q = queries[i:i + batch_size]
        d = [doc[:32000] for doc in docs[i:i + batch_size]]  # the model saw at most 32,000 characters
        enc = tok(q, d, truncation=True, max_length=8192, padding=True, return_tensors="pt").to(device)
        scores += model(**enc).logits.squeeze(-1).float().tolist()
    return scores


def score_many(queries, docs, batch_size=8):
    """Same scores as score() up to bfloat16 noise, faster on large inputs: batches pairs of similar length."""
    order = sorted(range(len(queries)), key=lambda i: len(queries[i]) + len(docs[i][:32000]))
    s = score([queries[i] for i in order], [docs[i] for i in order], batch_size)
    out = [0.0] * len(queries)
    for i, v in zip(order, s):
        out[i] = v
    return out


def label(s):
    """Default cuts: below 0.5 is a 0, 1.5 and above is a 2."""
    return 0 if s < 0.5 else (1 if s < 1.5 else 2)


query = "What is the remainder when 2^100 is divided by 7?"
same_problem = ("Powers of 2 modulo 7 repeat with period 3: 2^1 = 2, 2^2 = 4, 2^3 = 8 = 1 (mod 7). "
                "Since 100 = 3*33 + 1, 2^100 = (2^3)^33 * 2 = 2 (mod 7). "
                "So the remainder when 2^100 is divided by 7 is 2.")
changed_condition = ("Powers of 2 modulo 5 repeat with period 4: 2^1 = 2, 2^2 = 4, 2^3 = 3, 2^4 = 1 (mod 5). "
                     "Since 100 = 4*25, 2^100 = (2^4)^25 = 1 (mod 5). "
                     "So the remainder when 2^100 is divided by 5 is 1.")
s = score([query, query], [same_problem, changed_condition])
print([round(x, 2) for x in s], [label(x) for x in s])  # roughly [2.14, 0.31] [2, 0]
```

The second document uses the same method on the same kind of problem, but changes a condition, so it scores near 0.

## Using it on your data

**Input format.** It has to match training exactly:

- The question comes first and the document second, joined with no separator and no template. The tokenizer's pair encoding does exactly that. Do not add `Query:` / `Document:` markers or the base model's yes/no chat template; the scores will be off.
- The document is cut to its first 32,000 characters and the pair to 8,192 tokens. The tokenizer's `model_max_length` is 8,192, so `truncation=True` alone already stops there.
- Right padding with `<|endoftext|>`, which the tokenizer sets by default.

**What it was trained on.** Questions are self-contained math problems in English, mostly copied from questions that people posted on web pages. Documents are English math web pages cleaned to their mathematical content, with formulas as LaTeX or plain text. Keyword-style queries and other languages are outside that distribution; check a sample of your own data before relying on it there.

**Choosing thresholds.** The model regresses onto the 0 / 1 / 2 labels, so its scores pull toward the middle: a third of the LLM's 2s come out between 0.5 and 1.5. The default cuts make precise but incomplete positives and safe negatives. If recall matters, lower the positive threshold, ideally after labeling 100–200 of your own pairs with a stronger judge to see where your data falls.

**Compare within a question.** Most of the label variance lies between questions. The model is most reliable at ordering the candidates of one question; the absolute score of a pair is less comparable across questions.

**Speed.** On one A100-80GB with the default SDPA attention, `score_many` scored 2,000 held-out pairs averaging 3,251 tokens in 18 minutes: 1.8 pairs per second, about 6,000 tokens per second, 21 GiB peak memory. That is roughly 160,000 such pairs per GPU-day; shorter documents go faster. With FlashAttention-2 installed (`attn_implementation="flash_attention_2"`), padded batches ran about 1.5× faster in our runs.

## How well it agrees with an LLM judge

Every number below measures agreement with GPT-5.6-Sol, not with human annotators.

**Held-out pairs:** 31,830 pairs over 6,000 questions, disjoint by question from training. The LLM labeled 41.4% of them 0, 37.0% 1 and 21.6% 2.

| | |
|---|---:|
| Within-question R² [95% CI] | **0.364** [0.349, 0.380] |
| Within-question Spearman, median over questions | 0.671 |
| Exact agreement with the LLM label (cuts 0.5 / 1.5) | **73.2%** |
| LLM 2 scored as 2 | 63.3% |
| 0 ↔ 2 flips | 1.11% |
| AUC, LLM 2 vs LLM 0 (pooled) | 0.971 |
| AUC, LLM ≥ 1 vs LLM 0 (pooled) | 0.901 |

| LLM label | model → 0 | model → 1 | model → 2 |
|---|---:|---:|---:|
| 0 | 75.4% | 23.1% | 1.4% |
| 1 | 17.1% | 76.5% | 6.4% |
| 2 | 2.4% | 34.3% | 63.3% |

**Retrieval candidates it labeled in production:** 3,600 pairs sampled from three kinds of candidates, 400 for each kind and model label, judged by GPT-5.6-Sol on the document's first 30,000 characters and weighted back to the kinds' sizes. The 23 pairs the LLM judged unevaluable are left out.

| | Dense retriever, ranks 6–20 | Promoted by fusion, short documents | Promoted by fusion, long documents | All three |
|---|---:|---:|---:|---:|
| Exact agreement with the LLM label | 78.1% | 76.3% | 79.2% | 77.6% |
| LLM 2 scored as 2 | 56.6% | 73.6% | 70.6% | 65.0% |
| 0 ↔ 2 flips | 0.41% | 0.75% | 0.56% | 0.52% |
| Model 2 that the LLM also gives 2 | 83.2% | 83.7% | 83.1% | **83.5%** |
| Model 0 that the LLM gives 2 | 0.5% | 0.5% | 0.5% | **0.5%** |

"Promoted by fusion" means candidates that fusing the dense, BM25 and word 5-gram rankings brought into the top 20 although the dense retriever alone ranked them lower.

## How it was used

It labeled 24,105,846 of the 43,506,169 training judgments of [MathPinpoint](https://huggingface.co/datasets/oklenAI/MathPinpoint), a dataset for problem-level math retrieval. There the LLM judged the candidates that mattered most (each question's top 5 dense candidates, a reranker's top 5 among the next 15, and long documents), and this model judged the rest. Its raw score is in the dataset's `rm8b_score` column wherever it scored a pair, so you can check that your setup reproduces it. Run with the code above on one GPU, 1,981 of 2,000 held-out pairs got the same label as this checkpoint's training-time predictions (median score difference 0.008).

## Training

- **Base:** [Qwen/Qwen3-Reranker-8B](https://huggingface.co/Qwen/Qwen3-Reranker-8B), with a new single-output regression head (`Qwen3ForSequenceClassification`, `num_labels=1`).
- **Objective:** mean squared error against GPT-5.6-Sol's 0 / 1 / 2 labels.
- **Data:** a pool of 9,777,827 LLM-labeled pairs over 1,843,779 questions: dense-retrieval candidates at ranks 6–20 that a zero-shot reranker picked as its top 5, and long documents from the same ranks.
- **Input:** the document cut to 32,000 characters, the pair truncated to 8,192 tokens, as above.
- **Optimization:** AdamW, peak learning rate 6e-6, 3% warmup then cosine decay, weight decay 0.01, gradient clipping at 1.0, bfloat16, FlashAttention-2, gradient checkpointing. Micro-batches were packed to at most 16,384 padded tokens (up to 64 pairs, 4.8 on average) with 3 accumulation steps on 8× A100-80GB, about 114 pairs per optimizer step and 25 seconds per step.
- **Steps:** 10,000, in three segments: 1–2,500, 2,501–5,000 and 5,001–10,000. Each segment started from the previous weights with a fresh optimizer and its own cosine schedule. The batch order is seeded and each segment rebuilt it from the start, so the batches of the first 2,500 steps were seen three times and those of the next 2,500 once: this checkpoint has seen about 570,000 distinct pairs of the pool.
- **Why this checkpoint:** training continued to 15,000 steps and plateaued (within-question R² 0.373, agreement 73.7%). The 10,000-step checkpoint is the one that labeled MathPinpoint.
- A Qwen3-Reranker-0.6B trained with the same data and configuration reached a within-question R² of 0.140.

## Limitations

- **A student of one LLM judge.** It learned GPT-5.6-Sol's judgments, including that judge's leniency at the 0 / 1 boundary. Score 1 is the least stable label even between two LLM judges; treat it as "partially relevant, uncertain".
- **Correctness is part of the label, but not separately tested.** The rubric gives 0 to a document with a major mathematical error. How often the model catches such errors, as opposed to judging whether the problem matches, has not been measured.
- **Reads only the beginning of long documents:** the first 32,000 characters and at most 8,192 tokens of the pair. An answer deep inside a long page may be missed.
- **Domain.** English math web text with questions written as math problems. Other languages, other kinds of documents and other notions of relevance are untested.

## Citation

```bibtex
@misc{mathpinpoint2026,
  title     = {{MathPinpoint}: Training Data and a Test Set for Problem-Level Math Retrieval},
  author    = {Cai, Zefeng and Shippole, Enrico},
  year      = {2026},
  note      = {Version 2.0},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/oklenAI/MathPinpoint}
}
```

## License

Apache-2.0, as for the base model.