Blame for README.md

---
language:
- en
pipeline_tag: text-classification
library_name: transformers
tags:
- prm
- process-reward-model
- reward model
- math
- hallucination-detection
license: apache-2.0
base_model:
- Qwen/Qwen2.5-Math-7B-Instruct
datasets:
- ZaandaTeika/SHARP
---

# Qwen2.5-7B-SHARP-Step

## Introduction

**Qwen2.5-7B-SHARP-Step** is a Process Reward Model (PRM) for step-level hallucination detection in mathematical
reasoning. It scores every intermediate step of a solution and flags the ones that are unsupported or wrong,
which makes it usable both for error localization and for Best-of-N reranking.

The model is trained from [Qwen/Qwen2.5-Math-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Math-7B-Instruct) on the
[SHARP](https://huggingface.co/datasets/ZaandaTeika/SHARP) corpus. A `<extra_0>` marker is appended to every step,
and a two-way classification head predicts, at each marker, whether the step is correct.

This is the **Step** variant: it is supervised with step-level labels, where every reasoning step of the
solution carries its own correct / hallucinated label.
The companion **Span** variant, supervised with span-level hallucination annotations, is
[Qwen2.5-7B-SHARP-Span](https://huggingface.co/ZaandaTeika/Qwen2.5-7B-SHARP-Span).

## Model Details

| | |
|---|---|
| Base model | [Qwen/Qwen2.5-Math-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Math-7B-Instruct) |
| Architecture | `Qwen2ForProcessRewardModel` (custom code in `modeling_qwen2_rm.py`) |
| Reward head | `Linear(hidden, hidden) -> ReLU -> Linear(hidden, 2)` on top of every token |
| Hidden size / layers | 3584 / 28 |
| Max position embeddings | 4096 |
| Step marker | `<extra_0>` (single special token) |
| Supervision | step-level labels over the reasoning steps of the solution |
| Weights dtype | bfloat16 |

## Requirements

* `transformers>=4.40.0`. The latest version is recommended.
* `trust_remote_code=True` -- the PRM class ships with the checkpoint.

## Quick Start

> [!Important]
>
> **Qwen2.5-7B-SHARP-Step** is a process reward model used for scoring reasoning steps, not for generation.

### Prerequisites

- **Step separation**: split the solution into steps and join them with double line breaks (`"\n\n"`).
- **Step marker**: append `<extra_0>` to the end of every step.
- **Prompt format**: the model does not use a chat template. Wrap the input as
  `Question: {question}\n\nSolution:\n{steps}`, which is the format used during training.
- **Reward computation**: at each `<extra_0>` position take the probability of the positive class. The result is a
  value between 0 and 1, where low values mark a hallucinated or incorrect step.

### Hugging Face Transformers

```python
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer


def build_prompt(question, steps):
    body = "\n\n".join(f"{step.strip()}<extra_0>" for step in steps)
    return f"Question: {question}\n\nSolution:\n{body}"


def make_step_rewards(logits, token_masks):
    probabilities = F.softmax(logits, dim=-1)
    probabilities = probabilities * token_masks.unsqueeze(-1)  # bs, seq_len, num_labels

    all_scores_res = []
    for i in range(probabilities.size(0)):
        sample = probabilities[i]  # seq_len, num_labels
        positive_probs = sample[sample != 0].view(-1, 2)[:, 1]  # valid_tokens, num_labels
        all_scores_res.append(positive_probs.cpu().tolist())
    return all_scores_res


model_name = "ZaandaTeika/Qwen2.5-7B-SHARP-Step"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
).eval()

data = {
    "question": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?",
    "steps": [
        "In April, Natalia sold 48 clips.",
        "In May she sold half as many, so she sold 48 / 2 = 24 clips.",
        "Altogether she sold 48 + 24 = 72 clips. The answer is \\boxed{72}.",
    ],
}

prompt = build_prompt(data["question"], data["steps"])
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model(input_ids=input_ids)

step_sep_id = tokenizer.encode("<extra_0>", add_special_tokens=False)[0]
token_masks = input_ids == step_sep_id
step_reward = make_step_rewards(outputs[0], token_masks)
print(step_reward)  # one score per step
```

## Source Model and Attribution

This checkpoint is a fine-tuned derivative of
[Qwen/Qwen2.5-Math-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Math-7B-Instruct),
released under the [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) license.
The weights were modified: the language modelling head was replaced with a two-way
process reward head, and the model was further trained on span-derived step-level
supervision.

Training data comes from the [SHARP](https://huggingface.co/datasets/ZaandaTeika/SHARP)
corpus, whose reasoning traces and hallucination annotations are licensed under
CC BY 4.0. SHARP builds on GSM8K and MATH (both MIT); see the dataset card for the
full source attribution.

## Additional Information

### Licensing Information

This checkpoint is released under the
[Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) license, following its
base model.