| --- |
| language: |
| - en |
| pipeline_tag: text-classification |
| library_name: transformers |
| tags: |
| - prm |
| - process-reward-model |
| - reward model |
| - math |
| - hallucination-detection |
| license: apache-2.0 |
| base_model: |
| - Qwen/Qwen2.5-Math-7B-Instruct |
| datasets: |
| - ZaandaTeika/SHARP |
| --- |
| |
| # Qwen2.5-7B-SHARP-Step |
|
|
| ## Introduction |
|
|
| **Qwen2.5-7B-SHARP-Step** is a Process Reward Model (PRM) for step-level hallucination detection in mathematical |
| reasoning. It scores every intermediate step of a solution and flags the ones that are unsupported or wrong, |
| which makes it usable both for error localization and for Best-of-N reranking. |
|
|
| The model is trained from [Qwen/Qwen2.5-Math-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Math-7B-Instruct) on the |
| [SHARP](https://huggingface.co/datasets/ZaandaTeika/SHARP) corpus. A `<extra_0>` marker is appended to every step, |
| and a two-way classification head predicts, at each marker, whether the step is correct. |
|
|
| This is the **Step** variant: it is supervised with step-level labels, where every reasoning step of the |
| solution carries its own correct / hallucinated label. |
| The companion **Span** variant, supervised with span-level hallucination annotations, is |
| [Qwen2.5-7B-SHARP-Span](https://huggingface.co/ZaandaTeika/Qwen2.5-7B-SHARP-Span). |
|
|
| ## Model Details |
|
|
| | | | |
| |---|---| |
| | Base model | [Qwen/Qwen2.5-Math-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Math-7B-Instruct) | |
| | Architecture | `Qwen2ForProcessRewardModel` (custom code in `modeling_qwen2_rm.py`) | |
| | Reward head | `Linear(hidden, hidden) -> ReLU -> Linear(hidden, 2)` on top of every token | |
| | Hidden size / layers | 3584 / 28 | |
| | Max position embeddings | 4096 | |
| | Step marker | `<extra_0>` (single special token) | |
| | Supervision | step-level labels over the reasoning steps of the solution | |
| | Weights dtype | bfloat16 | |
|
|
| ## Requirements |
|
|
| * `transformers>=4.40.0`. The latest version is recommended. |
| * `trust_remote_code=True` -- the PRM class ships with the checkpoint. |
|
|
| ## Quick Start |
|
|
| > [!Important] |
| > |
| > **Qwen2.5-7B-SHARP-Step** is a process reward model used for scoring reasoning steps, not for generation. |
|
|
| ### Prerequisites |
|
|
| - **Step separation**: split the solution into steps and join them with double line breaks (`"\n\n"`). |
| - **Step marker**: append `<extra_0>` to the end of every step. |
| - **Prompt format**: the model does not use a chat template. Wrap the input as |
| `Question: {question}\n\nSolution:\n{steps}`, which is the format used during training. |
| - **Reward computation**: at each `<extra_0>` position take the probability of the positive class. The result is a |
| value between 0 and 1, where low values mark a hallucinated or incorrect step. |
|
|
| ### Hugging Face Transformers |
|
|
| ```python |
| import torch |
| import torch.nn.functional as F |
| from transformers import AutoModel, AutoTokenizer |
| |
| |
| def build_prompt(question, steps): |
| body = "\n\n".join(f"{step.strip()}<extra_0>" for step in steps) |
| return f"Question: {question}\n\nSolution:\n{body}" |
| |
| |
| def make_step_rewards(logits, token_masks): |
| probabilities = F.softmax(logits, dim=-1) |
| probabilities = probabilities * token_masks.unsqueeze(-1) # bs, seq_len, num_labels |
| |
| all_scores_res = [] |
| for i in range(probabilities.size(0)): |
| sample = probabilities[i] # seq_len, num_labels |
| positive_probs = sample[sample != 0].view(-1, 2)[:, 1] # valid_tokens, num_labels |
| all_scores_res.append(positive_probs.cpu().tolist()) |
| return all_scores_res |
| |
| |
| model_name = "ZaandaTeika/Qwen2.5-7B-SHARP-Step" |
| |
| tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) |
| model = AutoModel.from_pretrained( |
| model_name, |
| device_map="auto", |
| torch_dtype=torch.bfloat16, |
| trust_remote_code=True, |
| ).eval() |
| |
| data = { |
| "question": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?", |
| "steps": [ |
| "In April, Natalia sold 48 clips.", |
| "In May she sold half as many, so she sold 48 / 2 = 24 clips.", |
| "Altogether she sold 48 + 24 = 72 clips. The answer is \\boxed{72}.", |
| ], |
| } |
| |
| prompt = build_prompt(data["question"], data["steps"]) |
| input_ids = tokenizer.encode(prompt, return_tensors="pt").to(model.device) |
| |
| with torch.no_grad(): |
| outputs = model(input_ids=input_ids) |
| |
| step_sep_id = tokenizer.encode("<extra_0>", add_special_tokens=False)[0] |
| token_masks = input_ids == step_sep_id |
| step_reward = make_step_rewards(outputs[0], token_masks) |
| print(step_reward) # one score per step |
| ``` |
|
|
| ## Source Model and Attribution |
|
|
| This checkpoint is a fine-tuned derivative of |
| [Qwen/Qwen2.5-Math-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Math-7B-Instruct), |
| released under the [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) license. |
| The weights were modified: the language modelling head was replaced with a two-way |
| process reward head, and the model was further trained on span-derived step-level |
| supervision. |
|
|
| Training data comes from the [SHARP](https://huggingface.co/datasets/ZaandaTeika/SHARP) |
| corpus, whose reasoning traces and hallucination annotations are licensed under |
| CC BY 4.0. SHARP builds on GSM8K and MATH (both MIT); see the dataset card for the |
| full source attribution. |
|
|
| ## Additional Information |
|
|
| ### Licensing Information |
|
|
| This checkpoint is released under the |
| [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) license, following its |
| base model. |
|
|
