---
library_name: transformers
pipeline_tag: text-generation
base_model: FrenzyMath/Reaper-Base
base_model_relation: finetune
tags:
- qwen3
- reaper
- policy-value
- dual-head
---

# Reaper-SFT

Reaper-SFT is the supervised fine-tuned policy-value checkpoint in the Reaper model family. The repository stores a Qwen3 1.7B policy model together with a separate categorical value head.

This checkpoint is part of the [Reaper model collection](https://huggingface.co/collections/FrenzyMath/reaper-6ad03ad88957749d6114b8f6).

## Checkpoint structure

- `policy_model/`: `Qwen3ForCausalLM` policy checkpoint
- `value_head.pt`: state dictionary for a linear value head with 32 output bins
- `dual_head_meta.json`: policy/value loss weights and value-head configuration
- Tokenizer and chat-template files are stored at the repository root

The included metadata specifies `policy_loss_weight=1.0`, `value_loss_weight=0.1`, and `value_dropout=0.01`.

## Model family

| Checkpoint | Stage |
| --- | --- |
| [FrenzyMath/Reaper-Base](https://huggingface.co/FrenzyMath/Reaper-Base) | Base checkpoint |
| [FrenzyMath/Reaper-SFT](https://huggingface.co/FrenzyMath/Reaper-SFT) | Supervised fine-tuning |
| [FrenzyMath/Reaper-1](https://huggingface.co/FrenzyMath/Reaper-1) | Final policy-value checkpoint |

## Load the policy model

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "FrenzyMath/Reaper-SFT"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
policy_model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    subfolder="policy_model",
    torch_dtype="auto",
    device_map="auto",
)
```

## Load the value head

```python
import json
import torch
from huggingface_hub import hf_hub_download

meta_path = hf_hub_download(repo_id, "dual_head_meta.json")
head_path = hf_hub_download(repo_id, "value_head.pt")

with open(meta_path, encoding="utf-8") as handle:
    metadata = json.load(handle)

value_head = torch.nn.Linear(
    policy_model.config.hidden_size,
    metadata["num_value_bins"],
)
value_head.load_state_dict(
    torch.load(head_path, map_location="cpu", weights_only=True)
)
value_head.eval()
```

The policy and value head are stored separately. Integrating value predictions with policy hidden states depends on the calling application.

## Intended use and limitations

This checkpoint is intended for research with policy-value modeling and as the supervised stage preceding Reaper-1. Users should independently evaluate it for their target domain. Training-data details, benchmark results, and a license declaration are not included in the current release.