| --- |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: text-classification |
| tags: |
| - grt |
| - reward-model |
| - process-reward-model |
| - prm |
| - rlhf-verifier |
| - rotational-transformer |
| - givens-rotations |
| - clifford-algebra |
| - matterwave |
| base_model: MatterWaveAI/grt-30m |
| --- |
| |
| # GRT-30M-PRM (`MatterWaveAI/grt-30m-prm`) |
|
|
| **GRT-30M-PRM** is the **100% rotational Process Reward Model (PRM) and scalar RLHF Verifier** initialized from [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (`GRTForRewardModeling`, `29,306,021` parameters, **zero `nn.Linear` layers**, **zero `nn.Conv1d` layers**). |
|
|
| It equips the 12-layer Givens Rotational Transformer backbone with dual zero-`nn.Linear` rotational heads (`GRTCosineClassificationHead`): |
| - **`reward_head`**: Evaluates circular-pooled planar-pair sequence representations (`GRTForSequenceClassification._circular_pool`) to emit a sequence-level scalar reward (`logits` of shape `[B, 1]`). |
| - **`step_head`**: Evaluates `L2`-normalized token/step hidden states across reasoning traces to emit fine-grained per-token / per-step verification scores (`step_logits` of shape `[B, L, 1]`). |
|
|
| ## Process Verification & Preference Alignment Metrics |
|
|
| - **Held-Out Pairwise Preference Ranking Accuracy**: `37.50%` (untrained head baseline) → **`100.00%`** (across all 8 classical geometric optimizers) |
| - **Held-Out Step-Level Verification Accuracy**: `31.42%` → **`98.07%`** (`HuygensSymplecticAdam`; `97.55%` w/ `PolarNaturalAdam`, `97.21%` w/ `CyclideAdam`) |
| - **Held-Out Step-Level Verification F1**: **`98.59%`** (`HuygensSymplecticAdam`; `98.23%` w/ `PolarNaturalAdam`, `97.99%` w/ `CyclideAdam`) |
| - **Held-Out Total Verification Loss (Bradley-Terry + Step BCE)**: `15.4491` → **`0.0382`** (`HuygensSymplecticAdam`, w/ `0.0000` held-out Bradley-Terry preference loss) |
| - **Positive-vs-Flawed Sequence Reward Margin**: `-0.2095` → **`+25.1196`** (`CyclideAdam`) / **`+24.4569`** (`HuygensSymplecticAdam`) |
|
|
| --- |
|
|
| ## 8-Way Classical Geometric Optimizer Ablation |
|
|
| We benchmarked all 8 classical geometric, Riemannian, conic, symplectic, gyroscopic, and synchronization optimizers from `transformers.models.grt.optimization_grt` on `MatterWaveAI/grt-30m-prm` (`200` steps of joint Bradley-Terry pairwise preference margin + token/step-level binary cross-entropy process verification on `blocks[11:12]` + `final_norm` + `reward_head` + `step_head`, `812,423` trainable parameters, `batch_size = 8` pairs, `lr = 4.5e-3`, `max_angular_step = 0.08`, `seed = 42`): |
|
|
| | Rank | Optimizer (`optimizer_type`) | Final Train Loss | Held-Out Total Loss | Held-Out BT Loss | Preference Acc (%) | Step Verification Acc (%) | Step F1 (%) | Reward Margin | Wall Time (s) | |
| | :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | |
| | **1** | **`HuygensSymplecticAdam` (`huygens_symplectic_adam`)** | **`0.0464`** | **`0.0382`** | **`0.0000`** | **`100.00%`** | **`98.07%`** | **`98.59%`** | `+24.4569` | `320.77s` | |
| | **2** | **`PolarNaturalAdam` (`polar_natural_adam`)** | `0.0503` | `0.0533` | `0.0003` | **`100.00%`** | `97.55%` | `98.23%` | `+23.7497` | `301.44s` | |
| | **3** | **`CyclideAdam` (`cyclide_adam`)** | `0.0714` | `0.0588` | **`0.0000`** | **`100.00%`** | `97.21%` | `97.99%` | **`+25.1196`** | `303.28s` | |
| | **4** | **`SymplecticAdam` (`symplectic_adam`)** | `0.1077` | `0.0729` | `0.0003` | **`100.00%`** | `96.76%` | `97.65%` | `+21.5043` | `310.61s` | |
| | **5** | **`PairAdam` (`pair_adam`)** | `0.1109` | `0.1139` | `0.0074` | **`100.00%`** | `95.86%` | `97.06%` | `+17.8117` | **`257.80s`** | |
| | **6** | **`HessianPairAdam` (`hessian_pair_adam`)** | `0.1688` | `0.1637` | `0.0003` | **`100.00%`** | `92.83%` | `94.66%` | `+15.9064` | `261.16s` | |
| | **7** | **`HuygensAdam` (`huygens_adam`)** | `0.0911` | `0.1795` | `0.0265` | **`100.00%`** | `95.34%` | `96.63%` | `+13.5915` | `289.98s` | |
| | **8** | **`GyricAdam` (`gyric_adam`)** | `0.2236` | `0.2213` | `0.0057` | **`100.00%`** | `89.13%` | `91.58%` | `+14.9081` | `315.54s` | |
| |
| ### Key Geometric Findings on Process Reward Modeling |
| - **Two-Stage Huygens → Symplectic Curriculum (`HuygensSymplecticAdam`) Achieves #1 Step Verification & Lowest Loss**: Annealing circular synchronization splay torque (`splay_anneal_steps = 100`) into a Gromov–Liouville symplectic non-squeezing floor enables `HuygensSymplecticAdam` to separate valid derivation steps from mid-trace mathematical/algorithmic errors with **`98.07%` step accuracy (`98.59%` F1)** and **`0.0382` total validation loss**. |
| - **Riemannian Polar Metric (`PolarNaturalAdam`) & Dupin Cyclide Conic Metric (`CyclideAdam`) Excel at Sharp Step Boundaries**: `PolarNaturalAdam` (`97.55%` step acc, `0.0533` loss) and `CyclideAdam` (`97.21%` step acc, **`+25.1196` sequence reward margin**) leverage radial-scale preconditioning on `reward_head` and `step_head` to amplify the scalar reward separation between valid and flawed multi-step traces. |
| |
| --- |
| |
| ## Quickstart (`transformers`) |
| |
| ```python |
| import torch |
| from transformers import AutoTokenizer |
| from transformers.models.grt.modeling_grt import GRTForRewardModeling |
| |
| model_id = "MatterWaveAI/grt-30m-prm" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = GRTForRewardModeling.from_pretrained(model_id, dtype=torch.float32) |
| |
| trace = ( |
| "<|im_start|>user\nProblem: Solve for x: 4x - 9 = 19.<|im_end|>\n" |
| "<|im_start|>assistant\n" |
| "Step 1: Start with the linear equation 4x - 9 = 19.\n" |
| "Step 2: Add 9 to both sides to isolate the 4x term: 4x = 28.\n" |
| "Step 3: Divide both sides by 4: x = 28 / 4 = 7.\n" |
| "Step 4: Verify: 4(7) - 9 = 28 - 9 = 19. Answer: x = 7.<|im_end|>" |
| ) |
| inputs = tokenizer(trace, return_tensors="pt", add_special_tokens=False) |
| with torch.no_grad(): |
| out = model(**inputs) |
| seq_reward = out.logits.item() |
| step_probs = torch.sigmoid(out.step_logits[0, :, 0]) |
| print(f"Sequence Reward: {seq_reward:.4f} | Mean Step Verification Confidence: {step_probs.mean().item():.4f}") |
| ``` |
| |
| --- |
| |
| ## License |
| |
| Apache 2.0 |
| |
