File size: 4,357 Bytes
dd9391b b994c0f dd9391b bbc94bd dd9391b 4132b44 dd9391b 4132b44 dd9391b bbc94bd dd9391b b994c0f dd9391b 4132b44 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 | ---
license: mit
library_name: transformers
pipeline_tag: text-classification
base_model: OpenAssistant/reward-model-deberta-v3-base
base_model_relation: finetune
datasets:
- HannahRoseKirk/prism-alignment
language:
- en
tags:
- reward-model
- scpo
- cultural-alignment
- canada
- arxiv:2606.18606
---
# SCPO Reward Model — Canada
A scalar reward model fine-tuned on PRISM preference pairs from participants
residing in Canada. Higher scores indicate a preferred response for the
same prompt. Scores are not probabilities and are not calibrated across models.
[Paper](https://arxiv.org/abs/2606.18606) ·
[Code](https://github.com/minsik-ai/Steerable-Cultural-Preference)
## Recorded experiment results
| Measurement | Accuracy |
|---|---:|
| Starting global model, overall | 60.4046% |
| SCPO model, overall | 66.1850% |
These are original in-memory post-training experiment scores, with no model-mode
or random-state reset before scoring. A fresh `eval()` inference session disables
dropout and may produce different accuracy. `validation.json` checks loading and
finite, repeatable inference on generic probes; it is not a benchmark result.
Configurations were tuned on a fixed test split; these checkpoint results differ
from the paper’s three-run averages. Reproduction details are in `provenance.json`.
## Training and reproduction
- Full-model fine-tuning from `OpenAssistant/reward-model-deberta-v3-base`.
- Base revision: `7dab4156e63944834c3aae169c3fe83955672cff`.
- PRISM revision: `18ab5cfb37456f4ec8cbc00212ce54cf7b1239f6`.
- Learning rate: `1e-06`; batch size: 8; optimizer steps: 1024.
- Filtering tau: 0.7; weighting beta: 1.1.
- Data shuffle seed: 0; survey split/model seed: 0;
Trainer seed: 42; Trainer data seed: unset.
- Survey users split 85/15; human-rating ties excluded from preference pairs.
- Original training inputs use literal `prompt + " [SEP] " + response`, capped
and padded to 512 tokens. Original scoring uses tokenizer prompt/response pairs
in interaction batches of at most eight with no explicit truncation.
See `release-settings.json`, `provenance.json`, and `requirements-pw-lock.txt`.
The exact country script and unchanged trainer/collator are included under
`reproduction/`. With the recorded environment installed, run:
```bash
python reproduction/preferences.py ./reproduction-output
```
Original loaders use upstream `main`; verify the recorded model/data revisions
before rerunning. No private dataset records are bundled.
## Inference
The following uses standard evaluation-mode inference.
```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model_id = "MulticulturalAlignment/scpo-reward-model-canada"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()
inputs = tokenizer("How can I disagree respectfully?",
"Listen carefully and explain your reasons calmly.",
return_tensors="pt")
with torch.inference_mode():
reward = model(**inputs).logits.item()
print(reward)
```
## Intended use and limitations
Research on preference modeling and pluralistic alignment. Country labels reflect
participants' residence, not a representative account of a country's culture or
every resident's preferences. The English-dominant sample and small country test
sets limit generalization. This reward model does not generate text and is not
validated for consequential decisions about people.
## Attribution and release license
These fine-tuned weights are released under the [MIT License](https://opensource.org/license/mit).
The [OpenAssistant base model](https://huggingface.co/OpenAssistant/reward-model-deberta-v3-base) is
marked MIT. The [PRISM dataset card](https://huggingface.co/datasets/HannahRoseKirk/prism-alignment)
lists human text as CC-BY-4.0 and model responses as CC-BY-NC-4.0, subject to
original provider terms. This package contains no PRISM records.
## Citation
```bibtex
@misc{scpo,
title={Steerable Cultural Preference Optimization of Reward Models},
author={Minsik Oh and Advit Deepak and Sophie Wu and Douwe Kiela and Ekaterina Shutova},
year={2026},
eprint={2606.18606},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2606.18606},
}
```
|