HiddenSystemsAI/rem-matcher

REM matcher

A DeBERTa-v3-large cross-encoder that decides whether a predicted move by a named actor is the same response-equivalent move (REM) as a realized one. It is the learned matcher used as the reinforcement learning reward in Toward Simulating Real-World Actors: Training Small Language Models to Predict Their Next Moves.

Input

The model reads a text pair. Each side is the actor's name followed by a move:

Actor context: {actor}
Action: {move}

During training, actors with known aliases were written as {actor}; aliases: {alias1}; {alias2}. The predicted move goes first and the realized (target) move second. Pairs are truncated to 512 tokens. Labels are no (0) and yes (1).

Calibrated probability

Raw softmax scores are not calibrated. Use the Platt scaling in calibration.json on the logit margin:

p_match = sigmoid((logit_yes - logit_no) / temperature + bias)

import json, torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "HiddenSystemsAI/rem-matcher"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
cal = json.load(open(hf_hub_download(repo, "calibration.json")))

def side(actor, move):
    return f"Actor context: {actor}\nAction: {move}"

actor = "Bitget"
pairs = [
    (side(actor, "Bitget resumes withdrawals"), side(actor, "Bitget reopens withdrawals for users")),
    (side(actor, "Bitget offers a bounty for the stolen funds"), side(actor, "Bitget reopens withdrawals for users")),
]
batch = tokenizer([a for a, _ in pairs], [b for _, b in pairs], padding=True,
                  truncation=True, max_length=512, return_tensors="pt")
with torch.inference_mode():
    logits = model(**batch).logits.float()
p = torch.sigmoid((logits[:, 1] - logits[:, 0]) / cal["temperature"] + cal["bias"])
print(p.tolist())

In the paper's reward a predicted move credits a target when p_match >= 0.75, and two predicted moves count as duplicates when p_match >= 0.8.

Training

The model was initialized from an earlier ordinal matcher of the same architecture, with a new two-class head trained by cross-entropy for one epoch (learning rate 1e-5, effective batch 32). All pair labels come from Claude models (Sonnet labels, a blind second pass on borderline pairs, Opus adjudication). Splits are disjoint by scenario; the test split holds only real pairs of model-predicted and realized moves from held-out stories.

Split	Pairs	Positives	Scenarios
Train	81,439	43,904	397
Development	7,148	3,709	37
Calibration	7,937	4,367	45
Test (real pairs)	4,048	667	54

Evaluation

On the 4,048 held-out real test pairs, with calibrated probabilities at a 0.5 threshold: accuracy 94.4%, precision 83.4%, recall 82.6%, expected calibration error 0.017. The paper reports precision and recall at the 0.75 training gate and compares the matcher's credits with a three-judge panel on the benchmark lists.

Limitations

The matcher was trained and evaluated on news-derived moves of governments, companies and other public actors, written at the REM level. It undercounts matches that human-style judges accept (it misses about a fifth to a third of judged matches on the paper's benchmark), and optimizing a policy against it can raise its score faster than judged recall. Use it as a training signal or for grouping moves, not as an evaluation metric on its own.