jonasaise/laya-swedish-preference-gate

Laya Swedish Preference Gate (v2)

A ~11 ms pairwise judge for Swedish text. Give it a Swedish prompt and two candidate responses; it returns a calibrated probability that the first one is better, from a single forward pass of a fine-tuned Laya decision model (mmBERT-base backbone). It is built as a "System 1" filter: settle the confident comparisons in milliseconds and escalate the uncertain ones to a slow LLM judge ("System 2").

v2 answers two questions:

• preference: which response better answers the prompt. On 299 held-out pairs where both answers are fluent Swedish and only quality differs, it is right 75.9% of the time (v1: 64.2%; Kimi-k3 as judge: 87.3%) and its confidence is calibrated there too.
• naturalness: which response is written in more natural, idiomatic Swedish (the translationese detector that v1 effectively was, now as an explicit question).

v1 is still available: SwedishPreferenceGate("jonasaise/laya-swedish-preference-gate", revision="v1"). v1 is slightly better on the original translationese-vs-idiomatic pairs but much weaker and over-confident on fluent pairs; see the comparison below.

Code, pipeline and full results: github.com/joenaess/laya-swedish-preference-gate.

Results at a glance

Metric	v1	v2 (this revision)	Reference / target
Original held-out pairs: idiomatic vs translationese (one call)	98.5%	97.4%	—
Fluent held-out pairs: both natural Swedish (one call)	62.2%	77.9%	—
Probe: 299 fluent pairs, both orders averaged [95% CI]	64.2% [58.9%, 69.6%]	75.9% [70.9%, 80.6%]	Kimi-k3 judge: 87.3%
ECE, original / fluent held-out (one call, 15 bins)	0.004 / 0.288	0.014 / 0.029	✅ ≤ 0.085
ECE on the probe (both orders averaged)	0.230	0.033	—
Probe accuracy on pairs passing margin 0.2 (share passing)	65.0% (92.8%)	84.5% (66.9%)	—
Naturalness question: idiomatic vs translationese (one call)	—	97.8%	—
GPU forward, batch 1, median / p95	10.3 ms / 11.7 ms	11.0 ms / 12.6 ms	✅ < 35 ms

"One call" scores each pair once, in the stored order and in the swapped order, and counts both; "both orders averaged" averages the two probabilities per pair (more robust, two calls). CPU latency was measured for v1 only (607 ms per pair); v2 has the same architecture and size.

How to use

Install

pip install "git+https://github.com/joenaess/laya-swedish-preference-gate"
# or: uv pip install "git+https://github.com/joenaess/laya-swedish-preference-gate"

Which answer is better?

from laya_gate.predictor import SwedishPreferenceGate

gate = SwedishPreferenceGate("jonasaise/laya-swedish-preference-gate")  # margin=0.2, CUDA if available

result = gate.predict_preference(
    prompt="Hur fungerar fotosyntesen?",
    chosen_candidate="Växter tar upp ljus med klorofyll och använder energin för att bygga "
                     "socker av koldioxid och vatten. Syre bildas som en restprodukt.",
    rejected_candidate="Växter gör mat av solljus.",
)
print(result)
# {'preference': 'candidate_a', 'p_chosen': <float>, 'is_valid_margin': <bool>,
#  'probabilities': {'candidate_a': <float>, 'candidate_b': <float>}}

Key	Type	Meaning
preference	"candidate_a" / "candidate_b"	the response the model prefers
p_chosen	float in [0, 1]	calibrated probability that candidate_a (your chosen_candidate) is better
is_valid_margin	bool	abs(p_chosen - 0.5) >= margin; with the default margin=0.2, true when either side has ≥ 0.70
probabilities	dict	both option probabilities (sum to 1)

chosen_candidate / rejected_candidate are just slot A and slot B: the names describe your hypothesis, not ground truth. Batched: gate.predict_preference_batch(prompts, chosen_candidates, rejected_candidates, batch_size=8).

Which answer reads more naturally?

r = gate.predict_naturalness(
    prompt="Vad ska jag göra i Stockholm en regnig dag?",
    candidate_a="Gå på Fotografiska eller Vasamuseet, och fika på något mysigt kafé.",
    candidate_b="Du kan besöka ett museum, vänligen notera att det är en bra aktivitet för regn.",
)
print(r)  # {'more_natural': 'candidate_a', 'p_a_more_natural': <float>, 'probabilities': {...}}

Batched: gate.predict_naturalness_batch(prompts, candidates_a, candidates_b). When both answers are fluent, this question stays near 0.5 (within 0.2 of 0.5 for 83.3% of probe pairs), so a confident naturalness verdict is a sign that one side is translationese.

Score both orders (recommended for fluent answers)

ab = gate.predict_preference(prompt, resp_1, resp_2)["p_chosen"]
ba = gate.predict_preference(prompt, resp_2, resp_1)["p_chosen"]
p_resp_1_better = (ab + (1 - ba)) / 2

The model picks the same winner in both orders for 85.3% of probe pairs; averaging removes the rest of the position effect.

Cascade: gate first, LLM judge only when unsure

def judge_pair(prompt, a, b, llm_judge, gate=gate, margin=0.2):
    r = gate.predict_preference(prompt, a, b)
    if abs(r["p_chosen"] - 0.5) >= margin:
        return r["preference"], "gate"
    return llm_judge(prompt, a, b), "llm"     # your own (slow) LLM-as-judge call

On the probe, with Kimi-k3 as the judge, see the cascade table below: higher margins route more pairs to the judge and get closer to its accuracy. The pre-registered rule (cheapest threshold within 0.5 points of the best) picks 0.45, routing 88.0% for 88.1%.

Pin a version

gate_v1 = SwedishPreferenceGate("jonasaise/laya-swedish-preference-gate", revision="v1")   # first release

With plain laya: laya.load("jonasaise/laya-swedish-preference-gate") opens the checkpoint; the two question definitions it was trained on are in questions.json and must be used verbatim.

Intended use

• Fast first-pass ranking of Swedish LLM outputs (best-of-N, A/B comparisons of model versions), with low-margin pairs escalated to an LLM judge or a human.
• Detecting translationese, English loan-word overuse, literal phrasing or US-centric answers (naturalness question).
• Flagging suspicious pairs in Swedish preference datasets.

Out of scope: fact-checking; safety or toxicity classification; languages other than Swedish; any high-stakes decision without human review. On fluent pairs it is still 11.4 percentage points behind an LLM judge.

Training data

1. Original pairs (jonasaise/dpo-swedish-interpreted): idiomatic, localized Swedish (chosen) vs a deliberately poor translation (rejected). Used for both questions.

Source: jonasaise/dpo-swedish-interpreted. The repository's single train split is stored as parquet files with two different schemas (the dataset card declares only one), so datasets.load_dataset fails with a cast error; data.py reads the files one by one and keeps prompt, chosen, rejected.

Raw pairs	14986
Dropped (empty / identical / duplicate)	{"empty_field": 0, "identical_responses": 0, "duplicate": 0}
Train / val_calib / val_eval	12750 / 1118 / 1118
Val pairs moved to train (prompt also in train)	12
Fraction with chosen in slot B (train / calib / eval)	0.500 / 0.499 / 0.498
Pairs truncated to fit 1024 tokens	6336 (42.3%)
Untruncated input length, tokens (median / p95 / max)	858.0 / 2423.75 / 11276
Question head / state budget, tokens	32 / 991

How the pairs were made. The dataset card says kimi-k3 localized each UltraFeedback prompt and chosen response into idiomatic Swedish and deliberately generated the rejected response with poor translation, excessive English loan words or US-centric content. The preference signal is therefore mostly authentic vs. translated Swedish. Two non-neural baselines on val_eval show how lexical it is:

Baseline (val_eval)	Accuracy
Longer response wins	71.3%
Logistic regression on TF-IDF(A) − TF-IDF(B)	98.4%

Words the TF-IDF model associates with the chosen side: alltså, gärna, utifrån, innebär, stället, svenska, exempel, sverige, samt, däremot. With the rejected side: denna, detta, konfidens, såsom, kommer, av, baserat, inkluderar, hjälpa, är.

2. Fluent pairs (new in v2): UltraFeedback pairs with a clear GPT-4 rating gap, both answers rewritten into natural Swedish by Kimi-k3 with identical instructions, so the two sides differ in quality, not style. Preference target = the higher-rated answer; naturalness target = 50/50.

12000 more UltraFeedback pairs (disjoint from the probe; pool of 23244) were rewritten into natural Swedish with the probe's translation prompt, by kimi-k3, reasoning off, prompt = probe.TRANSLATE_SYSTEM. 11988 translated cleanly (12 failures); 316 were dropped because their Swedish prompt nearly duplicates a probe or validation prompt (char n-gram cosine ≥ 0.8), leaving 11672 pairs, split train 9921 / val_calib 875 / val_eval 876 with the better answer in slot B for 50.0% of train pairs.

Before generating, the first 100 probe pairs were re-translated with reasoning off (median 518 completion tokens per pair) and compared with the reasoning-on translations. Lengths are preserved (off/on length ratio 0.996 for the better answer, 1.000 for the worse), character similarity is 0.80 / 0.78, and Kimi-k3 recovers the UltraFeedback label on 86.5% of reasoning-off pairs vs 89.0% of reasoning-on pairs (both orders). The small drop was accepted for training data; the probe itself stays reasoning-on.

Throughput against the self-hosted Kimi-k3 endpoint (8× NVIDIA B300, HGX), 1,000 requests per chunk, reasoning off: concurrency 64: 1.15–1.79 req/s over 11 chunk(s), concurrency 128: 1.44–1.44 req/s over 1 chunk(s). Concurrency 128 was slower than 64 (longer queueing, more parse failures), so the remaining chunks ran at 64. Total generation wall-clock: 2.3 h. Variation between chunks at the same concurrency reflects other load on the shared endpoint.

Chunk	Concurrency	Requests/s	Output tokens/s	Median latency	Failed / parse failures
0	64	1.77	1161	33.2 s	0 / 0
1	128	1.44	990	50.2 s	0 / 2
2	64	1.22	820	32.7 s	0 / 1
3	64	1.15	798	34.0 s	0 / 2
4	64	1.79	1158	32.9 s	0 / 0
5	64	1.74	1175	34.5 s	0 / 0
6	64	1.77	1161	33.9 s	0 / 2
7	64	1.45	972	33.1 s	0 / 1
8	64	1.43	942	33.2 s	0 / 1
9	64	1.41	945	33.9 s	0 / 2
10	64	1.27	863	34.1 s	0 / 1
11	64	1.76	1158	33.8 s	0 / 0

Training, calibration and evaluation

Training

Initialized from the base convaiinnovations/laya multilingual checkpoint (not from v1), same recipe as v1 (lr 2e-05, micro-batch 8 × accumulation 2, warmup + cosine, bf16), 3 epochs over 45342 (pair, question) units; each pair encoded in both orders; one order drawn per item per epoch. Best epoch: 1 (lowest mean val_calib loss over (source, question) groups). Total 108.7 min on NVIDIA GeForce RTX 4080 Laptop GPU; peak GPU memory 10.73 GiB.

Accuracy is n/a for fluent/naturalness, whose target is 50/50.

Epoch	Train loss	original/preference loss / acc	original/naturalness loss / acc	fluent/preference loss / acc	fluent/naturalness loss / acc	Time
1	0.421	0.055 / 97.8%	0.066 / 98.1%	0.551 / 75.3%	0.748 / n/a	35.9 min
2	0.266	0.046 / 98.7%	0.042 / 98.7%	0.700 / 76.5%	0.743 / n/a	36.4 min
3	0.197	0.088 / 98.6%	0.046 / 98.6%	1.322 / 73.9%	0.755 / n/a	36.4 min

Temperatures were fitted per question on val_calib, both orders: preference T = 1.446 (NLL 0.280 → 0.265, fit on val_calib original+fluent, both orders), naturalness T = 0.575 (NLL 0.066 → 0.057, fit on val_calib original, both orders).

v1 vs v2

Both checkpoints were scored with the same code (python -m laya_gate.v2 evaluate, v1 via --checkpoint checkpoints/best_model --section v1_eval), on the held-out val_eval halves and the 299-pair probe, each pair in both orders.

Metric	v1	v2
Original val_eval, preference (order-averaged)	98.7%	97.6%
Original val_eval, ECE (order-averaged)	0.009	0.023
Fluent val_eval, preference (order-averaged)	63.4%	78.8%
Fluent val_eval, ECE (order-averaged)	0.224	0.024
Probe (reasoning-on), preference [95% CI]	64.2% [58.9%, 69.6%]	75.9% [70.9%, 80.6%]
Probe, ECE (order-averaged)	0.230	0.033
Probe, positional consistency	82.3%	85.3%
Probe, share passing margin 0.2	83.6%	65.6%
Probe, accuracy on pairs passing the margin	66.8%	85.7%
Probe, accuracy on pairs routed to the judge	51.0%	57.3%
GPU forward, batch 1, median	10.3 ms	11.0 ms

For reference, Kimi-k3 scores 87.3% on the probe.

Naturalness head (v2 only). On original val_eval it picks the idiomatic answer over the translationese one 98.1% of the time (ECE 0.017). On fluent pairs, where both answers are natural, its order-averaged probability stays within 0.2 of 0.5 for 85.4% of fluent val_eval pairs and 83.3% of probe pairs (mean distance from 0.5: 0.095 and 0.105).

Probe cascade. Gate first, Kimi-k3 (cached reasoning-on verdicts from the probe run, judge alone 87.3%) for instances with |p − 0.5| < threshold. Each order is a separate instance here, so threshold 0 differs slightly from the order-averaged accuracy above.

Threshold	v1 routed / accuracy	v2 routed / accuracy	v2 expected latency
0.00	0.0% / 64.7%	0.0% / 75.9%	11.0 ms
0.05	2.8% / 65.4%	7.9% / 78.8%	760.1 ms
0.10	4.2% / 65.6%	17.1% / 81.3%	1636.7 ms
0.15	5.5% / 66.1%	24.7% / 83.8%	2369.9 ms
0.20	7.2% / 66.7%	33.1% / 84.8%	3166.8 ms
0.25	9.4% / 67.1%	38.8% / 85.1%	3708.7 ms
0.30	11.9% / 68.9%	49.5% / 86.6%	4728.8 ms
0.35	16.2% / 70.4%	59.5% / 86.6%	5685.1 ms
0.40	21.6% / 71.1%	72.6% / 87.3%	6928.3 ms
0.45 (recommended)	31.9% / 73.7%	88.0% / 88.1%	8394.7 ms

Recommended for v2 (lowest routed fraction within 0.5% of the best cascade accuracy in the sweep): threshold 0.45, routing 88.0% for 88.1%; for v1 the same rule gives 0.45 → 31.9% routed, 73.7%.

Fitted temperatures: preference T = 1.446, naturalness T = 0.575 (in temperature.json).

Limitations

• Still behind an LLM judge on fluent pairs, and the probe has only 299 pairs (see the CI).
• Translated, not native, data. All training and test text was written or rewritten into Swedish by Kimi-k3. Native Swedish preferences were not measured.
• The judge is also the translator. Kimi-k3 generated all Swedish text and is the reference judge, so agreement with it is not independent evidence.
• Pairwise preference, not fact-checking. A fluent wrong answer can beat a terse correct one.
• 1024-token window. Long prompts keep their tail and long answers their head; differences late in long answers are invisible.
• Labels are GPT-4 ratings (UltraFeedback) and Kimi-k3's construction, not human judgements.
• Single seed, single run.

Files

File	Contents
model.safetensors	decision-model weights (encoder + head)
rl_agent_config.json	Laya config: max_len 1024, head_max_len 256
questions.json	the two choice questions (preference, naturalness)
temperature.json	fitted temperature per question, read by SwedishPreferenceGate
encoder/config.json	ModernBERT (mmBERT-base) encoder config
tokenizer/	tokenizer files
results/metrics.json	every number on this card

Citation and acknowledgements

• Laya decision models and training recipe: NandhaKishorM/laya, base checkpoint convaiinnovations/laya.
• Encoder: jhu-clsp/mmBERT-base.
• Data: jonasaise/dpo-swedish-interpreted and argilla/ultrafeedback-binarized-preferences-cleaned (UltraFeedback).
• Translation and judge model: Kimi-k3, self-hosted (8× NVIDIA B300, HGX).

@software{laya_swedish_preference_gate,
  title  = {Laya Swedish Preference Gate},
  author = {Lind, Jonas},
  year   = {2026},
  url    = {https://huggingface.co/jonasaise/laya-swedish-preference-gate}
}

License

Apache-2.0. Base model license: apache-2.0; dataset licenses: jonasaise/dpo-swedish-interpreted mit, argilla/ultrafeedback-binarized-preferences-cleaned mit. Check the upstream UltraFeedback terms before commercial use.