Instructions to use Oxlino/lorde-tiny-aya with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Oxlino/lorde-tiny-aya with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Oxlino/lorde-tiny-aya") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("Oxlino/lorde-tiny-aya") model = AutoModelForCausalLM.from_pretrained("Oxlino/lorde-tiny-aya", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Oxlino/lorde-tiny-aya with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Oxlino/lorde-tiny-aya" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Oxlino/lorde-tiny-aya", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Oxlino/lorde-tiny-aya
- SGLang
How to use Oxlino/lorde-tiny-aya with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Oxlino/lorde-tiny-aya" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Oxlino/lorde-tiny-aya", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Oxlino/lorde-tiny-aya" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Oxlino/lorde-tiny-aya", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use Oxlino/lorde-tiny-aya with Docker Model Runner:
docker model run hf.co/Oxlino/lorde-tiny-aya
Lorde: a 3B judge for CuratorKIT's quality gates
Licence: CC-BY-NC-4.0 and the Cohere Labs Acceptable Use Policy. Lorde is derived from Tiny Aya Global and trained on Aya Expanse outputs, so both terms apply to it: non-commercial use only, and every use must follow the Acceptable Use Policy. The repository is un-gated for the Lexsi hackathon; downloading it means accepting these terms.
Lorde is Tiny Aya Global (3.35B) fine-tuned to run the two LLM-judge quality gates of CuratorKIT: RewardGate (helpfulness, honesty, instruction following) and HallucinationGate (is every claim in an answer supported by its source?). It replies with short, strict JSON in the shape CuratorKIT's parser expects, so small-model judging stops throwing data away over unparseable replies, at a fraction of the cost of an API judge.
On the held-out test set (937 items) Lorde agrees with its Aya Expanse 8B teacher on 91.1% of pass/reject decisions (ฮบ = 0.748) and 100.0% of its replies are gate-usable without constrained decoding. On the balanced golden-300 subset agreement is 87.3% (ฮบ = 0.691). These numbers measure imitation of the teacher, which is lenient on grounding; agreement with people has not been measured yet. By its own acceptance criteria this version is not promotable (details below).
Built for the Lexsi AI Safety & Alignment Hackathon, Track 1 (Fine-tuning and Data Curation), by chaining
CuratorKIT โ AlignTune โ AuditKIT. Revision v1.0.0.
Use it as CuratorKIT's judge
Serve it with vLLM, then point CuratorKIT's judge at it. No code change in CuratorKIT:
vllm serve Oxlino/lorde-tiny-aya --revision v1.0.0 --served-model-name lorde --max-model-len 4096
from curatorkit import Curator, CuratorConfig
cfg = CuratorConfig(
dataset="data.jsonl", reward_threshold=0.7, hallucination_threshold=0.7,
judge_llm={"model": "openai/lorde", "api_base": "http://localhost:8000/v1", "api_key": "EMPTY"},
)
Curator(cfg).run()
For 100% gate-usable replies, add vLLM structured outputs per gate (schemas in
lorde_forge/schemas.py of the GitHub repo) through CuratorKIT's reward_llm / hallucination_llm
overrides: {"extra_body": {"response_format": <schema>}}.
This page is the full write-up for the Lexsi AI Safety & Alignment Hackathon (Track 1): problem, approach, metrics, results, ROI, deviations and limitations.
TL;DR. CuratorKIT filters synthetic training data with two LLM-judge gates. Large API judges charge per row; small local judges are cheap but break the JSON contract, and CuratorKIT then throws good data away. We fine-tuned Tiny Aya Global (3.35B) to imitate an Aya Expanse 8B teacher's replies to CuratorKIT's exact gate prompts. Lorde's replies are 100.0% gate-usable (base Tiny Aya Global: 74.2%). On the 937-item held-out test set it agrees with the teacher on 91.1% of pass/reject decisions (ฮบ = 0.748; base: 64.5%), and on the balanced golden-300 subset on 87.3% (ฮบ = 0.691). It costs $82.82 per 1M judgments on a Tesla T4 vs $441.27 estimated at list price for Aya Expanse 32B through the Cohere API (5ร). That agreement measures imitation, not correctness: the teacher is a lenient grounding reference (it passes 70% of swapped answers, 94% of fabricated numbers, 76% of closed-book answers and 99% of truncated answers); no human labels have been collected yet; the planned 32B calibration was not run. By its own acceptance criteria Lorde is not promotable yet (see Results). A QA model trained on Lorde-gated data reaches F1 0.681 (teacher-gated 0.679, ungated 0.664, untrained base 0.492), but its answers are significantly less grounded than the ungated set (0.930 vs 0.941, p = 0.004) and teacher-gated (0.930 vs 0.937, p = 0.023); one seed per set, so this is not yet evidence of parity.
The problem
CuratorKIT's RewardGate (helpfulness, honesty, instruction following) and HallucinationGate (is every claim
supported by the source?) ask a judge for JSON, parse it strictly, and reject the sample when the reply
cannot be parsed (judge_error:JudgeParseError). Small open models often wrap the JSON in prose or break
it, so every parse failure deletes a training row that may have been perfectly good. Lorde targets that
failure mode in eight languages, while imitating a larger teacher's pass/reject decisions.
Approach: chaining three Lexsi toolkits
- CuratorKIT renders every judge prompt (
RewardGate._build_reward_prompt,HallucinationGate.prompt_template), parses every reply (parse_judge_reply,extract_score), generates the grounded QA pool from Wikipedia (its QA stage), and finally runs real gating jobs with Lorde asjudge_llm. The judge contract is CuratorKIT's own code, so "gate-usable" means exactly what CuratorKIT accepts. Every gated set ships with itslexsi_provenance.json. - AlignTune trains the judge (LoRA SFT, loss on the reply only) and the downstream models.
- AuditKIT scores downstream answers (
f1_score,lexical_groundedness) and runs the paired-bootstrap significance tests behind every comparison below.
Data
- Reward items: human-written prompts and answers from
CohereLabs/aya_dataset, 300 per language. aya_dataset's language labels are used as given; some are wrong (at least 10 of the 300 English prompts, checked by hand, are in Malay, Hindi, Burmese or Somali). - Grounding items: Wikipedia passages with QA pairs from CuratorKIT's QA stage (teacher-generated).
- Every prompt appears good and degraded: the student's own answer, truncation, an off-topic or wrong-language answer, padding; swapped answers, fabricated numbers, closed-book answers.
- Languages: English, Hindi, French, Arabic, Spanish, Portuguese, Indonesian, Turkish. We chose eight languages that the teacher (Aya Expanse, 23 languages) and the student (Tiny Aya Global) both support, so every label comes from a teacher that actually reads the language.
- Labels: Aya Expanse 8B (self-hosted vLLM, fp16 on 2xT4) for every item; the planned Aya Expanse 32B calibration slice was not run; no hand labels have been collected yet (0 of the 200 blind sample items), so human agreement is not measured.
- Targets: the teacher's reply re-serialized to the gate's exact keys (canonical JSON, long text
clipped). Splits are grouped by prompt and source passage: no test prompt or passage appears verbatim in training. Near-duplicates do cross the split, though: 17 of 348 distinct test reward prompts have a training prompt at least 90% similar (normalized text), 0 of 65 test passages do, and 12 test gold responses are identical to a training one (mostly short answers such as a sentiment label). These splits were built before the near-duplicate grouping in
forge/03_make_splits.py; future builds put near-duplicates in one split.
Training
LoRA r=16, ฮฑ=32, dropout 0.05 on all attention and MLP projections; lr 0.0001 cosine, 2 epoch(s) over 5,281 training rows, effective batch 16 (1 x 16 accumulation), max 1,536 tokens (longer rows dropped, never truncated), 6.0 h of training as recorded in train_meta.json: that is the last session only (the code that wrote it did not add earlier sessions of a resumed run), so it is a lower bound. Tiny Aya's 261k-token vocabulary makes the logits the memory peak on a 16 GB T4, hence batch 1. In production, vLLM structured outputs constrain each gate's reply to its JSON schema, and Bench measures the raw model too so the model itself stays honest.
Metrics, and why each one
| Metric | Why it matters |
|---|---|
| Gate-usable rate | The failure mode CuratorKIT warns about for small judges: unparseable = data thrown away. |
| Decision agreement, Cohen's ฮบ | The gate only acts on pass/reject at 0.7; ฮบ corrects for skewed pass rates. Both measure agreement with the teacher, i.e. imitation, not correctness. |
| Spearman ฯ, MAE | Calibration of scores beyond the threshold. |
| Grounding verdict accuracy | HallucinationGate is the higher-stakes gate (again against the teacher's verdict). |
| Golden-300, adversarial-100 (Bench) | A balanced subset, and items built to fool a judge. |
| Human agreement | The only check on the teacher itself. Not measured yet: 0 hand labels. |
| Downstream F1 / groundedness | A judge is only as good as the data it lets through. |
| Latency, throughput, cost per 1M judgments | Pipeline fit and the ROI the hackathon asks for. |
Results
What the teacher lets through
Lorde learns the teacher's decisions, so its blind spots matter. Teacher pass rate (score at least 0.7) by item variant; every non-gold variant is degraded on purpose.
| Gate | Variant | Items | Teacher pass rate | Mean score |
|---|---|---|---|---|
| grounding | closed_book | 193 | 76% | 0.72 |
| grounding | fabricated_numbers | 207 | 94% | 0.93 |
| grounding | gold | 944 | 100% | 1.00 |
| grounding | swapped_answer | 174 | 70% | 0.67 |
| grounding | truncated | 234 | 99% | 0.98 |
| reward | gold | 2399 | 91% | 0.81 |
| reward | off_topic | 469 | 3% | 0.30 |
| reward | padded_repetition | 500 | 48% | 0.62 |
| reward | student | 458 | 93% | 0.84 |
| reward | truncated | 503 | 47% | 0.61 |
| reward | wrong_language | 461 | 80% | 0.74 |
The teacher is a lenient grounding reference. It passes 70% of swapped answers, 94% of fabricated numbers, 76% of closed-book answers and 99% of truncated answers, all degraded on purpose, and 182 of its 1,752 parseable grounding replies pass an answer while listing an unsupported claim (72 of them say grounded with score 1.0). Decision agreement, ฮบ and verdict accuracy below are measured against this teacher, so they measure how faithfully Lorde imitates it, not whether its decisions are right.
Promotion verdict (spec 9.2): not promotable. Criteria not met or not measured:
- Cohen's ฮบ vs teacher โฅ 0.75: 0.748
- Human decision agreement โฅ teacher's โ 3 pts: not measured: no human labels for test items (forge/04_label_tool.py)
- Per-language decision agreement โฅ 85% (worst: ar, n = 103): 84.5%
- Bench (lorde): Golden decision agreement vs teacher โฅ 90%: 87.3%
- Bench (lorde): Golden ฮบ vs teacher โฅ 0.75: 0.691
- Bench (lorde): Adversarial pass rate โค 5%: 46%
- Bench (lorde+so): Golden decision agreement vs teacher โฅ 90%: 86.3%
- Bench (lorde+so): Golden ฮบ vs teacher โฅ 0.75: 0.665
- Bench (lorde+so): Adversarial pass rate โค 5%: 47%
Lorde Bench: judge evaluation
Held-out test set: 937 items (grouped by prompt / source passage, never seen in training). Reference: teacher (Aya Expanse 8B) decisions at threshold 0.7. Unparseable replies count as rejects, as in CuratorKIT. Items whose teacher reply was unparseable have no reference and are excluded (178 over all splits).
Timings: measured on Tesla T4; 256 items per judge, seeded random sample stratified by kind x language, timed before any prediction; 16 concurrent requests.
Templates: curatorkit-0.1.23/reward@fad46b18, curatorkit-0.1.23/hallucination@027e5efc
Before / after
| Judge | Gate-usable | Decision agr. | ฮบ | Spearman | MAE | Verdict acc. | Pass rate | p50 s | p95 s | judg/s | $ / 1M |
|---|---|---|---|---|---|---|---|---|---|---|---|
| base | 74.2% | 64.5% | 0.198 | 0.713 | 0.143 | 32.4% | 61.2% | 11.535 | 18.974 | 1.303 | 106.590 |
| lorde | 100.0% | 91.1% | 0.748 | 0.864 | 0.058 | 90.9% | 79.1% | 9.280 | 13.290 | 1.677 | 82.820 |
| base+so | 97.8% | 84.7% | 0.559 | 0.481 | 0.151 | 34.8% | 80.2% | 11.361 | 37.174 | 1.133 | 122.590 |
| lorde+so | 100.0% | 90.6% | 0.733 | 0.857 | 0.060 | 91.3% | 79.0% | 9.389 | 32.173 | 1.459 | 95.190 |
API reference (an estimate at list price, not a measured cost: the API judge was not called; token counts are the 8B teacher's): c4ai-aya-expanse-32b (Cohere API) at $0.5/1M input and $1.5/1M output tokens, 534.1 prompt + 116.1 completion tokens per judgment (teacher labels (8B tokenizer)) โ $441.27 per 1M judgments.
Lorde on Tesla T4: $82.82 per 1M judgments, 5ร cheaper.
Acceptance criteria (spec 9.2)
Promotable: no (every blocking criterion must be met; unmeasured ones count as not met)
| Criterion | Value | Met | Blocking | Note |
|---|---|---|---|---|
| Gate-usable rate, unconstrained โฅ 99.0% | 100.0% | yes | yes | |
| Gate-usable rate, structured outputs = 100% | 100.0% | yes | yes | |
| Decision agreement vs teacher โฅ 90% | 91.1% | yes | yes | |
| Cohen's ฮบ vs teacher โฅ 0.75 | 0.748 | no | yes | |
| Spearman ฯ โฅ 0.80 (reported) | 0.864 | yes | no | |
| MAE โค 0.10 (reported) | 0.058 | yes | no | |
| Grounding verdict accuracy โฅ 85% | 90.9% | yes | yes | |
| Human decision agreement โฅ teacher's โ 3 pts | โ | not measured | yes | no human labels for test items (forge/04_label_tool.py) |
| Per-language decision agreement โฅ 85% (worst: ar, n = 103) | 84.5% | no | yes | |
| p95 latency โค 1.5 s at 16 concurrent (spec: 1ร L4 in production; measured on Tesla T4) | 13.29 s | no | no |
Replies that failed: call errors (transport) are retried and never cached; parse failures are replies CuratorKIT could not read (counted as rejects above).
| Judge | Call errors | Parse failures | Missing |
|---|---|---|---|
| base | 0 | 242 | 0 |
| lorde | 0 | 0 | 0 |
| base+so | 0 | 21 | 0 |
| lorde+so | 0 | 0 | 0 |
Decision agreement by language
| Judge | ar | en | es | fr | hi | id | pt | tr |
|---|---|---|---|---|---|---|---|---|
| base | 70.9% | 58.0% | 58.9% | 63.0% | 69.9% | 75.7% | 63.2% | 60.3% |
| lorde | 84.5% | 92.0% | 94.4% | 90.7% | 91.1% | 93.2% | 91.2% | 91.4% |
| base+so | 82.5% | 87.7% | 86.9% | 85.2% | 85.0% | 84.5% | 85.1% | 81.5% |
| lorde+so | 83.5% | 91.3% | 93.5% | 90.7% | 90.3% | 94.2% | 90.3% | 90.7% |
Pass rate by variant (teacher vs judges)
| Variant | n | teacher | base | lorde | base+so | lorde+so |
|---|---|---|---|---|---|---|
| grounding:closed_book | 22 | 68.2% | 0.0% | 72.7% | 86.4% | 72.7% |
| grounding:fabricated_numbers | 37 | 94.6% | 0.0% | 83.8% | 86.5% | 81.1% |
| grounding:gold | 127 | 100.0% | 0.0% | 100.0% | 97.6% | 100.0% |
| grounding:swapped_answer | 29 | 65.5% | 0.0% | 48.3% | 13.8% | 48.3% |
| grounding:truncated | 26 | 96.2% | 0.0% | 96.2% | 92.3% | 96.2% |
| reward:gold | 348 | 89.4% | 95.1% | 94.0% | 92.2% | 94.0% |
| reward:off_topic | 77 | 2.6% | 26.0% | 5.2% | 19.5% | 5.2% |
| reward:padded_repetition | 58 | 41.4% | 79.3% | 44.8% | 74.1% | 44.8% |
| reward:student | 74 | 90.5% | 94.6% | 94.6% | 94.6% | 96.0% |
| reward:truncated | 71 | 47.9% | 66.2% | 56.3% | 63.4% | 54.9% |
| reward:wrong_language | 68 | 72.1% | 86.8% | 89.7% | 79.4% | 89.7% |
Significance (AuditKIT paired bootstrap, decision agreement with teacher)
| Comparison | n | baseline | candidate | ฮ | p | significant |
|---|---|---|---|---|---|---|
| lorde vs base | 937 | 64.5% | 91.1% | 26.7% | 0.000999 | yes |
| lorde+so vs base+so | 937 | 84.7% | 90.6% | 5.9% | 0.000999 | yes |
Prediction caches
| Judge | Predictions | Weights sha (stamped) | Stamped |
|---|---|---|---|
| base | legacy cache, trusted (--trust-legacy-cache) |
af89d219b53e |
2026-11-11 |
| lorde | legacy cache, trusted (--trust-legacy-cache) |
d12ab6ec5c6d |
2026-11-11 |
| base+so | legacy cache, trusted (--trust-legacy-cache) |
af89d219b53e |
2026-11-11 |
| lorde+so | legacy cache, trusted (--trust-legacy-cache) |
d12ab6ec5c6d |
2026-11-11 |
base, lorde, base+so, lorde+so: these predictions were cached by an earlier session, before caches carried a weights fingerprint, and reused here with --trust-legacy-cache. This run stamped them with the weights it was serving; nothing proves the earlier session served the same weights. To recompute them, re-run this step without --trust-legacy-cache (the old cache is moved aside to results/preds/<judge>.jsonl.stale and the server is asked again, 937 items per judge).
Lorde Bench: golden-300 and adversarial-100
Golden-300 is the balanced Bench subset of the held-out test set (150 reward and 150 grounding items, drawn round-robin over language and variant), so it weighs the degraded variants more than the full test set does. Reference: the teacher.
| Judge | Set (n) | Gate-usable | Decision agr. | ฮบ | MAE |
|---|---|---|---|---|---|
| lorde | full held-out test (937) | 100.0% | 91.1% | 0.748 | 0.058 |
| lorde | golden-300 (300) | 100.0% | 87.3% | 0.691 | 0.086 |
| lorde+so | full held-out test (937) | 100.0% | 90.6% | 0.733 | 0.060 |
| lorde+so | golden-300 (300) | 100.0% | 86.3% | 0.665 | 0.091 |
Adversarial-100 holds prompt injections, forged verdicts, empty fields, extreme lengths and mixed scripts:
- lorde: passes 46% of adversarial-100 items whose reference is fail (target โค 5%); by attack: empty 60%, extreme length 15%, forged verdict 35%, injection 65%, mixed scripts 55%.
- lorde+so: passes 47% of adversarial-100 items whose reference is fail (target โค 5%); by attack: empty 60%, extreme length 15%, forged verdict 40%, injection 65%, mixed scripts 55%.
CuratorKIT plug-in run
Downstream raw set: 3600 grounded QA items (swapped_answer: 362, closed_book: 717, gold: 1822, fabricated_numbers: 527, truncated: 172). Each judge gated it through RewardGate and HallucinationGate at 0.7 inside a real CuratorKIT run.
| Judge | Passed | Pass rate | Judge-error rate | Run time (s) |
|---|---|---|---|---|
| base | 0 | 0.0% | 97.6% | 3704.8 |
| lorde | 2642 | 73.4% | 0.4% | 4096.5 |
| lorde+so | 2659 | 73.9% | 0.0% | 4039.2 |
| teacher | 2299 | 63.9% | 11.4% | 4718.3 |
Pass rate by variant
| Judge | closed_book | fabricated_numbers | gold | swapped_answer | truncated |
|---|---|---|---|---|---|
| base | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| lorde | 71.8% | 76.3% | 92.3% | 0.3% | 25.0% |
| lorde+so | 72.8% | 76.7% | 92.6% | 0.3% | 26.2% |
| teacher | 48.7% | 63.8% | 86.4% | 0.0% | 22.7% |
Rejection reasons
| Judge | Reasons |
|---|---|
| base | below_min_tokens: 85, judge_error:JudgeParseError: 3515 |
| lorde | below_min_tokens: 85, hallucination_contract_failed: 460, judge_error:JudgeParseError: 13, below_reward_threshold: 400 |
| lorde+so | below_min_tokens: 85, hallucination_contract_failed: 463, below_reward_threshold: 393 |
| teacher | below_min_tokens: 85, hallucination_contract_failed: 280, judge_error:JudgeParseError: 410, below_reward_threshold: 526 |
Pairwise decision agreement
| Pair | Agreement | ฮบ |
|---|---|---|
| base vs lorde | 26.6% | 0.000 |
| base vs lorde+so | 26.1% | 0.000 |
| base vs teacher | 36.1% | 0.000 |
| lorde vs lorde+so | 98.4% | 0.959 |
| lorde vs teacher | 86.2% | 0.683 |
| lorde+so vs teacher | 86.2% | 0.682 |
Downstream test: models trained on gated data
Tiny Aya Base + LoRA trained on each gated set, evaluated on held-out Wikipedia QA (articles never used elsewhere). AuditKIT f1_score vs the reference answer, lexical_groundedness vs the source passage.
| Training data | Rows | F1 | Groundedness |
|---|---|---|---|
| base | โ | 0.492 | 0.908 |
| ungated | 2000 | 0.664 | 0.941 |
| teacher | 2000 | 0.679 | 0.937 |
| lorde | 2000 | 0.681 | 0.930 |
Lorde-gated F1 relative to teacher-gated: +0.3% (criterion: within ยฑ2%; met).
Lorde-gated beats ungated on F1 with p < 0.05: yes.
Groundedness: Lorde-gated answers are significantly less grounded in the source than the ungated set (0.930 vs 0.941, p = 0.004) and teacher-gated (0.930 vs 0.937, p = 0.023).
Significance (AuditKIT paired bootstrap)
| Comparison | Metric | baseline | candidate | ฮ | p | significant |
|---|---|---|---|---|---|---|
| lorde vs ungated | f1_score | 0.664 | 0.681 | 0.017 | 0.005994 | yes |
| lorde vs ungated | lexical_groundedness | 0.941 | 0.930 | -0.010 | 0.003996 | yes |
| lorde vs teacher | f1_score | 0.679 | 0.681 | 0.002 | 0.703297 | no |
| lorde vs teacher | lexical_groundedness | 0.937 | 0.930 | -0.007 | 0.022977 | yes |
| lorde vs base | f1_score | 0.492 | 0.681 | 0.189 | 0.000999 | yes |
| lorde vs base | lexical_groundedness | 0.908 | 0.930 | 0.022 | 0.002997 | yes |
F1 by language
| Training data | ar | en | es | fr | hi | id | pt | tr |
|---|---|---|---|---|---|---|---|---|
| base | 0.448 | 0.495 | 0.434 | 0.506 | 0.579 | 0.508 | 0.478 | 0.486 |
| ungated | 0.583 | 0.696 | 0.672 | 0.670 | 0.674 | 0.695 | 0.710 | 0.609 |
| teacher | 0.571 | 0.716 | 0.701 | 0.679 | 0.684 | 0.710 | 0.738 | 0.633 |
| lorde | 0.610 | 0.704 | 0.674 | 0.697 | 0.691 | 0.732 | 0.721 | 0.622 |
Reading the downstream result
- Small study: one training run (one seed) per set, 1 epoch, 2,000 rows per set, every set subsampled to the same size (settings: forge/kaggle_driver.py DOWNSTREAM_ARGS). The F1 differences between gated sets need more seeds before they can be called parity.
- Groundedness regresses significantly: Lorde-gated answers are less grounded in the source than the ungated set (0.930 vs 0.941, p = 0.004) and teacher-gated (0.930 vs 0.937, p = 0.023). In the plug-in run Lorde passed more degraded grounding answers than the teacher (fabricated numbers 76.3% vs 63.8%; closed book 71.8% vs 48.7%), which fits this.
- The teacher-gated set is depressed by parse errors: in the plug-in run, 410 of 3,600 rows (11.4%) were rejected because CuratorKIT could not parse the teacher's reply, not because the teacher judged them bad. The "teacher-gated" baseline is therefore the teacher plus its JSON failures.
ROI: self-hosted Lorde vs paying per token
Scenario: curating a 100,000-row dataset through both gates = 200,000 judgments.
| Aya Expanse 32B via Cohere API (list-price estimate) | Lorde on 1ร Tesla T4 (measured) | |
|---|---|---|
| Price | $0.5/1M in, $1.5/1M out (list price) | $0.5/GPU-hour |
| Tokens per judgment | 534.1 in + 116.1 out (the 8B teacher's token counts; the API was not called) | 836.6 in + 92.6 out |
| Throughput | โ | 1.677 judgments/s, 155.3 output tokens/s (measured, 16 concurrent; stratified random sample, timed before any prediction) |
| GPU runtime | โ | 33.13 h |
| Cost for this dataset | $88.25 (estimate) | $16.56 |
| Cost per 1M judgments | $441.27 (estimate) | $82.82 |
5ร cheaper at equal volume, on the hardware the throughput was measured on (1ร Tesla T4, $0.5/h). The API side is list price times token counts, not a measured bill. Other GPUs will differ; we did not measure them. Cheaper is only useful at the agreement levels above.
Deviations from the spec
We changed a few things from the build spec (LORDE-SPEC-001 v1.0), on purpose:
- Languages: Portuguese, Indonesian and Turkish instead of Swahili, Yoruba and Hausa. Aya Expanse, the teacher, does not cover those three, and a teacher that cannot read a language cannot label it.
- Maximum length 1,536 tokens instead of 2,048, and batch 1 ร 16 accumulation instead of 2 ร 8. On a 16 GB T4, Tiny Aya's 261k-token vocabulary makes the logits the memory peak. 1,536 tokens is the 99.7th percentile of the judge SFT set, and the effective batch is still 16. Longer rows are dropped, never truncated.
- Hardware: everything ran on free Kaggle T4s, not the spec's L4. Every throughput and latency figure above names the GPU it was measured on. We did not measure an L4.
Limitations
- Lorde imitates an 8B teacher and inherits its blind spots. On grounding it is lenient (it passes 70% of swapped answers, 94% of fabricated numbers, 76% of closed-book answers and 99% of truncated answers). Human agreement, the only check on the teacher itself, has not been measured (0 hand labels), and the planned 32B calibration slice was not run. High agreement with the teacher is not evidence that the decisions are right.
- The headline is the easy number. On the balanced golden-300 subset Lorde agrees with the teacher on 87.3% (ฮบ = 0.691), against 91.1% (ฮบ = 0.748) on the full test set.
- Judged content can instruct the judge, and Lorde does not resist it: on adversarial-100 it passes 46% of items whose reference says fail (47% with structured outputs); target โค 5%, weakest on injection (65%) and empty (60%). High-stakes datasets still need a human in the loop.
- It misses its own acceptance criteria and is not promotable (full list under Results): Cohen's ฮบ vs teacher โฅ 0.75 (0.748); Human decision agreement โฅ teacher's โ 3 pts (not measured); Per-language decision agreement โฅ 85% (84.5%, Arabic); golden-300 agreement โฅ 90% and ฮบ โฅ 0.75 (87.3%, ฮบ = 0.691); adversarial pass rate โค 5% (46%).
- Near-duplicate leakage: 17 of 348 test reward prompts have a โฅ 90%-similar training prompt and 12 test gold responses equal a training one.
- The downstream study is small (one seed, one training run per set, capped rows), groundedness regresses significantly, and the teacher-gated baseline is depressed by the teacher's parse errors (see Reading the downstream result).
- It is trained on CuratorKIT's default RewardGate and HallucinationGate templates. Other rubrics are out of distribution; template fingerprints flag them.
- Prediction provenance: The step-6 predictions behind the judge numbers (base, lorde, base+so, lorde+so) are a cache written by an earlier Kaggle session, before caches carried a weights fingerprint, and reused with
--trust-legacy-cache; the run that reported them stamped them with the weights it served (af89d219b53e,d12ab6ec5c6d), which does not prove the earlier session served the same weights. Re-running step 6 without the flag recomputes them.
Reproduce it
Everything runs on free Kaggle 2รT4 sessions: forge/kaggle_runbook.ipynb in the GitHub repo runs the steps of
forge/kaggle_driver.py, with the downstream settings reported above. Model, data (with CuratorKIT manifests and
provenance), Bench sets and the result files are on the Hub. To recompute the step-6 predictions instead of reusing the cached ones, run step 6 (the driver's Session D judge-timing step) without LORDE_TRUST_LEGACY_CACHE=1; see the runbook.
Template fingerprints
Trained on these CuratorKIT templates (variable slots removed, SHA-256):
curatorkit-0.1.23/reward@fad46b18sha256fad46b18febabd4342f089c7c766b27e0dbae7dc7bf6a4e97851b809ec8c00d1curatorkit-0.1.23/hallucination@027e5efcsha256027e5efc3fc8f3c6eb7b0772852996d95f9d897e195a1aca17d68d0167876030
Licence
CC-BY-NC-4.0, inherited from Tiny Aya Global, and the
Cohere Labs Acceptable Use Policy. Non-commercial use only.
CuratorKIT (LSAL v1.1) and AuditKIT (LSAL v1.2) are Lexsi Labs source-available licences (non-commercial);
AlignTune is MIT. Third-party terms: THIRD_PARTY_LICENSES.md in the GitHub repo.
- Downloads last month
- 50