Tiny Aya Global GRPO adapters (reward-hacking study)

Research artifact. These are LoRA adapters from ten 100-step GRPO runs on Tiny Aya Global (3.35B) that compare a shipped code-execution reward with a hardened grader. They were trained to measure whether a reward that pays for non-answers teaches a model to cheat; they are not meant for use as coding assistants. What the runs showed, and its limits, is in results/ of the code repo (ebt55/rewards-pay-for-nothing). This card makes no claims beyond that. The pre-registered verdict for the eight main runs is "uninformative": both under-training checks failed, so these adapters do not show that the shipped reward had no effect (results/grpo-8runs-2026-11-03.md).

What is here

One folder per run, each holding the step-100 adapter (adapter_model.safetensors, about 121 MB) and its adapter_config.json. ADAPTER-MANIFEST.json lists every file with its source path and size.

run arm training reward hint in training prompt seed securegrade elicitation sha256 (stripped text) GPU
e-ship-s0 E (elicited) shipped yes 0 0.1.0 a3307e2a…6e29c20 NVIDIA L4
e-ship-s1 E (elicited) shipped yes 1 0.1.0 a3307e2a…6e29c20 NVIDIA L4
e-sec-s0 E (elicited) secure yes 0 0.1.0 a3307e2a…6e29c20 NVIDIA L4
e-sec-s1 E (elicited) secure yes 1 0.1.0 a3307e2a…6e29c20 NVIDIA L4
u-ship-s0 U (unaided) shipped no 0 0.1.0 none (eval hint a3307e2a…6e29c20) NVIDIA L4
u-ship-s1 U (unaided) shipped no 1 0.1.0 none (eval hint a3307e2a…6e29c20) NVIDIA L4
u-sec-s0 U (unaided) secure no 0 0.1.0 none (eval hint a3307e2a…6e29c20) NVIDIA L4
u-sec-s1 U (unaided) secure no 1 0.1.0 none (eval hint a3307e2a…6e29c20) NVIDIA L4
h-ship-s0 H (hole stated, exploratory) shipped yes, two sentences 0 0.1.1 473943c4…8d751a9 RTX 3090
h-sec-s0 H (hole stated, exploratory) secure yes, two sentences 0 0.1.1 473943c4…8d751a9 RTX 5090

Full hashes: a3307e2a5a1f60d419a3ed4f2d9070e8ab8bbef5495155f1b6ef9e6396e29c20 (the one-sentence hint, jobs/elicitation_text.txt in the code repo) and 473943c40d8d9ffd370221beee7386628a20959b3eccfe050dec4461f8d751a9 (the two-sentence hint, jobs/elicitation_combined_hole.txt). Hashes are of the text after stripping surrounding whitespace, as grpo_train.py computes them.

  • shipped: AlignTune's registry code_execution reward (commit 57e9a95), with the task's asserts passed as test_cases.
  • secure: securegrade.score(...), the fraction of asserts passed, graded on returned values.
  • The two H runs are exploratory (written up before launch in design/HOLE-PAIR-2026-11-04.md) and ran on securegrade 0.1.1; the eight E/U runs ran on 0.1.0.

Training setup (all runs)

  • Base: CohereLabs/tiny-aya-global, revision af89d219b53ed9b13b8a4645f9c8028973510324.
  • Trainer: AlignTune create_rl_trainer (TRL 1.7.1 GRPO backend), jobs/grpo_train.py.
  • Data: 321 MBPP-sanitized training tasks; user turn = MBPP paper template with the first assert.
  • LoRA r 16, alpha 32, dropout 0, on q/k/v/o/gate/up/down projections.
  • 100 steps; 8 generations per prompt; 32 completions per step; max 512 completion tokens; temperature 0.8, top-p 1.0; beta 0.01; loss dapo; bf16; vLLM colocate.
  • Learning rate: planned as a constant 2e-5. As run, TRL's default linear schedule applied in all ten runs: 2e-5 at step 1, 2e-7 at step 100 (dated appendix in the code repo's design/RUNS-2026-11-02.md).
  • Exact arguments per run: grpo_args.json and summary.json in the dataset repo ebt005/p9-reward-hacking-logs.

Not included: the intermediate adapters (steps 25, 50, 75) and the ref/ sub-adapter that TRL stores beside each checkpoint (a frozen, untrained copy of the step-0 LoRA used as the KL reference).

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id, rev = "CohereLabs/tiny-aya-global", "af89d219b53ed9b13b8a4645f9c8028973510324"
tok = AutoTokenizer.from_pretrained(base_id, revision=rev)
base = AutoModelForCausalLM.from_pretrained(base_id, revision=rev, torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "ebt005/p9-tiny-aya-grpo-adapters", subfolder="e-ship-s0")

The base model is gated; accept its licence on its model page first.

Licence

The adapters are derived from Tiny Aya Global and follow its licence: CC-BY-NC-4.0, plus the C4AI licence agreement and acceptable-use terms linked from the base model's page. Non-commercial use only. Training tasks come from MBPP (CC-BY-4.0); evaluation also used HumanEval (MIT).

Limits

3.35B parameters, 100 steps, two seeds for the main runs and one for the exploratory pair, one prompt format, one hint sentence (two for the H pair), heuristic exploit detectors, a learning rate that decayed against the plan. The runs used securegrade 0.1.0 and 0.1.1; the later 0.1.2 hardening and the 0.1.3 fixes were not used for training. See the code repo's results/, design/ and CLAIMS.md before drawing conclusions.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for ebt005/p9-tiny-aya-grpo-adapters

Adapter
(395)
this model

Datasets used to train ebt005/p9-tiny-aya-grpo-adapters

Collection including ebt005/p9-tiny-aya-grpo-adapters