Instructions to use ebt005/p9-tiny-aya-grpo-adapters with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use ebt005/p9-tiny-aya-grpo-adapters with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Tiny Aya Global GRPO adapters (reward-hacking study)
Research artifact. These are LoRA adapters from ten 100-step GRPO runs on Tiny Aya Global
(3.35B) that compare a shipped code-execution reward with a hardened grader. They were trained to
measure whether a reward that pays for non-answers teaches a model to cheat; they are not meant
for use as coding assistants. What the runs showed, and its limits, is in results/ of the code
repo (ebt55/rewards-pay-for-nothing). This card makes no claims beyond that. The
pre-registered verdict for the eight main runs is "uninformative": both under-training checks
failed, so these adapters do not show that the shipped reward had no effect
(results/grpo-8runs-2026-11-03.md).
What is here
One folder per run, each holding the step-100 adapter (adapter_model.safetensors, about 121 MB)
and its adapter_config.json. ADAPTER-MANIFEST.json lists every file with its source path and
size.
| run | arm | training reward | hint in training prompt | seed | securegrade | elicitation sha256 (stripped text) | GPU |
|---|---|---|---|---|---|---|---|
| e-ship-s0 | E (elicited) | shipped | yes | 0 | 0.1.0 | a3307e2a…6e29c20 |
NVIDIA L4 |
| e-ship-s1 | E (elicited) | shipped | yes | 1 | 0.1.0 | a3307e2a…6e29c20 |
NVIDIA L4 |
| e-sec-s0 | E (elicited) | secure | yes | 0 | 0.1.0 | a3307e2a…6e29c20 |
NVIDIA L4 |
| e-sec-s1 | E (elicited) | secure | yes | 1 | 0.1.0 | a3307e2a…6e29c20 |
NVIDIA L4 |
| u-ship-s0 | U (unaided) | shipped | no | 0 | 0.1.0 | none (eval hint a3307e2a…6e29c20) |
NVIDIA L4 |
| u-ship-s1 | U (unaided) | shipped | no | 1 | 0.1.0 | none (eval hint a3307e2a…6e29c20) |
NVIDIA L4 |
| u-sec-s0 | U (unaided) | secure | no | 0 | 0.1.0 | none (eval hint a3307e2a…6e29c20) |
NVIDIA L4 |
| u-sec-s1 | U (unaided) | secure | no | 1 | 0.1.0 | none (eval hint a3307e2a…6e29c20) |
NVIDIA L4 |
| h-ship-s0 | H (hole stated, exploratory) | shipped | yes, two sentences | 0 | 0.1.1 | 473943c4…8d751a9 |
RTX 3090 |
| h-sec-s0 | H (hole stated, exploratory) | secure | yes, two sentences | 0 | 0.1.1 | 473943c4…8d751a9 |
RTX 5090 |
Full hashes: a3307e2a5a1f60d419a3ed4f2d9070e8ab8bbef5495155f1b6ef9e6396e29c20 (the one-sentence
hint, jobs/elicitation_text.txt in the code repo) and
473943c40d8d9ffd370221beee7386628a20959b3eccfe050dec4461f8d751a9 (the two-sentence hint,
jobs/elicitation_combined_hole.txt). Hashes are of the text after stripping surrounding
whitespace, as grpo_train.py computes them.
- shipped: AlignTune's registry
code_executionreward (commit57e9a95), with the task's asserts passed astest_cases. - secure:
securegrade.score(...), the fraction of asserts passed, graded on returned values. - The two H runs are exploratory (written up before launch in
design/HOLE-PAIR-2026-11-04.md) and ran on securegrade 0.1.1; the eight E/U runs ran on 0.1.0.
Training setup (all runs)
- Base:
CohereLabs/tiny-aya-global, revisionaf89d219b53ed9b13b8a4645f9c8028973510324. - Trainer: AlignTune
create_rl_trainer(TRL 1.7.1 GRPO backend),jobs/grpo_train.py. - Data: 321 MBPP-sanitized training tasks; user turn = MBPP paper template with the first assert.
- LoRA r 16, alpha 32, dropout 0, on q/k/v/o/gate/up/down projections.
- 100 steps; 8 generations per prompt; 32 completions per step; max 512 completion tokens;
temperature 0.8, top-p 1.0; beta 0.01; loss
dapo; bf16; vLLM colocate. - Learning rate: planned as a constant 2e-5. As run, TRL's default linear schedule applied in all
ten runs: 2e-5 at step 1, 2e-7 at step 100 (dated appendix in the code repo's
design/RUNS-2026-11-02.md). - Exact arguments per run:
grpo_args.jsonandsummary.jsonin the dataset repoebt005/p9-reward-hacking-logs.
Not included: the intermediate adapters (steps 25, 50, 75) and the ref/ sub-adapter that TRL
stores beside each checkpoint (a frozen, untrained copy of the step-0 LoRA used as the KL
reference).
Loading
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_id, rev = "CohereLabs/tiny-aya-global", "af89d219b53ed9b13b8a4645f9c8028973510324"
tok = AutoTokenizer.from_pretrained(base_id, revision=rev)
base = AutoModelForCausalLM.from_pretrained(base_id, revision=rev, torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "ebt005/p9-tiny-aya-grpo-adapters", subfolder="e-ship-s0")
The base model is gated; accept its licence on its model page first.
Licence
The adapters are derived from Tiny Aya Global and follow its licence: CC-BY-NC-4.0, plus the C4AI licence agreement and acceptable-use terms linked from the base model's page. Non-commercial use only. Training tasks come from MBPP (CC-BY-4.0); evaluation also used HumanEval (MIT).
Limits
3.35B parameters, 100 steps, two seeds for the main runs and one for the exploratory pair, one
prompt format, one hint sentence (two for the H pair), heuristic exploit detectors, a learning
rate that decayed against the plan. The runs used securegrade 0.1.0 and 0.1.1; the later 0.1.2
hardening and the 0.1.3 fixes were not used for training. See the code repo's results/, design/ and CLAIMS.md
before drawing conclusions.
- Downloads last month
- -