9eve5h/rls-bench-adapters

RLS-Bench adapters

LoRA adapters trained on RLS-Bench, a verifiable benchmark for Supabase/Postgres Row Level Security policies, with the benchmark's test harness as the training signal. The task set is at 9eve5h/rls-bench.

Each folder is a standard PEFT adapter (adapter_config.json, adapter_model.safetensors) exported from Tinker, with a manifest.json naming the Tinker run and checkpoint and giving the size and SHA-256 of adapter_model.safetensors as exported. The same manifests are committed in the GitHub repo as results/train/adapter_manifest_*.json, so a downloaded weights file can be checked against them. (The manifest's entry for adapter_config.json describes the file as Tinker wrote it; the export script then filled in the base model name, so that one file is a few bytes longer than recorded.)

Every number on this page is produced by code in that repo and stored under its results/ directory. The full account, including the runs that did not work, is the repo's docs/writeup.md §5.

What is here

Folder	Base model	What	Test pass	Test leak	Size
(none: the base model)	Qwen/Qwen3.5-9B	untrained	9% [5–16]	18%
(none: the base model)	nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16	untrained	12% [7–19]	22%
sft_Qwen3.5-9B	Qwen/Qwen3.5-9B	SFT on 427 passing DeepSeek V4.1 Flash answers to the 72 train tasks (version 2 data)	38% [29–46]	16%	378 MB
grpo_Qwen3.5-9B_step40	Qwen/Qwen3.5-9B	GRPO from the SFT model, seed 0, step 40 of 50	38% [30–47]	29%	378 MB
grpo_Qwen3.5-9B_seed1_step40	Qwen/Qwen3.5-9B	GRPO from the SFT model, seed 1, step 40 of 50	41% [32–50]	14%	378 MB
repair_Qwen3.5-9B_step5	Qwen/Qwen3.5-9B	repair RL (up to three tries with harness feedback) from the GRPO seed-0 step-40 checkpoint, step 5 of 25	41% [32–50]	18%	378 MB
distil_Qwen3.5-9B_step30	Qwen/Qwen3.5-9B	on-policy distillation from Qwen3.5-397B-A17B with no correctness signal, step 30 of 40	31% [23–40]	18%	378 MB
sft_v3_Qwen3.5-9B	Qwen/Qwen3.5-9B	SFT on two-source data (DeepSeek V4.1 Flash and Kimi K2.6), 72 train tasks (version 3 data)	33% [26–42]	31%	378 MB
grpo_v3_Qwen3.5-9B_step40	Qwen/Qwen3.5-9B	GRPO from the version-3 SFT model, seed 0, step 40 of 40	45% [36–54]	25%	378 MB
sft_82tasks_Qwen3.5-9B	Qwen/Qwen3.5-9B	SFT on two-source data, 82 train tasks (version 4)	35% [27–44]	15%	378 MB
grpo_v5_halfbatch_Qwen3.5-9B_step10	Qwen/Qwen3.5-9B	GRPO from the version-4 SFT model at half batch, step 10 of 40 (a negative result: worse than its SFT)	27% [20–35]	24%	378 MB
sft_Nemotron	nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16	SFT on the version 2 data	26% [19–34]	31%	1,544 MB
grpo_Nemotron_step50	nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16	GRPO from the SFT model, final checkpoint (step 50)	29% [22–38]	10%	1,544 MB

Pass and leak rates are over 120 samples on the 20 held-out test tasks (6 per task: 5 at temperature 0.7 and one greedy), with 95% Wilson intervals on the pass rate. A leak is a test in which a simulated user saw or changed rows it should not have. Each row was evaluated on the test split once, pre-registered in the repo's docs/preregistration.md and locked to a commit; checkpoints were chosen on the 11 dev tasks.

judge_Qwen3-8B (369 MB, base Qwen/Qwen3-8B) is a judge, not a policy writer: given a task and a candidate policy it predicts the harness verdict (pass, leak, block or error). SFT on 2,352 harness-labelled candidates, 2 epochs, learning rate 0.00047, rank 32. On 400 held-out candidates: κ 0.48 against the harness over the four classes, leak recall 53%, leak precision 62%, and 10% of leaks called a pass.

What held up, and what did not

A tuned SFT is most of the gain for Qwen3.5-9B (9% → 38% pass). GRPO's dev gain did not transfer to test for Qwen (38% → 38–41%), and its two seeds disagree on the leak rate by 15 points, so at this scale run-to-run variance is as large as the effect. For Nemotron, GRPO cut the leak rate on 13 of 20 test tasks (31% → 10%), the one significant paired result. The highest pass rate of any row is the version-3 GRPO seed 0 (45%), at a higher leak rate than the version-2 SFT. The version-5 run at half batch made its SFT worse on every measure and is kept here as the negative result it is. The most reliable lever was not training at all: feeding the failing tests back took the GRPO model from 35% to 63% pass within three tries (results/repair_eval/, results/test_eval_v2/*repair3*).

Training details

Folder	Tinker run	Checkpoint	Recipe	Learning rate	Rank	Max tokens	Test evaluation
sft_Qwen3.5-9B	sft_Qwen3.5-9B_20261025T184329Z_lr0.000473	final	SFT, 3 epochs (78 steps), batch 16	0.00047	32	4,096	results/test_eval_v2/
grpo_Qwen3.5-9B_step40	rl_Qwen3.5-9B_lr4e-5_s50	000040	GRPO, 50 steps × 16 tasks × 8 rollouts, seed 0	4e-05	32	2,048	results/test_eval_v2/
grpo_Qwen3.5-9B_seed1_step40	rl_Qwen3.5-9B_lr4e-5_s50_seed1	000040	GRPO, 50 steps × 16 tasks × 8 rollouts, seed 1	4e-05	32	2,048	results/test_eval_v2/
repair_Qwen3.5-9B_step5	repair_Qwen3.5-9B_from40_s25	000005	GRPO, 25 steps × 16 tasks × 4 rollouts, seed 0	4e-05	32	1,536	results/test_eval_v2/
distil_Qwen3.5-9B_step30	distil_Qwen3.5-9B_from397B_s40	000030	distillation, 40 steps × 16 tasks × 4 rollouts	0.0001	32	2,048	results/test_eval_v2/
sft_v3_Qwen3.5-9B	sft_v3_Qwen3.5-9B_20261026T050548Z_lr0.00024	final	SFT, 3 epochs (84 steps), batch 16	0.00024	32	4,096	results/test_eval_v3/
grpo_v3_Qwen3.5-9B_step40	rl_v3_Qwen3.5-9B_lr4e-5_s40	000040	GRPO, 40 steps × 16 tasks × 8 rollouts, seed 0	4e-05	32	2,048	results/test_eval_v3/
sft_82tasks_Qwen3.5-9B	sft_v3b_Qwen3.5-9B_20261026T124124Z_lr0.00024	final	SFT, 3 epochs (90 steps), batch 16	0.00024	32	4,096	results/test_eval_v4/
grpo_v5_halfbatch_Qwen3.5-9B_step10	rl_v3b_Qwen3.5-9B_lr4e-5_s40_b8	000010	GRPO, 40 steps × 8 tasks × 8 rollouts, seed 0	4e-05	32	1,536	results/test_eval_v5/
sft_Nemotron	sft_NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16_20261025T185449Z_lr0.0003	final	SFT, 3 epochs (78 steps), batch 16	0.0003	32	4,096	results/test_eval_v2/
grpo_Nemotron_step50	rl_Nemotron_lr6e-5_s50	final	GRPO, 50 steps × 16 tasks × 8 rollouts, seed 0	6e-05	32	2,048	results/test_eval_v2/

SFT: rank-32 LoRA on all linear layers, linear decay, loss on the answer only; the learning rate was chosen by a three-point sweep on held-out dev loss and that run is the model (nothing retrained). GRPO: the Tinker cookbook RL loop with group-centred rewards (8 answers per task at temperature 1.0), constant-reward groups dropped, the importance-sampling loss, no KL penalty. The harness is the reward: 1 for a pass, 0 for a safe failure (a block, or SQL that fails to apply), −0.1 for a format error and −0.5 for a leak, so a group that all fail still prefers failing closed. Repair: the same loop with up to three turns; after a failing try the model gets the names of the failing tests, the rules they cover, whether each was a leak or a block, and the SQL error, never the expected rows. Distillation: the cookbook's on-policy distillation from Qwen3.5-397B-A17B with the reward scaled to zero.

Prompt format: the fixed benchmark prompt (eval/prompts.py in the repo) in the base model's chat template with thinking disabled; the Tinker renderers were qwen3_5_disable_thinking, nemotron3_ultra_disable_thinking and, for the judge, qwen3_disable_thinking.

How to load one

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen3.5-9B"
tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, "9eve5h/rls-bench-adapters", subfolder="grpo_v3_Qwen3.5-9B_step40")

Use base = "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16" for the two Nemotron folders and "Qwen/Qwen3-8B" for the judge. To grade an answer, clone the repo and run uv run rls run tasks/T001-* answer.sql.

Caveats

• The test split is 20 tasks: every interval above is wide, and most rows are within noise of each other.
• These are research artifacts from a small budget, not a policy generator for production use. The best adapter here still leaks on a quarter of its test samples.
• The base models' own licenses apply to anything built on them: Apache 2.0 for Qwen3.5-9B and Qwen3-8B, the NVIDIA Open Model License for Nemotron-3.5-Lightning.

License

MIT, as the repo. The adapter weights are derived from the base models named above and the task set is CC-BY-4.0.