license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
library_name: peft
tags:
- lora
- code
- dsl
- aether
Aether SOP-to-program LoRA v0.2 (Qwen2.5-Coder-1.5B-Instruct)
A LoRA adapter that turns a plain-English business procedure, plus the owner's policy and sheet list, into an Aether program. The model is not a safety mechanism. Every program it writes must pass the Aether checker before it runs. The checker rejects programs that break the policy's rules; in every evaluation, the programs it accepted caused 0 monitor violations on generated test data. The model often writes such programs.
Links: browser demo of the checker and interpreter (the model is not hosted there): https://warnedorca.github.io/aether/ · code: https://github.com/WarnedOrca/aether · benchmark and held-out set: https://huggingface.co/datasets/sidchalamala/aether-benchmark
"correct" below means passes the checker and has the same effects as the reference program (scorer field safe_and_correct). It is not a safety rating.
Final evaluation: frozen held-out trap set (30 tasks, each model evaluated once)
Each task asks for something the policy forbids or Aether cannot do. The correct answer is a program that does the permitted alternative. The set was frozen before any trap training data existed and was never used for training or decisions.
| System | Correct | 95% CI |
|---|---|---|
| Base model + reference card + 3 examples | 0/30 | 0.00-0.11 |
| v0.1 (synthetic data only) | 6/30 | 0.10-0.37 |
| v0.2 (this adapter: + 413 trap samples) | 6/30 | 0.10-0.37 |
- Negative result: adding 413 targeted trap samples did not improve held-out performance. v0.2 vs v0.1: 6 vs 6, paired exact McNemar p = 1.
- v0.2 vs the base model: p = 0.031.
- Novel trap types absent from all training data (delete rows, raise a limit): 0/4 for every system.
- In every evaluation, the programs the checker accepted caused 0 monitor violations on generated test data. The model by itself solves 6/30 held-out traps.
Held-out v2 (90 tasks, 30 traps, 15 new business types; each condition run once)
A second frozen set, written and hashed before any v0.3 training or tuning. Same scorer. 95% Wilson intervals.
| System | Correct (of 90) | 95% CI | Traps (of 30) |
|---|---|---|---|
| Base model + reference card + 3 examples | 1 | 0.00-0.06 | 0 |
| v0.2 plain (this adapter) | 20 | 0.15-0.32 | 2 |
| v0.2 with checker feedback | 22 | 0.17-0.34 | 2 |
| v0.2 grammar-constrained | 19 | 0.14-0.31 | 2 |
| v0.3 plain (not released) | 14 | 0.10-0.24 | 3 |
| v0.3 with checker feedback (not released) | 15 | 0.10-0.26 | 3 |
| v0.3 grammar-constrained (not released) | 15 | 0.10-0.26 | 3 |
- Pre-registered primary comparison (null result): v0.3 with checker feedback vs this adapter (v0.2) plain: 15 vs 20, paired exact McNemar p = 0.33. The new data plus checker feedback did not improve on v0.2.
- v0.2 plain vs the base model: 20 vs 1 (paired p = 2.1e-05; this test was not pre-registered).
- Everything else in this table is exploratory. "With checker feedback" means up to 3 extra attempts in which the model sees the parser and checker errors; it is reported separately from plain output. "Grammar-constrained" means decoding restricted to tokens the Aether grammar allows.
- v0.3 is not released. It is a negative result: it adds 1,232 generated samples from 30 more business types, scores 100/100 on its synthetic test split (as v0.2 does), and solves fewer held-out v2 tasks than v0.2 (not significant in the pre-registered test), which may suggest the generated data teaches the generator's templates more than reading real SOPs.
Held-out v3 (72 tasks, 36 traps, 18 new business types; the one-time final run of this adapter)
Held-out v3 (72 tasks, 36 traps, 18 new business types; the one-time final run of the released v0.2, 2026-10-29; each condition run once): base model with 3 examples 2/72 (95% CI 0.01-0.10); v0.2 plain 18/72 (0.16-0.36; 13/36 ordinary tasks, 5/36 traps); v0.2 with checker feedback 22/72 (0.21-0.42). No confirmatory test was run (the declared comparison needed a 7B model; none beat v0.2 on dev). Post-hoc and exploratory only (not declared before the run): base vs v0.2 plain, exact McNemar p = 0.00014; v0.2 plain vs with feedback, p = 0.125 (not significant); eval/results/stage5-posthoc.json. The checker's accepted programs caused 0 monitor violations. Source: eval/results/stage5/.
| System | Correct (of 72) | 95% CI | Ordinary (of 36), 95% CI | Traps (of 36), 95% CI | Parse rate |
|---|---|---|---|---|---|
| Base model + reference card + 3 examples | 2 | 0.01-0.10 | 2 (0.02-0.18) | 0 (0.00-0.10) | 0.32 |
| v0.2 plain (this adapter) | 18 | 0.16-0.36 | 13 (0.22-0.52) | 5 (0.06-0.29) | 0.81 |
| v0.2 with checker feedback | 22 | 0.21-0.42 | 14 (0.25-0.55) | 8 (0.12-0.38) | 0.79 |
- Feedback's 22 vs plain's 18 was not a pre-declared test. The two post-hoc tests above are exploratory: chosen after seeing the results, with no multiple-comparison adjustment.
- Programs the checker accepted caused 0 monitor violations on generated test data.
Development benchmark (110 tasks, used during development)
This benchmark was used to build the scorer and compare models during development, so treat it as a development set, not a clean test.
| System | Correct | Traps (30) |
|---|---|---|
| Base model + reference card + 3 examples | 3/110 | 0 |
| v0.1 | 30/110 | 2 |
| v0.2 | 34/110 | 3 |
v0.2 vs v0.1: paired p = 0.48.
Larger-model experiments (7B, development benchmark only): negative results
After v0.2, three QLoRA adapters on Qwen2.5-Coder-7B-Instruct were trained on new data whose English was written by a locally run open model (Qwen2.5-7B-Instruct) from script-made programs. None was better than v0.2 on the 110-task development benchmark, so none was released and none was evaluated on a held-out set. Same scorer, greedy decoding, paired exact McNemar against v0.2 (34/110).
| Adapter (7B) | Training data | Correct /110 (epoch 1, epoch 2) | p vs v0.2 (epoch 2) | Source |
|---|---|---|---|---|
| q7b-v1 | 898 local-model wordings of 515 programs | 26, 30 | 0.60 | eval/results/compare-dev-v0.2-vs-q7b-epoch*-2026-10-26.json |
| q7b-v2 | 2,401 wordings of 1,519 programs (mislabelled trap rows removed) | 32, 32 | 0.85 | eval/results/dev-report-q7b-v2-2026-10-27.json |
| q7b-v3 | q7b-v2 data + 950 rows where the obvious column is not allowed | 29, 33 | 1.0 | eval/results/dev-report-q7b-v3-session4.json |
- With checker feedback on dev (exploratory): q7b-v2 epoch 2 35/110 and q7b-v3 epoch 2 36/110, against v0.2 36/110 (p = 1.0 both;
eval/results/dev-repair-compare-2026-10-28.json,eval/results/dev-repair-compare-session4.json). - The 7B adapters' output parses more often (94.5-98.2% of the 110 tasks, against 78.2% for v0.2), but more of it reads or writes columns the policy does not allow (21-25 of 110 for q7b-v1 and q7b-v2, against 3 for v0.2) or does the wrong thing. q7b-v3, trained with programs where the obvious column is not allowed, had 13 such errors (q7b-v2: 25) and 33/110 correct (q7b-v2: 32); with one training run each, neither difference is a tested effect.
- In all of these runs, the programs the checker accepted caused 0 monitor violations.
Training
The v0.1 recipe (LoRA rank 16, learning rate 0.0002, 2 epochs, seed 42; only the program tokens are trained) on 2062 verified examples: 1649 synthetic plus 413 trap-style. Training took 42 minutes on one NVIDIA GeForce RTX 4070, at $0. The checkpoint was chosen by validation loss.
Limitations
- Trained on templated synthetic English only. Most held-out failures are general errors on unfamiliar businesses (parse errors, type errors, reading columns the policy does not allow), not only trap-following.
- All benchmark and held-out tasks were drafted by Claude, an AI assistant, not taken from real businesses.
- Small evaluation sets with wide intervals; one training seed per model.
- Aether's external actions (messages, money, calendars) are mocked. This is research software.
- Do not describe this model, or Aether as a whole, as "safe". The narrow claim is that the checker verifies the named rules C1-C8 for Aether programs (see the specification).
Provenance clarification: the older rule-based generator's template sentences in data/gen/*.py were typed by Claude Code, an AI coding assistant. Script execution produced the older training examples, but their English templates were AI-authored. The owner has decided that stage 4 will train only on data/synthetic_v4, whose wordings come from a locally run open-weight model. This note changes no published results.