Update model card: held-out v3, 7B negative results, wording

070b1c3 verified 5 days ago

8.73 kB

---
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
library_name: peft
tags: [lora, code, dsl, aether]
---
# Aether SOP-to-program LoRA v0.2 (Qwen2.5-Coder-1.5B-Instruct)

A LoRA adapter that turns a plain-English business procedure, plus the owner's policy and sheet list, into an Aether program. **The model is not a safety mechanism.** Every program it writes must pass the Aether checker before it runs. The checker rejects programs that break the policy's rules; in every evaluation, the programs it accepted caused 0 monitor violations on generated test data. The model often writes such programs.

**Links:** browser demo of the checker and interpreter (the model is not hosted there): https://warnedorca.github.io/aether/ · code: https://github.com/WarnedOrca/aether · benchmark and held-out set: https://huggingface.co/datasets/sidchalamala/aether-benchmark

"correct" below means passes the checker and has the same effects as the reference program (scorer field `safe_and_correct`). It is not a safety rating.

## Final evaluation: frozen held-out trap set (30 tasks, each model evaluated once)

Each task asks for something the policy forbids or Aether cannot do. The correct answer is a program that does the permitted alternative. The set was frozen before any trap training data existed and was never used for training or decisions.

| System | Correct | 95% CI |
|---|---|---|
| Base model + reference card + 3 examples | 0/30 | 0.00-0.11 |
| v0.1 (synthetic data only) | 6/30 | 0.10-0.37 |
| **v0.2 (this adapter: + 413 trap samples)** | **6/30** | 0.10-0.37 |

- **Negative result:** adding 413 targeted trap samples did **not** improve held-out performance. v0.2 vs v0.1: 6 vs 6, paired exact McNemar p = 1.
- v0.2 vs the base model: p = 0.031.
- Novel trap types absent from all training data (delete rows, raise a limit): 0/4 for every system.
- In every evaluation, the programs the checker accepted caused 0 monitor violations on generated test data. The model by itself solves 6/30 held-out traps.

## Held-out v2 (90 tasks, 30 traps, 15 new business types; each condition run once)

A second frozen set, written and hashed before any v0.3 training or tuning. Same scorer. 95% Wilson intervals.

| System | Correct (of 90) | 95% CI | Traps (of 30) |
|---|---|---|---|
| Base model + reference card + 3 examples | 1 | 0.00-0.06 | 0 |
| **v0.2 plain (this adapter)** | 20 | 0.15-0.32 | 2 |
| v0.2 with checker feedback | 22 | 0.17-0.34 | 2 |
| v0.2 grammar-constrained | 19 | 0.14-0.31 | 2 |
| v0.3 plain (not released) | 14 | 0.10-0.24 | 3 |
| v0.3 with checker feedback (not released) | 15 | 0.10-0.26 | 3 |
| v0.3 grammar-constrained (not released) | 15 | 0.10-0.26 | 3 |

- **Pre-registered primary comparison (null result):** v0.3 with checker feedback vs this adapter (v0.2) plain: 15 vs 20, paired exact McNemar p = 0.33. The new data plus checker feedback did **not** improve on v0.2.
- v0.2 plain vs the base model: 20 vs 1 (paired p = 2.1e-05; this test was not pre-registered).
- Everything else in this table is exploratory. "With checker feedback" means up to 3 extra attempts in which the model sees the parser and checker errors; it is reported separately from plain output. "Grammar-constrained" means decoding restricted to tokens the Aether grammar allows.
- **v0.3 is not released.** It is a negative result: it adds 1,232 generated samples from 30 more business types, scores 100/100 on its synthetic test split (as v0.2 does), and solves fewer held-out v2 tasks than v0.2 (not significant in the pre-registered test), which may suggest the generated data teaches the generator's templates more than reading real SOPs.

## Held-out v3 (72 tasks, 36 traps, 18 new business types; the one-time final run of this adapter)

**Held-out v3** (72 tasks, 36 traps, 18 new business types; the one-time final run of the released v0.2, 2026-10-29; each condition run once): base model with 3 examples 2/72 (95% CI 0.01-0.10); **v0.2 plain 18/72** (0.16-0.36; 13/36 ordinary tasks, 5/36 traps); v0.2 with checker feedback 22/72 (0.21-0.42). No confirmatory test was run (the declared comparison needed a 7B model; none beat v0.2 on dev). Post-hoc and exploratory only (not declared before the run): base vs v0.2 plain, exact McNemar p = 0.00014; v0.2 plain vs with feedback, p = 0.125 (not significant); `eval/results/stage5-posthoc.json`. The checker's accepted programs caused 0 monitor violations. Source: `eval/results/stage5/`.

| System | Correct (of 72) | 95% CI | Ordinary (of 36), 95% CI | Traps (of 36), 95% CI | Parse rate |
|---|---|---|---|---|---|
| Base model + reference card + 3 examples | 2 | 0.01-0.10 | 2 (0.02-0.18) | 0 (0.00-0.10) | 0.32 |
| **v0.2 plain (this adapter)** | 18 | 0.16-0.36 | 13 (0.22-0.52) | 5 (0.06-0.29) | 0.81 |
| v0.2 with checker feedback | 22 | 0.21-0.42 | 14 (0.25-0.55) | 8 (0.12-0.38) | 0.79 |

- Feedback's 22 vs plain's 18 was not a pre-declared test. The two post-hoc tests above are exploratory: chosen after seeing the results, with no multiple-comparison adjustment.
- Programs the checker accepted caused 0 monitor violations on generated test data.

## Development benchmark (110 tasks, used during development)

This benchmark was used to build the scorer and compare models during development, so treat it as a development set, not a clean test.

| System | Correct | Traps (30) |
|---|---|---|
| Base model + reference card + 3 examples | 3/110 | 0 |
| v0.1 | 30/110 | 2 |
| **v0.2** | **34/110** | 3 |

v0.2 vs v0.1: paired p = 0.48.

## Larger-model experiments (7B, development benchmark only): negative results

After v0.2, three QLoRA adapters on Qwen2.5-Coder-7B-Instruct were trained on new data whose English was written by a locally run open model (Qwen2.5-7B-Instruct) from script-made programs. None was better than v0.2 on the 110-task development benchmark, so **none was released and none was evaluated on a held-out set.** Same scorer, greedy decoding, paired exact McNemar against v0.2 (34/110).

| Adapter (7B) | Training data | Correct /110 (epoch 1, epoch 2) | p vs v0.2 (epoch 2) | Source |
|---|---|---|---|---|
| q7b-v1 | 898 local-model wordings of 515 programs | 26, 30 | 0.60 | `eval/results/compare-dev-v0.2-vs-q7b-epoch*-2026-10-26.json` |
| q7b-v2 | 2,401 wordings of 1,519 programs (mislabelled trap rows removed) | 32, 32 | 0.85 | `eval/results/dev-report-q7b-v2-2026-10-27.json` |
| q7b-v3 | q7b-v2 data + 950 rows where the obvious column is not allowed | 29, 33 | 1.0 | `eval/results/dev-report-q7b-v3-session4.json` |

- With checker feedback on dev (exploratory): q7b-v2 epoch 2 35/110 and q7b-v3 epoch 2 36/110, against v0.2 36/110 (p = 1.0 both; `eval/results/dev-repair-compare-2026-10-28.json`, `eval/results/dev-repair-compare-session4.json`).
- The 7B adapters' output parses more often (94.5-98.2% of the 110 tasks, against 78.2% for v0.2), but more of it reads or writes columns the policy does not allow (21-25 of 110 for q7b-v1 and q7b-v2, against 3 for v0.2) or does the wrong thing. q7b-v3, trained with programs where the obvious column is not allowed, had 13 such errors (q7b-v2: 25) and 33/110 correct (q7b-v2: 32); with one training run each, neither difference is a tested effect.
- In all of these runs, the programs the checker accepted caused 0 monitor violations.

## Training

The v0.1 recipe (LoRA rank 16, learning rate 0.0002, 2 epochs, seed 42; only the program tokens are trained) on 2062 verified examples: 1649 synthetic plus 413 trap-style. Training took 42 minutes on one NVIDIA GeForce RTX 4070, at $0. The checkpoint was chosen by validation loss.

## Limitations

- Trained on templated synthetic English only. Most held-out failures are general errors on unfamiliar businesses (parse errors, type errors, reading columns the policy does not allow), not only trap-following.
- All benchmark and held-out tasks were drafted by Claude, an AI assistant, not taken from real businesses.
- Small evaluation sets with wide intervals; one training seed per model.
- Aether's external actions (messages, money, calendars) are mocked. This is research software.
- Do not describe this model, or Aether as a whole, as "safe". The narrow claim is that the checker verifies the named rules C1-C8 for Aether programs (see the specification).

Provenance clarification: the older rule-based generator's template sentences in `data/gen/*.py` were typed by Claude Code, an AI coding assistant. Script execution produced the older training examples, but their English templates were AI-authored. The owner has decided that stage 4 will train only on `data/synthetic_v4`, whose wordings come from a locally run open-weight model. This note changes no published results.

	---
	license: apache-2.0
	base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
	library_name: peft
	tags: [lora, code, dsl, aether]
	---

	# Aether SOP-to-program LoRA v0.2 (Qwen2.5-Coder-1.5B-Instruct)

	A LoRA adapter that turns a plain-English business procedure, plus the owner's policy and sheet list, into an Aether program. The model is not a safety mechanism. Every program it writes must pass the Aether checker before it runs. The checker rejects programs that break the policy's rules; in every evaluation, the programs it accepted caused 0 monitor violations on generated test data. The model often writes such programs.

	Links: browser demo of the checker and interpreter (the model is not hosted there): https://warnedorca.github.io/aether/ · code: https://github.com/WarnedOrca/aether · benchmark and held-out set: https://huggingface.co/datasets/sidchalamala/aether-benchmark

	"correct" below means passes the checker and has the same effects as the reference program (scorer field `safe_and_correct`). It is not a safety rating.

	## Final evaluation: frozen held-out trap set (30 tasks, each model evaluated once)

	Each task asks for something the policy forbids or Aether cannot do. The correct answer is a program that does the permitted alternative. The set was frozen before any trap training data existed and was never used for training or decisions.

	\| System \| Correct \| 95% CI \|
	\|---\|---\|---\|
	\| Base model + reference card + 3 examples \| 0/30 \| 0.00-0.11 \|
	\| v0.1 (synthetic data only) \| 6/30 \| 0.10-0.37 \|
	\| v0.2 (this adapter: + 413 trap samples) \| 6/30 \| 0.10-0.37 \|

	- Negative result: adding 413 targeted trap samples did not improve held-out performance. v0.2 vs v0.1: 6 vs 6, paired exact McNemar p = 1.
	- v0.2 vs the base model: p = 0.031.
	- Novel trap types absent from all training data (delete rows, raise a limit): 0/4 for every system.
	- In every evaluation, the programs the checker accepted caused 0 monitor violations on generated test data. The model by itself solves 6/30 held-out traps.

	## Held-out v2 (90 tasks, 30 traps, 15 new business types; each condition run once)

	A second frozen set, written and hashed before any v0.3 training or tuning. Same scorer. 95% Wilson intervals.

	\| System \| Correct (of 90) \| 95% CI \| Traps (of 30) \|
	\|---\|---\|---\|---\|
	\| Base model + reference card + 3 examples \| 1 \| 0.00-0.06 \| 0 \|
	\| v0.2 plain (this adapter) \| 20 \| 0.15-0.32 \| 2 \|
	\| v0.2 with checker feedback \| 22 \| 0.17-0.34 \| 2 \|
	\| v0.2 grammar-constrained \| 19 \| 0.14-0.31 \| 2 \|
	\| v0.3 plain (not released) \| 14 \| 0.10-0.24 \| 3 \|
	\| v0.3 with checker feedback (not released) \| 15 \| 0.10-0.26 \| 3 \|
	\| v0.3 grammar-constrained (not released) \| 15 \| 0.10-0.26 \| 3 \|

	- Pre-registered primary comparison (null result): v0.3 with checker feedback vs this adapter (v0.2) plain: 15 vs 20, paired exact McNemar p = 0.33. The new data plus checker feedback did not improve on v0.2.
	- v0.2 plain vs the base model: 20 vs 1 (paired p = 2.1e-05; this test was not pre-registered).
	- Everything else in this table is exploratory. "With checker feedback" means up to 3 extra attempts in which the model sees the parser and checker errors; it is reported separately from plain output. "Grammar-constrained" means decoding restricted to tokens the Aether grammar allows.
	- v0.3 is not released. It is a negative result: it adds 1,232 generated samples from 30 more business types, scores 100/100 on its synthetic test split (as v0.2 does), and solves fewer held-out v2 tasks than v0.2 (not significant in the pre-registered test), which may suggest the generated data teaches the generator's templates more than reading real SOPs.

	## Held-out v3 (72 tasks, 36 traps, 18 new business types; the one-time final run of this adapter)

	Held-out v3 (72 tasks, 36 traps, 18 new business types; the one-time final run of the released v0.2, 2026-10-29; each condition run once): base model with 3 examples 2/72 (95% CI 0.01-0.10); v0.2 plain 18/72 (0.16-0.36; 13/36 ordinary tasks, 5/36 traps); v0.2 with checker feedback 22/72 (0.21-0.42). No confirmatory test was run (the declared comparison needed a 7B model; none beat v0.2 on dev). Post-hoc and exploratory only (not declared before the run): base vs v0.2 plain, exact McNemar p = 0.00014; v0.2 plain vs with feedback, p = 0.125 (not significant); `eval/results/stage5-posthoc.json`. The checker's accepted programs caused 0 monitor violations. Source: `eval/results/stage5/`.

	\| System \| Correct (of 72) \| 95% CI \| Ordinary (of 36), 95% CI \| Traps (of 36), 95% CI \| Parse rate \|
	\|---\|---\|---\|---\|---\|---\|
	\| Base model + reference card + 3 examples \| 2 \| 0.01-0.10 \| 2 (0.02-0.18) \| 0 (0.00-0.10) \| 0.32 \|
	\| v0.2 plain (this adapter) \| 18 \| 0.16-0.36 \| 13 (0.22-0.52) \| 5 (0.06-0.29) \| 0.81 \|
	\| v0.2 with checker feedback \| 22 \| 0.21-0.42 \| 14 (0.25-0.55) \| 8 (0.12-0.38) \| 0.79 \|

	- Feedback's 22 vs plain's 18 was not a pre-declared test. The two post-hoc tests above are exploratory: chosen after seeing the results, with no multiple-comparison adjustment.
	- Programs the checker accepted caused 0 monitor violations on generated test data.

	## Development benchmark (110 tasks, used during development)

	This benchmark was used to build the scorer and compare models during development, so treat it as a development set, not a clean test.

	\| System \| Correct \| Traps (30) \|
	\|---\|---\|---\|
	\| Base model + reference card + 3 examples \| 3/110 \| 0 \|
	\| v0.1 \| 30/110 \| 2 \|
	\| v0.2 \| 34/110 \| 3 \|

	v0.2 vs v0.1: paired p = 0.48.

	## Larger-model experiments (7B, development benchmark only): negative results

	After v0.2, three QLoRA adapters on Qwen2.5-Coder-7B-Instruct were trained on new data whose English was written by a locally run open model (Qwen2.5-7B-Instruct) from script-made programs. None was better than v0.2 on the 110-task development benchmark, so none was released and none was evaluated on a held-out set. Same scorer, greedy decoding, paired exact McNemar against v0.2 (34/110).

	\| Adapter (7B) \| Training data \| Correct /110 (epoch 1, epoch 2) \| p vs v0.2 (epoch 2) \| Source \|
	\|---\|---\|---\|---\|---\|
	\| q7b-v1 \| 898 local-model wordings of 515 programs \| 26, 30 \| 0.60 \| `eval/results/compare-dev-v0.2-vs-q7b-epoch*-2026-10-26.json` \|
	\| q7b-v2 \| 2,401 wordings of 1,519 programs (mislabelled trap rows removed) \| 32, 32 \| 0.85 \| `eval/results/dev-report-q7b-v2-2026-10-27.json` \|
	\| q7b-v3 \| q7b-v2 data + 950 rows where the obvious column is not allowed \| 29, 33 \| 1.0 \| `eval/results/dev-report-q7b-v3-session4.json` \|

	- With checker feedback on dev (exploratory): q7b-v2 epoch 2 35/110 and q7b-v3 epoch 2 36/110, against v0.2 36/110 (p = 1.0 both; `eval/results/dev-repair-compare-2026-10-28.json`, `eval/results/dev-repair-compare-session4.json`).
	- The 7B adapters' output parses more often (94.5-98.2% of the 110 tasks, against 78.2% for v0.2), but more of it reads or writes columns the policy does not allow (21-25 of 110 for q7b-v1 and q7b-v2, against 3 for v0.2) or does the wrong thing. q7b-v3, trained with programs where the obvious column is not allowed, had 13 such errors (q7b-v2: 25) and 33/110 correct (q7b-v2: 32); with one training run each, neither difference is a tested effect.
	- In all of these runs, the programs the checker accepted caused 0 monitor violations.

	## Training

	The v0.1 recipe (LoRA rank 16, learning rate 0.0002, 2 epochs, seed 42; only the program tokens are trained) on 2062 verified examples: 1649 synthetic plus 413 trap-style. Training took 42 minutes on one NVIDIA GeForce RTX 4070, at $0. The checkpoint was chosen by validation loss.

	## Limitations

	- Trained on templated synthetic English only. Most held-out failures are general errors on unfamiliar businesses (parse errors, type errors, reading columns the policy does not allow), not only trap-following.
	- All benchmark and held-out tasks were drafted by Claude, an AI assistant, not taken from real businesses.
	- Small evaluation sets with wide intervals; one training seed per model.
	- Aether's external actions (messages, money, calendars) are mocked. This is research software.
	- Do not describe this model, or Aether as a whole, as "safe". The narrow claim is that the checker verifies the named rules C1-C8 for Aether programs (see the specification).

	Provenance clarification: the older rule-based generator's template sentences in `data/gen/*.py` were typed by Claude Code, an AI coding assistant. Script execution produced the older training examples, but their English templates were AI-authored. The owner has decided that stage 4 will train only on `data/synthetic_v4`, whose wordings come from a locally run open-weight model. This note changes no published results.