ceselder/qwen3.6-27b-nla-rl

Qwen3.6-27B Natural Language Autoencoder (NLA)

This is a Natural Language Autoencoder (Anthropic, 2026) trained for Qwen/Qwen3.6-27B on layer-42 residual-stream activations using easyNLA.

This repo holds the full set of GRPO-RL checkpoints from the final training run, plus the base model they attach to and the reconstructor used to score them.

The model starts its RL at 53% FVE, and converges around 78% FVE after 800 steps

It was trained on batch size 522, with group size 8, to reduce noise and because it converged higher. Therefore, the steps aren't very comparable to Anthropics, who used a smaller batch size

I don't know what checkpoint is most useful, my guess is 300.

For issues, please DM me on twitter

Full held-out FVE trajectory (every 10 steps)

step	held-out FVE	reward
0	53.0%	-0.210
10	65.7%	-0.153
20	68.0%	-0.143
30	68.0%	-0.143
40	69.4%	-0.137
50	70.3%	-0.133
60	69.4%	-0.137
70	70.2%	-0.133
80	71.2%	-0.129
90	71.9%	-0.126
100	72.6%	-0.123
110	72.3%	-0.124
120	72.6%	-0.123
130	72.4%	-0.123
140	73.5%	-0.119
150	73.5%	-0.119
160	73.4%	-0.119
170	73.9%	-0.117
180	73.3%	-0.120
190	74.6%	-0.114
200	74.2%	-0.115
210	73.9%	-0.117
220	74.6%	-0.114
230	73.6%	-0.118
240	74.6%	-0.114
250	74.0%	-0.116
260	74.7%	-0.113
270	75.0%	-0.112
280	75.5%	-0.110
290	74.5%	-0.114
300	75.6%	-0.109
310	75.7%	-0.109
320	75.5%	-0.124
330	75.8%	-0.123
340	75.2%	-0.111
350	75.4%	-0.110
360	75.6%	-0.109
370	75.9%	-0.123
380	75.9%	-0.108
390	76.3%	-0.106
400	76.6%	-0.105
410	76.5%	-0.120
420	76.3%	-0.106
430	76.5%	-0.120
440	76.8%	-0.104
450	76.7%	-0.134
460	76.3%	-0.121
470	76.5%	-0.105
480	76.4%	-0.106
490	76.8%	-0.104
500	76.2%	-0.106
510	77.6%	-0.100
520	76.3%	-0.106
530	76.7%	-0.104
540	77.7%	-0.115
550	76.8%	-0.104
560	77.0%	-0.103
570	77.4%	-0.101
580	76.3%	-0.121
590	77.5%	-0.101

Claude-slop, that may be useful:

Repository contents

• av_base/ — base for every adapter. Qwen3.6-27B with the warmstart AV LoRA (r128, all-module) already merged in. A full Qwen3_5ForCausalLM in bf16 with tokenizer + chat template. Usable standalone as the pre-RL verbalizer.
• av_rl_adapters/ — the GRPO-RL verbalizer LoRA adapters (r64, rsLoRA, all-module), one saved every 100 steps. Each attaches on top of av_base:
  • av_rl_adapters/iter_000100 — RL step 100
  • av_rl_adapters/iter_000200 — RL step 200
  • av_rl_adapters/iter_000300 — RL step 300
  • av_rl_adapters/iter_000400 — RL step 400
  • av_rl_adapters/iter_000500 — RL step 500
  • av_rl_adapters/iter_000600 — RL step 600
• ar_reconstructor/ — the AR reconstructor at the end of the run (43-block truncated Qwen3_5ForCausalLM backbone + value_head.safetensors). Latest only — the AR is co-trained during RL but checkpointed once, not per-iter (see Caveats).
• nla_meta.yaml — extraction config, injection-token ids, and the exact AV/AR prompt templates.
• run_config.yaml — the exact RL hyperparameters.
• fve_trajectory.json — every held-out eval number behind the results above.
• data/example_activations.parquet — 64 sample layer-42 activations with their source text, so you can see what feeds the AV (and preview it in the HF viewer).

How to use

1. Load the verbalizer (base + a chosen RL adapter):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
from huggingface_hub import snapshot_download

repo = snapshot_download("ceselder/qwen3.6-27b-nla-rl")
tok  = AutoTokenizer.from_pretrained(f"{repo}/av_base")
base = AutoModelForCausalLM.from_pretrained(f"{repo}/av_base", torch_dtype=torch.bfloat16, device_map="cuda")
av   = PeftModel.from_pretrained(base, f"{repo}/av_rl_adapters/iter_000600")  # latest RL step
av.eval()

2. Verbalize an activation. The activation vector (d_model = 5120, extracted at layer 42) is injected norm-matched at layer 1 in place of the marker token ㈜ (id 158983). See nla_meta.yaml for the exact prompt template and injection ids, and the EasyNLA repo (scripts/show_nla_generations.py, register_karvonen_hook) for the injection hook. The AV then generates an <explanation>...</explanation>.

3. Score faithfulness with the AR (ar_reconstructor/): feed the explanation text through the reconstructor and read the value head → predicted activation → FVE vs the original.

Apply the chat template with enable_thinking=False — the Qwen3.6 template otherwise opens a bare <think> block.

Training pipeline

1. AV warmstart (SFT). LoRA --lora-scope all (attn + MLP + deltanet decay, r128) on activation→explanation pairs, merged → av_base.
2. AR SFT. Trained the 43-block reconstructor + value head to rebuild activations from explanations.
3. GRPO RL (this run). On-policy GRPO (group-relative advantage, no importance ratio), group size 8, temperature 1.0, k3 KL estimator (β=0.01), hinged length penalty (0.01/token over 192). Actor = fresh r64 rsLoRA (all-module) on av_base; critic (AR) co-trained (r64 LoRA). Batch 504 prompts, 6×B200, ~440 s/step. Reward = −reconstruction MSE. See run_config.yaml for everything.

Model / architecture notes

• Qwen3.6-27B is model_type qwen3_5: a hybrid 64-layer stack (48 gated-DeltaNet linear-attention + 16 full-attention), d_model 5120, vocab 248320.
• Activations extracted at layer 42 = ⌊2·64/3⌋. The AR backbone is 43 blocks.
• Injection char ㈜ (id 158983), neighbors (29, 510); no activation normalization (norm: none).

Caveats

• FVE is not comparable across base models. It is normalized by each model's predict-the-mean baseline variance, which differs per model. Compare FVE only within this model/layer.
• The AR is latest-only. Adapters are per-100-steps but the reconstructor is checkpointed once at the end. The exact per-step eval FVE was measured against the AR as it was at that step; to reproduce a number exactly you'd need that step's AR, which isn't saved. For use, pair any AV adapter with the provided (final) AR.
• base_model_name_or_path in the adapter configs points at this repo; the real base weights are in av_base/. Load av_base explicitly as shown above.

Provenance

• Warmstart data derived from ceselder/qwen3-8b-nla-L24-finefineweb-100k (texts + explanations reused; activations regenerated at L42).
• Built with EasyNLA (on nanoNLA).
• Interpretability research (MATS).