qwenagent-workers / README.md
Bobliu's picture
Upload folder using huggingface_hub
1ab87e5 verified
|
Raw History Blame Contribute Delete
2.9 kB
---
license: apache-2.0
base_model: Qwen/Qwen3-0.6B
library_name: pytorch
tags:
- qwen3
- interpretability
- recurrent-workers
- research
- negative-result
---
# qwenagent worker checkpoints (Qwen3-0.6B, layer 12)
Small trainable "worker" modules (private GRU slot states, a learned router, a return channel) that are injected into a **frozen** Qwen3-0.6B at the output of layer 12.
These files contain **only the worker parameters (about 1.8 MB each)**; they do not contain, and are not a re-upload of, the Qwen3-0.6B weights, which you load from `Qwen/Qwen3-0.6B`.
- Code, task generator, results and explainer video: https://github.com/Bobliuuu/qwenagent
- The license shown is the base model's (Apache-2.0); the GitHub repo does not state a separate license.
## What is here
| file | mode | what it is |
|---|---|---|
| `checkpoints/q7_adapter_s0.pt` | adapter | stateless per-token MLP at layer 12, parameter-matched |
| `checkpoints/q7_single_s0.pt` | single | ONE persistent GRU state, parameter-matched |
| `checkpoints/q7_uniform_s0.pt` | uniform | K private states, every token updates every slot, no routing |
| `checkpoints/q7_multi_s0.pt` | multi | K=4 private states with a learned router (the "subagent" claim) |
| `checkpoints/q7_oracle_s0.pt` | oracle | K private states, router replaced by true register ids |
All are the **seed-0** run of `q7`: query task on interleaved register programs, trained on 2-3 registers with random register names, 8000 steps, K=4, layer 12, occupancy gating on.
Each `.pt` is `{"state": state_dict, "args": training args}`; the matching per-run result JSON is in `results/`.
## Results of these exact checkpoints (accuracy %, chance 10%, 200 programs per size, from `results/q7_*_s0.json`)
| mode | P=2 | P=3 | P=4 | P=6 | P=8 | P=12 | P=16 |
|---|---|---|---|---|---|---|---|
| adapter | 60.5 | 59.0 | 49.5 | 47.5 | 41.0 | 33.5 | 27.5 |
| single | 61.0 | 58.5 | 55.5 | 46.0 | 41.0 | 39.0 | 25.0 |
| uniform | 57.0 | 55.5 | 54.5 | 24.0 | 12.5 | 11.0 | 10.0 |
| multi | 53.5 | 51.0 | 53.5 | 43.5 | 38.0 | 33.5 | 26.5 |
| oracle | 48.0 | 48.5 | 41.5 | 27.0 | 13.5 | 11.0 | 12.5 |
Single seed. Over 5 seeds in the GitHub repo, **private slots do not beat a stateless adapter or a single shared state**, and the slot version is less stable across seeds.
This is a research artifact of a mostly negative result, not a model for deployment.
## How to load / run
```bash
git clone https://github.com/Bobliuuu/qwenagent && cd qwenagent
pip install torch transformers
huggingface-cli download Bobliu/qwenagent-workers --local-dir hf
python eval_ckpt.py hf/checkpoints/q7_multi_s0.pt --out /tmp/probe_q7_multi_s0.json # reloads Qwen/Qwen3-0.6B plus the workers
```
`eval_ckpt.py` rebuilds the model and the layer-12 hook from `train.py`, so the frozen base model is downloaded from `Qwen/Qwen3-0.6B` on first use. Training needs a CUDA GPU.