--- license: apache-2.0 base_model: Qwen/Qwen3-0.6B library_name: pytorch tags: - qwen3 - interpretability - recurrent-workers - research - negative-result --- # qwenagent worker checkpoints (Qwen3-0.6B, layer 12) Small trainable "worker" modules (private GRU slot states, a learned router, a return channel) that are injected into a **frozen** Qwen3-0.6B at the output of layer 12. These files contain **only the worker parameters (about 1.8 MB each)**; they do not contain, and are not a re-upload of, the Qwen3-0.6B weights, which you load from `Qwen/Qwen3-0.6B`. - Code, task generator, results and explainer video: https://github.com/Bobliuuu/qwenagent - The license shown is the base model's (Apache-2.0); the GitHub repo does not state a separate license. ## What is here | file | mode | what it is | |---|---|---| | `checkpoints/q7_adapter_s0.pt` | adapter | stateless per-token MLP at layer 12, parameter-matched | | `checkpoints/q7_single_s0.pt` | single | ONE persistent GRU state, parameter-matched | | `checkpoints/q7_uniform_s0.pt` | uniform | K private states, every token updates every slot, no routing | | `checkpoints/q7_multi_s0.pt` | multi | K=4 private states with a learned router (the "subagent" claim) | | `checkpoints/q7_oracle_s0.pt` | oracle | K private states, router replaced by true register ids | All are the **seed-0** run of `q7`: query task on interleaved register programs, trained on 2-3 registers with random register names, 8000 steps, K=4, layer 12, occupancy gating on. Each `.pt` is `{"state": state_dict, "args": training args}`; the matching per-run result JSON is in `results/`. ## Results of these exact checkpoints (accuracy %, chance 10%, 200 programs per size, from `results/q7_*_s0.json`) | mode | P=2 | P=3 | P=4 | P=6 | P=8 | P=12 | P=16 | |---|---|---|---|---|---|---|---| | adapter | 60.5 | 59.0 | 49.5 | 47.5 | 41.0 | 33.5 | 27.5 | | single | 61.0 | 58.5 | 55.5 | 46.0 | 41.0 | 39.0 | 25.0 | | uniform | 57.0 | 55.5 | 54.5 | 24.0 | 12.5 | 11.0 | 10.0 | | multi | 53.5 | 51.0 | 53.5 | 43.5 | 38.0 | 33.5 | 26.5 | | oracle | 48.0 | 48.5 | 41.5 | 27.0 | 13.5 | 11.0 | 12.5 | Single seed. Over 5 seeds in the GitHub repo, **private slots do not beat a stateless adapter or a single shared state**, and the slot version is less stable across seeds. This is a research artifact of a mostly negative result, not a model for deployment. ## How to load / run ```bash git clone https://github.com/Bobliuuu/qwenagent && cd qwenagent pip install torch transformers huggingface-cli download Bobliu/qwenagent-workers --local-dir hf python eval_ckpt.py hf/checkpoints/q7_multi_s0.pt --out /tmp/probe_q7_multi_s0.json # reloads Qwen/Qwen3-0.6B plus the workers ``` `eval_ckpt.py` rebuilds the model and the layer-12 hook from `train.py`, so the frozen base model is downloaded from `Qwen/Qwen3-0.6B` on first use. Training needs a CUDA GPU.