qwenagent-workers / README.md
Bobliu's picture
Upload folder using huggingface_hub
1ab87e5 verified
|
Raw History Blame Contribute Delete
2.9 kB
metadata
license: apache-2.0
base_model: Qwen/Qwen3-0.6B
library_name: pytorch
tags:
  - qwen3
  - interpretability
  - recurrent-workers
  - research
  - negative-result

qwenagent worker checkpoints (Qwen3-0.6B, layer 12)

Small trainable "worker" modules (private GRU slot states, a learned router, a return channel) that are injected into a frozen Qwen3-0.6B at the output of layer 12. These files contain only the worker parameters (about 1.8 MB each); they do not contain, and are not a re-upload of, the Qwen3-0.6B weights, which you load from Qwen/Qwen3-0.6B.

  • Code, task generator, results and explainer video: https://github.com/Bobliuuu/qwenagent
  • The license shown is the base model's (Apache-2.0); the GitHub repo does not state a separate license.

What is here

file mode what it is
checkpoints/q7_adapter_s0.pt adapter stateless per-token MLP at layer 12, parameter-matched
checkpoints/q7_single_s0.pt single ONE persistent GRU state, parameter-matched
checkpoints/q7_uniform_s0.pt uniform K private states, every token updates every slot, no routing
checkpoints/q7_multi_s0.pt multi K=4 private states with a learned router (the "subagent" claim)
checkpoints/q7_oracle_s0.pt oracle K private states, router replaced by true register ids

All are the seed-0 run of q7: query task on interleaved register programs, trained on 2-3 registers with random register names, 8000 steps, K=4, layer 12, occupancy gating on. Each .pt is {"state": state_dict, "args": training args}; the matching per-run result JSON is in results/.

Results of these exact checkpoints (accuracy %, chance 10%, 200 programs per size, from results/q7_*_s0.json)

mode P=2 P=3 P=4 P=6 P=8 P=12 P=16
adapter 60.5 59.0 49.5 47.5 41.0 33.5 27.5
single 61.0 58.5 55.5 46.0 41.0 39.0 25.0
uniform 57.0 55.5 54.5 24.0 12.5 11.0 10.0
multi 53.5 51.0 53.5 43.5 38.0 33.5 26.5
oracle 48.0 48.5 41.5 27.0 13.5 11.0 12.5

Single seed. Over 5 seeds in the GitHub repo, private slots do not beat a stateless adapter or a single shared state, and the slot version is less stable across seeds. This is a research artifact of a mostly negative result, not a model for deployment.

How to load / run

git clone https://github.com/Bobliuuu/qwenagent && cd qwenagent
pip install torch transformers
huggingface-cli download Bobliu/qwenagent-workers --local-dir hf
python eval_ckpt.py hf/checkpoints/q7_multi_s0.pt --out /tmp/probe_q7_multi_s0.json   # reloads Qwen/Qwen3-0.6B plus the workers

eval_ckpt.py rebuilds the model and the layer-12 hook from train.py, so the frozen base model is downloaded from Qwen/Qwen3-0.6B on first use. Training needs a CUDA GPU.