File size: 4,153 Bytes
e29f88c 01b4b8a e29f88c dfc0949 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | ---
language: en
base_model: Qwen/Qwen2.5-0.5B-Instruct
tags:
- loopformer
- recurrent-transformer
- pointer-chasing
- custom-code
---
# LoopFormer: recurrent pointer execution
Frozen final checkpoint: `controller-shared-number-precision-seed83`.
This is a custom recurrent Qwen model for following a supplied A–Z rule table.
It requires the included Python inference code; ordinary `AutoModelForCausalLM`
generation does not implement its recurrence.
## Download and run
Use Python 3.11 or 3.12. Install a PyTorch build suitable for your device.
The requirements record the validated research dependency versions; the Space
adds Gradio and `spaces` for hosting. Export a graph request from the playground,
or create JSON with `mapping` (26 ordered `[source, destination]` pairs), `start`
(one A–Z letter), and `steps` (a positive integer).
```bash
hf download mukunds/loopformer --local-dir loopformer-model
cd loopformer-model
python -m pip install -r requirements.txt
python -m scripts.demo --checkpoint . --request graph.json --device cuda --download
```
The explicit `--download` option permits downloading the pinned Qwen base
revision. It is needed on the first run. `--device cpu` and `--device mps` are
also available, but their latency and final-checkpoint fidelity must be measured
on the selected device. Each output line is a cumulative JSON snapshot of actual
model predictions; the final line includes separate correctness/stopping metrics.
The default requested-step ceiling is 256 and the safety cap is 272. To explore
longer rollouts, explicitly increase both `--max-steps` and `--loop-cap`. This
does not establish graph-execution accuracy beyond the evaluated 256 steps.
The symbol vocabulary remains exactly A–Z; 256 steps does not mean 256 nodes.
## Architecture and inference
The base is `Qwen/Qwen2.5-0.5B-Instruct`, revision
`7ae557604adf67be50417f59c2c2f167def9a775`. Six prelude layers prepare the prompt;
twelve shared middle layers update a recurrent working state; six coda layers
read out a distribution over the 26 letters after every loop. A trained bridge
re-enters the shared middle block. Fixed prefix memory is reused within a run.
The final executor adapts all R weights and freezes P/C.
The Steps field is routed away from the executor. A shared learned number reader
initializes a separate scalar timer, whose learned affine recurrence and stop
readout determine completion. Stop probability is a timing signal, not a
correctness estimate. Decoded predictions and reference states are never fed
back into recurrence. Default inference is FP32 with SDPA and threshold 0.5.
The downloadable runtime uses research PyTorch 2.14.0; HF ZeroGPU requires
PyTorch 2.13.0 for the live Space. All 280 checked intermediate predictions,
stopping summaries and stop probabilities matched the audited saved runs on
that hosted runtime. This bounded check is not full-population equivalence.
`adapter_model.pt` contains all adapted executor/controller tensors, not merely
LoRA weights. SHA-256:
`0f8a3f2572ec2caf4fafbcdd074b5283bd8c610602502763b3c04487ce1f123d`.
Optimizer state and training data are excluded from this inference release.
## Evidence and limitations
On the frozen final panel of 1,350 new graphs and requests 1–256:
- Final letter plus exact stopping: 97.81% over 345,600 graph/count queries.
- Complete trajectories through depth 256: 96.89% (1,308/1,350 graphs).
- Exact stopping: 100% across that finite requested-count panel.
Executor supervision used depths 1–6, 8, 10, 12. Controller fitting used
counts 1–63 excluding 9, 17, 29, 41, 53, with supervised unrolls capped at 12.
Controller-only numerical results at longer horizons are distinct from executing
graphs correctly. This is one task family, not general reasoning or transfer
evidence. Arbitrary hand-edited graphs can fail. Changes to precision, runtime,
hardware or kernels need validation against native intermediate readouts.
See the [source and full evidence](https://github.com/MukundShankar-dev/loopformer)
and the [interactive playground](https://mukundshankar-dev.github.io/projects/loopformer/demo/).
|