| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3.5-0.8B |
| library_name: mlx |
| tags: |
| - lora |
| - bend |
| - program-synthesis |
| - in-weights-execution |
| --- |
| |
| # QwenBend: LoRA adapter |
|
|
| **Qwen3.5-0.8B writes a Bend program. Then the same model type-checks it and runs it inside its own |
| forward pass.** Code, design notes and the 15-second demo: **https://github.com/voidash/qwenbend** |
|
|
| This repository holds the **LoRA adapter** that teaches Qwen3.5-0.8B to write BendLite, a subset |
| of [Bend2](https://github.com/HigherOrderCO/Bend). |
|
|
| The engine half of QwenBend is not trained. Its weights are set by construction, from code in the |
| GitHub repo: |
| - a 16-register CPU with 8,192 words of memory held in Gated DeltaNet heads; |
| - the CPU applies Bend2's static rules (types, affine use, termination) and executes the program. |
|
|
| `python bendvm/qwenbend.py "your task"` there builds the full 3.97B model in memory in about 15 s |
| from Qwen3.5-0.8B and this adapter: |
| - 0.753B nonzero weights are Qwen's; |
| - 52.9M are the engine's; |
| - the rest is zero padding from widening the hidden size to 4096. |
|
|
| ## The adapter |
|
|
| - `adapter/adapters.safetensors` + `adapter/adapter_config.json`: **mlx_lm** LoRA format. |
| - rank 8, scale 20, the last 16 layers; |
| - trained with lr 1e-4, batch 2; |
| - the 400-step checkpoint was best by validation loss (training to 1,200 steps overfit). |
| - `qwen_fused.py` in the GitHub repo fuses it into the HF weights in torch: W += 20·(A·B)ᵀ. |
| - sha256 of `adapters.safetensors`: `4577f177fdaabd10ae0f920f71a93d8f89603236b05484822bf0393e4caf315c`. |
| |
| ## Measured |
| |
| On 200 test items: greedy, each program checked and run with real Bend2. |
| |
| | model | valid | correct | in-distribution correct | held-out families correct | |
| |---|---:|---:|---:|---:| |
| | Qwen3.5-0.8B (no adapter) | 0% | 0% | 0% | 0% | |
| | **+ this adapter** | 58% | **42%** | **80%** | 4% | |
|
|
| End to end in the merged model (Apple M4 Pro, MPS, bf16): **5/5** programs correct, and every |
| execution trace equals the reference interpreter's token for token. About 20 tokens/s. |
|
|
| ## Limits |
|
|
| The writer is a 0.8B model: it often writes the wrong program for unfamiliar tasks. The engine |
| guarantees the answer is exactly what the program computes, not that the program fits the question. |
|
|
| License: Apache-2.0 (same as the base model). Built at [Ampixa](https://ampixa.com). |
|
|
