---
license: apache-2.0
language:
  - en
base_model: Qwen/Qwen3.5-4B
library_name: transformers
pipeline_tag: text-generation
tags:
  - qwen3.5
  - tinyllama
  - riscv
  - int4
  - in-weights
  - compiled-weights
  - exact-arithmetic
  - qwenti
---

# Qwenti inference: TinyLLama-v0 inside Qwen3.5-4B

This is [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B) with a token-level RV32 machine
compiled into its transformer stack, and TinyLLama-v0 compiled into that machine's weight memory.
Thirty-six circuit layers sit in front of Qwen's thirty-two. No weight in them was trained: a
compiler computes all of them.

The method comes from [Qwenputer](https://github.com/divagr18/qwenputer) by Divyansh Agrawal, which
compiles a computer into a model's weights and drives it through the model's own token stream. This
repository applies it to a 4B base and puts a language model inside the machine. The smaller sibling
[`OpceanAI/Qwen3.5-0.8B-abliterated-Qwenti`](https://huggingface.co/OpceanAI/Qwen3.5-0.8B-abliterated-Qwenti)
does the same with a CHIP-8 machine, and its `overlay` field is what showed how to keep the machine's
readout out of the text path.

## Measured

| | |
|---|---|
| Parameters inside the machine | 4,621,392 |
| Weight memory | 5,229,568 cells of 4 bits, 2.49 MB, 639 of 768 units |
| Circuit | 36 layers, 457 heads, 3,259 dims |
| Merged model | 68 layers, hidden 4,096, 27.4 GB in bf16 |
| Text agreement with the base, first token, 6 prompts | 6 of 6 |
| KL(base against this model), 96 positions | 6.5e-4 |
| Top-1 agreement over the same positions | 100% |
| `The capital of France is` | ` Paris.` |
| Decode on one Colab G4 | 23.2 tokens/s |
| Peak VRAM | 61.9 GB |
| Driven with a chip token | emits a chip token, best text logit at 0.0 |
| Chip stream after `GO`, no state injected | 17 of 24 tokens stay in the machine |
| TinyLLama tokens from the verified kernel | 46 in 618 s, 37 of 40 equal to fp32 |

![40-token agreement](figures/fig7_forty_tokens.png)

## What works and what does not

The text path matches the base model. The machine's own readout is gated by `mu`, so at a text
position the head sees no output code and answers text, and at a chip position it sees the code and
answers in the machine's vocabulary.

The machine executes TinyLLama correctly on the reference implementation and on the circuit: five
checkpoints of the real program compared token by token, zero disagreements, and forty greedy tokens
that match a float model holding the same int4 weights on all forty.

![minutes per TinyLLama token on the G4](figures/fig8_speed.png)

Running the guest program inside this checkpoint is correct but slow. One TinyLLama token costs
1,821,283 machine instructions, or 2,619,430 trace tokens, and this model processes about 23 chip
positions per second, so a token takes roughly 29 hours. The Triton executor runs the circuit alone
at 1,263 positions per second, which brings it to 33 minutes per token. Neither is a chat path.

An earlier build of this checkpoint answered `!` instead of text. The cause was the readout: the text
bridge writes an echo code on text tokens, the final norm scales the code dimensions by 2^20, and a
residue of 1e-2 in them became a 1e4 logit. Section 11 of the paper records the four probes that
found it and the fix.

![instruction mix of one model token](figures/fig5_instruction_mix.png)

## How to use

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("OpceanAI/qwenti-inference")
model = AutoModelForCausalLM.from_pretrained(
    "OpceanAI/qwenti-inference", dtype=torch.bfloat16, device_map="cuda").eval()

ids = tok("The capital of France is", return_tensors="pt").to("cuda")
out = model.generate(**ids, max_new_tokens=20, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
```

That prints ` Paris.` followed by the same completion the base model gives. The machine's vocabulary
is appended after Qwen's, and `chipformer.json` records the layout (`chip_token_offset`,
`chip_layers`, `chip_dims_offset`, `slots`, `code_dims`, `tau_dim`, `mu_dim`, `tool_call`,
`printable`). Driving the guest program needs the machine state injected into the cache, which the
`chipformer` package does with `core.snapshot` and `RvChat`.

## Why there is no bf16 build of the machine

| storage | size for 4,621,392 parameters |
|---|---:|
| int4, the weight memory | 2.20 MB |
| bf16 or fp16 | 8.81 MB |
| fp32 | 17.63 MB |

![storage against the weight memory](figures/fig9_storage.png)

The weight memory holds 2.49 MB, which in bf16 is 1,307,392 weights. The model is 3.53 times larger,
so a bf16 build has to live as model tensors, and the `VDOT` instruction stops being the dot product.

## Files

| Path | What |
|---|---|
| `PAPER-tinyllama-rv32.pdf` | the paper, 13 pages, figures and tables typeset |
| `PAPER-tinyllama-rv32.md` | the paper source |
| `DESIGN-tinyllama.md` | the working design log, including every wall and its fix |
| `figures/` | instruction mix, quantization cost, 40-token agreement, speed, storage |
| `results.json` | the raw result document of the G4 run |
| `scripts/` | kernel, assembly matvec, exporter, Python mirror, circuit checker, G4 driver, PDF builder |
| `colab_bundle.zip` | the patched tree and scripts, ready for a Colab VM |

## Hardware

Colab G4: NVIDIA RTX PRO 6000 Blackwell Server Edition, 95.6 GB of VRAM, driver 580.82.07, torch
2.14.1+cu130, CUDA 13.0, capability 12.0, 48 vCPU, 176.9 GB of RAM. Built with clang and ld.lld
targeting `riscv32-unknown-elf` on a machine with no GPU.

## Citation

```bibtex
@misc{qwenti-inference-2026,
  title  = {A 4.6M-parameter language model as an int4 workload on a token-level RISC-V machine},
  author = {OpceanAI},
  year   = {2026},
  url    = {https://huggingface.co/OpceanAI/qwenti-inference}
}
```