Instructions to use OpceanAI/qwenti-inference with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OpceanAI/qwenti-inference with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="OpceanAI/qwenti-inference")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("OpceanAI/qwenti-inference")
model = AutoModelForCausalLM.from_pretrained("OpceanAI/qwenti-inference", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use OpceanAI/qwenti-inference with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OpceanAI/qwenti-inference"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpceanAI/qwenti-inference",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/OpceanAI/qwenti-inference

SGLang

How to use OpceanAI/qwenti-inference with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OpceanAI/qwenti-inference" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpceanAI/qwenti-inference",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OpceanAI/qwenti-inference" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpceanAI/qwenti-inference",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use OpceanAI/qwenti-inference with Docker Model Runner:
```
docker model run hf.co/OpceanAI/qwenti-inference
```

qwenti-inference / README.md

OpceanAI

Card: rewritten, credits Qwenputer, figures restored

5f77090 verified 2 days ago

preview code

Raw History Blame Contribute Delete

5.91 kB

---
license: apache-2.0
language:
  - en
base_model: Qwen/Qwen3.5-4B
library_name: transformers
pipeline_tag: text-generation
tags:
  - qwen3.5
  - tinyllama
  - riscv
  - int4
  - in-weights
  - compiled-weights
  - exact-arithmetic
  - qwenti
---
# Qwenti inference: TinyLLama-v0 inside Qwen3.5-4B

This is [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B) with a token-level RV32 machine
compiled into its transformer stack, and TinyLLama-v0 compiled into that machine's weight memory.
Thirty-six circuit layers sit in front of Qwen's thirty-two. No weight in them was trained: a
compiler computes all of them.

The method comes from [Qwenputer](https://github.com/divagr18/qwenputer) by Divyansh Agrawal, which
compiles a computer into a model's weights and drives it through the model's own token stream. This
repository applies it to a 4B base and puts a language model inside the machine. The smaller sibling
[`OpceanAI/Qwen3.5-0.8B-abliterated-Qwenti`](https://huggingface.co/OpceanAI/Qwen3.5-0.8B-abliterated-Qwenti)
does the same with a CHIP-8 machine, and its `overlay` field is what showed how to keep the machine's
readout out of the text path.

## Measured

| | |
|---|---|
| Parameters inside the machine | 4,621,392 |
| Weight memory | 5,229,568 cells of 4 bits, 2.49 MB, 639 of 768 units |
| Circuit | 36 layers, 457 heads, 3,259 dims |
| Merged model | 68 layers, hidden 4,096, 27.4 GB in bf16 |
| Text agreement with the base, first token, 6 prompts | 6 of 6 |
| KL(base against this model), 96 positions | 6.5e-4 |
| Top-1 agreement over the same positions | 100% |
| `The capital of France is` | ` Paris.` |
| Decode on one Colab G4 | 23.2 tokens/s |
| Peak VRAM | 61.9 GB |
| Driven with a chip token | emits a chip token, best text logit at 0.0 |
| Chip stream after `GO`, no state injected | 17 of 24 tokens stay in the machine |
| TinyLLama tokens from the verified kernel | 46 in 618 s, 37 of 40 equal to fp32 |

![40-token agreement](figures/fig7_forty_tokens.png)

## What works and what does not

The text path matches the base model. The machine's own readout is gated by `mu`, so at a text
position the head sees no output code and answers text, and at a chip position it sees the code and
answers in the machine's vocabulary.

The machine executes TinyLLama correctly on the reference implementation and on the circuit: five
checkpoints of the real program compared token by token, zero disagreements, and forty greedy tokens
that match a float model holding the same int4 weights on all forty.

![minutes per TinyLLama token on the G4](figures/fig8_speed.png)

Running the guest program inside this checkpoint is correct but slow. One TinyLLama token costs
1,821,283 machine instructions, or 2,619,430 trace tokens, and this model processes about 23 chip
positions per second, so a token takes roughly 29 hours. The Triton executor runs the circuit alone
at 1,263 positions per second, which brings it to 33 minutes per token. Neither is a chat path.

An earlier build of this checkpoint answered `!` instead of text. The cause was the readout: the text
bridge writes an echo code on text tokens, the final norm scales the code dimensions by 2^20, and a
residue of 1e-2 in them became a 1e4 logit. Section 11 of the paper records the four probes that
found it and the fix.

![instruction mix of one model token](figures/fig5_instruction_mix.png)

## How to use

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("OpceanAI/qwenti-inference")
model = AutoModelForCausalLM.from_pretrained(
    "OpceanAI/qwenti-inference", dtype=torch.bfloat16, device_map="cuda").eval()
ids = tok("The capital of France is", return_tensors="pt").to("cuda")
out = model.generate(**ids, max_new_tokens=20, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
```

That prints ` Paris.` followed by the same completion the base model gives. The machine's vocabulary
is appended after Qwen's, and `chipformer.json` records the layout (`chip_token_offset`,
`chip_layers`, `chip_dims_offset`, `slots`, `code_dims`, `tau_dim`, `mu_dim`, `tool_call`,
`printable`). Driving the guest program needs the machine state injected into the cache, which the
`chipformer` package does with `core.snapshot` and `RvChat`.

## Why there is no bf16 build of the machine

| storage | size for 4,621,392 parameters |
|---|---:|
| int4, the weight memory | 2.20 MB |
| bf16 or fp16 | 8.81 MB |
| fp32 | 17.63 MB |

![storage against the weight memory](figures/fig9_storage.png)

The weight memory holds 2.49 MB, which in bf16 is 1,307,392 weights. The model is 3.53 times larger,
so a bf16 build has to live as model tensors, and the `VDOT` instruction stops being the dot product.

## Files

| Path | What |
|---|---|
| `PAPER-tinyllama-rv32.pdf` | the paper, 13 pages, figures and tables typeset |
| `PAPER-tinyllama-rv32.md` | the paper source |
| `DESIGN-tinyllama.md` | the working design log, including every wall and its fix |
| `figures/` | instruction mix, quantization cost, 40-token agreement, speed, storage |
| `results.json` | the raw result document of the G4 run |
| `scripts/` | kernel, assembly matvec, exporter, Python mirror, circuit checker, G4 driver, PDF builder |
| `colab_bundle.zip` | the patched tree and scripts, ready for a Colab VM |

## Hardware

Colab G4: NVIDIA RTX PRO 6000 Blackwell Server Edition, 95.6 GB of VRAM, driver 580.82.07, torch
2.14.1+cu130, CUDA 13.0, capability 12.0, 48 vCPU, 176.9 GB of RAM. Built with clang and ld.lld
targeting `riscv32-unknown-elf` on a machine with no GPU.

## Citation

```bibtex
@misc{qwenti-inference-2026,
  title  = {A 4.6M-parameter language model as an int4 workload on a token-level RISC-V machine},
  author = {OpceanAI},
  year   = {2026},
  url    = {https://huggingface.co/OpceanAI/qwenti-inference}
}
```

	---
	license: apache-2.0
	language:
	- en
	base_model: Qwen/Qwen3.5-4B
	library_name: transformers
	pipeline_tag: text-generation
	tags:
	- qwen3.5
	- tinyllama
	- riscv
	- int4
	- in-weights
	- compiled-weights
	- exact-arithmetic
	- qwenti
	---

	# Qwenti inference: TinyLLama-v0 inside Qwen3.5-4B

	This is [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B) with a token-level RV32 machine
	compiled into its transformer stack, and TinyLLama-v0 compiled into that machine's weight memory.
	Thirty-six circuit layers sit in front of Qwen's thirty-two. No weight in them was trained: a
	compiler computes all of them.

	The method comes from [Qwenputer](https://github.com/divagr18/qwenputer) by Divyansh Agrawal, which
	compiles a computer into a model's weights and drives it through the model's own token stream. This
	repository applies it to a 4B base and puts a language model inside the machine. The smaller sibling
	[`OpceanAI/Qwen3.5-0.8B-abliterated-Qwenti`](https://huggingface.co/OpceanAI/Qwen3.5-0.8B-abliterated-Qwenti)
	does the same with a CHIP-8 machine, and its `overlay` field is what showed how to keep the machine's
	readout out of the text path.

	## Measured

	\| \| \|
	\|---\|---\|
	\| Parameters inside the machine \| 4,621,392 \|
	\| Weight memory \| 5,229,568 cells of 4 bits, 2.49 MB, 639 of 768 units \|
	\| Circuit \| 36 layers, 457 heads, 3,259 dims \|
	\| Merged model \| 68 layers, hidden 4,096, 27.4 GB in bf16 \|
	\| Text agreement with the base, first token, 6 prompts \| 6 of 6 \|
	\| KL(base against this model), 96 positions \| 6.5e-4 \|
	\| Top-1 agreement over the same positions \| 100% \|
	\| `The capital of France is` \| ` Paris.` \|
	\| Decode on one Colab G4 \| 23.2 tokens/s \|
	\| Peak VRAM \| 61.9 GB \|
	\| Driven with a chip token \| emits a chip token, best text logit at 0.0 \|
	\| Chip stream after `GO`, no state injected \| 17 of 24 tokens stay in the machine \|
	\| TinyLLama tokens from the verified kernel \| 46 in 618 s, 37 of 40 equal to fp32 \|

	![40-token agreement](figures/fig7_forty_tokens.png)

	## What works and what does not

	The text path matches the base model. The machine's own readout is gated by `mu`, so at a text
	position the head sees no output code and answers text, and at a chip position it sees the code and
	answers in the machine's vocabulary.

	The machine executes TinyLLama correctly on the reference implementation and on the circuit: five
	checkpoints of the real program compared token by token, zero disagreements, and forty greedy tokens
	that match a float model holding the same int4 weights on all forty.

	![minutes per TinyLLama token on the G4](figures/fig8_speed.png)

	Running the guest program inside this checkpoint is correct but slow. One TinyLLama token costs
	1,821,283 machine instructions, or 2,619,430 trace tokens, and this model processes about 23 chip
	positions per second, so a token takes roughly 29 hours. The Triton executor runs the circuit alone
	at 1,263 positions per second, which brings it to 33 minutes per token. Neither is a chat path.

	An earlier build of this checkpoint answered `!` instead of text. The cause was the readout: the text
	bridge writes an echo code on text tokens, the final norm scales the code dimensions by 2^20, and a
	residue of 1e-2 in them became a 1e4 logit. Section 11 of the paper records the four probes that
	found it and the fix.

	![instruction mix of one model token](figures/fig5_instruction_mix.png)

	## How to use

	```python
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer

	tok = AutoTokenizer.from_pretrained("OpceanAI/qwenti-inference")
	model = AutoModelForCausalLM.from_pretrained(
	"OpceanAI/qwenti-inference", dtype=torch.bfloat16, device_map="cuda").eval()

	ids = tok("The capital of France is", return_tensors="pt").to("cuda")
	out = model.generate(**ids, max_new_tokens=20, do_sample=False)
	print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
	```

	That prints ` Paris.` followed by the same completion the base model gives. The machine's vocabulary
	is appended after Qwen's, and `chipformer.json` records the layout (`chip_token_offset`,
	`chip_layers`, `chip_dims_offset`, `slots`, `code_dims`, `tau_dim`, `mu_dim`, `tool_call`,
	`printable`). Driving the guest program needs the machine state injected into the cache, which the
	`chipformer` package does with `core.snapshot` and `RvChat`.

	## Why there is no bf16 build of the machine

	\| storage \| size for 4,621,392 parameters \|
	\|---\|---:\|
	\| int4, the weight memory \| 2.20 MB \|
	\| bf16 or fp16 \| 8.81 MB \|
	\| fp32 \| 17.63 MB \|

	![storage against the weight memory](figures/fig9_storage.png)

	The weight memory holds 2.49 MB, which in bf16 is 1,307,392 weights. The model is 3.53 times larger,
	so a bf16 build has to live as model tensors, and the `VDOT` instruction stops being the dot product.

	## Files

	\| Path \| What \|
	\|---\|---\|
	\| `PAPER-tinyllama-rv32.pdf` \| the paper, 13 pages, figures and tables typeset \|
	\| `PAPER-tinyllama-rv32.md` \| the paper source \|
	\| `DESIGN-tinyllama.md` \| the working design log, including every wall and its fix \|
	\| `figures/` \| instruction mix, quantization cost, 40-token agreement, speed, storage \|
	\| `results.json` \| the raw result document of the G4 run \|
	\| `scripts/` \| kernel, assembly matvec, exporter, Python mirror, circuit checker, G4 driver, PDF builder \|
	\| `colab_bundle.zip` \| the patched tree and scripts, ready for a Colab VM \|

	## Hardware

	Colab G4: NVIDIA RTX PRO 6000 Blackwell Server Edition, 95.6 GB of VRAM, driver 580.82.07, torch
	2.14.1+cu130, CUDA 13.0, capability 12.0, 48 vCPU, 176.9 GB of RAM. Built with clang and ld.lld
	targeting `riscv32-unknown-elf` on a machine with no GPU.

	## Citation

	```bibtex
	@misc{qwenti-inference-2026,
	title = {A 4.6M-parameter language model as an int4 workload on a token-level RISC-V machine},
	author = {OpceanAI},
	year = {2026},
	url = {https://huggingface.co/OpceanAI/qwenti-inference}
	}
	```