Instructions to use OpceanAI/qwenti-inference with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OpceanAI/qwenti-inference with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="OpceanAI/qwenti-inference")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("OpceanAI/qwenti-inference")
model = AutoModelForCausalLM.from_pretrained("OpceanAI/qwenti-inference", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use OpceanAI/qwenti-inference with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OpceanAI/qwenti-inference"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpceanAI/qwenti-inference",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/OpceanAI/qwenti-inference

SGLang

How to use OpceanAI/qwenti-inference with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OpceanAI/qwenti-inference" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpceanAI/qwenti-inference",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OpceanAI/qwenti-inference" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpceanAI/qwenti-inference",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use OpceanAI/qwenti-inference with Docker Model Runner:
```
docker model run hf.co/OpceanAI/qwenti-inference
```

Configuration Parsing Warning:Config file tokenizer_config.json cannot be fetched (too big)

Qwenti inference: TinyLLama-v0 inside Qwen3.5-4B

This is Qwen/Qwen3.5-4B with a token-level RV32 machine compiled into its transformer stack, and TinyLLama-v0 compiled into that machine's weight memory. Thirty-six circuit layers sit in front of Qwen's thirty-two. No weight in them was trained: a compiler computes all of them.

The method comes from Qwenputer by Divyansh Agrawal, which compiles a computer into a model's weights and drives it through the model's own token stream. This repository applies it to a 4B base and puts a language model inside the machine. The smaller sibling OpceanAI/Qwen3.5-0.8B-abliterated-Qwenti does the same with a CHIP-8 machine, and its overlay field is what showed how to keep the machine's readout out of the text path.

Measured


Parameters inside the machine	4,621,392
Weight memory	5,229,568 cells of 4 bits, 2.49 MB, 639 of 768 units
Circuit	36 layers, 457 heads, 3,259 dims
Merged model	68 layers, hidden 4,096, 27.4 GB in bf16
Text agreement with the base, first token, 6 prompts	6 of 6
KL(base against this model), 96 positions	6.5e-4
Top-1 agreement over the same positions	100%
`The capital of France is`	`Paris.`
Decode on one Colab G4	23.2 tokens/s
Peak VRAM	61.9 GB
Driven with a chip token	emits a chip token, best text logit at 0.0
Chip stream after `GO`, no state injected	17 of 24 tokens stay in the machine
TinyLLama tokens from the verified kernel	46 in 618 s, 37 of 40 equal to fp32

What works and what does not

The text path matches the base model. The machine's own readout is gated by mu, so at a text position the head sees no output code and answers text, and at a chip position it sees the code and answers in the machine's vocabulary.

The machine executes TinyLLama correctly on the reference implementation and on the circuit: five checkpoints of the real program compared token by token, zero disagreements, and forty greedy tokens that match a float model holding the same int4 weights on all forty.

Running the guest program inside this checkpoint is correct but slow. One TinyLLama token costs 1,821,283 machine instructions, or 2,619,430 trace tokens, and this model processes about 23 chip positions per second, so a token takes roughly 29 hours. The Triton executor runs the circuit alone at 1,263 positions per second, which brings it to 33 minutes per token. Neither is a chat path.

An earlier build of this checkpoint answered ! instead of text. The cause was the readout: the text bridge writes an echo code on text tokens, the final norm scales the code dimensions by 2^20, and a residue of 1e-2 in them became a 1e4 logit. Section 11 of the paper records the four probes that found it and the fix.

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("OpceanAI/qwenti-inference")
model = AutoModelForCausalLM.from_pretrained(
    "OpceanAI/qwenti-inference", dtype=torch.bfloat16, device_map="cuda").eval()

ids = tok("The capital of France is", return_tensors="pt").to("cuda")
out = model.generate(**ids, max_new_tokens=20, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

That prints Paris. followed by the same completion the base model gives. The machine's vocabulary is appended after Qwen's, and chipformer.json records the layout (chip_token_offset, chip_layers, chip_dims_offset, slots, code_dims, tau_dim, mu_dim, tool_call, printable). Driving the guest program needs the machine state injected into the cache, which the chipformer package does with core.snapshot and RvChat.

Why there is no bf16 build of the machine

storage	size for 4,621,392 parameters
int4, the weight memory	2.20 MB
bf16 or fp16	8.81 MB
fp32	17.63 MB

The weight memory holds 2.49 MB, which in bf16 is 1,307,392 weights. The model is 3.53 times larger, so a bf16 build has to live as model tensors, and the VDOT instruction stops being the dot product.

Files

Path	What
`PAPER-tinyllama-rv32.pdf`	the paper, 13 pages, figures and tables typeset
`PAPER-tinyllama-rv32.md`	the paper source
`DESIGN-tinyllama.md`	the working design log, including every wall and its fix
`figures/`	instruction mix, quantization cost, 40-token agreement, speed, storage
`results.json`	the raw result document of the G4 run
`scripts/`	kernel, assembly matvec, exporter, Python mirror, circuit checker, G4 driver, PDF builder
`colab_bundle.zip`	the patched tree and scripts, ready for a Colab VM

Hardware

Colab G4: NVIDIA RTX PRO 6000 Blackwell Server Edition, 95.6 GB of VRAM, driver 580.82.07, torch 2.14.1+cu130, CUDA 13.0, capability 12.0, 48 vCPU, 176.9 GB of RAM. Built with clang and ld.lld targeting riscv32-unknown-elf on a machine with no GPU.

Citation

@misc{qwenti-inference-2026,
  title  = {A 4.6M-parameter language model as an int4 workload on a token-level RISC-V machine},
  author = {OpceanAI},
  year   = {2026},
  url    = {https://huggingface.co/OpceanAI/qwenti-inference}
}

Downloads last month: 217

Safetensors

Model size

14B params

Tensor type

BF16

F32

Model tree for OpceanAI/qwenti-inference

Base model

Qwen/Qwen3.5-4B-Base

Finetuned

Qwen/Qwen3.5-4B

Finetuned

(1,348)

this model