trymirai/Qwen3.8-27B-S-experimental

Qwen3.8-27B, Mirai S, experimental

Qwen3.8-27B quantized with the Mirai S codec: 2.4 bits per weight, 8.45 GB on disk.

This is a work in progress and the numbers here are not final. Work on both runtimes is unfinished. Treat anything measured with it as provisional.

On Apple silicon it runs on a prebuilt uzu runtime that ships in this repo. On NVIDIA it runs on stock vLLM with a small plugin that also ships in this repo.

Apple silicon — uzu

You need an Apple silicon Mac on macOS 26.4 or newer with 24 GB of memory or more. The uzu/ folder holds a prebuilt runtime, so there is nothing to build:

hf download trymirai/Qwen3.8-27B-S-experimental --exclude "vllm/*" --local-dir qwen3.8-s
chmod +x qwen3.8-s/uzu/uzu
qwen3.8-s/uzu/uzu server --model qwen3.8-s

That starts an OpenAI-compatible endpoint on port 8000:

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen3.8-s","messages":[{"role":"user","content":"Capital of Australia? One word."}],"enable_thinking":false,"max_tokens":8}'

To chat in the terminal instead, leave out server. With the server running, python3 qwen3.8-s/speedcheck.py measures your Mac in under a minute; on an M5 Pro it prints 52 tokens/s decoding code (17 on prose), 440 tokens/s prefill, and 0.4 s to start answering a follow-up in a 7k-token chat. Follow-ups stay fast because uzu keeps the conversation cached between turns. More in uzu/README.md.

To build uzu yourself instead, install Rust with rustup, clone uzu's ryan/qtip-s-agent branch, and run cargo run --release -p cli -- server --model <model dir> inside it. If the build stops because the Metal toolchain is missing, run xcodebuild -downloadComponent MetalToolchain and try again.

Thinking is on by default. Turn it off per request with "enable_thinking": false, or server-wide by starting the server with UZU_SERVER_THINKING=0. Things to know:

• uzu's levels are low / medium / high / xhigh.
• With UZU_SERVER_THINKING=0, send enable_thinking together with reasoning_effort; reasoning_effort on its own returns 400.
• Streaming replies carry the thinking as reasoning_content deltas, then the answer as content, the way vLLM's reasoning parser does.

NVIDIA — vLLM

The vllm/ folder (9.3 GB download) holds a vLLM-ready copy of this checkpoint and the mirai_s plugin, which runs the compressed weights directly on the GPU: 8.2 GiB of GPU memory, so a 12 GB card runs it and 16 GB leaves room for speculative decoding. On an RTX 3090 it answers one chat at 79 (prose) to 122 (code) tokens/s with speculative decoding, serves 517 tokens/s across 32 parallel requests, and has room for about 160k tokens of context. Install and serve commands, including one for drivers older than 580, are in vllm/README.md.

Known issues

• Both runtimes: a tight max_tokens with thinking on can spend the whole budget inside the trace and return an empty answer. Give reasoning requests at least 4000 tokens.