Image-Text-to-Text
Transformers
Safetensors
qwen3_5
qwen3_8
qwopus
coding
tool-use
gptq
int4
mtp
speculative-decoding
intel
xpu
arc
vllm
conversational
4-bit precision
Instructions to use Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked") model = AutoModelForMultimodalLM.from_pretrained("Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked
- SGLang
How to use Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked with Docker Model Runner:
docker model run hf.co/Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked
|
Download README.md from Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked: direct link, hf CLI and curl.
- Browser
- Download file 11.2 kB
-
https://huggingface.co/Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked/resolve/main/README.md
- Command line
-
hf download hf://Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked/README.md
-
curl -L -o README.md https://huggingface.co/Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked/resolve/main/README.md
11.2 kB
| license: apache-2.0 | |
| base_model: Jackrong/Qwopus3.8-27B-Coder-Flash | |
| base_model_relation: quantized | |
| pipeline_tag: image-text-to-text | |
| library_name: transformers | |
| tags: [qwen3_8, qwopus, coding, tool-use, gptq, int4, mtp, speculative-decoding, intel, xpu, arc, vllm] | |
| # Qwopus3.8-27B-Coder-Flash, GPTQ INT4 baked (v1.1.1) | |
| [Jackrong/Qwopus3.8-27B-Coder-Flash](https://huggingface.co/Jackrong/Qwopus3.8-27B-Coder-Flash) quantized from BF16 to | |
| INT4 by Launch80. The `lm_head` and the MTP draft head are INT4 **on disk** too, so a single **Intel Arc Pro B65 32 GB** | |
| serves it at the full **262,144-token** context, and stock vLLM loads it with no weight patches. | |
| **This model is a work in progress, and the full story is public:** | |
| - **The study, with every number, the bug we found and how we fixed it:** https://launch80.com/blog/qwopus-coder-int4-bake-study | |
| - **Follow along and ask questions in the Launch80 Discord:** https://discord.gg/launch80 | |
| | version | what changed | status | | |
| |---|---|---| | |
| | v1.0 (tag `v1.0`) | GPTQ INT4 body calibrated with tool-call data, baked INT4 head and MTP | released 2 Nov 2026 | | |
| | v1.0.1 (tag `v1.0.1`) | card only, weights unchanged: all seven tasks measured on this exact checkpoint, served as shipped (MTP k=6) | released 2 Nov 2026 | | |
| | v1.1 (tag `v1.1`) | weights unchanged; adds a fast-decode serving profile with our INT4 DFlash2 drafter (k=7, 128k): 89.3 tok/s on the B65 | released 3 Nov 2026 | | |
| | **v1.1.1** (this revision, tag `v1.1.1`) | card only, weights unchanged: one more vLLM-XPU boot patch removes a per-step host sync in the verify step; fast profile **92.5 / 96.0 tok/s** (two quick runs) | released 3 Nov 2026 | | |
| ## What's in it | |
| | tensor | source (BF16) | this repo | | |
| |---|---|---| | |
| | 400 body linears | BF16 | GPTQ INT4 sym g128 (GPTQModel 7.3.2) | | |
| | `lm_head` (248,320 × 5,120) | BF16, 2.54 GB | GPTQ INT4 sym g128, Hessian-calibrated | | |
| | `mtp.*` draft: `fc`, q/k/v/o, gate/up/down | BF16 | GPTQ INT4 sym g128, Hessian-calibrated | | |
| | embeddings, norms, vision tower, GDN `in_proj_a`/`in_proj_b` | BF16 | unchanged (byte-identical) | | |
| 17.0 GB on disk. The tokenizer, chat template and processor files are the source repo's, unchanged. | |
| **Body.** GPTQModel 7.3.2 ran on the Arc Pro B65 itself with these settings: | |
| - 4-bit, symmetric, group 128 | |
| - `desc_act` off, `act_group_aware` on | |
| - damping 0.05, seed 608 | |
| - 512 × 2048-token calibration samples: 153 C4, 204 code, 53 chat and 102 tool-call conversations. The tool-call samples are from [NousResearch/hermes-function-calling-v1](https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1) `@dae3e1d`, rendered through the model's own chat template with `tools=`, half of them parallel calls. | |
| The calibration sha256 is in `quant-meta.json`. | |
| **Head and MTP bake.** These steps follow the layout of our earlier bakes: | |
| 1. Forward hooks inside vLLM captured Hessians on wikitext, chat and code slices. | |
| 2. Each of the nine linears was GPTQ-quantized under a grid of dampings and slice mixes. | |
| 3. The setting with the lowest output error on held-out rows was kept. | |
| Every baked linear beats round-to-nearest (RTN): | |
| | linear | baked | RTN | | |
| |---|--:|--:| | |
| | `lm_head` | **4.68%** | 8.18% | | |
| | `mtp.fc` | **6.98%** | 11.47% | | |
| | `mtp…q_proj` | **2.68%** | 4.44% | | |
| | `mtp…k_proj` | **6.05%** | 10.08% | | |
| | `mtp…v_proj` | **5.11%** | 8.28% | | |
| | `mtp…o_proj` | **6.67%** | 13.04% | | |
| | `mtp…gate_proj` | **3.67%** | 6.35% | | |
| | `mtp…up_proj` | **5.70%** | 9.56% | | |
| | `mtp…down_proj` | **7.57%** | 11.96% | | |
| ## Quality vs BF16 (pre-registered) | |
| We wrote the thresholds down and hashed them before any quantized model was evaluated. The BF16 reference ran on a rented | |
| RTX PRO 6000 (vLLM 0.30, CUDA); this checkpoint ran on the Arc Pro B65 (vLLM 0.30, XPU), served exactly as shipped, | |
| with MTP k=6 on for every task and for the logprobs. Full study: | |
| https://launch80.com/a/84829a3a-a0b2-407f-ab38-59cd2d5006fa | |
| | # | measure | bar | this checkpoint | | | |
| |---|---|---|---|---| | |
| | T1 | wikitext-2 perplexity | ≤ +3% | +1.72% | pass | | |
| | T2 | code-corpus perplexity | ≤ +3% | +1.83% | pass | | |
| | T3 | mean next-token KL divergence (1.05M tokens, top-64) | ≤ 0.05 | 0.0412 | pass | | |
| | T4 | top-1 next-token agreement | ≥ 93% | 92.84% | **fail** (by 0.16 pt) | | |
| | T5 | seven tasks, paired pass@1, greedy | Δ ≥ −2 pts or CI includes 0 | 7/7 tasks (table below) | pass | | |
| | T6 | KLD vs Jackrong's Q4_K_M GGUF | ours ≤ theirs | 0.0412 vs 0.0250 | **fail** | | |
| | T7 | greedy, spec decoding on = off | identical | 26/32; all 6 differences are at a tie (3 exact, 3 a 0.0156 FP16 step) | fail as written | | |
| | task | BF16 | this checkpoint | Δ pts [95% CI] | | |
| |---|--:|--:|---| | |
| | GSM8K | 86.96 | 86.66 | −0.30 [−1.21, +0.61] | | |
| | IFEval | 80.22 | 80.78 | +0.55 [−2.22, +3.14] | | |
| | MMLU-Pro subset | 71.80 | 70.90 | −0.90 [−2.70, +0.90] | | |
| | HumanEval+ | 90.24 | 88.41 | −1.83 [−5.49, +1.83] | | |
| | MBPP+ | 76.98 | 75.66 | −1.32 [−3.17, +0.53] | | |
| | BFCL subset (400) | 89.00 | 88.25 | −0.75 [−3.00, +1.50] | | |
| | LiveCodeBench v6 slice | 41.14 | 40.57 | −0.57 [−5.14, +4.00] | | |
| **T6, against [Jackrong's Q4_K_M GGUF](https://huggingface.co/Jackrong/Qwopus3.8-27B-Coder-Flash-GGUF): fail.** On the same | |
| 512 × 2048 tokens, Q4_K_M stays closer to BF16: mean KLD 0.0250 vs our 0.0412, top-1 agreement 94.92% vs 92.84%, perplexity | |
| +0.73% vs +1.72% (wikitext-2). The files are about the same size (16.8 vs 17.1 GB); Q4_K_M keeps 67 tensors at 6 bits, while | |
| this checkpoint carries a BF16 vision tower and fp16 embeddings and a uniform 4-bit g128 body. Method: the GGUF ran in llama.cpp | |
| (b11723, SYCL) on the B65, its top-64 logprobs were scored with the same code against the same BF16 reference. This differs from | |
| the registered method (a BF16 GGUF base, which needed a rental) and is dated in the thresholds file. Q8_0 measured the same way | |
| gives 0.0024, so the cross-engine term is small. Pick Q4_K_M for fidelity on llama.cpp; pick this build for vLLM with | |
| speculative decoding. | |
| Needle retrieval: 100% at 32k and at 128k. MTP draft acceptance over the whole suite: 46.8% of drafted tokens | |
| (80.2 / 62.2 / 48.0 / 37.4 / 29.5 / 23.3% by draft position), about 3.8 tokens per verify step under greedy decoding. | |
| **Disclosure: the tool-call calibration is a post-hoc fix.** The first bake, with no tool calls in its calibration set, | |
| passed six of the seven tasks but lost 7.75 points on BFCL. It ended its turn after the first of several parallel calls. | |
| We traced that to the calibration data, recalibrated the body, and rebaked the head and MTP. This checkpoint is that rebake, | |
| and the table above measures it on all seven tasks. The change was made after seeing results, so treat it as post hoc. | |
| The blog post has the whole investigation. | |
| ## Speed (Arc Pro B65 32 GB, vLLM 0.30.0 XPU, fp8 KV, single stream) | |
| Two serving profiles, same weights. Both are quick BetterBench runs (1 warmup + 5 passes per category): | |
| | profile | drafter | context | weighted decode | page | | |
| |---|---|---|--:|---| | |
| | **fast decode** + GDN no-sync patch (v1.1.1) | [Launch80/Qwen3.8-27B-DFlash2-Int4-baked-drafter](https://huggingface.co/Launch80/Qwen3.8-27B-DFlash2-Int4-baked-drafter), k=7 | 131,072 | **92.5 / 96.0 tok/s** (two runs) | https://launch80.com/a/81dea056-a018-4e48-9f5e-b89b039f368d | | |
| | fast decode, v1.1 (no patch) | same | 131,072 | 89.3 tok/s | https://launch80.com/a/54608011-fd4d-4c68-9d32-868a2fcdd03b | | |
| | full context | built-in MTP head, k=6 | 262,144 | 75.8 tok/s | https://launch80.com/a/30aa8ae2-f00c-409a-a4a9-2e298fc81f67 | | |
| The fast profile gains most on structured output (5.2 accepted tokens per step on JSON, 4.7 on summaries) and least on | |
| long thinking and prose (2.8 and 2.5). It costs context: with the drafter loaded, the 32 GB card fits 128k, not 262k. | |
| Spec-decoding parity (T7) for the fast profile with the v1.1.1 patch: 30 of 32 greedy outputs identical to spec-off, and both | |
| differences are at exact top-2 ties (v1.1 without the patch: 28 of 32, all ties), the same pattern as the MTP profile. | |
| What v1.1.1 fixes: on every speculative step, vLLM's GDN (linear-attention) metadata builder indexed GPU tensors with a CPU | |
| boolean mask. That forces the host to wait for the GPU, which cancelled async scheduling and left ~7 ms of host time in each | |
| step at k=7. `patch_gdn_spec_nosync.py` replaces the mask with the equivalent slice: 47.9 → 44.4 ms per step on a single | |
| prompt. We also tried a Triton INT4 GEMM for the 9-to-16-row verify shapes; it ran 3.5× slower than oneDNN, so the fast profile | |
| stays at k=7. Quick BetterBench runs vary by a few tok/s between runs, so both runs are shown. | |
| The MTP head is byte-identical to base Qwen3.8-27B (the fine-tune never trained it). We also fine-tuned the DFlash2 | |
| drafter on 10,000 of this model's own answers; no checkpoint beat the stock drafter, so v1.1 uses the stock drafter, INT4-baked. | |
| Drafter study: https://launch80.com/a/a7b8a95a-e2c5-473c-b017-310232fc56e0 | |
| ## Serving | |
| Full-context profile (75.8 tok/s, 262k context): | |
| ```bash | |
| vllm serve Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked \ | |
| --quantization gptq --dtype float16 --kv-cache-dtype fp8 --max-model-len 262144 \ | |
| --speculative-config '{"method":"mtp","num_speculative_tokens":6}' \ | |
| --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 | |
| ``` | |
| Fast-decode profile (92.5 / 96.0 tok/s, 128k context): | |
| ```bash | |
| vllm serve Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked \ | |
| --quantization gptq --dtype float16 --kv-cache-dtype fp8 --max-model-len 131072 \ | |
| --speculative-config '{"method":"dflash","model":"Launch80/Qwen3.8-27B-DFlash2-Int4-baked-drafter","num_speculative_tokens":7,"quantization":"gptq"}' \ | |
| --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 | |
| ``` | |
| The drafter must run in bf16 while this target runs in fp16: on XPU that needs the `patch_dflash_mixed_dtype.py` boot | |
| patch from [launch80/B65](https://github.com/launch80/B65) (on CUDA, serve both in bf16 with `--dtype bfloat16`). | |
| Every number above was measured with these configurations (vLLM 0.30.0 XPU image), plus three vLLM-XPU correctness | |
| patches for speculative decoding (MTP nightly, MTP boundary, GDN mixed-split) from the same repository. The fast profile's | |
| v1.1.1 number also needs `patch_gdn_spec_nosync.py` (run it after the MTP boundary patch); without it the same command gives 89.3. Tool calling | |
| needs `qwen3_xml`, not `hermes`. Pin a revision (`--revision v1.1.1`) if you want this exact build; `main` moves as new | |
| versions ship. | |
| ## License | |
| Apache-2.0, as the source model and the Qwen3.8-27B base are. | |
| ## Credits | |
| Qwen (Qwen3.8-27B) · Unsloth (the base Jackrong built on) · Jackrong (Qwopus3.8-27B-Coder-Flash) · NousResearch | |
| (hermes-function-calling-v1, used for calibration) · GPTQModel · Inco AI / z-lab (the DFlash2 drafter) · SpecForge (the drafter fine-tune | |
| experiment) · Wrapzii (asked for this bake, and the on-disk bake | |
| layout it follows). | |
| ## Community | |
| Questions, results, or a model you want tested? Join the Launch80 Discord: https://discord.gg/launch80 | |
| If this work is useful to you, please support the server: https://launch80.com/supporter | |