talisma/Qwen-3.5-4B-Conversational

🐉 Qwen‑3.5‑4B‑Conversational

A thinking model that fits in your pocket — now in GGUF.

📄 Base Model Card · 📘 Qwen3.5 Blog · 💬 Talisma AI Studio

Qwen3.5‑4B outscores the 30B‑parameter Qwen3‑30B‑A3B‑Thinking‑2507 on agentic tool‑use — at roughly 1/7th the size.

---

🍿 TL;DR — what even is this? (for the non‑techie reader)

Qwen3.5‑4B is a small, sharp "thinking" AI model built by the Qwen Team — it reads text, images and even video, reasons step‑by‑step before answering, and speaks 201 languages. Talisma AI Studio took the original model and shrank it down using a technique called quantization, exporting it as several GGUF files of different sizes — the same brain, packaged at different levels of compression.

Think of it like saving the same photo as a tiny thumbnail, a normal JPEG, or a full‑resolution RAW file — same picture, different file sizes for different needs. Smaller file = fits on more modest hardware and runs faster; larger file = closer to the original model's full sharpness. Pick the one that matches your laptop, server, or phone.

This checkpoint keeps Qwen3.5's thinking‑by‑default behaviour, its vision understanding, and its native 262K‑token context (extendable to 1M tokens) — just packaged to actually run on the hardware you already own.

---

✨ Why this model is actually exciting

🧠 Thinks before it speaks Generates internal <think>...</think> reasoning before the final answer — strong on math, coding, and multi‑step logic.	👀 Sees text, images & video Native multimodal training (not bolted‑on) — reads documents, screenshots, charts, photos, and can summarize video clips.	🪶 Punches way above its weight On agentic tool‑use (TAU2‑Bench) this 4B model beats the 30B‑parameter Qwen3‑30B‑A3B‑Thinking‑2507 — see the bar above.
🌍 Speaks 201 languages Broad multilingual coverage with strong scores on MMMLU, MMLU‑ProX, and WMT24++ translation benchmarks.	🛠️ Built for agents Native tool/function‑calling, strong BFCL‑V4 and TAU2‑Bench scores — plugs into agent frameworks and automations.	📚 Remembers a lot 262,144‑token native context, extensible to 1,010,000 tokens via YaRN — feed it entire codebases or long transcripts.

---

🎯 Who is this for & what can you build with it?

🖱️ Click to expand real‑world use cases

If you are a...	You can use it to...
🧑‍💻 Developer	Run a private, local coding assistant or chatbot — no external API calls
🏢 Enterprise / CXM team	Power support agents, ticket triage, and document‑parsing pipelines
🎓 Student / Researcher	Get step‑by‑step reasoning on math, science, and logic problems
📄 Ops / Back‑office teams	Extract data from scanned documents, forms, screenshots, and charts
🌐 Global product teams	Build multilingual support bots across 201 languages and dialects
🎬 Media / content teams	Summarize or answer questions about video content
🤖 Automation engineers	Wire it into agentic workflows via native function‑calling

---

🔬 Under the hood — model specifications

Extracted from the official Qwen/Qwen3.5-4B model card.

Property	Qwen3.5‑4B
Type	Causal Language Model with Vision Encoder
Total Parameters	4B (≈5B including vision components)
Hidden Dimension	2,560
Number of Layers	32
Vocabulary Size	248,320 (padded, tied to LM head)
Context Length	262,144 native · up to 1,010,000 with YaRN
Modalities Supported	Text, Image, Video
Languages	201
License	Apache 2.0

📎 Architecture notes (click to expand)

Qwen3.5 uses a hybrid architecture repeated 8 times, where each block is 3 layers of Gated DeltaNet (a linear‑attention variant) followed by FFN, then 1 layer of Gated Attention (full attention) followed by FFN:

8 × ( 3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN) )

• Gated DeltaNet: 32 linear‑attention heads for V, 16 for Q/K, head dimension 128 — cheap, fast, and good at long sequences.
• Gated Attention: 16 heads for Q, 4 for KV, head dimension 256, with rotary position embeddings (dim 64) — brings back full‑sequence awareness periodically so nothing gets lost over long context.
• Multi‑Token Prediction (MTP): trained to predict several tokens ahead, enabling speculative decoding for faster inference.

This mix gives Qwen3.5 the speed of a linear‑attention model with the long‑range awareness of full attention — a big part of why the 4B model competes with much larger dense/thinking models on agentic and long‑context tasks.

---

📊 Benchmark performance (Qwen3.5‑4B, base model scores)

Agentic tool‑use — small model, big result

_{TAU2‑Bench (avg. of 3 domains) — higher is better}

📈 Full language benchmark table (click to expand)

Category	Benchmark	GPT‑OSS‑120B	GPT‑OSS‑20B	Qwen3‑Next‑80B‑A3B‑Thinking	Qwen3‑30BA3B‑Thinking‑2507	Qwen3.5‑9B	Qwen3.5‑4B
Knowledge & STEM	MMLU‑Pro	80.8	74.8	82.7	80.9	82.5	79.1
Knowledge & STEM	GPQA Diamond	80.1	71.5	77.2	73.4	81.7	76.2
Instruction Following	IFEval	88.9	88.2	88.9	88.9	91.5	89.8
Long Context	AA‑LCR	50.7	30.7	51.7	49.0	63.0	57.0
Reasoning & Coding	HMMT Nov 25	90.0	81.8	81.2	73.8	82.9	76.8
Reasoning & Coding	LiveCodeBench v6	82.7	74.6	68.7	66.0	65.6	55.8
General Agent	BFCL‑V4	--	--	49.7	42.4	66.1	50.3
General Agent	TAU2‑Bench	--	--	57.4	41.9	79.1	79.9
Multilingual	MMMLU	78.2	69.7	81.3	78.4	81.2	76.1

Empty cells (--) indicate scores not yet available or not applicable. Full table with all 40+ benchmarks is on the base model card.

🖼️ Vision‑language benchmark table (click to expand)

Category	Benchmark	GPT‑5‑Nano	Gemini‑2.5‑Flash‑Lite	Qwen3‑VL‑30B‑A3B	Qwen3.5‑9B	Qwen3.5‑4B
STEM & Puzzle	MMMU	75.8	73.4	76.0	78.4	77.6
STEM & Puzzle	MMMU‑Pro	57.2	59.7	63.0	70.1	66.3
STEM & Puzzle	MathVision	62.2	52.1	65.7	78.9	74.6
General VQA	RealWorldQA	71.8	72.2	77.4	80.3	79.5
Document Understanding	OCRBench	75.3	82.5	83.9	89.2	85.0
Video Understanding	VideoMME (w/ sub.)	71.7	74.6	79.9	84.5	83.5

Full vision‑language table on the base model card.

---

🧊 Why we quantized this model

Full‑precision Qwen3.5‑4B ships as ~9 GB of BF16 weights — great on a workstation GPU, painful on a laptop or edge box. Quantization compresses those weights (from 16‑bit down to as low as 2‑bit) using calibrated, block‑wise scaling ("K‑quants"), trading a small amount of precision for large practical wins:

Benefit	What it means in practice
🪶 Smaller footprint	Down to ~1.96 GB — fits on machines with 4–6 GB of RAM/VRAM
⚡ Faster inference	Less data to move through memory = lower latency per token
💻 Runs on modest hardware	No dedicated GPU required for the lower‑bit variants
📦 Single‑file, portable format	GGUF bundles weights + metadata + tokenizer in one file
💰 Cheaper to serve	Lower RAM/VRAM per instance means more concurrent users per machine

The trade‑off: very aggressive quantization (Q2_K) can soften factual precision and coding accuracy slightly. K‑quants (the _K_M and _K variants below) mitigate this with smarter, non‑uniform bit allocation, so quality holds up much better than naive quantization at the same file size.

---

📦 Files in this repository

_{Relative file sizes of each quantization (main language‑model weights)}

File	Size	Quality	Speed	Recommended for
Qwen3.5-4B-Q2_K.gguf	1.96 GB	⭐⭐	⚡⚡⚡⚡⚡	Extremely constrained devices, quick testing only
Qwen3.5-4B-Q3_K_M.gguf	2.32 GB	⭐⭐⭐	⚡⚡⚡⚡	Low‑RAM laptops, fast prototyping
Qwen3.5-4B-Q4_K_M.gguf	2.78 GB	⭐⭐⭐⭐	⚡⚡⚡⚡	✅ Best all‑round balance — most users start here
Qwen3.5-4B-Q5_K_M.gguf	3.16 GB	⭐⭐⭐⭐	⚡⚡⚡	Better quality, still light
Qwen3.5-4B-Q6_K.gguf	3.56 GB	⭐⭐⭐⭐⭐	⚡⚡⚡	Near full‑precision quality
Qwen3.5-4B-Q8_0.gguf	4.61 GB	⭐⭐⭐⭐⭐	⚡⚡	Max‑quality quantization, workstation GPUs
Qwen3.5-4B-BF16.gguf	8.67 GB	🏆 Full precision	⚡	Research, fine‑tuning, maximum fidelity
mmproj-Qwen3.5-4B-BF16.gguf	676 MB	—	—	Required alongside any of the above for image/video input

💡 Rule of thumb: allocate roughly file size + 2 GB of RAM/VRAM for comfortable inference (extra headroom for context and KV‑cache at longer sequence lengths). Add another ~1 GB if you're also loading the mmproj vision projector.

🤔 Not sure which one to pick? Click here

• Just exploring / laptop with 4–8 GB RAM → go with Q4_K_M
• Have a decent GPU (6 GB+ VRAM) and want the best answers → go with Q6_K or Q8_0
• Fine‑tuning, research, or need bit‑for‑bit fidelity → use BF16
• Running on something really small (SBCs, older phones) → Q2_K or Q3_K_M, but expect softer reasoning quality
• Need image or video understanding → always download mmproj-Qwen3.5-4B-BF16.gguf alongside your chosen quant

---

🚀 Quick start

🦙 Using llama.cpp

# Download your preferred quantization + the vision projector
huggingface-cli download talisma/Qwen-3.5-4B-Conversational \
  Qwen3.5-4B-Q4_K_M.gguf mmproj-Qwen3.5-4B-BF16.gguf --local-dir .

# Text chat
./llama-cli -m Qwen3.5-4B-Q4_K_M.gguf -p "Explain photosynthesis to a 10 year old." -cnv

# With an image
./llama-cli -m Qwen3.5-4B-Q4_K_M.gguf --mmproj mmproj-Qwen3.5-4B-BF16.gguf \
  -p "Describe this image." --image photo.jpg

💻 Using Ollama

ollama run hf.co/talisma/Qwen-3.5-4B-Conversational:Q4_K_M

🐍 Using Python (transformers, base model reference)

from transformers import AutoProcessor, AutoModelForCausalLM

MODEL_ID = "Qwen/Qwen3.5-4B"

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, dtype="auto", device_map="auto"
)

messages = [{"role": "user", "content": "Type \"I love Qwen3.5\" backwards"}]

inputs = processor.apply_chat_template(
    messages, tokenize=True, return_dict=True, return_tensors="pt",
    add_generation_prompt=True
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=2048)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

---

⚙️ Recommended settings

Mode	temperature	top_p	top_k	presence_penalty
Thinking (general tasks)	1.0	0.95	20	1.5
Thinking (precise coding)	0.6	0.95	20	0.0
Non‑thinking (general tasks)	0.7	0.80	20	1.5
Non‑thinking (reasoning tasks)	1.0	0.95	20	1.5

• Qwen3.5 thinks by default, generating <think>...</think> content before its final answer. Disable it via enable_thinking: False in your chat template if you just want direct responses.
• Recommended output length: 32,768 tokens for most queries; up to 81,920 tokens for hard math/coding benchmarking.
• For long‑context work beyond 262K tokens, enable YaRN RoPE scaling (see the base model card for the exact config) — but keep at least 128K tokens of context to preserve full thinking capability.

🧵 Multi‑turn tip: in conversation history, keep only the model's final answers — strip out prior "thinking" content before the next user turn. This is handled automatically by the provided chat template.

---

🧬 Provenance & attribution

Base model:      Qwen/Qwen3.5-4B        (Qwen Team, Alibaba)
Quantized by:    Talisma AI Studio
Format:          GGUF
Method:          llama.cpp quantization
License:         Apache 2.0 (inherited from base model)

Talisma AI Studio evaluates and benchmarks open‑weight models (Qwen, Gemma, Llama, Mistral, Phi, and embedding models) to power Talisma's enterprise CXM/CRM and Agentic AI products. This repository packages the Qwen Team's Qwen3.5‑4B for efficient, on‑premise, and edge deployment.

---

⚠️ Good to know

Limitations & responsible use

• Like all LLMs, this model can produce incorrect or outdated factual statements — verify important outputs.
• Lower‑bit quantizations (especially Q2_K) can show a noticeable drop in coding and precise‑math accuracy compared to BF16.
• Video input is processed as sampled frames (default 2 fps); very long videos should use a lower frame rate.
• Refer to the Qwen Team's official usage guidance and Apache 2.0 license terms for deployment considerations.

---

📖 Citation

@misc{qwen3.5,
    title  = {{Qwen3.5}: Towards Native Multimodal Agents},
    author = {{Qwen Team}},
    month  = {February},
    year   = {2026},
    url    = {https://qwen.ai/blog?id=qwen3.5}
}

---

Made with 💜 by Talisma AI Studio · Base model by Qwen Team (Alibaba)

🤗 Talisma on Hugging Face · 📄 License: Apache 2.0