Instructions to use Kwaipilot/KAT-Coder-V2.5-Dev with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Kwaipilot/KAT-Coder-V2.5-Dev with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Kwaipilot/KAT-Coder-V2.5-Dev")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Kwaipilot/KAT-Coder-V2.5-Dev")
model = AutoModelForMultimodalLM.from_pretrained("Kwaipilot/KAT-Coder-V2.5-Dev", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Kwaipilot/KAT-Coder-V2.5-Dev with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Kwaipilot/KAT-Coder-V2.5-Dev"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Kwaipilot/KAT-Coder-V2.5-Dev",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Kwaipilot/KAT-Coder-V2.5-Dev

SGLang

How to use Kwaipilot/KAT-Coder-V2.5-Dev with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Kwaipilot/KAT-Coder-V2.5-Dev" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Kwaipilot/KAT-Coder-V2.5-Dev",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Kwaipilot/KAT-Coder-V2.5-Dev" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Kwaipilot/KAT-Coder-V2.5-Dev",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Kwaipilot/KAT-Coder-V2.5-Dev with Docker Model Runner:
```
docker model run hf.co/Kwaipilot/KAT-Coder-V2.5-Dev
```

My experience using and evaluating this model in coding tasks (the results are simply amazing).

by Am0gusIsSus - opened 14 days ago

Discussion

Am0gusIsSus

14 days ago

Here’s what I noticed when comparing other “fine-tuned” QWEN 3.5 models: 27B and 35B-A3B. Finally, this model doesn’t break at 32K of context; my working context size is 1M, and it works correctly at that size—there are no issues with YARN scaling! The others break immediately (they lose context, tool calling fails) as soon as the context size exceeds 32–64K. I need to generate documentation and tests for a huge project—no problem! Your work is simply amazing! I’d love to see your version of the 122B-A10B. People have also mentioned in other discussions that "This model has been trained on broken Qwen3.5-35B-A3B official weight release that Alibaba Group shipped to people. It's not issue with training or learning data. It's issue with model itself. Same problem exists in Qwen3.6-35B-A3B base weights. It's main reason why this model gets stuck in a recursive loop even after full finetune in BF16 safetensors - expert tensors in block 0 has 40% of noisy zero blocks that are visible only on Q8_0 quantization. Three ssm_con1vd tensors have enormous big signal scale and accumulate cascade error distributing during training on entire neural network.". How did you manage to get around that? In any case, your work is simply wonderful—thank you so much!

liushiyang

Kwaipilot org 14 days ago

During RL training, we observed that within a certain range, the increase in sequence length is accompanied by rising reward values. Therefore, accuracy does not degrade at larger lengths, indicating that the model is capable of handling tasks within this length range. The open-sourced checkpoint in this work supports a slightly longer sequence length compared to the original checkpoint while maintaining favorable metrics, which may account for this phenomenon.

Zeioth

8 days ago

•

edited 8 days ago

It's impressive how it fixes the structural hallucination repetitions of qwen 3.6 3A. And bonus points, it seems to improve perplexity. For domestic hardware, it's the perfect coding model.

nonetrix

7 days ago

•

edited 7 days ago

I might have just gotten lucky, but this is first model I've been able to run not to mess this prompt up:

Write a Minecraft clone in Three.js in one HTML file with embedded JavaScript and CSS with the following requirements met: Block face culling, infinite world generation, block placing, block breaking, hot bar of blocks, flying

KAT:

Qwen 27B:

Qwen got it wrong to spacial reasoning pretty sure. Completely wrong cube geometry\winding order, and got some math wrong for culling

Edit: now that I play with it more still made a few issues

Slightly broken hot-bar
Block breaking doesn't update visually, seems to delete it from the world though

https://gist.github.com/nonetrix/6306f6aa110d306e3c62e320de6f9541

I would still consider that way better than a mess of polygons, easily fixable with a follow up likely where Qwen for the life of it couldn't fix it's issues

Edit: seems like I just got lucky? It made the same mistake as Qwen when I tried again, maybe not messing it up as badly though? Still entirely possible it gets this right more than base model, I would have to test way more than I have to know for sure, really encouraging it got it right the first go around mostly

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment