README.md

---
license: apache-2.0
base_model: sahilchachra/KAT-Coder-V2.5-Dev-NVFP4A16
base_model_relation: quantized
library_name: freetoken
pipeline_tag: text-generation
language:
- en
- zh
tags:
- freetoken
- ftw
- qwen3_5_moe
- qwen3.6
- mixture-of-experts
- coding
- agentic-coding
- nvfp4
- nvfp4a16
- w4a16
- text-only
- conversational
arxiv: 2607.05471
---

# KAT-Coder-V2.5-Dev-FTW

**A FreeToken-ready FTW build of the KAT-Coder-V2.5-Dev coding fine-tune.**

This repository packages the 35B-A3B KAT-Coder-V2.5-Dev Mixture-of-Experts
fine-tune in FreeToken's native FTW fast-load format. It is intended to load
directly in [FreeToken](https://github.com/FlashML-org/FreeToken) for local,
heterogeneous GPU/CPU inference.

KAT-Coder-V2.5-Dev was created by Kwaipilot by post-training
Qwen3.6-35B-A3B with supervised fine-tuning and reinforcement learning for
coding and agentic software-engineering tasks. This repository does **not**
claim a new fine-tuning run: it makes that existing fine-tune usable in
FreeToken.

## Model lineage

1. [Qwen3.6-35B-A3B](https://huggingface.co/Qwen) — base architecture.
2. [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev)
   — 35B total / approximately 3B active coding fine-tune, trained with SFT and RL.
3. [sahilchachra/KAT-Coder-V2.5-Dev-NVFP4A16](https://huggingface.co/sahilchachra/KAT-Coder-V2.5-Dev-NVFP4A16)
   — data-free, weight-only NVFP4A16 quantization.
4. **remixie/KAT-Coder-V2.5-Dev-FTW** — FreeToken FTW conversion and packaging.

The model has 40 hybrid Qwen3.5/3.6-style layers, 256 routed experts plus a
shared expert, and 8 routed experts active per token. It supports reasoning,
coding, agent workflows, and tool calling through the Qwen chat template.

## What is FTW?

FTW is FreeToken's self-contained, sharded fast-load checkpoint format. The
conversion pre-packs weights and routed-expert banks so FreeToken does not need
to reinterpret the original Hugging Face safetensors every time the model is
started.

This repository contains:

- `freetoken_weight.json` — FTW tensor manifest and shard map
- `freetoken-00000.ftw`
- `freetoken-00001.ftw`
- `freetoken-00002.ftw`
- model configuration, tokenizer, chat template, and generation metadata

The three FTW shards contain 22,898,896,896 bytes (approximately 21.3 GiB) of
tensor data. Keep the manifest and all three shards together.

> **Runtime compatibility:** `.ftw` files are not safetensors. This repository
> is for FreeToken and is not directly loadable by Transformers, vLLM,
> SGLang, llama.cpp, Ollama, or LM Studio. Use the linked upstream repositories
> for those runtimes.

## Precision layout

`A16` means that NVFP4 weights are multiplied by 16-bit activations on the GPU;
it does not mean the stored weights are 16-bit.

| Component | Storage / execution |
|---|---|
| Routed expert weights | NVFP4 E2M1, group size 16 |
| Shared expert weights | NVFP4 E2M1, group size 16 |
| GatedDeltaNet / linear-attention projections | BF16 |
| Full self-attention projections | BF16 in this FTW build |
| Routers, gates, embeddings, LM head, and norms | BF16 |
| GPU NVFP4 activation path | W4A16: FP4 weights × BF16 activations |
| CPU expert path in FreeToken hybrid mode | W4A8 for CPU-computed expert work |
| KV cache quantization | None declared by the source checkpoint; runtime-managed by FreeToken |

The source A16 checkpoint stores the full-attention projections as weight-only
NVFP4 and the GatedDeltaNet projections as BF16. FreeToken currently expects a
uniform attention representation for this hybrid architecture, so the ten
full-attention layers were dequantized to BF16 during conversion. The expert
and shared-expert FP4 payloads remain packed NVFP4.

## Download

Install the Hugging Face CLI if needed:

```bash
pip install -U "huggingface_hub[cli]"
```

### Windows / FreeToken Desktop

```powershell
hf download remixie/KAT-Coder-V2.5-Dev-FTW `
  --local-dir "$env:USERPROFILE\.freetoken\models\KAT-Coder-V2.5-Dev-FTW"
```

Then restart or refresh FreeToken Desktop, open **Models**, and start
`KAT-Coder-V2.5-Dev-FTW`.

### Linux

```bash
hf download remixie/KAT-Coder-V2.5-Dev-FTW \
  --local-dir ~/.freetoken/models/KAT-Coder-V2.5-Dev-FTW
```

## Serve with FreeToken

The command below matches the FreeToken engine version used for validation:

```powershell
ft serve --model-path "$env:USERPROFILE\.freetoken\models\KAT-Coder-V2.5-Dev-FTW" `
  --host 127.0.0.1 `
  --port 8000
```

FreeToken automatically chooses its attention and MoE backends. On the tested
24 GB GPU it selected Triton attention plus the hybrid GPU/CPU MoE backend.

### OpenAI-compatible API

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="KAT-Coder-V2.5-Dev-FTW",
    messages=[
        {
            "role": "user",
            "content": "Write a Python function that returns the n-th Fibonacci number.",
        }
    ],
    temperature=0.7,
    top_p=0.8,
    max_tokens=1024,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": False},
    },
)

print(response.choices[0].message.content)
```

Thinking is enabled by the model template by default. Remove
`"enable_thinking": False` to allow a reasoning trace. FreeToken automatically
selects the `qwen3` reasoning parser and `qwen3_coder` tool-call parser for this
model family.

## Conversion details

The conversion was performed from
`sahilchachra/KAT-Coder-V2.5-Dev-NVFP4A16` without additional training,
calibration, or another quantization pass.

The conversion process:

- validated all 13 source safetensor shards;
- adapted compressed-tensors NVFP4 tensor names to FreeToken's expected layout;
- converted compressed-tensors global-scale divisor semantics to FreeToken's
  multiplier convention;
- fused shared-expert gate/up projections into FreeToken's layout;
- dequantized the ten full-attention layers to BF16 so all attention layers use
  a representation supported uniformly by FreeToken;
- preserved routed and shared-expert NVFP4 packed weights and their FP8 block
  scales;
- packed all 40 routed-expert layers into FTW expert banks; and
- copied the tokenizer, chat template, generation config, and relevant model
  metadata.

No fine-tuning or benchmark claims originate from the FTW conversion itself.

## Validation

This artifact was smoke-tested end to end with:

- FreeToken Desktop `0.2.0-beta.13`
- FreeToken engine `0.1.1+g30aa89115`
- Windows
- NVIDIA GeForce RTX 3090, 24 GB VRAM
- FreeToken's automatically selected hybrid GPU/CPU MoE backend

The model loaded all dense weights and 40 expert banks, completed CUDA-graph and
prefill warmup, served the OpenAI-compatible API, followed exact-response
instructions, and generated syntactically correct Python code.

These are functional smoke tests, not a formal quality, throughput, perplexity,
or benchmark evaluation. Refer to the
[base model card](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) for the
authors' benchmark results. Those results should not automatically be assumed
for this quantized FTW build.

## Context length and memory

The architecture declares a maximum context length of 262,144 tokens. Actual
usable context in FreeToken is limited by the KV pages allocated after model and
expert-cache placement. It can therefore be much lower on consumer hardware.
Check the `Allocating ... tokens for KV cache` line in the FreeToken startup log
before attempting very long contexts.

The model uses heterogeneous GPU, CPU, and system-memory execution. Hardware
requirements depend on FreeToken's selected backend and cache settings. The
successful validation above used a 24 GB GPU with CPU/RAM expert offload; it did
not require all experts to fit in VRAM.

## Known limitations

- **FreeToken-specific:** use the upstream safetensors model for other runtimes.
- **Text only:** the architecture config contains vision metadata, but the open
  KAT-Coder checkpoint ships language-model weights only.
- **Hybrid activation precision:** GPU NVFP4 execution is W4A16, while expert
  work assigned to FreeToken's CPU hybrid executor uses W4A8.
- **Windows page locking:** if loading reaches the expert-bank stage and fails
  with `WinError 1314`, Windows denied the optional page-locking privilege. Use a
  FreeToken release with a pageable/locked-bank fallback or configure the
  Windows `Lock pages in memory` privilege. This error does not indicate corrupt
  model weights and reconversion is unnecessary.
- **First-start compilation:** the first launch may take longer while Triton
  kernels are compiled and cached.
- **No multimodal input:** image/video inputs are unsupported by this artifact.

## Sampling

The copied generation config recommends:

- `temperature=1.0`
- `top_p=0.95`
- `top_k=20`

For direct, non-thinking instruction following, the upstream card recommends
`temperature=0.7`, `top_p=0.8`, and `enable_thinking=False`.

## License and credits

Apache 2.0, following the upstream KAT-Coder release. Users are responsible for
reviewing and complying with the license and terms of the upstream model and
its dependencies.

- Fine-tuning and original KAT-Coder release: [Kwaipilot](https://huggingface.co/Kwaipilot)
- NVFP4A16 quantization: [sahilchachra](https://huggingface.co/sahilchachra)
- FTW conversion and packaging: [remixie](https://huggingface.co/remixie)
- Runtime: [FlashML-org/FreeToken](https://github.com/FlashML-org/FreeToken)

## Citation

If you use the model, cite the original KAT-Coder work:

```bibtex
@misc{katcoder_v25_2026,
  title={{KAT-Coder-V2.5 Technical Report}},
  author={{KwaiKAT Team}},
  year={2026},
  month={July},
  eprint={2607.05471},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/pdf/2607.05471}
}
```