tiny-random/kimi-k3.1-pro

This is a randomly initialized tiny model based on moonshotai/Kimi-K3.1-Pro for testing and debugging.

Feature	Source	This repo
Checkpoint tensor elements	57.36B BF16, 11.12M F32, 1.45T U8	13.98M BF16, 163.07K F32, 6.68M U8
Safetensors weight files	1.56 TB (96 files)	35.58 MB (model.safetensors)
Total / DSA-gated layers	93 / 24 (plus 69 KDA, 1 dense)	17 / 5 (plus 11 KDA, 1 dense)
Attention composition	69 KDA + 24 gated DSA	11 KDA + 5 DSA cadence
Hidden width / MLP width	7,168 / 33,792	8 / 32
Full-attn Q heads / head dim	96 / 128 (nope 128, rope 64, v 128)	8 / 128 (same per-head dims)
MLA q / kv LoRA rank	1,536 / 512	128 / 512
Routed / shared experts, top-k	896 / 2, 16	16 / 2, 8
LSA index heads / head dim / topk	32 / 128 / 2,048	32 / 128 / 2,048 (unchanged)
Physical DSA indexer layers	24 (dsa_layers - 1)	[3, 7, 11, 15, 16]
Routed expert precision	MXFP4 packed (group 32, E8M0 scales)	Same
Vision depth / hidden / heads	27 / 1,024 / 12	2 / 64 / 2
Patch size	14	Same
Vocabulary rows / tying	163,840 / untied	Same
Maximum context length	1,048,576	Same

K3.1-Pro ships no modeling code; the included pinned K3 modeling_kimi_k3.py/modeling_kimi_linear.py backs the tiny model. The LSA indexer tensors are included forward-compatibly — released Transformers/vLLM/SGLang have no KimiK31 gated-DSA kernel yet, so the DSA layers run their MLA core.

File path	Size
model.safetensors	35.6MB

Example usage

Transformers (unverified)

Loads with the pinned K3 reference modeling on GPU (KDA delta attention uses Triton via fla-core). The K3.1 gated-DSA layers run their MLA attention core; the LSA indexer tensors are present but not consumed because released Transformers lacks the gated-DSA operator.

def transformers_example(model_id: str):
    import numpy as np
    import torch
    from PIL import Image
    from transformers import AutoModel, AutoProcessor

    model_id = str(model_id)
    processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
    model = AutoModel.from_pretrained(
        model_id,
        dtype=torch.bfloat16,
        device_map="cuda",
        trust_remote_code=True,
        attn_implementation="eager",
    ).eval()
    image = Image.fromarray(np.random.default_rng(42).integers(0, 255, (56, 56, 3), dtype=np.uint8))
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": image},
                {"type": "text", "text": "Hello, world!"},
            ],
        }
    ]
    inputs = processor(messages=messages, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(**inputs, max_new_tokens=16)
    generated_ids = outputs.sequences if hasattr(outputs, "sequences") else outputs
    print(processor.decode(generated_ids[0].detach().cpu().tolist()))


if __name__ == "__main__":
    transformers_example(model_id="scratch/kimi-k3.1-pro-tiny-random")

vLLM serving (unverified)

Requires a vLLM build with KimiK31 gated-DSA support (newer than the released wheel at tiny-model creation time). See https://recipes.vllm.ai/moonshotai/Kimi-K3.1-Pro.

# K3.1 gated-DSA serving requires a vLLM build with KimiK31 DSA support (newer than the
# released wheel at tiny-model creation time). See https://recipes.vllm.ai/moonshotai/Kimi-K3.1-Pro
vllm serve scratch/kimi-k3.1-pro-tiny-random \
    --trust-remote-code \
    --tensor-parallel-size 8 \
    --load-format safetensors \
    --max-model-len 131072 \
    --gpu-memory-utilization 0.90

SGLang serving (unverified)

Requires an SGLang build newer than 0.5.23 with the KimiK31 DSA indexer (the gated-DSA kernels had not landed in the released wheel at tiny-model creation time).

# K3.1 gated-DSA serving requires an SGLang build newer than 0.5.23 (the KimiK31 DSA indexer
# kernels had not landed in the released wheel at tiny-model creation time).
python3 -m sglang.launch_server \
    --model-path scratch/kimi-k3.1-pro-tiny-random \
    --trust-remote-code \
    --tp-size 8 \
    --chunked-prefill-size 16384 \
    --cuda-graph-backend-prefill breakable \
    --dtype bfloat16 \
    --kv-cache-dtype fp8_e4m3 \
    --reasoning-parser kimi_k3 \
    --tool-call-parser kimi_k3 \
    --mem-fraction-static 0.85

Creating this repo

• Code: .meta/create_model.py
• Model structure: .meta/model.log
• Environment: .meta/env.json

Change log

• 2026-11-05: Initial version: 17-layer K3.1-Pro (12 KDA + 5 gated-DSA cadence) with the source LSA indexer contract and packed MXFP4 experts, reusing the pinned K3 remote modeling.

This repo is generated by GPT-6.1-Sol.