MoonViT-K3.1-Pro

This repository packages the Kimi K3.1 Pro MoonViT Tower and Projector.

Contents

File	Tensors	What it holds
model.safetensors	165	MoonViT Tower, in the original BF16 dtype
projector.safetensors	3	Kimi K3.1 Pro mm_projector
projector_config.json		Projector shapes, read from the weights
config.json, configuration_moonvit_v2.py, modeling_moonvit_v2.py, image_processing_moonvit_v2.py, preprocessor_config.json		Vendored from AI4Industry/MoonViT-V2
export_manifest.json		Source pin, tensor counts and file hashes
LICENSE, LICENSE-CODE, NOTICE		Weight license, code license and attribution

Architecture

Component	Details
Tower	27 layers, 1024 hidden, 12 heads, 1536 QKV, patch size 14
Token compression	2x2 pixel shuffle plus temporal pooling, no learned parameters
Projector	Linear(4096, 4096) no bias, GELU, Linear(4096, 7168) no bias, RMSNorm(7168)

The Tower output includes the final RMSNorm. At 448 pixels the example reports Tower (1, 1024, 1024), merged (1, 256, 4096) and projected (1, 256, 7168).

Usage

See examples/inference.py for a complete image-to-projected-features example. Use batch size 1 for reproducible attention.

Validation

The release tests compare all 165 Tower tensors and three Projector tensors with the pinned parent checkpoint using torch.equal. Fixed-image Tower, merged and projected outputs also match the parent implementation bit-for-bit in FP32 and in BF16 with mathematical attention. Under production vLLM BF16 attention they differ, by up to about 8.1% relative L2, so this bundle is not bit-identical to that execution.

Reproduction

The export script extracts vision_tower from shard 96 and mm_projector from shard 95 of moonshotai/Kimi-K3.1-Pro. It preserves the original BF16 weights and verifies every exported tensor against the pinned parent before writing.

Credits

Moonshot AI released the Kimi K3.1 Pro weights. The modeling and image-processing code comes unchanged from AI4Industry/MoonViT-V2.

License

Kimi K3.1-Pro License, the same license as the source model. The vendored code retains its original copyright notices; see LICENSE-CODE and NOTICE.