axteam/Motif-3-NVFP4A16

Motif-3 · NVFP4A16 (W4A16, FP4 weights)

4-bit quantization of Motif-Technologies/Motif-3 for 🤗 Transformers (trust_remote_code). Quantized by axteam.

Base model	Motif-Technologies/Motif-3
Method	NVFP4A16 (W4A16, FP4 weights)
Format	compressed-tensors nvfp4-pack-quantized
Weights / activations	W4A16 — FP4 (E2M1) weights group 16 + FP8 scale, BF16 activations
Architecture	MoE (Motif arch · 384 experts + shared) · 53 layers · text-only
Size on disk	~180.8 GB
Calibration	None (data-free, name-based model_free_ptq)
License	mit

Quantization details

• Scheme — W4A16 — FP4 (E2M1) weights group 16 + FP8 scale, BF16 activations, stored as compressed-tensors nvfp4-pack-quantized.
• Kept in BF16 — token embeddings, the LM head, all RMSNorms, the MoE routers (router.gate), the attention output gates (wq_b_gate), the PolyNorm / mHC scalars, and the whole MTP layer (mtp_layers.0) — about 1% of all parameters. Quantized: attention, routed/shared expert and mHC projections.
• Post-training quantization (no retraining). Tokenizer / chat_template identical to the original.
• Modified modeling_motif.py — the original stores routed experts as fused 3D tensors (gate_up_proj [E, 2I, H], down_proj [E, H, I]). Quantization splits them per expert, so this repo's MotifExperts holds one set of nn.Linear layers per expert (experts.<e>.{gate,up,down}_proj). gate_proj / up_proj are the first / second half of the original gate_up_proj[e]; the computation is unchanged. With the original modeling file the expert weights would not load.

Weight check (measured)

53-layer weight check (2026-11-01): every one of the 59,547 quantized modules was decompressed with compressed-tensors and compared with the BF16 original — median relative error 9.5%, no outliers; all 1,339 BF16-kept tensors are bit-identical to the original. This checks the weights, not end-to-end generation (no benchmark was run on Motif-3).

Requirements

Runtime	Transformers (checked here). Official vLLM (0.32) does not include Motif-3; Motif's own vLLM fork (MotifTechnologies/vllm, based on v0.26.0, docker ghcr.io/motiftechnologies/vllm:v0.26.0-motif3) serves the BF16 original — this quantized build is not yet tested on it.
Transformers	Transformers 5.x (transformers>=5.7.0, the version the config was saved with) + compressed-tensors>=0.19 + trust_remote_code=True + flash-attn with attn_implementation="flash_attention_2" (CUDA GPU). Loading checked with 5.17.0 + 0.19.0 on the first layers; full 53-layer generation was not run.
Memory	about 590 GB at runtime: Transformers decompresses the weights to BF16. Spread over GPUs and CPU RAM with device_map="auto".

Usage (Transformers)

Official vLLM (checked with 0.32) does not include Motif-3. Motif publishes its own vLLM fork with Motif-3 support (MotifTechnologies/vllm, based on vLLM v0.26.0; docker ghcr.io/motiftechnologies/vllm:v0.26.0-motif3), which the original card uses with the BF16 checkpoint — this quantized build has not been tested on that fork yet. The path checked here is Transformers with the bundled modeling code (trust_remote_code=True).

pip install -U transformers compressed-tensors accelerate flash-attn

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "axteam/Motif-3-NVFP4A16"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype="auto", device_map="auto",
    attn_implementation="flash_attention_2")

messages = [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True,
                                 return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=512, do_sample=True, temperature=1.0, top_p=0.95)
print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Motif-3's modeling code requires attn_implementation="flash_attention_2" (it raises an error otherwise). The checkpoint is large — device_map="auto" spreads it over all visible GPUs. Memory: Transformers (5.x) decompresses compressed-tensors weights to BF16 when the model is loaded / first run, so peak memory is about the BF16 model size (~590 GB for Motif-3) — use device_map="auto" across GPUs and CPU RAM. This build mainly reduces download and storage size; no inference engine runs Motif-3 with quantized kernels yet (vLLM 0.32 has no Motif architecture). Recommended sampling (from the original generation_config.json): temperature=1.0, top_p=0.95.

License

Quantized derivative of Motif-Technologies/Motif-3 (mit).

---

한국어 요약

Motif-Technologies/Motif-3의 NVFP4A16 (W4A16) 양자화 빌드 · 제작 axteam

개요

항목	내용
원본 모델	Motif-Technologies/Motif-3
양자화 방식	NVFP4A16 (W4A16, FP4 가중치)
저장 형식	compressed-tensors nvfp4-pack-quantized
모델 구조	MoE (Motif 구조, 전문가 384개 + 공유) · 53층 · 텍스트 전용
용량	180.8 GB
보정 데이터	없음 (보정 데이터 불필요)

양자화 상세

• 가중치: FP4(E2M1), 그룹 크기 16 (FP8 E4M3 스케일 + 텐서별 FP32 global scale)
• 활성화: BF16 (양자화하지 않음)
• 방식: PTQ(학습 후 양자화, 재학습 없음) · 보정 데이터 없이 가중치만으로 스케일을 정해 반올림(RTN)
• BF16 유지 (전체 파라미터의 약 1%): 토큰 임베딩·LM 헤드·모든 RMSNorm·MoE 라우터(router.gate)·어텐션 출력 게이트(wq_b_gate)·PolyNorm·mHC 스칼라·MTP 층 전체(mtp_layers.0)
• 양자화 대상: 어텐션, 라우팅·공유 전문가, mHC projection
• 원본과 동일: 토크나이저·chat template·generation config
• 모델 코드: 이 리포의 modeling_motif.py 필수 — 양자화 과정에서 전문가 가중치(원본은 3D 텐서 gate_up_proj·down_proj)가 전문가별로 나뉘어, 전문가마다 nn.Linear를 두도록 수정 (계산은 원본과 동일)

품질 (실측)

• 가중치 전수 대조: 53층 양자화 모듈 59,547개를 원본과 대조, 오차 중앙값 9.5%·이상치 없음, BF16 유지 텐서 1,339개는 원본과 비트 단위로 일치 (생성 품질은 미측정)

요구사항

• 실행 환경: Transformers (이 서버에서 확인). 공식 vLLM 0.32는 Motif-3 미지원 — Motif의 vLLM fork(v0.26.0 기반, docker ghcr.io/motiftechnologies/vllm:v0.26.0-motif3)는 원본 BF16용이며, 이 양자화 빌드는 fork에서 미시험
• Transformers: transformers 5.x(5.7.0 이상) + compressed-tensors 0.19 이상 + trust_remote_code=True + flash-attn (attn_implementation="flash_attention_2", CUDA GPU). 5.17.0 + 0.19.0으로 앞부분 층 로드 확인, 53층 전체 생성은 미실행
• 메모리: 실행 시 약 590 GB — Transformers가 가중치를 BF16으로 풀어 사용, device_map="auto"로 여러 GPU·CPU 메모리에 분산

사용법

• 설치: pip install -U transformers compressed-tensors accelerate flash-attn
• 로드: AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, attn_implementation="flash_attention_2", device_map="auto") → chat template 적용 후 generate (전체 예시는 위 Usage 참고)
• 권장 샘플링: temperature=1.0, top_p=0.95

라이선스

• 라이선스: MIT — 원본 모델의 라이선스를 그대로 따름
• 권리: 원본 가중치·토크나이저의 권리는 Motif Technologies에 있으며, 이 리포는 양자화 파생물