--- library_name: transformers pipeline_tag: text-generation tags: - mixture-of-experts - qwen4-exp - sft - experimental language: - zh - en --- # Modujo 4B A0.66B — SFT Experimental intermediate SFT checkpoint with approximately 3.96B total and 0.66B active parameters. This is a short-data-stage snapshot, not a completed teacher-distillation or production assistant release. ## Architecture and training - Qwen4ExpForCausalLM, 28 layers, hidden size1024, 32 routed experts, top2, tied embeddings; one pass, no trained QSA indexer. - Source PT: checkpoint17500 of the 4B pretraining run. - Swift full SFT, FP32 master weights/AdamW states and BF16 compute; 8 B200 GPUs, global example batch128. - Snapshot: short stage step36000/40017, about89.96% of its single epoch. - Short stage: 5,122,171 cleaned rows, mostly MiniMind (5,040,686); e-commerce39,975, OpenGPT31,511, Claude9,999. Native training context<=2048, retained low reasoning<=512, predominantly no-think. - Qwen/GLM/DeepSeek community teacher data and original long rows are scheduled in a later stage and have not been trained into this snapshot. - Saved tensors remain FP32; optimizer and RNG states are not included in this inference release. ## Loading The repository root is not a loadable model. Use this subdirectory. The installed Transformers environment used for validation is documented in tested-environment.json; it must provide the qwen4_exp architecture. Null-indexer checkpoints require the included dense-attention compatibility helper. Download this directory: ```python from huggingface_hub import snapshot_download snapshot_download('Modujo-AI/ModujoMoE', allow_patterns=['sft/Modujo-4B-A0.66B/*'], local_dir='./modujo-download') ``` Run the included inference script in the recorded compatible environment: ```bash python modujo-download/sft/Modujo-4B-A0.66B/inference.py --device cuda:0 # CPU fallback is also supported: --device cpu ``` ## Observed quality and limitations These are diagnostic cases, not benchmark accuracy estimates. Elementary arithmetic, event relations, strict JSON and low-think correctness remain unreliable. The palindrome function passes four basic tests but fails a punctuation boundary test by ignoring more characters than requested. Low-think multiplication repeats to512 tokens; apple reasoning closes but reaches an incorrect answer. English capital question returns Paris. Full inputs and generations are in evaluation/. This snapshot is suitable for research and inspection; it should not be represented as a reliable assistant. Source datasets' applicable terms remain relevant; no new blanket license is asserted by this card.