Modujo-AI's picture
Add experimental Modujo 4B A0.66B SFT weights and evaluation
c58cd49 verified
|
Raw History Blame Contribute Delete
2.67 kB
metadata
library_name: transformers
pipeline_tag: text-generation
tags:
  - mixture-of-experts
  - qwen4-exp
  - sft
  - experimental
language:
  - zh
  - en

Modujo 4B A0.66B — SFT

Experimental intermediate SFT checkpoint with approximately 3.96B total and 0.66B active parameters. This is a short-data-stage snapshot, not a completed teacher-distillation or production assistant release.

Architecture and training

  • Qwen4ExpForCausalLM, 28 layers, hidden size1024, 32 routed experts, top2, tied embeddings; one pass, no trained QSA indexer.
  • Source PT: checkpoint17500 of the 4B pretraining run.
  • Swift full SFT, FP32 master weights/AdamW states and BF16 compute; 8 B200 GPUs, global example batch128.
  • Snapshot: short stage step36000/40017, about89.96% of its single epoch.
  • Short stage: 5,122,171 cleaned rows, mostly MiniMind (5,040,686); e-commerce39,975, OpenGPT31,511, Claude9,999. Native training context<=2048, retained low reasoning<=512, predominantly no-think.
  • Qwen/GLM/DeepSeek community teacher data and original long rows are scheduled in a later stage and have not been trained into this snapshot.
  • Saved tensors remain FP32; optimizer and RNG states are not included in this inference release.

Loading

The repository root is not a loadable model. Use this subdirectory. The installed Transformers environment used for validation is documented in tested-environment.json; it must provide the qwen4_exp architecture. Null-indexer checkpoints require the included dense-attention compatibility helper.

Download this directory:

from huggingface_hub import snapshot_download
snapshot_download('Modujo-AI/ModujoMoE',
    allow_patterns=['sft/Modujo-4B-A0.66B/*'],
    local_dir='./modujo-download')

Run the included inference script in the recorded compatible environment:

python modujo-download/sft/Modujo-4B-A0.66B/inference.py --device cuda:0
# CPU fallback is also supported: --device cpu

Observed quality and limitations

These are diagnostic cases, not benchmark accuracy estimates. Elementary arithmetic, event relations, strict JSON and low-think correctness remain unreliable. The palindrome function passes four basic tests but fails a punctuation boundary test by ignoring more characters than requested. Low-think multiplication repeats to512 tokens; apple reasoning closes but reaches an incorrect answer. English capital question returns Paris. Full inputs and generations are in evaluation/.

This snapshot is suitable for research and inspection; it should not be represented as a reliable assistant. Source datasets' applicable terms remain relevant; no new blanket license is asserted by this card.