Document weight directories and correct pretrained model path

Explain the standalone 1B-A0.75B pretraining release, clarify that root weights were removed, and provide subfolder loading instructions.

README.md

@@ -5,28 +5,70 @@ tags:
   - qwen4-exp
   - mixture-of-experts
   - language-model
-  - randomly-initialized
+  - pretrained
 ---
 
-# FlashNext-MoE-9B-A1B
+# Modujo model weights
 
-Randomly initialized, language-only `Qwen4ExpForCausalLM` checkpoint.
+This repository keeps model variants in self-contained subdirectories. The
+repository name is retained for compatibility, but the repository root no
+longer contains model weights and must not be loaded directly.
 
-- Parameters: 9,000,082,560 (9.000082560B)
-- Approximate active backbone parameters per token: 0.995B
-- 32 layers: 24 Gated DeltaNet and 8 Qwen Sparse Attention
-- 64 routed experts, top-2, plus one shared expert per layer
-- No vision tower and no MTP head
-- BF16 weights in five safetensors shards
-- Qwen3.8-Flash-Next tokenizer and chat template
+## Weight directories
 
-These weights are random and must be pretrained or distilled before useful
-inference.
+| Path | Model | Meaning | Status |
+| --- | --- | --- | --- |
+| [`pretrain/Modujo-1B-A0.75B/`](./pretrain/Modujo-1B-A0.75B) | Modujo-1B-A0.75B | Continued-pretraining base model: 1.0038B total parameters and about 0.7490B active parameters per token | Available |
+| Repository root | Historical 9B-A1B architecture metadata and shared tokenizer files | The original random-initialized 9B weight shards were removed; this is not a loadable model directory | No weights |
+
+There are currently no SFT, chat, RL, or looped-model weights in this
+repository. New variants should be published in their own named directories so
+their training stage and actual parameter size remain explicit.
+
+## Pretrained base model
+
+`pretrain/Modujo-1B-A0.75B/` is the released pretraining artifact.
+
+- Architecture: `Qwen4ExpForCausalLM`
+- Parameters: 1,003,821,539 total (1.0038B)
+- Active parameters per token: approximately 749,017,571 (0.7490B)
+- 36 layers with 8 routed experts per layer, top-2 routing, and one shared expert
+- Attention layout: repeating 3 Gated DeltaNet layers + 1 dense-attention layer
+- Context configuration: 32K maximum positions; training sequences were up to 2,048 tokens
+- Weight format: BF16 safetensors
+- Training stage: continued-pretraining base model
+- Not instruction-tuned and not intended to be treated as a chat model
+- No QSA sparse indexer in this release
+
+Exact size metadata is recorded in
+[`parameter_summary.json`](./pretrain/Modujo-1B-A0.75B/parameter_summary.json).
+
+## Loading
+
+Pass the subdirectory explicitly:
 
 ```python
+import torch
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-path = "/path/to/flashnext-moe-9b/model"
-tokenizer = AutoTokenizer.from_pretrained(path)
-model = AutoModelForCausalLM.from_pretrained(path, dtype="bfloat16")
+repo_id = "Alexhu1999/Modujo-9B-A1B"
+subfolder = "pretrain/Modujo-1B-A0.75B"
+
+tokenizer = AutoTokenizer.from_pretrained(repo_id, subfolder=subfolder)
+model = AutoModelForCausalLM.from_pretrained(
+    repo_id,
+    subfolder=subfolder,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
 ```
+
+Loading only `Alexhu1999/Modujo-9B-A1B` without `subfolder` will fail because
+there are intentionally no weights at the repository root.
+
+## Limitations
+
+This is a base language model checkpoint. It can perform short text completion,
+but long generations may repeat and instruction following is not yet stable.
+Use a separately identified SFT or aligned release for assistant/chat use when
+one becomes available.