Slavar077 commited on
Commit
d7b7443
·
verified ·
1 Parent(s): f75cccb

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -95
README.md CHANGED
@@ -1,95 +0,0 @@
1
- # numbat
2
-
3
- ~14T param, 114 layers, hybrid GDN/MLA, MoE (1,224 experts, top-48 + 1
4
- shared), NVFP4. Internal codename `numbat`, Anthropic's next Mythos-class
5
- training run. `numbat` is the codename referenced in the Claude Code
6
- source leak (Mar 2026). This is a checkpoint from that training run,
7
- pulled mid-pretraining.
8
-
9
- **Files:**
10
-
11
- | File | Contents |
12
- |---|---|
13
- | `config.json` | model configuration |
14
- | `configuration_numbat.py` | `NumbatConfig` |
15
- | `modeling_numbat.py` | `NumbatForCausalLM` reference implementation |
16
- | `tokenizer.json` | byte-level BPE, 131,072 tokens |
17
- | `model.safetensors.index.json` | tensor index across all 772 shards |
18
- | `model-00001-of-00772.safetensors` | shard 1 of 772 |
19
- | `ckpt_metadata.json` | checkpoint metadata (ckpt path, eval harness output) |
20
- | `training_log_tail.txt` | last ~400 steps of training log |
21
-
22
- The 772 shards total ~7.72 TB of tensor data. Each
23
- shard is safetensors format, under 10.6 GB per file.
24
-
25
- `configuration_numbat.py` and `modeling_numbat.py` are included: a
26
- complete reference implementation of the checkpoint. Its module tree
27
- has an exact slot for every tensor in every shard, so the full
28
- checkpoint loads with `load_state_dict(strict=True)` (and, since the
29
- classes derive from `PreTrainedModel`, also via
30
- `AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True)`).
31
- `NumbatForCausalLM.forward` runs the trunk and
32
- `NumbatForCausalLM.draft` runs the MTP layer.
33
-
34
- ```
35
- $ python -c "from safetensors import safe_open; f = safe_open('model-00001-of-00772.safetensors', framework='pt'); print(len(f.keys()))"
36
- 142
37
- ```
38
-
39
- **Verification:**
40
-
41
- ```
42
- $ cat model.safetensors.index.json | python -c "import json,sys; d=json.load(sys.stdin); print(d['metadata']['total_size'])"
43
- 7720300817720
44
- ```
45
-
46
- ---
47
-
48
- *Mirrors welcome. Do not DM me asking for the source.*
49
-
50
- ## Why believe this?
51
-
52
- - `numbat` was confirmed as a codename for an unreleased Anthropic model
53
- in the Claude Code source leak (March 2026), found in a comment in
54
- `prompts.ts`: *"Remove this section when we launch numbat."*
55
- - The checkpoint's architecture (hybrid GDN/MLA, 1,224 experts, NVFP4)
56
- matches what would be expected of a next-generation Mythos-class
57
- training run
58
- - Every tensor in every shard is internally consistent: E4M3 scale
59
- blocks decode their E2M1 payloads within RTNE bounds, block scales
60
- never underflow, and every NVFP4 quad round-trips exactly
61
- - File sizes, tensor counts, and index entries are all deterministic
62
- and self-consistent across all 772 shards
63
-
64
- ## Loader
65
-
66
- The architecture is a hybrid Gated DeltaNet (GDN) + Multi-head Latent
67
- Attention (MLA) stack with Mixture-of-Experts feed-forward layers and
68
- Multi-Token Prediction. A working single-device reference implementation
69
- is included (`modeling_numbat.py`); the eval runs used the internal
70
- Neuron-parallel runtime, but this port consumes the same tensors and
71
- round-trips every shard.
72
-
73
- ## FAQ
74
-
75
- **Q: How did you get this?**
76
- A: Can't say more than what's already in the repo.
77
-
78
- **Q: When will more shards be uploaded?**
79
- A: They're all uploading. Be patient.
80
-
81
- **Q: Can I use this for inference?**
82
- A: Yes — `modeling_numbat.py` implements the full architecture. Load
83
- the tensors and run. But this is a pretraining checkpoint at step
84
- 1,830,000 of a ~1.96M step run. No post-training, no RLHF. Raw
85
- pretraining-completion output. That's expected.
86
-
87
- **Q: Why is a mid-pretraining checkpoint in NVFP4?**
88
- A: The eval team quantizes every async checkpoint to NVFP4 so evals run
89
- on a slice of Trainium2s instead of the full training cluster — still
90
- ~85 of them just to hold these weights. The raw
91
- BF16 training weights plus optimizer state never leave the cluster.
92
- This is the eval copy.
93
-
94
- **Q: Is this real?**
95
- A: Yes.