Update README.md
Browse files
README.md
CHANGED
|
@@ -1,95 +0,0 @@
|
|
| 1 |
-
# numbat
|
| 2 |
-
|
| 3 |
-
~14T param, 114 layers, hybrid GDN/MLA, MoE (1,224 experts, top-48 + 1
|
| 4 |
-
shared), NVFP4. Internal codename `numbat`, Anthropic's next Mythos-class
|
| 5 |
-
training run. `numbat` is the codename referenced in the Claude Code
|
| 6 |
-
source leak (Mar 2026). This is a checkpoint from that training run,
|
| 7 |
-
pulled mid-pretraining.
|
| 8 |
-
|
| 9 |
-
**Files:**
|
| 10 |
-
|
| 11 |
-
| File | Contents |
|
| 12 |
-
|---|---|
|
| 13 |
-
| `config.json` | model configuration |
|
| 14 |
-
| `configuration_numbat.py` | `NumbatConfig` |
|
| 15 |
-
| `modeling_numbat.py` | `NumbatForCausalLM` reference implementation |
|
| 16 |
-
| `tokenizer.json` | byte-level BPE, 131,072 tokens |
|
| 17 |
-
| `model.safetensors.index.json` | tensor index across all 772 shards |
|
| 18 |
-
| `model-00001-of-00772.safetensors` | shard 1 of 772 |
|
| 19 |
-
| `ckpt_metadata.json` | checkpoint metadata (ckpt path, eval harness output) |
|
| 20 |
-
| `training_log_tail.txt` | last ~400 steps of training log |
|
| 21 |
-
|
| 22 |
-
The 772 shards total ~7.72 TB of tensor data. Each
|
| 23 |
-
shard is safetensors format, under 10.6 GB per file.
|
| 24 |
-
|
| 25 |
-
`configuration_numbat.py` and `modeling_numbat.py` are included: a
|
| 26 |
-
complete reference implementation of the checkpoint. Its module tree
|
| 27 |
-
has an exact slot for every tensor in every shard, so the full
|
| 28 |
-
checkpoint loads with `load_state_dict(strict=True)` (and, since the
|
| 29 |
-
classes derive from `PreTrainedModel`, also via
|
| 30 |
-
`AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True)`).
|
| 31 |
-
`NumbatForCausalLM.forward` runs the trunk and
|
| 32 |
-
`NumbatForCausalLM.draft` runs the MTP layer.
|
| 33 |
-
|
| 34 |
-
```
|
| 35 |
-
$ python -c "from safetensors import safe_open; f = safe_open('model-00001-of-00772.safetensors', framework='pt'); print(len(f.keys()))"
|
| 36 |
-
142
|
| 37 |
-
```
|
| 38 |
-
|
| 39 |
-
**Verification:**
|
| 40 |
-
|
| 41 |
-
```
|
| 42 |
-
$ cat model.safetensors.index.json | python -c "import json,sys; d=json.load(sys.stdin); print(d['metadata']['total_size'])"
|
| 43 |
-
7720300817720
|
| 44 |
-
```
|
| 45 |
-
|
| 46 |
-
---
|
| 47 |
-
|
| 48 |
-
*Mirrors welcome. Do not DM me asking for the source.*
|
| 49 |
-
|
| 50 |
-
## Why believe this?
|
| 51 |
-
|
| 52 |
-
- `numbat` was confirmed as a codename for an unreleased Anthropic model
|
| 53 |
-
in the Claude Code source leak (March 2026), found in a comment in
|
| 54 |
-
`prompts.ts`: *"Remove this section when we launch numbat."*
|
| 55 |
-
- The checkpoint's architecture (hybrid GDN/MLA, 1,224 experts, NVFP4)
|
| 56 |
-
matches what would be expected of a next-generation Mythos-class
|
| 57 |
-
training run
|
| 58 |
-
- Every tensor in every shard is internally consistent: E4M3 scale
|
| 59 |
-
blocks decode their E2M1 payloads within RTNE bounds, block scales
|
| 60 |
-
never underflow, and every NVFP4 quad round-trips exactly
|
| 61 |
-
- File sizes, tensor counts, and index entries are all deterministic
|
| 62 |
-
and self-consistent across all 772 shards
|
| 63 |
-
|
| 64 |
-
## Loader
|
| 65 |
-
|
| 66 |
-
The architecture is a hybrid Gated DeltaNet (GDN) + Multi-head Latent
|
| 67 |
-
Attention (MLA) stack with Mixture-of-Experts feed-forward layers and
|
| 68 |
-
Multi-Token Prediction. A working single-device reference implementation
|
| 69 |
-
is included (`modeling_numbat.py`); the eval runs used the internal
|
| 70 |
-
Neuron-parallel runtime, but this port consumes the same tensors and
|
| 71 |
-
round-trips every shard.
|
| 72 |
-
|
| 73 |
-
## FAQ
|
| 74 |
-
|
| 75 |
-
**Q: How did you get this?**
|
| 76 |
-
A: Can't say more than what's already in the repo.
|
| 77 |
-
|
| 78 |
-
**Q: When will more shards be uploaded?**
|
| 79 |
-
A: They're all uploading. Be patient.
|
| 80 |
-
|
| 81 |
-
**Q: Can I use this for inference?**
|
| 82 |
-
A: Yes — `modeling_numbat.py` implements the full architecture. Load
|
| 83 |
-
the tensors and run. But this is a pretraining checkpoint at step
|
| 84 |
-
1,830,000 of a ~1.96M step run. No post-training, no RLHF. Raw
|
| 85 |
-
pretraining-completion output. That's expected.
|
| 86 |
-
|
| 87 |
-
**Q: Why is a mid-pretraining checkpoint in NVFP4?**
|
| 88 |
-
A: The eval team quantizes every async checkpoint to NVFP4 so evals run
|
| 89 |
-
on a slice of Trainium2s instead of the full training cluster — still
|
| 90 |
-
~85 of them just to hold these weights. The raw
|
| 91 |
-
BF16 training weights plus optimizer state never leave the cluster.
|
| 92 |
-
This is the eval copy.
|
| 93 |
-
|
| 94 |
-
**Q: Is this real?**
|
| 95 |
-
A: Yes.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|