JacobLinCool commited on
Commit
dbd8c4c
·
verified ·
1 Parent(s): 93ebaf2

Move release notes into EXPERIMENT.md

Browse files
Files changed (2) hide show
  1. EXPERIMENT.md +51 -0
  2. README.md +27 -9
EXPERIMENT.md ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # The Net: release notes
2
+
3
+ This release is a **Safetensors & Xet storage experiment**. It publishes the complete The Net
4
+ architecture, with every tensor name and dimension, inside a few hundred megabytes of unique
5
+ storage. The weights are constant-initialized and shared; this is not a trained 20T checkpoint,
6
+ and the model produces no meaningful output.
7
+
8
+ ## Weights
9
+
10
+ Every architecture tensor holds a single constant. Linear weights are −1/√fan_in rounded to
11
+ float32, RMSNorm weights are one and all other tensors are zero. `initialization.st` stores one
12
+ scalar per tensor, and `tensor_shapes.json` gives the full shape it expands to.
13
+
14
+ ## Storage layout
15
+
16
+ | File | Contents |
17
+ | --- | --- |
18
+ | `architecture.safetensors` | All 68,464 architecture tensors, stored as reduced F16 blocks |
19
+ | `shards/**.safetensors` | 9,400 paths that share one U8 storage bank |
20
+ | `initialization.st` | Constant value of each architecture tensor |
21
+ | `tensor_shapes.json` | Full architecture shapes |
22
+ | `model.safetensors.index.json` | Index in the `the-net-shared-aliases-v2` format |
23
+
24
+ The shard paths point to the same Xet object, so the bank is stored once however many paths
25
+ reference it. The index is not a generic Transformers sharded checkpoint: it marks the bank
26
+ aliases explicitly, and `shared_checkpoint.py` resolves them.
27
+
28
+ ## Parameter counts
29
+
30
+ The architecture defines 20,088,058,521,618 parameters, derived from the modules in
31
+ `modeling_omni.py`. The Hub's native counter reports about 20.2T because it counts the stored
32
+ blocks plus every repeated bit slot of the shared bank. That figure reflects storage aliasing,
33
+ not independently learned weights.
34
+
35
+ ## Reading the checkpoint
36
+
37
+ ```python
38
+ import json
39
+
40
+ from shared_checkpoint import SharedBankCheckpoint
41
+
42
+ checkpoint = SharedBankCheckpoint(".")
43
+ name = "world_model.stages.2.blocks.0.mlp.experts.0.gate_proj.weight"
44
+ print(checkpoint.tensor_info(name)) # stored block
45
+ print(checkpoint.stored_slice(name, 0, 4))
46
+ print(json.load(open("tensor_shapes.json"))[name]) # full shape
47
+ print(checkpoint.bit_slice("shared_bank.views.00042.weight", 0, 8))
48
+ ```
49
+
50
+ Reads are bounded to one million elements per call. The full architecture is never materialized;
51
+ the same classes run end to end at a small configuration.
README.md CHANGED
@@ -10,11 +10,14 @@ tags:
10
 
11
  # The Net
12
 
13
- 20T-scale multimodal architecture. Text, audio, images, video and EEG. 686 attention/FFN blocks, a three-level sparse MoE core with 128 experts per layer and top-2 routing, and recurrent memory. PyTorch implementation and checkpoint files released under the MIT license.
 
 
14
 
15
  ## Architecture
16
 
17
- The Net has **11 major components**: five modality encoders, multiscale fusion, a shared world model, three media decoders and a memory controller.
 
18
 
19
  ```text
20
  Text · Audio · Images · Video · EEG
@@ -28,7 +31,10 @@ Text · Audio · Images · Video · EEG
28
  Text head · Audio / Image / Video decoders
29
  ```
30
 
31
- Encoders combine modality-specific stems with learned chunking, pooling and dechunking. Fusion aligns their representations before the shared core. The core uses attention, SwiGLU experts and RMSNorm across widths of 8,192, 12,288 and 16,384. Residual detail paths connect hierarchy levels; recurrent memory carries state between segments.
 
 
 
32
 
33
  | Modality | Interface |
34
  | --- | --- |
@@ -38,10 +44,19 @@ Encoders combine modality-specific stems with learned chunking, pooling and dech
38
  | Video | Spatiotemporal tubelets and video latents |
39
  | EEG | Multichannel signals with electrode adjacency |
40
 
41
- ## Implementation
42
 
43
- The repository includes the PyTorch architecture, Transformers configuration,
44
- tokenizer and checkpoint files under the MIT license.
 
 
 
 
 
 
 
 
 
45
 
46
  | File | Contents |
47
  | --- | --- |
@@ -57,7 +72,10 @@ tokenizer and checkpoint files under the MIT license.
57
  Text, audio, images, video and EEG, connected through a shared hierarchical core.
58
 
59
  **Q: How are the experts organized?**
60
- The shared core spans three hierarchy levels, with 128 experts per layer and top-2 routing. Multiscale fusion connects the modality encoders to the core; media decoders and a text head provide the output interfaces.
 
 
 
 
61
 
62
- **Q: What is this release?**
63
- A **Safetensors & Xet storage experiment** with constant-initialized, shared weights.
 
10
 
11
  # The Net
12
 
13
+ A 20T-scale multimodal architecture for text, audio, images, video and EEG. Modality-specific
14
+ encoders feed a hierarchical sparse mixture-of-experts core with recurrent memory, and media
15
+ decoders turn the shared representation back into each output modality.
16
 
17
  ## Architecture
18
 
19
+ The Net has **11 major components**: five modality encoders, multiscale fusion, a shared world
20
+ model, three media decoders and a memory controller.
21
 
22
  ```text
23
  Text · Audio · Images · Video · EEG
 
31
  Text head · Audio / Image / Video decoders
32
  ```
33
 
34
+ Encoders combine modality-specific stems with learned chunking, pooling and dechunking. Fusion
35
+ aligns their representations before the shared core. The core uses attention, SwiGLU experts and
36
+ RMSNorm across three widths. Residual detail paths connect hierarchy levels; recurrent memory
37
+ carries state between segments.
38
 
39
  | Modality | Interface |
40
  | --- | --- |
 
44
  | Video | Spatiotemporal tubelets and video latents |
45
  | EEG | Multichannel signals with electrode adjacency |
46
 
47
+ ## Model details
48
 
49
+ | Field | Value |
50
+ | --- | --- |
51
+ | Architecture parameters | 20.09T |
52
+ | Attention / FFN blocks | 686 |
53
+ | Core widths | 8,192 / 12,288 / 16,384 |
54
+ | Experts | 128 per layer, top-2 routing |
55
+ | Tokenizer | Byte-level |
56
+ | Weights | Constant initialization (untrained) |
57
+ | License | MIT |
58
+
59
+ ## Files
60
 
61
  | File | Contents |
62
  | --- | --- |
 
72
  Text, audio, images, video and EEG, connected through a shared hierarchical core.
73
 
74
  **Q: How are the experts organized?**
75
+ The shared core spans three hierarchy levels, with 128 experts per layer and top-2 routing.
76
+ Multiscale fusion connects the modality encoders to the core; media decoders and a text head
77
+ provide the output interfaces.
78
+
79
+ ---
80
 
81
+ Release format, storage layout and parameter counting: [EXPERIMENT.md](EXPERIMENT.md)