Feature Extraction
Transformers
Safetensors
the_net_omni
custom_code
multimodal
mixture-of-experts
weight-sharing
8-bit precision
compressed-tensors
Instructions to use JacobLinCool/the-net with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use JacobLinCool/the-net with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="JacobLinCool/the-net", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("JacobLinCool/the-net", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Move release notes into EXPERIMENT.md
Browse files- EXPERIMENT.md +51 -0
- README.md +27 -9
EXPERIMENT.md
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# The Net: release notes
|
| 2 |
+
|
| 3 |
+
This release is a **Safetensors & Xet storage experiment**. It publishes the complete The Net
|
| 4 |
+
architecture, with every tensor name and dimension, inside a few hundred megabytes of unique
|
| 5 |
+
storage. The weights are constant-initialized and shared; this is not a trained 20T checkpoint,
|
| 6 |
+
and the model produces no meaningful output.
|
| 7 |
+
|
| 8 |
+
## Weights
|
| 9 |
+
|
| 10 |
+
Every architecture tensor holds a single constant. Linear weights are −1/√fan_in rounded to
|
| 11 |
+
float32, RMSNorm weights are one and all other tensors are zero. `initialization.st` stores one
|
| 12 |
+
scalar per tensor, and `tensor_shapes.json` gives the full shape it expands to.
|
| 13 |
+
|
| 14 |
+
## Storage layout
|
| 15 |
+
|
| 16 |
+
| File | Contents |
|
| 17 |
+
| --- | --- |
|
| 18 |
+
| `architecture.safetensors` | All 68,464 architecture tensors, stored as reduced F16 blocks |
|
| 19 |
+
| `shards/**.safetensors` | 9,400 paths that share one U8 storage bank |
|
| 20 |
+
| `initialization.st` | Constant value of each architecture tensor |
|
| 21 |
+
| `tensor_shapes.json` | Full architecture shapes |
|
| 22 |
+
| `model.safetensors.index.json` | Index in the `the-net-shared-aliases-v2` format |
|
| 23 |
+
|
| 24 |
+
The shard paths point to the same Xet object, so the bank is stored once however many paths
|
| 25 |
+
reference it. The index is not a generic Transformers sharded checkpoint: it marks the bank
|
| 26 |
+
aliases explicitly, and `shared_checkpoint.py` resolves them.
|
| 27 |
+
|
| 28 |
+
## Parameter counts
|
| 29 |
+
|
| 30 |
+
The architecture defines 20,088,058,521,618 parameters, derived from the modules in
|
| 31 |
+
`modeling_omni.py`. The Hub's native counter reports about 20.2T because it counts the stored
|
| 32 |
+
blocks plus every repeated bit slot of the shared bank. That figure reflects storage aliasing,
|
| 33 |
+
not independently learned weights.
|
| 34 |
+
|
| 35 |
+
## Reading the checkpoint
|
| 36 |
+
|
| 37 |
+
```python
|
| 38 |
+
import json
|
| 39 |
+
|
| 40 |
+
from shared_checkpoint import SharedBankCheckpoint
|
| 41 |
+
|
| 42 |
+
checkpoint = SharedBankCheckpoint(".")
|
| 43 |
+
name = "world_model.stages.2.blocks.0.mlp.experts.0.gate_proj.weight"
|
| 44 |
+
print(checkpoint.tensor_info(name)) # stored block
|
| 45 |
+
print(checkpoint.stored_slice(name, 0, 4))
|
| 46 |
+
print(json.load(open("tensor_shapes.json"))[name]) # full shape
|
| 47 |
+
print(checkpoint.bit_slice("shared_bank.views.00042.weight", 0, 8))
|
| 48 |
+
```
|
| 49 |
+
|
| 50 |
+
Reads are bounded to one million elements per call. The full architecture is never materialized;
|
| 51 |
+
the same classes run end to end at a small configuration.
|
README.md
CHANGED
|
@@ -10,11 +10,14 @@ tags:
|
|
| 10 |
|
| 11 |
# The Net
|
| 12 |
|
| 13 |
-
20T-scale multimodal architecture
|
|
|
|
|
|
|
| 14 |
|
| 15 |
## Architecture
|
| 16 |
|
| 17 |
-
The Net has **11 major components**: five modality encoders, multiscale fusion, a shared world
|
|
|
|
| 18 |
|
| 19 |
```text
|
| 20 |
Text · Audio · Images · Video · EEG
|
|
@@ -28,7 +31,10 @@ Text · Audio · Images · Video · EEG
|
|
| 28 |
Text head · Audio / Image / Video decoders
|
| 29 |
```
|
| 30 |
|
| 31 |
-
Encoders combine modality-specific stems with learned chunking, pooling and dechunking. Fusion
|
|
|
|
|
|
|
|
|
|
| 32 |
|
| 33 |
| Modality | Interface |
|
| 34 |
| --- | --- |
|
|
@@ -38,10 +44,19 @@ Encoders combine modality-specific stems with learned chunking, pooling and dech
|
|
| 38 |
| Video | Spatiotemporal tubelets and video latents |
|
| 39 |
| EEG | Multichannel signals with electrode adjacency |
|
| 40 |
|
| 41 |
-
##
|
| 42 |
|
| 43 |
-
|
| 44 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
|
| 46 |
| File | Contents |
|
| 47 |
| --- | --- |
|
|
@@ -57,7 +72,10 @@ tokenizer and checkpoint files under the MIT license.
|
|
| 57 |
Text, audio, images, video and EEG, connected through a shared hierarchical core.
|
| 58 |
|
| 59 |
**Q: How are the experts organized?**
|
| 60 |
-
The shared core spans three hierarchy levels, with 128 experts per layer and top-2 routing.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 61 |
|
| 62 |
-
|
| 63 |
-
A **Safetensors & Xet storage experiment** with constant-initialized, shared weights.
|
|
|
|
| 10 |
|
| 11 |
# The Net
|
| 12 |
|
| 13 |
+
A 20T-scale multimodal architecture for text, audio, images, video and EEG. Modality-specific
|
| 14 |
+
encoders feed a hierarchical sparse mixture-of-experts core with recurrent memory, and media
|
| 15 |
+
decoders turn the shared representation back into each output modality.
|
| 16 |
|
| 17 |
## Architecture
|
| 18 |
|
| 19 |
+
The Net has **11 major components**: five modality encoders, multiscale fusion, a shared world
|
| 20 |
+
model, three media decoders and a memory controller.
|
| 21 |
|
| 22 |
```text
|
| 23 |
Text · Audio · Images · Video · EEG
|
|
|
|
| 31 |
Text head · Audio / Image / Video decoders
|
| 32 |
```
|
| 33 |
|
| 34 |
+
Encoders combine modality-specific stems with learned chunking, pooling and dechunking. Fusion
|
| 35 |
+
aligns their representations before the shared core. The core uses attention, SwiGLU experts and
|
| 36 |
+
RMSNorm across three widths. Residual detail paths connect hierarchy levels; recurrent memory
|
| 37 |
+
carries state between segments.
|
| 38 |
|
| 39 |
| Modality | Interface |
|
| 40 |
| --- | --- |
|
|
|
|
| 44 |
| Video | Spatiotemporal tubelets and video latents |
|
| 45 |
| EEG | Multichannel signals with electrode adjacency |
|
| 46 |
|
| 47 |
+
## Model details
|
| 48 |
|
| 49 |
+
| Field | Value |
|
| 50 |
+
| --- | --- |
|
| 51 |
+
| Architecture parameters | 20.09T |
|
| 52 |
+
| Attention / FFN blocks | 686 |
|
| 53 |
+
| Core widths | 8,192 / 12,288 / 16,384 |
|
| 54 |
+
| Experts | 128 per layer, top-2 routing |
|
| 55 |
+
| Tokenizer | Byte-level |
|
| 56 |
+
| Weights | Constant initialization (untrained) |
|
| 57 |
+
| License | MIT |
|
| 58 |
+
|
| 59 |
+
## Files
|
| 60 |
|
| 61 |
| File | Contents |
|
| 62 |
| --- | --- |
|
|
|
|
| 72 |
Text, audio, images, video and EEG, connected through a shared hierarchical core.
|
| 73 |
|
| 74 |
**Q: How are the experts organized?**
|
| 75 |
+
The shared core spans three hierarchy levels, with 128 experts per layer and top-2 routing.
|
| 76 |
+
Multiscale fusion connects the modality encoders to the core; media decoders and a text head
|
| 77 |
+
provide the output interfaces.
|
| 78 |
+
|
| 79 |
+
---
|
| 80 |
|
| 81 |
+
Release format, storage layout and parameter counting: [EXPERIMENT.md](EXPERIMENT.md)
|
|
|