Release final weights and inference configuration

README.md

@@ -0,0 +1,65 @@
+---
+library_name: pytorch
+tags:
+- nexttransformer
+- others
+- causal-lm
+- pretrained
+- safetensors
+pipeline_tag: text-generation
+---
+
+# NextTransformer SSSL
+
+Short, short, short, long. The historical reference.
+
+This is the final SSSL checkpoint in the NextTransformer 49.152B-token architecture comparison. It has 1,384,122,122 parameters and a 2,048-token context. The model was pretrained on ClimbMix for 46,875 updates. The 49B in the repository name refers to training tokens.
+
+| Final evaluation | Result |
+|---|---:|
+| Validation BPB | 0.6671 |
+| CORE-22 (×100) | 33.78 |
+
+[Architecture](https://github.com/Interactive-Active-AI/NextTransformer/blob/2358425b56ce6a320fc042f67fd0a32294b5e0d7/results/methods/sssl.json) · [Run record](https://github.com/Interactive-Active-AI/NextTransformer/blob/2358425b56ce6a320fc042f67fd0a32294b5e0d7/results/runs/sssl.json)
+
+## Load the weights
+
+Install the tested NextTransformer implementation and the safetensors reader in a Python 3.10–3.13 environment.
+
+```bash
+pip install "nexttransformer @ git+https://github.com/Interactive-Active-AI/NextTransformer.git@2358425b56ce6a320fc042f67fd0a32294b5e0d7" safetensors
+```
+
+The example encodes a prompt with the checkpoint's tokenizer and prints its most likely next token.
+
+```python
+import base64
+import json
+from pathlib import Path
+
+import tiktoken
+import torch
+from huggingface_hub import snapshot_download
+from safetensors.torch import load_file
+from nexttransformer.models import build_model, config_from_dict
+
+root = Path(snapshot_download("NextTransformer/SSSL-49B-Tokens"))
+config = config_from_dict(json.loads((root / "model_config.json").read_text()))
+model = build_model(config, device="cpu")
+model.load_state_dict(load_file(root / "model.safetensors"), strict=True)
+if hasattr(model, "set_step"):
+    model.set_step(46875)
+model.eval()
+
+spec = json.loads((root / "tokenizer.tiktoken.json").read_text())
+encoding = tiktoken.Encoding(
+    name=spec["name"],
+    pat_str=spec["pat_str"],
+    mergeable_ranks={base64.b64decode(b): rank for b, rank in spec["mergeable_ranks"]},
+    special_tokens=spec["special_tokens"],
+)
+prompt = [encoding.encode_single_token("<|bos|>")] + encoding.encode_ordinary("The sky is")
+with torch.inference_mode():
+    logits = model(torch.tensor([prompt]))
+print(encoding.decode([int(logits[0, -1].argmax())]))
+```

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9dfa66cd94cf3d4de4973b4e67cc639f4390c7216e8b974114fa6bcafbe3f8a1
+size 4227884608

model_config.json

@@ -0,0 +1,44 @@
+{
+  "architecture": "dense",
+  "dense_layers": 24,
+  "pre_layers": 3,
+  "core_layers": 9,
+  "post_layers": 3,
+  "recursions": 1,
+  "activation_checkpointing": false,
+  "fourier_dim": 256,
+  "loop_modulation_bound": 0.5,
+  "loop_condition_lr_multiplier": 0.5,
+  "loop_condition_grad_clip": 1.0,
+  "vocab_size": 32768,
+  "sequence_len": 2048,
+  "width": 1536,
+  "heads": 12,
+  "mlp_width": 6144,
+  "window_pattern": "SSSL",
+  "circle_max_steps": 27,
+  "circle_target_steps": 17.5,
+  "circle_method": "sparse_gate",
+  "circle_budget_mode": "mean_survival",
+  "circle_halter": "delta_depth",
+  "circle_halter_eps": 1e-06,
+  "circle_exit_interval": 1,
+  "circle_threshold": 0.5,
+  "circle_routing_seed": 1729,
+  "circle_budget_beta": 0.9,
+  "circle_random_warmup": 256,
+  "circle_route_transition": 768,
+  "circle_depth_rho": 1.0,
+  "circle_shape_weight": 0.1,
+  "circle_dual_lr": 0.05,
+  "circle_dual_limit": 2.0,
+  "circle_halter_lr_multiplier": 0.5,
+  "circle_halter_grad_clip": 1.0,
+  "mor_target_survival": [
+    0.64,
+    0.32
+  ],
+  "mor_budget_strength": 1.0,
+  "mor_dual_lr": 0.05,
+  "mor_dual_limit": 10.0
+}