File size: 2,348 Bytes
7262f08 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | ---
library_name: pytorch
tags:
- nexttransformer
- others
- causal-lm
- pretrained
- safetensors
pipeline_tag: text-generation
---
# NextTransformer SSSL
Short, short, short, long. The historical reference.
This is the final SSSL checkpoint in the NextTransformer 49.152B-token architecture comparison. It has 1,384,122,122 parameters and a 2,048-token context. The model was pretrained on ClimbMix for 46,875 updates. The 49B in the repository name refers to training tokens.
| Final evaluation | Result |
|---|---:|
| Validation BPB | 0.6671 |
| CORE-22 (×100) | 33.78 |
[Architecture](https://github.com/Interactive-Active-AI/NextTransformer/blob/2358425b56ce6a320fc042f67fd0a32294b5e0d7/results/methods/sssl.json) · [Run record](https://github.com/Interactive-Active-AI/NextTransformer/blob/2358425b56ce6a320fc042f67fd0a32294b5e0d7/results/runs/sssl.json)
## Load the weights
Install the tested NextTransformer implementation and the safetensors reader in a Python 3.10–3.13 environment.
```bash
pip install "nexttransformer @ git+https://github.com/Interactive-Active-AI/NextTransformer.git@2358425b56ce6a320fc042f67fd0a32294b5e0d7" safetensors
```
The example encodes a prompt with the checkpoint's tokenizer and prints its most likely next token.
```python
import base64
import json
from pathlib import Path
import tiktoken
import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from nexttransformer.models import build_model, config_from_dict
root = Path(snapshot_download("NextTransformer/SSSL-49B-Tokens"))
config = config_from_dict(json.loads((root / "model_config.json").read_text()))
model = build_model(config, device="cpu")
model.load_state_dict(load_file(root / "model.safetensors"), strict=True)
if hasattr(model, "set_step"):
model.set_step(46875)
model.eval()
spec = json.loads((root / "tokenizer.tiktoken.json").read_text())
encoding = tiktoken.Encoding(
name=spec["name"],
pat_str=spec["pat_str"],
mergeable_ranks={base64.b64decode(b): rank for b, rank in spec["mergeable_ranks"]},
special_tokens=spec["special_tokens"],
)
prompt = [encoding.encode_single_token("<|bos|>")] + encoding.encode_ordinary("The sky is")
with torch.inference_mode():
logits = model(torch.tensor([prompt]))
print(encoding.decode([int(logits[0, -1].argmax())]))
```
|