| --- |
| library_name: pytorch |
| tags: |
| - nexttransformer |
| - residual |
| - causal-lm |
| - pretrained |
| - safetensors |
| pipeline_tag: text-generation |
| --- |
| |
| # NextTransformer HC |
|
|
| Four residual streams; every sublayer reads from and writes back to them through learned connections. |
|
|
| This is the final HC checkpoint in the NextTransformer 49.152B-token architecture comparison. It has 1,356,235,700 parameters and a 2,048-token context. The model was pretrained on ClimbMix for 46,875 updates. The 49B in the repository name refers to training tokens. |
|
|
| | Final evaluation | Result | |
| |---|---:| |
| | Validation BPB | 0.6644 | |
| | CORE-22 (×100) | 35.09 | |
|
|
| [Architecture](https://github.com/Interactive-Active-AI/NextTransformer/blob/2358425b56ce6a320fc042f67fd0a32294b5e0d7/results/methods/hc.json) · [Run record](https://github.com/Interactive-Active-AI/NextTransformer/blob/2358425b56ce6a320fc042f67fd0a32294b5e0d7/results/runs/hc.json) |
|
|
| ## Load the weights |
|
|
| Install the tested NextTransformer implementation and the safetensors reader in a Python 3.10–3.13 environment. |
|
|
| ```bash |
| pip install "nexttransformer @ git+https://github.com/Interactive-Active-AI/NextTransformer.git@2358425b56ce6a320fc042f67fd0a32294b5e0d7" safetensors |
| ``` |
|
|
| The example encodes a prompt with the checkpoint's tokenizer and prints its most likely next token. |
|
|
| ```python |
| import base64 |
| import json |
| from pathlib import Path |
| |
| import tiktoken |
| import torch |
| from huggingface_hub import snapshot_download |
| from safetensors.torch import load_file |
| from nexttransformer.models import build_model, config_from_dict |
| |
| root = Path(snapshot_download("NextTransformer/HC-49B-Tokens")) |
| config = config_from_dict(json.loads((root / "model_config.json").read_text())) |
| model = build_model(config, device="cpu") |
| model.load_state_dict(load_file(root / "model.safetensors"), strict=True) |
| if hasattr(model, "set_step"): |
| model.set_step(46875) |
| model.eval() |
| |
| spec = json.loads((root / "tokenizer.tiktoken.json").read_text()) |
| encoding = tiktoken.Encoding( |
| name=spec["name"], |
| pat_str=spec["pat_str"], |
| mergeable_ranks={base64.b64decode(b): rank for b, rank in spec["mergeable_ranks"]}, |
| special_tokens=spec["special_tokens"], |
| ) |
| prompt = [encoding.encode_single_token("<|bos|>")] + encoding.encode_ordinary("The sky is") |
| with torch.inference_mode(): |
| logits = model(torch.tensor([prompt])) |
| print(encoding.decode([int(logits[0, -1].argmax())])) |
| ``` |
|
|
