How to use from the
Use from the
Transformers library
# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="gszauer/Gab10M", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("gszauer/Gab10M", trust_remote_code=True, device_map="auto")
Quick Links

Gab 10M

Gab 10M is a ~10 million parameter byte-level language model trained from scratch. It is the smallest of the Gab models (Gab 100M, Gab 1B), and the first with its own architecture: a looped transformer whose nine middle blocks run twice, so its 13 blocks of weights make 22 block passes per byte. There is no tokenizer to learn: every UTF-8 byte is a token.

It is a decoder-only transformer with pre-normalization (RMSNorm), short causal convolutions ("canon layers") before attention and the MLP, QK-norm, rotary position embeddings, a SwiGLU MLP, and tied input/output embeddings. The canonical model definition is reference.js (included in this repo), a dependency-free JavaScript implementation that the training code is matched against.

It speaks English and holds short conversations.

Setting Value
Parameters 10,302,720
Blocks 2 prelude + 9 core + 2 coda (13 unique)
Loop the 9 core blocks run twice: 22 block passes
Feature dimension 256
Attention heads 4
Head dimension 64
MLP hidden dimension 680 (SwiGLU)
Canon layer kernel 4
Context length 2,048 bytes (sliding)
Vocabulary size 256 (UTF-8 bytes)
RoPE base 10,000 (interleaved pairs)
RMSNorm epsilon 1e-5
Precision float32

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "gszauer/Gab10M"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

messages = [{"role": "user", "content": "Why is the sky blue?"}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
)
output = model.generate(
    inputs["input_ids"],
    do_sample=True,
    temperature=0.7,
    top_k=40,
    max_new_tokens=400,  # bytes
    stop_strings=["<|e|>"],
    tokenizer=tokenizer,
)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:]).split("<|e|>")[0])

Or chat in the terminal: python chat.py gszauer/Gab10M.

Tokens are bytes, so max_new_tokens counts bytes. The Transformers port is a simple, faithful implementation that recomputes the last 2,048 bytes each generation step (no KV cache), fine for a 10M model. reference.js implements KV-cached generation.

Files

File What it is
model.safetensors float32 weights for Transformers
gab10m.gab the native format: a GAB_ file (a 12-byte preamble, a JSON header with the shape and chat markers, then the float32 weights in reference.js parameters() order), loadable via GabGPT.fromArrayBuffer
model.json the shape and the weight order, as every engine reads it
reference.js the canonical implementation: tensor library, autograd, model with the loop, KV cache and trainer, in dependency-free JavaScript
modeling_gabgpt.py etc. the Transformers port (trust_remote_code)
chat.py a terminal chat through Transformers

Architecture notes

The middle blocks loop. The two prelude blocks run once, then the nine core blocks run twice with the same weights, then the two coda blocks run once. Before the second core pass, the prelude's output is added back into the residual stream, so the loop never loses sight of the input:

x = prelude(embed(bytes))
p = x
x = core(x)
x = core(x + p)
logits = RMSNorm(coda(x)) @ E.T

That buys the depth of a 22-block model for the weights of 13. A KV cache keeps one slot per block pass (22).

Canon layers. Before attention and before the MLP, each block mixes every position with the three before it, a 4-tap causal convolution per feature, added back as a residual: a = a + sum_j w_j * a[t - j].

Attention. Four heads of 64, with RMSNorm on queries and keys (QK-norm) before RoPE. RoPE uses interleaved pairs: adjacent channels (2i, 2i+1) rotate together with freq_i = 1/10000^(2i/64). There is no position embedding table.

The MLP is SwiGLU. MLP(x) = (SiLU(x @ Wgate) * (x @ Wup)) @ Wdown, 256 -> 680 -> 256, no biases.

The unembedding is tied. logits = RMSNorm(h) @ E.T, with no separate output head.

Bytes are tokens. The vocabulary is the 256 byte values. There are no special tokens; the chat markers are plain text the model reads and writes byte by byte.

Chat format

There is no system role. Turns are concatenated directly, with short markers:

<|u|>QUESTION<|e|><|a|>ANSWER<|e|>

A reply ends when the model writes <|e|>. The context slides: in a long conversation the model sees the latest 2,048 bytes, which it was trained for (fine-tuning windows start anywhere in a conversation).

Training

Pre-trained on all of Cosmopedia v2 (smollm-corpus), one pass: 147,277,920,173 bytes, 1,123,641 steps of 64 windows of 2,048 bytes, on one NVIDIA RTX PRO 6000 (about 220 hours). 1 document in 10,000 was held out for validation and another for test.

Then fine-tuned into a chat model on smol-smoltalk: 1,776,061,826 bytes, 27,100 steps of 32 windows of 2,048 bytes (about 3 hours), learning only from assistant replies.

Muon for the attention and MLP matrices, AdamW (betas 0.9/0.95, weight decay 0.1) for the rest, global-norm gradient clipping at 1.0. The learning rate warms up over 0.1% of the steps, stays constant, and decays linearly to zero over the last 10% (warmup-stable-decay): peak 2e-3 for pre-training and 4e-4 for fine-tuning, both chosen by pilot runs. FP32 weights with TF32 matrix products, seed 20261027.

Results, in bits per byte (lower is better; the chat test grades assistant bytes only):

Cosmopedia test smol-smoltalk test
End of pre-training 0.687
Gab 10M (this model, fine-tuned) 1.000 0.593

Fine-tuning on chat alone costs plain-text skill, as the Cosmopedia column shows.

Limitations

A 10M-parameter model. It writes fluent-looking English and follows the chat format, but it knows very little and makes things up; expect factual errors and quirks. It has no safety tuning beyond what SmolTalk's replies teach.

Links

Downloads last month
22
Safetensors
Model size
10.3M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train gszauer/Gab10M