Text Generation
Transformers
Safetensors
English
gabgpt
gift-of-gab
small-language-model
trained-from-scratch
byte-level
looped-transformer
browser
conversational
custom_code
Instructions to use gszauer/Gab10M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use gszauer/Gab10M with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="gszauer/Gab10M", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("gszauer/Gab10M", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use gszauer/Gab10M with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "gszauer/Gab10M" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "gszauer/Gab10M", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/gszauer/Gab10M
- SGLang
How to use gszauer/Gab10M with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "gszauer/Gab10M" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "gszauer/Gab10M", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "gszauer/Gab10M" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "gszauer/Gab10M", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use gszauer/Gab10M with Docker Model Runner:
docker model run hf.co/gszauer/Gab10M
|
Download README.md from gszauer/Gab10M: direct link, hf CLI and curl.
- Browser
- Download file 6.69 kB
-
https://huggingface.co/gszauer/Gab10M/resolve/main/README.md
- Command line
-
hf download hf://gszauer/Gab10M/README.md
-
curl -L -o README.md https://huggingface.co/gszauer/Gab10M/resolve/main/README.md
6.69 kB
| license: mit | |
| language: | |
| - en | |
| pipeline_tag: text-generation | |
| library_name: transformers | |
| tags: | |
| - gift-of-gab | |
| - small-language-model | |
| - trained-from-scratch | |
| - byte-level | |
| - looped-transformer | |
| - browser | |
| datasets: | |
| - HuggingFaceTB/smollm-corpus | |
| - HuggingFaceTB/smol-smoltalk | |
| # Gab 10M | |
| Gab 10M is a ~10 million parameter byte-level language model trained from scratch. It is the smallest of the Gab models ([Gab 100M](https://huggingface.co/gszauer/Gab100M), [Gab 1B](https://huggingface.co/gszauer/Gab1B)), and the first with its own architecture: a looped transformer whose nine middle blocks run twice, so its 13 blocks of weights make 22 block passes per byte. There is no tokenizer to learn: every UTF-8 byte is a token. | |
| It is a decoder-only transformer with pre-normalization (RMSNorm), short causal convolutions ("canon layers") before attention and the MLP, QK-norm, rotary position embeddings, a SwiGLU MLP, and tied input/output embeddings. The canonical model definition is `reference.js` (included in this repo), a dependency-free JavaScript implementation that the training code is matched against. | |
| It speaks English and holds short conversations. | |
| | Setting | Value | | |
| | --- | ---: | | |
| | Parameters | 10,302,720 | | |
| | Blocks | 2 prelude + 9 core + 2 coda (13 unique) | | |
| | Loop | the 9 core blocks run twice: 22 block passes | | |
| | Feature dimension | 256 | | |
| | Attention heads | 4 | | |
| | Head dimension | 64 | | |
| | MLP hidden dimension | 680 (SwiGLU) | | |
| | Canon layer kernel | 4 | | |
| | Context length | 2,048 bytes (sliding) | | |
| | Vocabulary size | 256 (UTF-8 bytes) | | |
| | RoPE base | 10,000 (interleaved pairs) | | |
| | RMSNorm epsilon | 1e-5 | | |
| | Precision | float32 | | |
| ## Usage | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| repo = "gszauer/Gab10M" | |
| tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True) | |
| model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True) | |
| messages = [{"role": "user", "content": "Why is the sky blue?"}] | |
| inputs = tokenizer.apply_chat_template( | |
| messages, | |
| add_generation_prompt=True, | |
| return_tensors="pt", | |
| return_dict=True, | |
| ) | |
| output = model.generate( | |
| inputs["input_ids"], | |
| do_sample=True, | |
| temperature=0.7, | |
| top_k=40, | |
| max_new_tokens=400, # bytes | |
| stop_strings=["<|e|>"], | |
| tokenizer=tokenizer, | |
| ) | |
| print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:]).split("<|e|>")[0]) | |
| ``` | |
| Or chat in the terminal: `python chat.py gszauer/Gab10M`. | |
| Tokens are bytes, so `max_new_tokens` counts bytes. The Transformers port is a simple, faithful implementation that recomputes the last 2,048 bytes each generation step (no KV cache), fine for a 10M model. `reference.js` implements KV-cached generation. | |
| ## Files | |
| | File | What it is | | |
| | --- | --- | | |
| | `model.safetensors` | float32 weights for Transformers | | |
| | `gab10m.gab` | the native format: a `GAB_` file (a 12-byte preamble, a JSON header with the shape and chat markers, then the float32 weights in `reference.js` `parameters()` order), loadable via `GabGPT.fromArrayBuffer` | | |
| | `model.json` | the shape and the weight order, as every engine reads it | | |
| | `reference.js` | the canonical implementation: tensor library, autograd, model with the loop, KV cache and trainer, in dependency-free JavaScript | | |
| | `modeling_gabgpt.py` etc. | the Transformers port (`trust_remote_code`) | | |
| | `chat.py` | a terminal chat through Transformers | | |
| ## Architecture notes | |
| **The middle blocks loop.** The two prelude blocks run once, then the nine core blocks run twice with the same weights, then the two coda blocks run once. Before the second core pass, the prelude's output is added back into the residual stream, so the loop never loses sight of the input: | |
| ```text | |
| x = prelude(embed(bytes)) | |
| p = x | |
| x = core(x) | |
| x = core(x + p) | |
| logits = RMSNorm(coda(x)) @ E.T | |
| ``` | |
| That buys the depth of a 22-block model for the weights of 13. A KV cache keeps one slot per block pass (22). | |
| **Canon layers.** Before attention and before the MLP, each block mixes every position with the three before it, a 4-tap causal convolution per feature, added back as a residual: `a = a + sum_j w_j * a[t - j]`. | |
| **Attention.** Four heads of 64, with RMSNorm on queries and keys (QK-norm) before RoPE. RoPE uses interleaved pairs: adjacent channels `(2i, 2i+1)` rotate together with `freq_i = 1/10000^(2i/64)`. There is no position embedding table. | |
| **The MLP is SwiGLU.** `MLP(x) = (SiLU(x @ Wgate) * (x @ Wup)) @ Wdown`, 256 -> 680 -> 256, no biases. | |
| **The unembedding is tied.** `logits = RMSNorm(h) @ E.T`, with no separate output head. | |
| **Bytes are tokens.** The vocabulary is the 256 byte values. There are no special tokens; the chat markers are plain text the model reads and writes byte by byte. | |
| ## Chat format | |
| There is no system role. Turns are concatenated directly, with short markers: | |
| ```text | |
| <|u|>QUESTION<|e|><|a|>ANSWER<|e|> | |
| ``` | |
| A reply ends when the model writes `<|e|>`. The context slides: in a long conversation the model sees the latest 2,048 bytes, which it was trained for (fine-tuning windows start anywhere in a conversation). | |
| ## Training | |
| Pre-trained on all of Cosmopedia v2 ([smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus)), one pass: 147,277,920,173 bytes, 1,123,641 steps of 64 windows of 2,048 bytes, on one NVIDIA RTX PRO 6000 (about 220 hours). 1 document in 10,000 was held out for validation and another for test. | |
| Then fine-tuned into a chat model on [smol-smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk): 1,776,061,826 bytes, 27,100 steps of 32 windows of 2,048 bytes (about 3 hours), learning only from assistant replies. | |
| Muon for the attention and MLP matrices, AdamW (betas 0.9/0.95, weight decay 0.1) for the rest, global-norm gradient clipping at 1.0. The learning rate warms up over 0.1% of the steps, stays constant, and decays linearly to zero over the last 10% (warmup-stable-decay): peak 2e-3 for pre-training and 4e-4 for fine-tuning, both chosen by pilot runs. FP32 weights with TF32 matrix products, seed 20261027. | |
| Results, in bits per byte (lower is better; the chat test grades assistant bytes only): | |
| | | Cosmopedia test | smol-smoltalk test | | |
| | --- | ---: | ---: | | |
| | End of pre-training | 0.687 | | | |
| | Gab 10M (this model, fine-tuned) | 1.000 | 0.593 | | |
| Fine-tuning on chat alone costs plain-text skill, as the Cosmopedia column shows. | |
| ## Limitations | |
| A 10M-parameter model. It writes fluent-looking English and follows the chat format, but it knows very little and makes things up; expect factual errors and quirks. It has no safety tuning beyond what SmolTalk's replies teach. | |
| ## Links | |
| - Blog/site: [gabormakesgames.com](https://gabormakesgames.com) | |