--- license: mit language: - en pipeline_tag: text-generation library_name: transformers tags: - gift-of-gab - small-language-model - trained-from-scratch - byte-level - game-boy-advance - browser datasets: - HuggingFaceTB/smollm-corpus - HuggingFaceTB/smol-smoltalk --- # ChatGBA > Try it in your browser: [gabormakesgames.com/ChatGBA](https://gabormakesgames.com/ChatGBA/) ChatGBA is a ~4 million parameter byte-level chat model trained from scratch, small enough to run on a Game Boy Advance. The cartridge (`ChatGBA.gba`) holds the whole model as int8 weights and runs it in integer arithmetic on the GBA's 16.78 MHz ARM7TDMI, with an on-screen keyboard. It is the console member of the Gab models ([Gab 10M](https://huggingface.co/gszauer/Gab10M), [Gab 100M](https://huggingface.co/gszauer/Gab100M), [Gab 1B](https://huggingface.co/gszauer/Gab1B)), with Gab 10M's block design shrunk to fit. It is a decoder-only transformer with pre-normalization (RMSNorm), short causal convolutions ("canon layers"), grouped-query attention with QK-norm and rotary position embeddings, short attention windows in most blocks, a SwiGLU MLP, and tied input/output embeddings. There is no tokenizer to learn: every UTF-8 byte is a token. The canonical model definition is `reference.js` (included in this repo), a dependency-free JavaScript implementation that the training code is matched against. It speaks English and holds short conversations. | Setting | Value | | --- | ---: | | Parameters | 4,003,264 | | Transformer blocks | 10 | | Feature dimension | 192 | | Attention heads | 6 query heads, 2 key/value heads (grouped-query) | | Head dimension | 32 | | Attention windows | 64 bytes in blocks 1–4 and 6–9, the whole context in blocks 5 and 10 | | MLP hidden dimension | 512 (SwiGLU) | | Canon layer kernel | 4 | | Context length | 512 bytes (sliding) | | Vocabulary size | 256 (UTF-8 bytes) | | RoPE base | 10,000 (interleaved pairs) | | RMSNorm epsilon | 1e-5 | | Precision | float32; int8 on the Game Boy Advance | ## Usage On a Game Boy Advance: load `ChatGBA.gba` in an emulator (mGBA works) or on a flash cartridge. At real Game Boy speed it writes about one letter every 2 seconds. In the browser: [gabormakesgames.com/ChatGBA](https://gabormakesgames.com/ChatGBA/) runs the cartridge in an emulator, and a JavaScript version of the model at full speed. With Transformers: ```python from transformers import AutoModelForCausalLM, AutoTokenizer repo = "gszauer/ChatGBA" tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True) messages = [{"role": "user", "content": "Why is the sky blue?"}] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, return_tensors="pt", return_dict=True, ) output = model.generate( inputs["input_ids"], do_sample=True, temperature=0.7, top_k=40, max_new_tokens=400, # bytes stop_strings=["<|e|>"], tokenizer=tokenizer, ) print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:]).split("<|e|>")[0]) ``` Or chat in the terminal: `python chat.py gszauer/ChatGBA`. Tokens are bytes, so `max_new_tokens` counts bytes. The Transformers port is a simple, faithful implementation that recomputes the last 512 bytes each generation step (no KV cache), fine for a 4M model. `reference.js` and the Game Boy Advance engine implement KV-cached generation. ## Files | File | What it is | | --- | --- | | `ChatGBA.gba` | the Game Boy Advance cartridge: the program and the int8 weights (4.03 MiB) | | `chatgba-int8.bin` | the int8 weights the cartridge carries: every matrix as int8 with one scale per output row, norm scales and canon filters in Q16, and lookup tables for RoPE, softmax and SiLU | | `chatgba-float32.gab` | the full float32 weights in the native format: a `GAB_` file (a 12-byte preamble, a JSON header with the shape and chat markers, then the weights in `reference.js` `parameters()` order), loadable via `GabGPT.fromArrayBuffer` | | `model.safetensors` | the same float32 weights for Transformers | | `model.json` | the shape and the weight order, as every engine reads it | | `reference.js` | the canonical implementation: tensor library, autograd, model, KV cache and trainer, in dependency-free JavaScript | | `source/` | the cartridge's source: the integer inference engine (C, with an ARM assembly inner loop), the chat program and its build (`make` rebuilds `ChatGBA.gba` byte for byte) | | `modeling_gabgpt.py` etc. | the Transformers port (`trust_remote_code`) | | `chat.py` | a terminal chat through Transformers | ## Architecture notes **Grouped-query attention.** Six query heads of 32 share two key/value heads, three query heads to each, so the cache that remembers the conversation is a third the size. Queries and keys get an RMSNorm each (QK-norm) before RoPE. RoPE uses interleaved pairs: adjacent channels `(2i, 2i+1)` rotate together with `freq_i = 1/10000^(2i/32)`. There is no position embedding table, so the context can slide: past 512 bytes the oldest byte falls out and nothing else changes. **Short windows.** Eight of the ten blocks look back only 64 bytes; blocks 5 and 10 see the whole context. Local blocks are cheap and their cache is small; the two global ones carry the conversation. **Canon layers.** Before attention and before the MLP, each block mixes every position with the three before it, a 4-tap causal convolution per feature, added back as a residual: `a = a + sum_j w_j * a[t - j]`. Bytes are tiny tokens, and this lets every position see its neighbours for 768 parameters a block. **The MLP is SwiGLU.** `MLP(x) = (SiLU(x @ Wgate) * (x @ Wup)) @ Wdown`, 192 -> 512 -> 192, no biases. **The unembedding is tied.** `logits = RMSNorm(h) @ E.T`, with no separate output head. **On the Game Boy Advance.** The GBA has no floating point, no divide instruction, 32 KB of fast RAM and 256 KB of slow RAM. The weights are read straight from cartridge ROM as int8; activations are 32-bit fixed point with 16 fractional bits; exp, sigmoid and RoPE's angles come from lookup tables; and the key/value cache is int8 in rings (64 positions for the windowed blocks, 512 for the global ones), 192 KB of the 256 KB. Quantization costs about 0.001 bits per byte, and the cartridge in an emulator gives byte-for-byte the same replies as the same integer engine compiled for a desktop. Each byte takes about 37 million cycles: 2.2 seconds. ## Chat format There is no system role. Turns are concatenated directly, with short markers: ```text <|u|>QUESTION<|e|><|a|>ANSWER<|e|> ``` A reply ends when the model writes `<|e|>`. The context slides: in a long conversation the model sees the latest 512 bytes, which it was trained for (fine-tuning windows start anywhere in a conversation). ## Training Pre-trained on all of Cosmopedia v2 ([smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus)), one pass: 147,277,920,173 bytes, 2,247,282 steps of 128 windows of 512 bytes, on one NVIDIA RTX PRO 6000 (about 94 hours). 1 document in 10,000 was held out for validation and another for test. Then fine-tuned into a chat model on [smol-smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk): 54,201 steps of 128 windows of 512 bytes, the windows starting every 256 bytes so that every conversation is also seen cut off at the top (about 2.3 hours), learning only from assistant replies. Muon for the attention and MLP matrices, AdamW (betas 0.9/0.95, weight decay 0.1) for the rest, global-norm gradient clipping at 1.0. The learning rate warms up over 0.1% of the steps, stays constant, and decays linearly to zero over the last 10% (warmup-stable-decay): peak 1e-3 for pre-training and 2e-4 for fine-tuning, both chosen by pilot runs. FP32 weights with TF32 matrix products, seed 20261027. Results, in bits per byte (lower is better; the chat test grades assistant bytes only): | | Cosmopedia test | smol-smoltalk test | | --- | ---: | ---: | | End of pre-training | 0.901 | | | ChatGBA (this model, fine-tuned) | 1.217 | 0.824 | Fine-tuning on chat alone costs plain-text skill, as the Cosmopedia column shows. ## Limitations A 4M-parameter model. It writes fluent-looking English and follows the chat format, but it knows very little and makes things up; expect factual errors and quirks. It has no safety tuning beyond what SmolTalk's replies teach. Game Boy Advance is a trademark of Nintendo. ChatGBA is not affiliated with or endorsed by Nintendo. ## Links - Try it out: [gabormakesgames.com/ChatGBA](https://gabormakesgames.com/ChatGBA/) - Blog/site: [gabormakesgames.com](https://gabormakesgames.com)