How to use from
Docker Model Runner
docker model run hf.co/gszauer/ChatGBA
Quick Links

ChatGBA

Try it in your browser: gabormakesgames.com/ChatGBA

ChatGBA is a ~4 million parameter byte-level chat model trained from scratch, small enough to run on a Game Boy Advance. The cartridge (ChatGBA.gba) holds the whole model as int8 weights and runs it in integer arithmetic on the GBA's 16.78 MHz ARM7TDMI, with an on-screen keyboard. It is the console member of the Gab models (Gab 10M, Gab 100M, Gab 1B), with Gab 10M's block design shrunk to fit.

It is a decoder-only transformer with pre-normalization (RMSNorm), short causal convolutions ("canon layers"), grouped-query attention with QK-norm and rotary position embeddings, short attention windows in most blocks, a SwiGLU MLP, and tied input/output embeddings. There is no tokenizer to learn: every UTF-8 byte is a token. The canonical model definition is reference.js (included in this repo), a dependency-free JavaScript implementation that the training code is matched against.

It speaks English and holds short conversations.

Setting Value
Parameters 4,003,264
Transformer blocks 10
Feature dimension 192
Attention heads 6 query heads, 2 key/value heads (grouped-query)
Head dimension 32
Attention windows 64 bytes in blocks 1–4 and 6–9, the whole context in blocks 5 and 10
MLP hidden dimension 512 (SwiGLU)
Canon layer kernel 4
Context length 512 bytes (sliding)
Vocabulary size 256 (UTF-8 bytes)
RoPE base 10,000 (interleaved pairs)
RMSNorm epsilon 1e-5
Precision float32; int8 on the Game Boy Advance

Usage

On a Game Boy Advance: load ChatGBA.gba in an emulator (mGBA works) or on a flash cartridge. At real Game Boy speed it writes about one letter every 2 seconds.

In the browser: gabormakesgames.com/ChatGBA runs the cartridge in an emulator, and a JavaScript version of the model at full speed.

With Transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "gszauer/ChatGBA"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

messages = [{"role": "user", "content": "Why is the sky blue?"}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
)
output = model.generate(
    inputs["input_ids"],
    do_sample=True,
    temperature=0.7,
    top_k=40,
    max_new_tokens=400,  # bytes
    stop_strings=["<|e|>"],
    tokenizer=tokenizer,
)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:]).split("<|e|>")[0])

Or chat in the terminal: python chat.py gszauer/ChatGBA.

Tokens are bytes, so max_new_tokens counts bytes. The Transformers port is a simple, faithful implementation that recomputes the last 512 bytes each generation step (no KV cache), fine for a 4M model. reference.js and the Game Boy Advance engine implement KV-cached generation.

Files

File What it is
ChatGBA.gba the Game Boy Advance cartridge: the program and the int8 weights (4.03 MiB)
chatgba-int8.bin the int8 weights the cartridge carries: every matrix as int8 with one scale per output row, norm scales and canon filters in Q16, and lookup tables for RoPE, softmax and SiLU
chatgba-float32.gab the full float32 weights in the native format: a GAB_ file (a 12-byte preamble, a JSON header with the shape and chat markers, then the weights in reference.js parameters() order), loadable via GabGPT.fromArrayBuffer
model.safetensors the same float32 weights for Transformers
model.json the shape and the weight order, as every engine reads it
reference.js the canonical implementation: tensor library, autograd, model, KV cache and trainer, in dependency-free JavaScript
source/ the cartridge's source: the integer inference engine (C, with an ARM assembly inner loop), the chat program and its build (make rebuilds ChatGBA.gba byte for byte)
modeling_gabgpt.py etc. the Transformers port (trust_remote_code)
chat.py a terminal chat through Transformers

Architecture notes

Grouped-query attention. Six query heads of 32 share two key/value heads, three query heads to each, so the cache that remembers the conversation is a third the size. Queries and keys get an RMSNorm each (QK-norm) before RoPE. RoPE uses interleaved pairs: adjacent channels (2i, 2i+1) rotate together with freq_i = 1/10000^(2i/32). There is no position embedding table, so the context can slide: past 512 bytes the oldest byte falls out and nothing else changes.

Short windows. Eight of the ten blocks look back only 64 bytes; blocks 5 and 10 see the whole context. Local blocks are cheap and their cache is small; the two global ones carry the conversation.

Canon layers. Before attention and before the MLP, each block mixes every position with the three before it, a 4-tap causal convolution per feature, added back as a residual: a = a + sum_j w_j * a[t - j]. Bytes are tiny tokens, and this lets every position see its neighbours for 768 parameters a block.

The MLP is SwiGLU. MLP(x) = (SiLU(x @ Wgate) * (x @ Wup)) @ Wdown, 192 -> 512 -> 192, no biases.

The unembedding is tied. logits = RMSNorm(h) @ E.T, with no separate output head.

On the Game Boy Advance. The GBA has no floating point, no divide instruction, 32 KB of fast RAM and 256 KB of slow RAM. The weights are read straight from cartridge ROM as int8; activations are 32-bit fixed point with 16 fractional bits; exp, sigmoid and RoPE's angles come from lookup tables; and the key/value cache is int8 in rings (64 positions for the windowed blocks, 512 for the global ones), 192 KB of the 256 KB. Quantization costs about 0.001 bits per byte, and the cartridge in an emulator gives byte-for-byte the same replies as the same integer engine compiled for a desktop. Each byte takes about 37 million cycles: 2.2 seconds.

Chat format

There is no system role. Turns are concatenated directly, with short markers:

<|u|>QUESTION<|e|><|a|>ANSWER<|e|>

A reply ends when the model writes <|e|>. The context slides: in a long conversation the model sees the latest 512 bytes, which it was trained for (fine-tuning windows start anywhere in a conversation).

Training

Pre-trained on all of Cosmopedia v2 (smollm-corpus), one pass: 147,277,920,173 bytes, 2,247,282 steps of 128 windows of 512 bytes, on one NVIDIA RTX PRO 6000 (about 94 hours). 1 document in 10,000 was held out for validation and another for test.

Then fine-tuned into a chat model on smol-smoltalk: 54,201 steps of 128 windows of 512 bytes, the windows starting every 256 bytes so that every conversation is also seen cut off at the top (about 2.3 hours), learning only from assistant replies.

Muon for the attention and MLP matrices, AdamW (betas 0.9/0.95, weight decay 0.1) for the rest, global-norm gradient clipping at 1.0. The learning rate warms up over 0.1% of the steps, stays constant, and decays linearly to zero over the last 10% (warmup-stable-decay): peak 1e-3 for pre-training and 2e-4 for fine-tuning, both chosen by pilot runs. FP32 weights with TF32 matrix products, seed 20261027.

Results, in bits per byte (lower is better; the chat test grades assistant bytes only):

Cosmopedia test smol-smoltalk test
End of pre-training 0.901
ChatGBA (this model, fine-tuned) 1.217 0.824

Fine-tuning on chat alone costs plain-text skill, as the Cosmopedia column shows.

Limitations

A 4M-parameter model. It writes fluent-looking English and follows the chat format, but it knows very little and makes things up; expect factual errors and quirks. It has no safety tuning beyond what SmolTalk's replies teach.

Game Boy Advance is a trademark of Nintendo. ChatGBA is not affiliated with or endorsed by Nintendo.

Links

Downloads last month
31
Safetensors
Model size
4M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train gszauer/ChatGBA