Instructions to use gszauer/Gab1B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use gszauer/Gab1B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="gszauer/Gab1B", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("gszauer/Gab1B", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use gszauer/Gab1B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "gszauer/Gab1B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gszauer/Gab1B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/gszauer/Gab1B

SGLang

How to use gszauer/Gab1B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "gszauer/Gab1B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gszauer/Gab1B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "gszauer/Gab1B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gszauer/Gab1B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use gszauer/Gab1B with Docker Model Runner:
```
docker model run hf.co/gszauer/Gab1B
```

Gab1B / README.md

gszauer

Gab 1B: weights, Transformers port, model card

c3e462e verified 6 days ago

preview code

Raw History Blame Contribute Delete

7.24 kB

---
license: mit
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- gift-of-gab
- small-language-model
- trained-from-scratch
- mixture-of-experts
- browser
datasets:
- gszauer/Gab100MPretrain
- HuggingFaceTB/smollm-corpus
- gszauer/Gab100MFinetune
- HuggingFaceTB/smoltalk
---
# Gab 1B

Gab 1B is a ~1 billion parameter mixture-of-experts language model trained from scratch on a Mac Studio. It is the [Gab 100M](https://huggingface.co/gszauer/Gab100M) architecture with every block's feed-forward network replaced by 32 experts, of which each token uses 2. It has ten times the parameters, but each token costs about the same compute as Gab 100M: ~108M parameters are active per token.

It is a decoder-only transformer with pre-normalization (RMSNorm), rotary position embeddings, causal self-attention, tied input/output embeddings, and a top-2 mixture of exact-GeLU experts in every block. The canonical model definition is `MiniGPT.js` (included in this repo). It is a dependency-free JavaScript implementation, and the whole training suite is matched against it.

It speaks English, holds multi-turn conversations, and supports an optional `<think>` reasoning trace inside assistant turns.

| Setting | Value |
| --- | ---: |
| Parameters | 1,028,477,184 |
| Active parameters per token | 107,752,704 |
| Transformer blocks | 13 |
| Feature dimension | 768 |
| Attention heads | 12 |
| Head dimension | 64 |
| Experts per block | 32 (top-2 routing) |
| Expert hidden dimension | 1536 (with a per-neuron bias) |
| Context length | 2048 |
| Vocabulary size | 20,000 |
| RoPE theta | 10,000 (interleaved pairs) |
| RMSNorm epsilon | 1e-5 |
| Precision | float32 weights (trained in bfloat16 mixed precision) |

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "gszauer/Gab1B"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
messages = [{"role": "user", "content": "Hello, how are you?"}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    # enable_thinking=True,  # seed the reply with a <think> trace
    return_tensors="pt",
    return_dict=True,
)
output = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

The Transformers port is a simple, faithful implementation that recomputes the full sequence each generation step (no KV cache), and runs the experts one at a time. Its float32 logits match the training suite's to within rounding. The browser runtime (`MiniGPT.js`) and the training suite's `inference.py` both implement KV-cached generation.

## Files

| File | What it is |
| --- | --- |
| `model.safetensors` | float32 weights with HF-style tensor names (per-head attention matrices fused, see below) |
| `gab1b.weights` | the native format: a headerless flat float32 buffer in `MiniGPT.js` `parameters()` order, loadable via `deserializeFromArrayBuffer` |
| `vocab.json` | the tokenizer: `{reserved, merges}` |
| `MiniGPT.js` | the canonical implementation: tensor library, autograd, tokenizer, model with mixture of experts, KV cache and AdamW trainer, in dependency-free JavaScript |
| `modeling_gift_of_gab.py` etc. | the Transformers port (`trust_remote_code`) |
| `convert.py` | converts `gab1b.weights` to `model.safetensors` |

## Architecture notes

**Each block's MLP is a mixture of experts.** A router, a bias-free `[768, 32]` linear layer, scores every expert for each token. The token goes to its two most probable experts, and their outputs are summed, weighted by the router probabilities renormalized over the two choices:

```text
p = softmax(x @ Wrouter)
MoE(x) = sum over the top-2 experts e of (p_e / (p_a + p_b)) * Expert_e(x)
```

Each expert is a Gab 100M-style MLP at half width: `Expert(x) = GeLU(x @ Wup + b) @ Wdown` with exact (erf) GeLU, `768 -> 1536 -> 768`. Two half-width experts cost the same as Gab 100M's one `768 -> 3072 -> 768` MLP. Inference is dropless: every token always reaches both of its experts.

**Attention is trained per head.** Each of the 12 heads owns its own `Wq`, `Wk` and `Wv` of shape `[768, 64]`, *and its own output projection* `Wo` of shape `[64, 768]`. Heads are never concatenated. Each head projects its result back up to 768 on its own, and the per-head results are summed:

```text
Attention(x) = sum_i( softmax(causal(q_i @ k_i.T / sqrt(64))) @ v_i @ Wo_i )
```

This is algebraically identical to a concat followed by one big `[768, 768]` output projection, so `model.safetensors` ships fused `[768, 768]` tensors and the port runs the standard formulation. The unfused per-head matrices are preserved in `gab1b.weights`.

**RoPE uses interleaved pairs.** Adjacent channels `(2i, 2i+1)` rotate together with `freq_i = 1/10000^(2i/64)`, unlike Llama-style split-half RoPE. There is no position embedding table, so position enters only through RoPE.

**The unembedding is tied.** `logits = RMSNorm(h) @ E.T`, with no separate output head.

**The tokenizer is byte-level BPE with sequential merge application.** Ids 0–255 are raw bytes, and every other id is a merge of two earlier ids, learned in order. Text is first split into pre-tokens (words with one leading space, digit groups of up to 3, and whitespace and punctuation runs of up to 4), and encoding applies every merge rule in sequence over each pre-token's UTF-8 bytes. Reserved tokens (`<|user|>`, `<|assistant|>`, `<|end|>`, `<|endoftext|>`, `<think>`, `</think>`, `<|pad|>`) are chained merges taught before training, so each encodes to a single id and is atomic.

## Chat format

There is no system role. Turns are concatenated directly:

```text
<|user|>QUESTION<|end|><|assistant|>ANSWER<|end|>
```

Thinking is optional and lives inside the assistant turn:

```text
<|user|>QUESTION<|end|><|assistant|><think>PRIVATE_REASONING</think>ANSWER<|end|>
```

The chat template supports `enable_thinking=True` to seed the reply with `<think>`.

## Training

Pre-trained on 31,047,744,657 tokens ([Gab100MPretrain](https://huggingface.co/datasets/gszauer/Gab100MPretrain) plus [Cosmopedia v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus)): 947,501 steps of 16 sequences in 514 hours. The context grew from 512 to 1024 to 2048 tokens, at 60% and 85% of training. The final validation loss is 1.270, measured on a held-out 0.02% of the corpus.

Then fine-tuned into a chat model on 941,343,647 tokens ([Gab100MFinetune](https://huggingface.co/datasets/gszauer/Gab100MFinetune) plus [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk)), learning only from assistant replies.

AdamW with linear warmup and cosine decay: peak learning rate 3e-4 for pre-training and 3e-5 for fine-tuning, betas 0.9/0.999, weight decay 0.01 (decoupled), and global-norm gradient clipping at 1.0. Trained with MLX on a Mac Studio (M3 Ultra). Training routing uses a fixed expert capacity (capacity factor 1.25), with a load-balancing loss (coefficient 0.01) and a router z-loss (coefficient 0.001).

## Links

- Blog/site: [gabormakesgames.com](https://gabormakesgames.com)

	---
	license: mit
	language:
	- en
	pipeline_tag: text-generation
	library_name: transformers
	tags:
	- gift-of-gab
	- small-language-model
	- trained-from-scratch
	- mixture-of-experts
	- browser
	datasets:
	- gszauer/Gab100MPretrain
	- HuggingFaceTB/smollm-corpus
	- gszauer/Gab100MFinetune
	- HuggingFaceTB/smoltalk
	---

	# Gab 1B

	Gab 1B is a ~1 billion parameter mixture-of-experts language model trained from scratch on a Mac Studio. It is the [Gab 100M](https://huggingface.co/gszauer/Gab100M) architecture with every block's feed-forward network replaced by 32 experts, of which each token uses 2. It has ten times the parameters, but each token costs about the same compute as Gab 100M: ~108M parameters are active per token.

	It is a decoder-only transformer with pre-normalization (RMSNorm), rotary position embeddings, causal self-attention, tied input/output embeddings, and a top-2 mixture of exact-GeLU experts in every block. The canonical model definition is `MiniGPT.js` (included in this repo). It is a dependency-free JavaScript implementation, and the whole training suite is matched against it.

	It speaks English, holds multi-turn conversations, and supports an optional `<think>` reasoning trace inside assistant turns.

	\| Setting \| Value \|
	\| --- \| ---: \|
	\| Parameters \| 1,028,477,184 \|
	\| Active parameters per token \| 107,752,704 \|
	\| Transformer blocks \| 13 \|
	\| Feature dimension \| 768 \|
	\| Attention heads \| 12 \|
	\| Head dimension \| 64 \|
	\| Experts per block \| 32 (top-2 routing) \|
	\| Expert hidden dimension \| 1536 (with a per-neuron bias) \|
	\| Context length \| 2048 \|
	\| Vocabulary size \| 20,000 \|
	\| RoPE theta \| 10,000 (interleaved pairs) \|
	\| RMSNorm epsilon \| 1e-5 \|
	\| Precision \| float32 weights (trained in bfloat16 mixed precision) \|

	## Usage

	```python
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer

	repo = "gszauer/Gab1B"
	tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
	model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

	messages = [{"role": "user", "content": "Hello, how are you?"}]
	inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	# enable_thinking=True, # seed the reply with a <think> trace
	return_tensors="pt",
	return_dict=True,
	)
	output = model.generate(**inputs, max_new_tokens=128)
	print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
	```

	The Transformers port is a simple, faithful implementation that recomputes the full sequence each generation step (no KV cache), and runs the experts one at a time. Its float32 logits match the training suite's to within rounding. The browser runtime (`MiniGPT.js`) and the training suite's `inference.py` both implement KV-cached generation.

	## Files

	\| File \| What it is \|
	\| --- \| --- \|
	\| `model.safetensors` \| float32 weights with HF-style tensor names (per-head attention matrices fused, see below) \|
	\| `gab1b.weights` \| the native format: a headerless flat float32 buffer in `MiniGPT.js` `parameters()` order, loadable via `deserializeFromArrayBuffer` \|
	\| `vocab.json` \| the tokenizer: `{reserved, merges}` \|
	\| `MiniGPT.js` \| the canonical implementation: tensor library, autograd, tokenizer, model with mixture of experts, KV cache and AdamW trainer, in dependency-free JavaScript \|
	\| `modeling_gift_of_gab.py` etc. \| the Transformers port (`trust_remote_code`) \|
	\| `convert.py` \| converts `gab1b.weights` to `model.safetensors` \|

	## Architecture notes

	Each block's MLP is a mixture of experts. A router, a bias-free `[768, 32]` linear layer, scores every expert for each token. The token goes to its two most probable experts, and their outputs are summed, weighted by the router probabilities renormalized over the two choices:

	```text
	p = softmax(x @ Wrouter)
	MoE(x) = sum over the top-2 experts e of (p_e / (p_a + p_b)) * Expert_e(x)
	```

	Each expert is a Gab 100M-style MLP at half width: `Expert(x) = GeLU(x @ Wup + b) @ Wdown` with exact (erf) GeLU, `768 -> 1536 -> 768`. Two half-width experts cost the same as Gab 100M's one `768 -> 3072 -> 768` MLP. Inference is dropless: every token always reaches both of its experts.

	Attention is trained per head. Each of the 12 heads owns its own `Wq`, `Wk` and `Wv` of shape `[768, 64]`, and its own output projection `Wo` of shape `[64, 768]`. Heads are never concatenated. Each head projects its result back up to 768 on its own, and the per-head results are summed:

	```text
	Attention(x) = sum_i( softmax(causal(q_i @ k_i.T / sqrt(64))) @ v_i @ Wo_i )
	```

	This is algebraically identical to a concat followed by one big `[768, 768]` output projection, so `model.safetensors` ships fused `[768, 768]` tensors and the port runs the standard formulation. The unfused per-head matrices are preserved in `gab1b.weights`.

	RoPE uses interleaved pairs. Adjacent channels `(2i, 2i+1)` rotate together with `freq_i = 1/10000^(2i/64)`, unlike Llama-style split-half RoPE. There is no position embedding table, so position enters only through RoPE.

	The unembedding is tied. `logits = RMSNorm(h) @ E.T`, with no separate output head.

	The tokenizer is byte-level BPE with sequential merge application. Ids 0–255 are raw bytes, and every other id is a merge of two earlier ids, learned in order. Text is first split into pre-tokens (words with one leading space, digit groups of up to 3, and whitespace and punctuation runs of up to 4), and encoding applies every merge rule in sequence over each pre-token's UTF-8 bytes. Reserved tokens (`<\|user\|>`, `<\|assistant\|>`, `<\|end\|>`, `<\|endoftext\|>`, `<think>`, `</think>`, `<\|pad\|>`) are chained merges taught before training, so each encodes to a single id and is atomic.

	## Chat format

	There is no system role. Turns are concatenated directly:

	```text
	<\|user\|>QUESTION<\|end\|><\|assistant\|>ANSWER<\|end\|>
	```

	Thinking is optional and lives inside the assistant turn:

	```text
	<\|user\|>QUESTION<\|end\|><\|assistant\|><think>PRIVATE_REASONING</think>ANSWER<\|end\|>
	```

	The chat template supports `enable_thinking=True` to seed the reply with `<think>`.

	## Training

	Pre-trained on 31,047,744,657 tokens ([Gab100MPretrain](https://huggingface.co/datasets/gszauer/Gab100MPretrain) plus [Cosmopedia v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus)): 947,501 steps of 16 sequences in 514 hours. The context grew from 512 to 1024 to 2048 tokens, at 60% and 85% of training. The final validation loss is 1.270, measured on a held-out 0.02% of the corpus.

	Then fine-tuned into a chat model on 941,343,647 tokens ([Gab100MFinetune](https://huggingface.co/datasets/gszauer/Gab100MFinetune) plus [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk)), learning only from assistant replies.

	AdamW with linear warmup and cosine decay: peak learning rate 3e-4 for pre-training and 3e-5 for fine-tuning, betas 0.9/0.999, weight decay 0.01 (decoupled), and global-norm gradient clipping at 1.0. Trained with MLX on a Mac Studio (M3 Ultra). Training routing uses a fixed expert capacity (capacity factor 1.25), with a load-balancing loss (coefficient 0.01) and a router z-loss (coefficient 0.001).

	## Links

	- Blog/site: [gabormakesgames.com](https://gabormakesgames.com)