MiniGrimm / README.md
Bayhakl1's picture
Update README.md
e2b91ed verified
|
Raw History Blame Contribute Delete
5.95 kB
---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- reasoning
- muon
datasets:
- openai/gsm8k
- HuggingFaceTB/smoltalk
- HuggingFaceH4/ultrachat_200k
- allenai/tulu-3-sft-personas-instruction-following
- HuggingFaceH4/no_robots
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/smollm-corpus
- wikimedia/wikipedia
- Open-Orca/OpenOrca
- TIGER-Lab/WebInstructSub
- nvidia/OpenMathInstruct-2
- kaist-ai/CoT-Collection
- microsoft/orca-math-word-problems-200k
- allenai/WildChat-1M
- Magpie-Align/Llama-3-Magpie-Pro-1M-v0.1
- openbmb/UltraInteract_sft
- WizardLMTeam/WizardLM_evol_instruct_V2_196k
---
# MiniGrimm (48M SFT)
**MiniGrimm** is a tiny **48.29M-parameter** (42.00M non-embedding) decoder-only Transformer trained from scratch and fine-tuned with Supervised Fine-Tuning (SFT).
This is a small-scale experimental model built to test a deep-and-thin Llama-style architecture trained with the **Muon** optimizer. Because it is under 50M parameters and has a 512-token context window, **keep your expectations realistic**—it is not a general-purpose assistant and will hallucinate facts or stumble on complex logic. That said, it can hold short casual chats and do basic step-by-step arithmetic surprisingly well for its size.
---
## Benchmarks
Evaluated after SFT on **ArithMark 3.0** (arithmetic reasoning):
| Benchmark | Metric | Score |
| :--- | :--- | :---: |
| **ArithMark 3.0** | Raw Accuracy | **38.1%** |
| **ArithMark 3.0** | Normalized Accuracy | **38.2%** |
*(Not bad for a 42M non-embedding model, though it will still make plenty of mistakes on multi-step word problems or larger numbers.)*
---
Other Benchmarks:
| Benchmark | Accuracy (Raw) | Accuracy (Norm) | Notes |
| :--- | :---: | :---: | :--- |
| **SciQ** | 70.7% (±1.44) | 63.0% (±1.53) | Science QA |
| **PIQA** | 61.7% (±1.13) | 62.0% (±1.13) | Physical commonsense |
| **Winogrande** | 52.4% (±1.40) | — | Pronoun resolution |
| **BoolQ** | 48.6% (±0.87) | — | Yes/No reading comprehension |
| **ARC-Easy** | 48.6% (±1.03) | 43.8% (±1.02) | Grade-school science |
| **HellaSwag** | 29.1% (±0.45) | 31.7% (±0.46) | Sentence completion |
| **OpenBookQA** | 19.2% (±1.76) | 30.0% (±2.05) | Elementary science facts |
| **ARC-Challenge** | 22.3% (±1.22) | 27.2% (±1.30) | Harder science reasoning |
| **LAMBADA (OpenAI)** | 26.9% (±0.62) | — | Perplexity: 83.15 (±3.71) |
---
## Architecture & Training Summary
Instead of a wide, shallow network, MiniGrimm uses a tall, narrow layout (28 layers at `d_model=384`) paired with modern stability tweaks:
| Spec | Detail |
| :--- | :--- |
| **Parameters** | 48.29M total (~42.00M non-embedding, tied embeddings) |
| **Layers / Hidden Dim** | 28 layers, `d_model = 384`, `ffn_hidden = 960` (SwiGLU) |
| **Attention** | Grouped-Query Attention (6 query heads, 2 KV heads) + QK-Norm |
| **Positional Encoding** | RoPE (`theta = 10000.0`) |
| **Logit Soft-Capping** | `30.0` |
| **Context Window** | `512` tokens |
| **Tokenizer** | Custom 16,384-vocab Byte-Level BPE (with single-digit number splitting) |
| **Pretraining** | ~12.55B tokens (FineWeb-Edu, Cosmopedia-v2, Wikipedia, CoT/math, chat) using Muon + AdamW |
| **SFT** | Assistant-only loss masking across GSM8K, SmolTalk, UltraChat-200k, Tülu-3, and No Robots |
---
## Usage
Because MiniGrimm uses custom modeling code (for QK-Norm and logit soft-capping), you need to pass `trust_remote_code=True` when loading the model.
### Quickstart with `transformers`
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "Bayhakl1/MiniGrimm"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
trust_remote_code=True,
).to(device)
model.eval()
# Format the prompt using MiniGrimm's special chat tokens
prompt = (
"<|system|>You are a helpful assistant.<|end|>"
"<|user|>If I have 3 apples and buy 5 more, then eat 2, how many apples do I have?<|end|>"
"<|assistant|>"
)
inputs = tokenizer(prompt, return_tensors="pt").to(device)
# Stop when the model emits either <|end|> or <|endoftext|>
stop_token_ids = [
tokenizer.convert_tokens_to_ids("<|end|>"),
tokenizer.eos_token_id,
]
with torch.no_grad():
output_ids = model.generate(
**inputs,
max_new_tokens=150,
do_sample=True,
temperature=0.6,
top_p=0.9,
eos_token_id=stop_token_ids,
pad_token_id=tokenizer.pad_token_id,
)
# Slice off the prompt tokens so only the assistant's reply is printed
reply_ids = output_ids[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(reply_ids, skip_special_tokens=True))
```
### Important Usage Notes
1. **Chat Template Format:** Always wrap turns using the exact single-token delimiters `<|system|>`, `<|user|>`, `<|assistant|>`, and `<|end|>`:
```text
<|system|>Optional system prompt.<|end|><|user|>First message<|end|><|assistant|>First reply<|end|><|user|>Follow-up<|end|><|assistant|>
```
*(The `<|system|>...<|end|>` block is optional—you can start directly with `<|user|>`.)*
2. **512-Token Context Limit:** The model was trained with a maximum sequence length of **512 tokens** (prompt + generated tokens combined). Keep prompts short and `max_new_tokens` around `100–250` so the total length stays within 512 tokens.
3. **Greedy vs. Sampling:** For math questions, `do_sample=False` (greedy decoding) usually gives cleaner step-by-step arithmetic. For casual chat, `do_sample=True` with `temperature=0.6`–`0.7` helps avoid repetitive loops.
---
## Limitations
* **Small Model Scale (~48M):** Limited factual recall—it will confidently make up dates, names, or trivia.
* **Short Memory:** Anything past 512 tokens is outside its training window.
* **English Only:** Trained almost exclusively on English text and basic math.