--- license: apache-2.0 language: - en library_name: transformers pipeline_tag: text-generation tags: - reasoning - muon datasets: - openai/gsm8k - HuggingFaceTB/smoltalk - HuggingFaceH4/ultrachat_200k - allenai/tulu-3-sft-personas-instruction-following - HuggingFaceH4/no_robots - HuggingFaceFW/fineweb-edu - HuggingFaceTB/smollm-corpus - wikimedia/wikipedia - Open-Orca/OpenOrca - TIGER-Lab/WebInstructSub - nvidia/OpenMathInstruct-2 - kaist-ai/CoT-Collection - microsoft/orca-math-word-problems-200k - allenai/WildChat-1M - Magpie-Align/Llama-3-Magpie-Pro-1M-v0.1 - openbmb/UltraInteract_sft - WizardLMTeam/WizardLM_evol_instruct_V2_196k --- # MiniGrimm (48M SFT) **MiniGrimm** is a tiny **48.29M-parameter** (42.00M non-embedding) decoder-only Transformer trained from scratch and fine-tuned with Supervised Fine-Tuning (SFT). This is a small-scale experimental model built to test a deep-and-thin Llama-style architecture trained with the **Muon** optimizer. Because it is under 50M parameters and has a 512-token context window, **keep your expectations realistic**—it is not a general-purpose assistant and will hallucinate facts or stumble on complex logic. That said, it can hold short casual chats and do basic step-by-step arithmetic surprisingly well for its size. --- ## Benchmarks Evaluated after SFT on **ArithMark 3.0** (arithmetic reasoning): | Benchmark | Metric | Score | | :--- | :--- | :---: | | **ArithMark 3.0** | Raw Accuracy | **38.1%** | | **ArithMark 3.0** | Normalized Accuracy | **38.2%** | *(Not bad for a 42M non-embedding model, though it will still make plenty of mistakes on multi-step word problems or larger numbers.)* --- Other Benchmarks: | Benchmark | Accuracy (Raw) | Accuracy (Norm) | Notes | | :--- | :---: | :---: | :--- | | **SciQ** | 70.7% (±1.44) | 63.0% (±1.53) | Science QA | | **PIQA** | 61.7% (±1.13) | 62.0% (±1.13) | Physical commonsense | | **Winogrande** | 52.4% (±1.40) | — | Pronoun resolution | | **BoolQ** | 48.6% (±0.87) | — | Yes/No reading comprehension | | **ARC-Easy** | 48.6% (±1.03) | 43.8% (±1.02) | Grade-school science | | **HellaSwag** | 29.1% (±0.45) | 31.7% (±0.46) | Sentence completion | | **OpenBookQA** | 19.2% (±1.76) | 30.0% (±2.05) | Elementary science facts | | **ARC-Challenge** | 22.3% (±1.22) | 27.2% (±1.30) | Harder science reasoning | | **LAMBADA (OpenAI)** | 26.9% (±0.62) | — | Perplexity: 83.15 (±3.71) | --- ## Architecture & Training Summary Instead of a wide, shallow network, MiniGrimm uses a tall, narrow layout (28 layers at `d_model=384`) paired with modern stability tweaks: | Spec | Detail | | :--- | :--- | | **Parameters** | 48.29M total (~42.00M non-embedding, tied embeddings) | | **Layers / Hidden Dim** | 28 layers, `d_model = 384`, `ffn_hidden = 960` (SwiGLU) | | **Attention** | Grouped-Query Attention (6 query heads, 2 KV heads) + QK-Norm | | **Positional Encoding** | RoPE (`theta = 10000.0`) | | **Logit Soft-Capping** | `30.0` | | **Context Window** | `512` tokens | | **Tokenizer** | Custom 16,384-vocab Byte-Level BPE (with single-digit number splitting) | | **Pretraining** | ~12.55B tokens (FineWeb-Edu, Cosmopedia-v2, Wikipedia, CoT/math, chat) using Muon + AdamW | | **SFT** | Assistant-only loss masking across GSM8K, SmolTalk, UltraChat-200k, Tülu-3, and No Robots | --- ## Usage Because MiniGrimm uses custom modeling code (for QK-Norm and logit soft-capping), you need to pass `trust_remote_code=True` when loading the model. ### Quickstart with `transformers` ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer repo_id = "Bayhakl1/MiniGrimm" device = "cuda" if torch.cuda.is_available() else "cpu" tokenizer = AutoTokenizer.from_pretrained(repo_id) model = AutoModelForCausalLM.from_pretrained( repo_id, trust_remote_code=True, ).to(device) model.eval() # Format the prompt using MiniGrimm's special chat tokens prompt = ( "<|system|>You are a helpful assistant.<|end|>" "<|user|>If I have 3 apples and buy 5 more, then eat 2, how many apples do I have?<|end|>" "<|assistant|>" ) inputs = tokenizer(prompt, return_tensors="pt").to(device) # Stop when the model emits either <|end|> or <|endoftext|> stop_token_ids = [ tokenizer.convert_tokens_to_ids("<|end|>"), tokenizer.eos_token_id, ] with torch.no_grad(): output_ids = model.generate( **inputs, max_new_tokens=150, do_sample=True, temperature=0.6, top_p=0.9, eos_token_id=stop_token_ids, pad_token_id=tokenizer.pad_token_id, ) # Slice off the prompt tokens so only the assistant's reply is printed reply_ids = output_ids[0][inputs["input_ids"].shape[1]:] print(tokenizer.decode(reply_ids, skip_special_tokens=True)) ``` ### Important Usage Notes 1. **Chat Template Format:** Always wrap turns using the exact single-token delimiters `<|system|>`, `<|user|>`, `<|assistant|>`, and `<|end|>`: ```text <|system|>Optional system prompt.<|end|><|user|>First message<|end|><|assistant|>First reply<|end|><|user|>Follow-up<|end|><|assistant|> ``` *(The `<|system|>...<|end|>` block is optional—you can start directly with `<|user|>`.)* 2. **512-Token Context Limit:** The model was trained with a maximum sequence length of **512 tokens** (prompt + generated tokens combined). Keep prompts short and `max_new_tokens` around `100–250` so the total length stays within 512 tokens. 3. **Greedy vs. Sampling:** For math questions, `do_sample=False` (greedy decoding) usually gives cleaner step-by-step arithmetic. For casual chat, `do_sample=True` with `temperature=0.6`–`0.7` helps avoid repetitive loops. --- ## Limitations * **Small Model Scale (~48M):** Limited factual recall—it will confidently make up dates, names, or trivia. * **Short Memory:** Anything past 512 tokens is outside its training window. * **English Only:** Trained almost exclusively on English text and basic math.