| --- |
| language: en |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - mini-llm |
| - from-scratch |
| - small-language-model |
| --- |
| |
| # Mini-LLM Assistant 150M |
|
|
| A ~150M parameter conversational assistant with a full training pipeline implemented |
| end-to-end in PyTorch: BPE tokenizer, pretraining and supervised fine-tuning, no |
| pretrained backbone. |
|
|
| This repository hosts both the SFT assistant (repository root) and the base model it |
| was fine-tuned from ([`base/`](base)). |
|
|
| ## Model details |
|
|
| | | | |
| | -------------- | ------------------------------------------------------------------------------------------------------------------------------- | |
| | Parameters | 152,596,224 (input embedding shared with the output head) | |
| | Architecture | Transformer decoder: RMSNorm, RoPE, causal attention, SwiGLU | |
| | Context length | 512 tokens | |
| | Vocabulary | 32,768 (BPE) | |
| | Pretraining | ~4.50B tokens (FineWeb-Edu + Wikipedia), 34,332 steps, base val loss 2.941 (32k vocabulary) | |
| | Fine-tuning | 2 epochs of SFT on Dolly-15k, No Robots, OpenAssistant oasst2, alpaca-cleaned and identity/small-talk examples, val loss 2.780 | |
|
|
| ## Base model |
|
|
| The [`base/`](base) subfolder contains the pretrained checkpoint |
| (`mini-llm-base-150M-34332steps.pt`) before SFT: 152,596,224 parameters, 32,768-token |
| BPE vocabulary, ~4.50B tokens (FineWeb-Edu + Wikipedia), 34,332 steps, base val loss |
| 2.941. It is a raw completion model, not an assistant: `base/` ships without a chat |
| template and without the assistant's sampling defaults. |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| repo = "GautierP/MiniLLM-150M" |
| tok = AutoTokenizer.from_pretrained(repo, subfolder="base", trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained(repo, subfolder="base", trust_remote_code=True) |
| |
| inputs = tok("The capital of France is", return_tensors="pt") |
| out = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.8, top_k=50) |
| print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| repo = "GautierP/MiniLLM-150M" |
| tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True) |
| |
| messages = [{"role": "user", "content": "What is the capital of France?"}] |
| inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt") |
| out = model.generate(inputs, max_new_tokens=128) |
| print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| `trust_remote_code=True` is required, the architecture ships with the repository |
| (`modeling_mini_llm.py`, `configuration_mini_llm.py`). Sampling defaults |
| (temperature 0.5, top_k 40, repetition penalty 1.15) live in `generation_config.json`. |
|
|
| The training format is single-turn and fixed; `chat_template.jinja` implements it |
| exactly: |
|
|
| ``` |
| User: <question> |
| Assistant: <answer> |
| ``` |
|
|
| Each question is processed independently, no conversation history is attached. |
|
|
| ## Limitations |
|
|
| **Factual accuracy** is limited by model scale: fluent, well-formatted answers do not |
| imply verified facts, and any factual statement should be treated as unconfirmed. |
| Generation quality is tracked by the |
| [Mini-Eval](https://github.com/Mini-llm-lab/Mini-Eval) suite. English only. |
|
|
| ## Links |
|
|
| - Base model weights: https://huggingface.co/GautierP/MiniLLM-150M/tree/main/base |
| - Code, training pipeline and full documentation: https://github.com/Mini-llm-lab/ |
|
|
