---
language: en
library_name: transformers
pipeline_tag: text-generation
tags:
  - mini-llm
  - from-scratch
  - small-language-model
---

# Mini-LLM Assistant 150M

A ~150M parameter conversational assistant with a full training pipeline implemented
end-to-end in PyTorch: BPE tokenizer, pretraining and supervised fine-tuning, no
pretrained backbone.

This repository hosts both the SFT assistant (repository root) and the base model it
was fine-tuned from ([`base/`](base)).

## Model details

|                |                                                                                                                                 |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| Parameters     | 152,596,224 (input embedding shared with the output head)                                                                       |
| Architecture   | Transformer decoder: RMSNorm, RoPE, causal attention, SwiGLU                                                                    |
| Context length | 512 tokens                                                                                                                      |
| Vocabulary     | 32,768 (BPE)                                                                                                                    |
| Pretraining    | ~4.50B tokens (FineWeb-Edu + Wikipedia), 34,332 steps, base val loss 2.941 (32k vocabulary)                                    |
| Fine-tuning    | 2 epochs of SFT on Dolly-15k, No Robots, OpenAssistant oasst2, alpaca-cleaned and identity/small-talk examples, val loss 2.780 |

## Base model

The [`base/`](base) subfolder contains the pretrained checkpoint
(`mini-llm-base-150M-34332steps.pt`) before SFT: 152,596,224 parameters, 32,768-token
BPE vocabulary, ~4.50B tokens (FineWeb-Edu + Wikipedia), 34,332 steps, base val loss
2.941. It is a raw completion model, not an assistant: `base/` ships without a chat
template and without the assistant's sampling defaults.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "GautierP/MiniLLM-150M"
tok = AutoTokenizer.from_pretrained(repo, subfolder="base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="base", trust_remote_code=True)

inputs = tok("The capital of France is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.8, top_k=50)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "GautierP/MiniLLM-150M"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

messages = [{"role": "user", "content": "What is the capital of France?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(inputs, max_new_tokens=128)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
```

`trust_remote_code=True` is required, the architecture ships with the repository
(`modeling_mini_llm.py`, `configuration_mini_llm.py`). Sampling defaults
(temperature 0.5, top_k 40, repetition penalty 1.15) live in `generation_config.json`.

The training format is single-turn and fixed; `chat_template.jinja` implements it
exactly:

```
User: <question>
Assistant: <answer>
```

Each question is processed independently, no conversation history is attached.

## Limitations

**Factual accuracy** is limited by model scale: fluent, well-formatted answers do not
imply verified facts, and any factual statement should be treated as unconfirmed.
Generation quality is tracked by the
[Mini-Eval](https://github.com/Mini-llm-lab/Mini-Eval) suite. English only.

## Links

- Base model weights: https://huggingface.co/GautierP/MiniLLM-150M/tree/main/base
- Code, training pipeline and full documentation: https://github.com/Mini-llm-lab/