Manprot/MP-01-100M-base

MP-01 100M base

A 100.6M-parameter decoder-only language model by Manprot
Author: Dominik Andrzejewski

Author	Dominik Andrzejewski
Organization	Manprot
Parameters	100.6M (bf16, tied embeddings)
Architecture	Custom VBQ decoder (18 layers, GQA, SwiGLU, QK-norm)
Tokens seen	about 11.3B in total across all stages
7-task average	48.43, the best of the models compared below
Type	Base model on chat-tuned roots, not an instruction model

Note: this is not a pure base model. Its lineage includes a chat fine-tune, so it can show chat-style behaviour. It continues text but does not follow instructions reliably.

Contents

• Training lineage
• Datasets
• Architecture
• Evaluation
• Compute
• Loading
• Limitations
• Repository contents
• Author and citation

Training lineage

1. Pretraining (75M): a 75M-parameter model was pretrained from scratch on English text, single pass.
2. Growth to 100M: grown from 12 to 18 layers (identity-initialised) and pretrained further on new documents.
3. Chat fine-tune: fine-tuned for chat. These weights are the starting point of the expansion.
4. Expansion: continued pretraining on new documents from the same sources, to step 19,000 with a 262k-token batch.
5. Cooldown: the learning rate was decayed to its minimum over 8,000 steps (2.10B tokens) on a second, unseen corpus, starting from step 19,000. This is the checkpoint released here.

Tokens and hardware for each stage are in Compute.

Two other cooldowns are not in this release's lineage: a local 170-step experiment from step 13,000, and a 4,000-step cooldown from step 15,000 that was the previous version of this repository (7-task average 47.29). Both are in the progress table below.

Datasets

In total: 16 training datasets and 8 evaluation datasets. All 16 training datasets are also in this page's metadata; the Hugging Face sidebar shows the first three and folds the rest behind "+13". Every dataset used to train or evaluate this checkpoint is listed here. Each stage after the first used documents the earlier stages had not seen; shards were excluded between corpora. Training data is not re-released. It was built from the public datasets below.

Pretraining, growth, expansion and cooldown (stages 1, 2, 4, 5)

Source	Dataset	Subset / filter	75M pretraining	Growth, expansion, cooldown
FineWeb-Edu	HuggingFaceTB/smollm-corpus	fineweb-edu-dedup, edu score ≥ 3.5	37%	34%
Wikipedia	wikimedia/wikipedia	20231101.en	6%	24%
Cosmopedia	HuggingFaceTB/smollm-corpus	cosmopedia-v2	20%	18%
DCLM	mlfoundations/dclm-baseline-1.0-parquet	fastText ≥ 0.10, lang ≥ 0.9	20%	12%
FineMath	HuggingFaceTB/finemath	finemath-4plus	7%	5%
Stack Exchange	HuggingFaceTB/stackexchange_2025_md	answer score ≥ 5, selected sites	5%	4%
Python code	allenai/dolma3_mix-6T-1025	stack_edu-Python, score ≥ 3.5	5%	3%

Shares are of each stage's tokens. The tokenizer was trained on a sample drawn from these same sources in the 75M mix proportions.

Chat fine-tune (stage 3)

Dataset	Used for
HuggingFaceTB/smol-smoltalk	General chat (main mix)
HuggingFaceTB/everyday-conversations-llama3.1-2k	Everyday small talk
OpenAssistant/oasst2	Human-written assistant conversations
HuggingFaceH4/ultrachat_200k	Multi-turn questions about the world
databricks/databricks-dolly-15k	Human-written QA, brainstorming, summaries
wikimedia/wikipedia	"Who was / what is" answers from article leads
jondurbin/airoboros-3.2	Instructions and roleplay
anthracite-org/kalo-opus-instruct-22k-no-refusal	Instructions
Gryphe/Sonnet3.5-Charcard-Roleplay	Character roleplay
hieunguyenminh/roleplay	Character roleplay
PygmalionAI/PIPPA	Character roleplay

The earlier 75M chat models (not in this checkpoint's weights; the 100M was grown from the 75M base before any chat tuning) used a subset of these: smol-smoltalk, everyday-conversations, oasst2, airoboros, kalo, Sonnet3.5 Charcard, hieunguyenminh/roleplay and PIPPA.

Chat data was filtered: refusals, boilerplate, other models' identities and over-length chats were dropped, and any sexual content involving minors was removed.

Used in later MP-01 runs (not in this checkpoint)

These datasets were used in training runs that continue from this model. They are listed for completeness: none of them is in the weights released here.

Dataset	Used for
SlayerLab/plwiki-history-rag-20260920	Polish history text
open-web-math/open-web-math	Mathematical web pages
sedthh/gutenberg_english	Public-domain books
RealTimeData/bbc_news_alltime	Recent business and economics news
meta-math/MetaMathQA	Step-by-step math answers (chat data)
sinhal/Charcard-Roleplay-ShareGPT-NSFW	Adult character roleplay (chat data)

Evaluation benchmarks

Benchmark	Dataset
ARC-Challenge, ARC-Easy	allenai/ai2_arc
HellaSwag	Rowan/hellaswag
OpenBookQA	allenai/openbookqa
PIQA	baber/piqa
SciQ	allenai/sciq
WinoGrande	allenai/winogrande
BLiMP	nyu-mll/blimp

Architecture

Component	Value
Layers	18
Width (d_model)	640
Attention heads / KV heads	10 / 5 (grouped-query attention, QK-norm)
MLP hidden	1664 (SwiGLU)
Vocabulary	32,768 (byte-level BPE, tokenizer.json)
Context length	2048 tokens (the final cooldown trained at 2048; the expansion before it at 1024)
Parameters	100.6M, tied input/output embeddings

This is a custom architecture, not a transformers model. Load it with the vbq/ code in this repository (see Loading).

Evaluation

Scored by this repository's harness (scripts/eval_checkpoint.py, scripts/eval_hf.py): zero-shot, 500 questions per task (the same sample every time). BLiMP is zero-shot minimal pairs (67 paradigms). With 500 questions a task, differences under about 2 points are within noise. Models use their own tokenizers, so validation loss is not comparable across models and is not compared.

Comparison with other small models

Sorted by the 7-task average. Bold is the best score in each column, whichever model it is.

Rank	Model	Params	ARC-C	ARC-E	HellaSwag	OBQA	PIQA	SciQ	Wino	Avg (7)	BLiMP
1	MP-01 100M base (this model)	100.6M	29.4	56.4	31.8	30.2	64.4	73.0	53.8	48.43	78.72
2	GPT-neo-125M	125.2M	25.2	41.8	27.8	25.6	63.4	73.4	50.4	43.94	79.26
3	OPT-125M	125.2M	25.4	39.8	30.6	27.8	61.0	70.0	50.8	43.63	81.41
4	GPT-2	124.4M	24.8	39.6	29.6	25.8	64.2	67.6	50.2	43.11	82.51
5	Pythia-160M	162.3M	25.4	41.8	26.0	26.4	62.8	69.0	47.8	42.74	78.53
6	DistilGPT2	81.9M	25.2	36.0	25.4	26.0	59.0	64.8	50.2	40.94	79.53
7	GPT-1	116.5M	25.8	34.4	22.8	26.6	55.8	59.0	53.6	39.71	77.59

Where this model stands

• 7-task average: first of the models measured, about 4.5 points above the best comparison model, and best or tied-best on 6 of the 7 tasks (SciQ: GPT-neo-125M is 0.4 ahead).
• BLiMP: behind GPT-2 (82.5), OPT-125M (81.4), DistilGPT2 (79.5) and GPT-neo-125M (79.3). Ahead of Pythia-160M (78.5) and GPT-1 (77.6).
• The comparison models are older and trained on different data and budgets, so this is not a training-budget-matched comparison.
• Cerebras-GPT-111M could not be loaded from the Hub and is not included.

Progress across checkpoints of this model

Checkpoint	Val loss	ARC-C	ARC-E	HellaSwag	OBQA	PIQA	SciQ	Wino	Avg (7)	BLiMP
Expansion step 9,500	2.7780	27.6	52.8	29.8	27.6	62.4	71.0	51.6	46.11	76.88
Expansion step 13,000	2.7722	28.8	49.8	28.4	29.0	61.6	70.4	51.8	45.69	78.03
Expansion step 15,000	2.7730	26.8	52.6	29.0	30.0	62.0	69.8	53.0	46.17	77.66
Earlier cooldown from 15,000, step 4,000 (previous release)	2.6531	29.2	54.0	29.4	29.4	62.8	72.2	54.0	47.29	78.70
Expansion step 19,000 (cooldown start)	2.7830	29.0	50.2	29.8	30.2	62.0	72.8	49.8	46.26	77.97
Cooldown step 5,000	2.6281	27.6	53.8	30.2	30.6	62.4	71.6	51.6	46.83	78.43
Cooldown step 6,000	2.6082	28.2	55.4	30.0	30.4	61.8	72.0	53.4	47.31	78.08
Cooldown step 7,000	2.5877	28.4	55.4	31.2	30.6	64.0	71.4	53.6	47.80	78.66
Cooldown step 8,000 (this release)	2.5723	29.4	56.4	31.8	30.2	64.4	73.0	53.8	48.43	78.72

Bold is the best value in each column (lowest for val loss). Validation loss is measured on the held-out split of the first expansion corpus, which the model trained on. It is not a clean test, so the cooldown's lower loss should not be read as general improvement on its own. The benchmark average rose at every saved step of the cooldown and is the more useful signal.

Compute

Stage	Hardware	GPU hours	Tokens
Pretraining, 75M base	Local AMD Radeon RX 7600 (8 GB)	not totalled from logs	4.0B
Growth to 100M, first stage	Local AMD Radeon RX 7600 (8 GB)	3.94 (one segment)	0.61B
Chat fine-tune	Local AMD Radeon RX 7600 (8 GB)	2.78	0.18B
Expansion, steps 2,314 to 5,040	Local AMD Radeon RX 7600 (8 GB)	8.01	0.71B new
Expansion, steps 5,040 to 19,000	Cloud A100 40 GB	6.24	3.66B new
Cooldown, 8,000 steps (this release)	Cloud H200 141 GB	1.38	2.10B
Earlier cooldown from step 15,000 (previous release, not in this lineage)	Cloud H200 141 GB	0.68	1.05B
Benchmarks of checkpoints	Local GPU	a few minutes each
Comparison benchmarks (6 models)	Cloud H200	a few minutes each, in parallel

Hours are wall-clock GPU time from the run logs. Costs in credits are not listed. The local GPU's times are per recorded segment; long runs were restarted, so the 75M pretraining total is not reliable from the logs.

Running it

Download the repository and continue a prompt (needs Python with torch, tokenizers, safetensors and huggingface_hub):

hf download Manprot/MP-01-100M-base --local-dir MP-01-100M-base
python MP-01-100M-base/run.py "The history of the printing press" --tokens 150

run.py uses a CUDA GPU when one is available and the CPU otherwise. Options: --tokens, --temperature, --top-k, --top-p, --repetition-penalty, --seed.

Loading

import json
from tokenizers import Tokenizer
from safetensors.torch import load_file
from vbq.model import VBQ, VBQConfig

cfg = VBQConfig(**json.load(open("config.json"))["model_cfg"])
model = VBQ(cfg)
model.load_state_dict(load_file("model.safetensors"))  # the tied output weight is stored once
model.eval()
tok = Tokenizer.from_file("tokenizer.json")

Limitations

• Not an instruction model. It can produce factually wrong, repetitive or unfocused text.
• Chat-tuned starting weights: expect some chat-style behaviour. It has not been checked for safety.
• Small size and small benchmarks. These numbers are indicative and are not measured on the full test sets.
• Trained on web and educational text that can contain bias and errors.
• Training data is not re-released (the source datasets are linked above).

Repository contents

• model.safetensors: weights (bf16)
• config.json: architecture and lineage
• tokenizer.json: tokenizer
• vbq/: model code
• run.py: command-line text generation
• eval_results.json: full evaluation output for this checkpoint and the comparison models
• manprot-logo.png: logo

Author and citation

Made by Dominik Andrzejewski (Manprot).

@misc{andrzejewski2026mp01,
  title        = {MP-01 100M base},
  author       = {Andrzejewski, Dominik},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Manprot/MP-01-100M-base}}
}

The datasets listed above belong to their authors and keep their own licenses.