xxparthparekhxx/pluto-124m-chat

Pluto 124M Chat

Pluto is a small chat model with 121M parameters. xxparthparekhxx trained it from scratch on one RTX 5060 Laptop GPU (8 GB): pretraining on 1B tokens of FineWeb-Edu, then chat fine-tuning in two stages. The pretrained model without chat tuning is pluto-124m-base.

It is named after the dwarf planet, since it is a small model. It can answer simple questions, rewrite and summarize text, and follow formatting instructions such as "use exactly three bullet points". At this size it often gets facts wrong, so check anything it tells you.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "xxparthparekhxx/pluto-124m-chat"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

messages = [{"role": "user", "content": "Give me three tips for staying focused while studying."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_dict=True, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

trust_remote_code=True is needed for both the tokenizer and the model. The architecture (gated attention with QK-norm and a logit soft-cap) is not a stock transformers class; its code is modeling_pluto.py in this repo. Greedy decoding (do_sample=False) gives the most reliable answers. The default generation config samples with temperature 0.8 and top-k 50.

The chat format is:

<|endoftext|><|user|>Who are you?<|end|><|assistant|>I'm Pluto, ...<|end|>

A <|system|> turn is also supported.

Evaluation

Compared with GPT-2 124M

All three models were scored with the same code: lm-evaluation-harness 0.4.13, zero-shot, fp32. GPT-2 is openai-community/gpt2.

Task (metric)	Pluto base	Pluto chat	GPT-2 124M
HellaSwag (acc_norm)	30.2	31.1	31.1
ARC-Easy (acc_norm)	45.2	41.9	39.5
ARC-Challenge (acc_norm)	24.3	26.7	22.7
OpenBookQA (acc_norm)	29.4	27.2	27.2
PIQA (acc_norm)	60.3	59.1	62.5
Winogrande (acc)	52.3	51.7	51.6
LAMBADA (acc)	23.3	16.8	32.6

The standard errors are about ±0.5 points on HellaSwag and LAMBADA, ±1 to 1.5 on ARC, PIQA and Winogrande, and ±2 on OpenBookQA.

Pluto saw 1B training tokens, far fewer than GPT-2, and it is ahead on the science and school-knowledge questions: ARC-Easy, ARC-Challenge and OpenBookQA. That matches its educational pretraining data. It is behind on PIQA, and well behind on LAMBADA, which asks for the last word of passages from novels; FineWeb-Edu contains little fiction. Winogrande is near chance (50%) for all three models.

On the same 4M held-out FineWeb-Edu tokens, the validation loss is 3.203 for Pluto base (perplexity 24.6) and 3.274 for GPT-2 (perplexity 26.4). This text is in-distribution for Pluto, so the comparison favors it.

Chat evaluation

These tests use held-out data from the fine-tuning sets:

• Formatting rules kept: 77 held-out prompts from smol-smoltalk's constraint subset. Every rule in them can be checked automatically, such as "exactly 3 bullet points", "all lowercase", "no commas" or a word limit. Answers are greedy, and an answer counts only if it keeps every rule.
• Identity: 8 questions about the model, none of which appear in training.
• Validation loss: on held-out distilled answers and held-out smol-smoltalk conversations, counting answer tokens only.

Fine-tuning from the base model	Rules kept	Says it is Pluto	Val loss, distilled	Val loss, smol-smoltalk
distilled set only	35.1%	–	2.034	2.372
smol-smoltalk only	50.6%	–	2.448	1.398
smol-smoltalk, then distilled	53.2%	0/8	1.950	1.646
Pluto chat: smol-smoltalk, then distilled + identity	46.8%	8/8	1.951	1.648

Training on smol-smoltalk first is what makes the formatting rules stick. The drop from 53.2% to 46.8% after adding the identity set is 7 prompts lost and 2 gained out of 77. That is within the noise of a test this size (paired McNemar test, p ≈ 0.18), and none of those answers mention Pluto.

Some greedy answers, including wrong ones:

Prompt	Pluto chat
Who made you?	I'm Pluto, and I was created by xxparthparekhxx, not by a big AI company.
Rewrite this sentence to sound more formal: we gotta finish the report by friday.	We must finish the report by Friday.
Can you check the weather in Mumbai for me right now?	I do not have access to your real-time location or a live weather app to check the weather for you. ...
Explain why the sky is blue in two sentences.	The sky is blue in two sentences because the blue light is being reflected off the Earth's surface. (wrong)

All of its evaluation answers are in eval/eval.json.

Training

Training ran in three stages, all on the same RTX 5060 Laptop GPU. Both fine-tuning stages compute the loss on assistant tokens only.

1. Pretraining: this is pluto-124m-base.

• Data: 1B tokens from FineWeb-Edu sample-10BT, tokenized with GPT-2's BPE.
• Steps: 1,907 steps of 524,288 tokens (512 sequences of 1,024), in bf16 with torch.compile.
• Optimizer: Muon for the 2-D weight matrices and AdamW for the embeddings and norms. Muon uses the DeepSeek-V4 variant: Newton–Schulz orthogonalization with the update rescaled to RMS 0.18, so both optimizers share one learning rate.
• Learning rate: peak 3e-3, weight decay 0.1. 200 warmup steps, then flat, then a linear decay to 10% of the peak over the last 25% of steps (warmup-stable-decay).
• Compute: about 10 hours on one RTX 5060 Laptop GPU, at roughly 26.7k tokens/s.

2. Fine-tuning on smol-smoltalk:

• The data is HuggingFaceTB/smol-smoltalk, cut to the turns that fit in 1,024 tokens: 437,234 conversations.
• Two kinds of conversation were removed: 14,179 whose first prompt is also in the distilled set, and one where the assistant says it is SmolLM.
• One epoch: 6,832 steps of 64 conversations.
• Peak learning rate 3e-4, 30 warmup steps, linear decay to zero. About 6 hours.

3. Fine-tuning on distilled answers and identity:

• Distilled set: 24,283 conversations. The prompts come from seven smol-smoltalk subsets; the answers were written by larger open models, Gemma 4 26B-A4B-it (18,864) and dots.3-note-preview (5,419).
• Filtering: answers that broke a prompt's formatting rules were dropped, and the teachers' descriptions of themselves were rewritten.
• Identity set: 304 template conversations about the model itself: its name, its maker, its size and its limits.
• Schedule: 3 passes over the distilled set and 4 over the identity set (74,065 examples, 1,158 steps of 64). Peak learning rate 1.5e-4, linear decay to zero. 32 minutes.
• Checkpoint: the one with the lowest validation loss.

Architecture

Parameters	121.2M (embeddings tied with the output layer)
Layers	12, pre-norm, RMSNorm
Width	768
Attention	grouped-query: 12 query heads, 4 key/value heads, head size 64
	RoPE (θ = 10,000), RMSNorm on queries and keys, sigmoid output gate
Feed-forward	SwiGLU, hidden size 2,048
Output	tanh logit soft-cap at 30
Vocabulary	GPT-2 BPE plus 4 chat tokens, padded to 50,304
Context	1,024 tokens

Limitations

• English only, with a 1,024-token context.
• Facts are unreliable: it states wrong facts confidently.
• No preference tuning or safety training beyond what is in the fine-tuning data.
• It can't browse the web and has no knowledge of current events.

It is a research and learning model, not for production use.

License

Apache-2.0. The training data is FineWeb-Edu (ODC-By 1.0) and smol-smoltalk (Apache-2.0). The distilled answers come from Gemma 4 26B-A4B-it and dots.3-note-preview, both Apache-2.0.