Instructions to use xxparthparekhxx/pluto-124m-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use xxparthparekhxx/pluto-124m-base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="xxparthparekhxx/pluto-124m-base", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("xxparthparekhxx/pluto-124m-base", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use xxparthparekhxx/pluto-124m-base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "xxparthparekhxx/pluto-124m-base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xxparthparekhxx/pluto-124m-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/xxparthparekhxx/pluto-124m-base

SGLang

How to use xxparthparekhxx/pluto-124m-base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "xxparthparekhxx/pluto-124m-base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xxparthparekhxx/pluto-124m-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "xxparthparekhxx/pluto-124m-base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xxparthparekhxx/pluto-124m-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use xxparthparekhxx/pluto-124m-base with Docker Model Runner:
```
docker model run hf.co/xxparthparekhxx/pluto-124m-base
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Pluto 124M Base

Pluto Base is a 121M-parameter language model. xxparthparekhxx trained it from scratch on 4.35B tokens of FineWeb-Edu: the first 750M tokens on one RTX 5060 Laptop GPU (8 GB), the rest on one RTX PRO 6000. It continues text and is not tuned for chat; for a chat model, use pluto-124m-chat.

This is version 2. Version 1, trained on 1B tokens, is the v1 revision of this repo (revision="v1" in from_pretrained).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "xxparthparekhxx/pluto-124m-base"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

inputs = tokenizer("The most important thing to know about the ocean is", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=60)
print(tokenizer.decode(output[0], skip_special_tokens=True))

trust_remote_code=True is needed for both the tokenizer and the model. The architecture (gated attention with QK-norm and a logit soft-cap) is not a stock transformers class; its code is modeling_pluto.py in this repo.

Evaluation

Compared with GPT-2 124M

All four models were scored with the same code: lm-evaluation-harness 0.4.13, zero-shot, fp32. GPT-2 is openai-community/gpt2.

Task (metric)	Pluto base v2	Pluto chat v2	GPT-2 124M	Pluto base v1
HellaSwag (acc_norm)	33.9	34.2	31.1	30.2
ARC-Easy (acc_norm)	47.6	44.6	39.5	45.2
ARC-Challenge (acc_norm)	27.0	26.5	22.7	24.3
OpenBookQA (acc_norm)	30.4	30.6	27.2	29.4
PIQA (acc_norm)	63.4	62.4	62.5	60.3
Winogrande (acc)	52.3	51.9	51.6	52.3
LAMBADA (acc)	28.1	20.6	32.6	23.3

The standard errors are about ±0.5 points on HellaSwag and LAMBADA, ±1 to 1.5 on ARC, PIQA and Winogrande, and ±2 on OpenBookQA.

Pluto base v2 is ahead of GPT-2 on HellaSwag and on the science and school-knowledge questions (ARC-Easy, ARC-Challenge and OpenBookQA), which matches its educational pretraining data. It is level on PIQA and still behind on LAMBADA, which asks for the last word of passages from novels; FineWeb-Edu contains little fiction. Winogrande is near chance (50%) for all of them. Compared with v1, the 4.35× longer training raised every score except Winogrande.

On the same 4M held-out FineWeb-Edu tokens, the validation loss is 2.991 for Pluto base v2 (perplexity 19.9), 3.203 for v1 (perplexity 24.6) and 3.274 for GPT-2 (perplexity 26.4). This text is in-distribution for Pluto, so the comparison favors it. The raw results are in eval/.

Training

Data: 4.35B tokens from FineWeb-Edu sample-10BT, tokenized with GPT-2's BPE.
Steps: 8,297 steps of 524,288 tokens (512 sequences of 1,024), in bf16 with torch.compile.
Optimizer: Muon for the 2-D weight matrices and AdamW for the embeddings and norms. Muon uses the DeepSeek-V4 variant: Newton–Schulz orthogonalization with the update rescaled to RMS 0.18, so both optimizers share one learning rate.
Learning rate: peak 3e-3, weight decay 0.1. 200 warmup steps, then flat, then a linear decay to 10% of the peak over the last 25% of steps (warmup-stable-decay).
How it was extended: v1 ran this same schedule for 1B tokens. Because the learning rate stays flat until the decay, v2 resumed from v1's checkpoint at step 1,430 (750M tokens, saved before the decay started) and continued the flat phase to 4.35B tokens before decaying.
Compute: steps 1–1,430 took about 7.6 hours on one RTX 5060 Laptop GPU (roughly 26.7k tokens/s); steps 1,431–8,297 took about 3.9 hours on one RTX PRO 6000 (roughly 258k tokens/s).

Architecture


Parameters	121.2M (embeddings tied with the output layer)
Layers	12, pre-norm, RMSNorm
Width	768
Attention	grouped-query: 12 query heads, 4 key/value heads, head size 64
	RoPE (θ = 10,000), RMSNorm on queries and keys, sigmoid output gate
Feed-forward	SwiGLU, hidden size 2,048
Output	tanh logit soft-cap at 30
Vocabulary	GPT-2 BPE plus 4 chat tokens (unused here), padded to 50,304
Context	1,024 tokens

Limitations

English only, with a 1,024-token context.
It continues text; it does not follow instructions.
It was trained on educational web pages only, so it knows little about other kinds of text, such as code or dialogue.
Its facts are unreliable.

License

Apache-2.0. The training data, FineWeb-Edu, is ODC-By 1.0.

Downloads last month: 20

Safetensors

Model size

0.1B params

Tensor type

F32

Model tree for xxparthparekhxx/pluto-124m-base

Quantizations

1 model

xxparthparekhxx
/

pluto-124m-base