MinimaLabs/min-spark-1.7

min-spark 1.7

min-spark 1.7 is a 9.98M-parameter base language model with native effort levels. It runs five shared transformer blocks two to five times in a row, and it was distilled from KLM 1.5, our 99M-parameter base model, over 34B tokens of next-token training.

What's new

• Distillation from KLM 1.5. Every next-token step trains min-spark to match the predictions of KLM 1.5, our own 99M-parameter model, as well as the text itself. The two models share a tokenizer, so the student learns from the teacher's full distribution over the next token.
• Longer training. min-spark 1.7 was trained on 34B tokens of next-token prediction, against 20B for min-spark 1.6, and the learning-rate decay alone covers 18B of them.
• A fourth effort level. max runs five passes through the shared blocks, two more than the model was trained with, and it gives the best HellaSwag and ArithMark-3 scores.

Benchmarks

All scores are zero-shot with length-normalized accuracy.

• ARC, HellaSwag and PIQA were scored using lm-eval 0.4.12.
• ArithMark-3 was scored using the official ArithMark-3.0 script.
• min-spark 1.7 is scored at each benchmark's best effort level.
• Peer scores are taken from the Open SLM leaderboard as of 2026-10-26.

min-spark 1.7's Intelligence Index on that leaderboard's formula is 13.18, which is the highest of any model under 10M parameters.

Each chart's axis starts at random-guess accuracy, which is 25% for most benchmarks and 50% for PIQA, so the height of a bar shows how far a model scores above chance. ARC-Challenge scores sit close to chance, so its axis starts at 20%.

Model	Params	ARC-Easy	ARC-Challenge	HellaSwag	PIQA	ArithMark-3
min-spark 1.7	9.98M	40.82%	24.15%	29.97%	60.55%	37.00%
min-spark 1.6	9.96M	38.30%	22.44%	28.52%	59.14%	36.90%
ForgePlex-M2.2-10M	9.99M	39.77%	22.70%	29.19%	59.79%	37.40%
heron1i-flash-lite-v2	9.92M	38.89%	23.38%	28.86%	58.54%	36.70%
LS3-Micro-Base	9.97M	34.76%	23.29%	27.31%	57.34%	38.90%
GPT-S3-3M	2.97M	34.76%	23.12%	27.68%	57.02%	39.10%

Effort levels

The effort level sets how many passes the model makes through its shared blocks. Large reasoning models usually do this by changing a thinking-token budget, whereas min-spark changes the computation inside the network.

Effort	Passes	Use
low	2	Fastest completion
medium	3 (default)	General use, and the most passes the model was trained with
high	4	One pass more than training, which reuses the third pass's adapters. Best for ARC-Challenge
max	5	Two passes more than training. Best for HellaSwag and ArithMark-3

Training

min-spark 1.7 was trained from scratch for 96,600 steps of 768 sequences × 512 tokens, in four stages.

1. Superposition warm-up for 9,000 steps, after 64 steps on synthetic bracket matching. Following Token-Superposition Training, each input position is the average of 7 consecutive tokens and the model predicts the next 7 tokens as an unordered group. The warm-up covers 24.8B tokens of ClimbMix for the compute of 3.5B, and it is the only stage without distillation.
2. Stable training on 16.0B tokens at a constant learning rate.
3. Decay on 16.0B tokens of ClimbMix, while the learning rate falls linearly to 10%.
4. Final decay on 2.4B tokens, while the learning rate falls from 10% to 1%. This stage mixes 10% math back in with the ClimbMix.

Each next-token loss is 0.1 × the loss on the text plus 0.9 × the KL divergence from KLM 1.5's prediction, at temperature 1. As in min-spark 1.6, the training objective is that loss at 3 passes, plus 0.1 × the loss at 2 passes and 0.1 × a term that keeps the 2-pass state close to the 3-pass one. The weight matrices were trained with Muon at a learning rate of 0.01 and the other parameters with NAdamW, in bf16. Each sequence packs several documents, and attention is masked at every document boundary.

The data mix shifted between the stable and decay stages as follows.

Data	Sources	Stable	Decay
Web text	ClimbMix, TxT360-v2	48%	99%
Q&A	TxT360-v2 Q&A, Dolmino Q&A, passages with questions and flashcards	16%
Textbooks	Cosmopedia v2	15%
Commonsense	DCLM-baseline filtered for physical commonsense, wikiHow, ATOMIC 2020, ANLI, GLUCOSE, CREAK, SciQ, QASC	10%
Math	OpenMathInstruct-2, Dolmino TinyMATH	7%	1%
Science web	Dolmino	4%

Architecture

Field	Value
Parameters	9,975,188
Blocks	1 prelude, 5 shared blocks run 2 to 5 times, 1 coda
Per-pass adapters	LoRA rank 33 on attention QKV, attention output, FFN gate/up and FFN down
Embedding width	288
Attention	6 query heads, 2 KV heads
Feed-forward	SwiGLU, hidden size 832
Context length	512 tokens
Vocabulary	4,096-token byte-level BPE, in a tied embedding table of 4,160 rows
Effort (loop count)	low = 2, medium = 3, high = 4, max = 5 (trained up to 3)

Intended use

min-spark 1.7 is a base model. It continues text but has not been trained to follow instructions or chat, so it is meant to be fine-tuned for your own use case.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MinimaLabs/min-spark-1.7"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).to(device)

inputs = tokenizer("The meaning of life is", return_tensors="pt").to(device)
outputs = model.generate(**inputs, effort="medium", max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

effort takes low, medium, high or max, and it works the same way with the pipeline API. To skip Transformers, use the bundled script.

python generate.py -p "The meaning of life is" -e medium

Reproducing the evaluation

python run_lmeval.py --effort low,medium,high,max
python bencharithmark-3.py --model MinimaLabs/min-spark-1.7 --dtype float32

The first command scores ARC, HellaSwag and PIQA at each effort and prints each task's best. The second is the official ArithMark-3 script, which runs at the default medium effort and gives 36.30. The 37.00 in the tables comes from the max effort, so to reproduce it, download the repo, set "effort": "max" in its config.json, and pass the local path to --model.

Limitations

• The model's knowledge is thin, and the facts it states are often wrong.
• It only works in English and has a 512-token context window.
• It has no safety alignment, so add your own filtering before putting it in front of users.

License

min-spark 1.7 is released under the Apache-2.0 license, which you can find in LICENSE.