min-spark

min-spark is a 5.76M-parameter language model with native effort levels. It introduces controllable depth to the sub-10M model class. The same checkpoint can produce a quick completion or spend more computation on the same prompt, selected with one inference argument.

The model was trained on 10.01B tokens. Its looped decoder reuses a compact transformer core across multiple passes, giving a small model the computation of a deeper network while keeping the parameter count fixed.

Introducing native effort levels

Effort levels have usually been associated with large reasoning models, where they adjust the thinking token budget or modify the system prompt to change how hard the model thinks. min-spark brings the idea directly into the language model. Each effort level changes the model's internal computation by selecting a different number of passes through its shared core.

Effort	Character	Recommended use
low	Fastest	High-throughput completion
medium	Balanced	General generation
high	Most compute	Highest available quality

Effort is a generation-time choice.

Usage

min-spark is compatible with Transformers and requires remote code loading.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "MinimaLabs/min-spark",
    trust_remote_code=True,
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(
    "MinimaLabs/min-spark",
    trust_remote_code=True,
)

prompt = "The meaning of life is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(
    **inputs,
    effort="high",
    max_new_tokens=64,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The same model works with the Transformers pipeline API:

from transformers import pipeline

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
result = pipe(prompt, effort="high", max_new_tokens=64)
print(result[0]["generated_text"])

A lightweight Transformers-free generation script is included in the repository:

python generate.py -p "The meaning of life is" -e high

Generation currently runs one sequence at a time, and right-padded batches are supported for evaluation. This model currently does not have KV cache support so long generations recompute the prompt at each step which may result in slightly longer generations. The context window is 512 tokens.

Evaluation

Scores below come from zero-shot evaluation with lm-eval 0.4.12. BLiMP uses accuracy. ARC-Easy, ARC-Challenge, HellaSwag, and PIQA use length-normalized accuracy. WikiText-2 is reported as byte-level perplexity, where lower is better.

min-spark reaches 69.19% on BLiMP at medium effort. ARC-Easy reaches 37.08%. ARC-Challenge reaches 23.21%. HellaSwag reaches 27.92%. PIQA reaches 54.35%. Its best WikiText-2 byte perplexity is 2.7747.

Effort levels

The table shows how the model responds to additional internal computation. Grammar improves most clearly from low to medium effort while the common-sense tasks remain close across the three settings.

Effort	BLiMP	ARC-Easy	ARC-Challenge	HellaSwag	PIQA	WikiText-2 byte-ppl
min-spark-low	67.11%	35.10%	23.21%	27.91%	54.13%	2.8783
min-spark-medium	69.19%	37.08%	22.78%	27.92%	54.30%	2.7747
min-spark-high	69.18%	37.08%	22.87%	27.91%	54.35%	2.7747

Benchmarks

The comparison places min-spark alongside published results for GPT-S2-5M, SLM-10M, and michel-nano-v2. It covers the benchmarks reported across this group.

Model	Params	BLiMP	ARC-Easy	ARC-Challenge	HellaSwag	PIQA
min-spark	5.76M	69.19%	37.08%	23.21%	27.92%	54.35%
GPT-S2-5M	5M	—	33.92%	22.87%	27.87%	57.56%
SLM-10M	10M	—	35.52%	23.46%	27.40%	57.07%
michel-nano-v2	8M	72.52%	35.90%	21.84%	27.40%	56.75%

min-spark reaches 37.08% on ARC-Easy, the highest score in this comparison. Its 23.21% ARC-Challenge result is close to SLM-10M at 23.46%. HellaSwag reaches 27.92%, and PIQA reaches 54.35%.

More in Less Parameters

KeyLM was the previous model made by us, and is a 75.25M-parameter language model trained on 18B tokens. min-spark uses less than one thirteenth of its parameter count while reaching a similar range on several small-model evaluations.

KeyLM reports 29.9% on its ARC average, 29.7% on HellaSwag, and 60.0% on PIQA. min-spark reaches 37.08% on ARC-Easy, 23.21% on ARC-Challenge, 27.92% on HellaSwag, and 54.35% on PIQA. The comparison highlights the value of repeated computation in a compact model. min-spark closes much of the size gap on these tasks while retaining a 5.76M parameter footprint.

Architecture

Field	Value
Parameters	5,758,572
Architecture	Tied-embedding looped decoder
Vocabulary	4,096-token byte-level BPE
Embedding width	288
Heads	6 query · 2 KV (GQA)
FFN hidden size	768
LoRA rank	16
Blocks	1 prelude · 3 shared body blocks · 1 coda · final RMSNorm
Context window	512 tokens
Effort (loop count)	low = 2 · medium = 3 · high = 4

Training

Field	Value
Training tokens	10.01B
Data mix	~90% filtered FineWeb-Edu · ~10% Finemath-4plus
Precision	fp16 autocast with gradient scaling
Context length	512 tokens
Global batch size	32
Optimizer	Muon (matrices) + NAdamW (auxiliary parameters), weight decay 0.01
Peak learning rate	Muon 0.01 · NAdamW 3e-3
Learning-rate schedule	2,000-step warmup → stable phase → 20% cooldown to 10% of peak
Attention masking	Intra-document
Checkpoint	Final cooldown checkpoint of the 10.01B-token run

Reproducing the evaluation

The evaluation runner is included in the repository. It supports each effort level and the benchmark suite used for this card.

python run_lmeval.py \
  --effort medium \
  --tasks blimp,arc_easy,arc_challenge,hellaswag,piqa,wikitext

Use --limit N to run a smaller evaluation during development.

Limitations

min-spark is a small base language model. It is not instruction-tuned and does not provide conversational alignment or safety filtering. Factual recall, multi-step reasoning, and long-form coherence are limited by its scale and training objective.

The model supports a 512-token context window. Generation accepts one sequence at a time and currently has no KV cache. Results were collected with a 512-token context and should not be assumed to transfer to longer inputs.

License

Apache-2.0. See LICENSE.