Inferno-Exp-1.0

Inferno-Exp-1.0 is a 76.2M-parameter decoder-only base language model trained from scratch by AlgoDriveAI. It is a math-first model: its training data centers on arithmetic and math word problems, and it is geared toward the AxiomicLabs small-language-model benchmarks used in SLM challenges, ArithMark 3.0 and ArithMark 2.0 (results below). Future versions will build on this math foundation with improvements aimed at broader language understanding and logical reasoning.

Parameters	76,163,200
Layers × width	14 × 640
Attention	grouped-query, 8 query / 2 KV heads, head dim 80, full-head RoPE
Feed-forward	plain SwiGLU, inner width 1536
Context	2048 tokens
Tokenizer	Mistral-7B-v0.1 (padded to 32,064); `<
Embeddings	input and output tied
Training	AlgoDriveAI/SLM_v3, 2 epochs; AdamW, WSD schedule, label smoothing 0.05
Checkpoint	step 290,000 (vanilla_gqa_epochs_r1)

This is a base model (no instruction tuning). The model code ships in the repo, so load it with trust_remote_code=True.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("AlgoDriveAI/Inferno-Exp-1.0")
model = AutoModelForCausalLM.from_pretrained("AlgoDriveAI/Inferno-Exp-1.0", trust_remote_code=True)

ids = tok("Once upon a time, a little fox", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=80, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

Evaluation

All scores are zero-shot on the checkpoint in this repo. The ArithMark benchmarks were scored with AxiomicLabs' official scripts, unmodified; the language benchmarks with the lm-evaluation-harness.

Benchmark	Metric	Score	Standard error	Chance
ArithMark 3.0 (word problems)	acc_norm	54.20%	±1.58	25%
ArithMark 2.0 (integer arithmetic)	acc	67.32%	±0.94	25%
HellaSwag	acc_norm	27.46%	±0.45	25%
ARC-Easy	acc_norm	40.24%	±1.01	25%
ARC-Challenge	acc_norm	24.66%	±1.26	25%
PIQA	acc_norm	56.58%	±1.16	50%

On the Open SLM Leaderboard's Intelligence Index formula these scores give 14.16.

Inferno-Exp-1.0 is math-first, and the scores show it: it is strong on arithmetic but modest on general commonsense and reading benchmarks, where its narrower training data gives it less to draw on. Upcoming versions keep the math foundation and broaden the training mix to lift these language and reasoning scores.

ArithMark 3.0

Scored with bencharithmark-3.py on ArithMark 3.0 (1,000 four-choice continuation word problems, grades 1–5) at commit 6f6e59dd: float32, batch size 32, max context 1,024, no BOS token, length-normalized log-likelihood as the primary metric.

Metric	Score
Length-normalized accuracy (acc_norm, primary)	54.20%
Raw continuation accuracy	54.20%
Standard error (acc_norm)	±1.58 points
Random-choice baseline	25.00%

ArithMark 2.0

Scored with benchmark_arithmark-2.0.py on ArithMark 2.0 (2,500 four-choice integer arithmetic continuations such as (16 / 4) + 44 =) at commit 5bd986c7: the script's standard GPU setting (bfloat16), batch size 16, max context 1,024, no BOS token, raw summed continuation log-likelihood (the benchmark's only metric).

Operators per expression	Items	Accuracy
1	1,250	95.04%
2	750	46.93%
3	500	28.60%
Overall	2,500	67.32% (1,683 / 2,500)

Standard error ±0.94 points; random-choice baseline 25.00%.

Language understanding (lm-evaluation-harness)

Zero-shot, acc_norm, float32, batch size 16:

lm_eval --model hf \
  --model_args "pretrained=AlgoDriveAI/Inferno-Exp-1.0,trust_remote_code=True,dtype=float32" \
  --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0 --batch_size 16

Task	Items	acc_norm
HellaSwag	10,042	27.46% ± 0.45
ARC-Easy	2,376	40.24% ± 1.01
ARC-Challenge	1,172	24.66% ± 1.26
PIQA	1,838	56.58% ± 1.16

Notes

• Generation re-reads the full context each step (no KV cache); fine at this size.
• The tokenizer does not add a BOS token; its bos_token is set to <|endoftext|>, the document boundary the model saw during training, for tools that prepend one.