Inferno-Exp-1.6

Inferno-Exp-1.6 is a 393.6M-parameter decoder-only base language model trained from scratch by AlgoDriveAI on about 4B tokens. It is a math-first model: its training emphasizes arithmetic and math word problems, and it is geared toward the AxiomicLabs small-language-model benchmarks used in SLM challenges, ArithMark 3.0. Future versions will build on this math foundation with improvements aimed at broader language understanding and logical reasoning.

Parameters	393,610,240 (393.6M)
Layers × width	32 × 1024 (deep and narrow)
Attention	grouped-query, 16 query / 4 KV heads, head dim 64, full-head RoPE
Feed-forward	plain SwiGLU, inner width 2816
Context	2048 tokens
Tokenizer	Mistral-7B-v0.1 (padded to 32,064); `<
Embeddings	input and output tied
Training	from scratch on AlgoDriveAI/SLM_v3 (expanded), ~4B tokens (one pass); AdamW, WSD schedule

Parameter accounting: 32,833,536 tied token-embedding parameters + 360,775,680 parameters across 32 transformer blocks + 1,024 final normalization parameters = 393,610,240 unique trainable parameters. The tied LM head is not counted twice.

This is a base model (no instruction tuning). The model code ships in the repo, so load it with trust_remote_code=True.

Compared with Inferno-Exp-1.5 (76M)

Same architecture family, about 5.17× the parameters and 32 layers instead of 14.

Benchmark	Inferno-Exp-1.5	Inferno-Exp-1.6	Change
ArithMark 3.0 (acc_norm)	71.00%	83.10%	+12.10
HellaSwag	27.85%	29.80%	+1.95
ARC-Easy	43.39%	47.01%	+3.62
ARC-Challenge	26.62%	30.12%	+3.50
PIQA	58.49%	60.45%	+1.96
Open SLM Leaderboard Intelligence Index	20.27	26.23	+5.96

Word-problem arithmetic and every language benchmark improve; bare-equation arithmetic (ArithMark 2.0) is lower than in Inferno-Exp-1.5.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("AlgoDriveAI/Inferno-Exp-1.6")
model = AutoModelForCausalLM.from_pretrained("AlgoDriveAI/Inferno-Exp-1.6", trust_remote_code=True)

ids = tok("Once upon a time, a little fox", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=80, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

Try it: interactive demo (Gradio)

The easiest way to play with the model is a small web app with a prompt box, sampling sliders, live streaming, and a Stop button:

1. Open a new notebook at colab.research.google.com and choose a GPU (Runtime → Change runtime type → T4 GPU). It also runs on CPU, just slower.
2. Paste the script below into one cell and run it.
3. Click the *.gradio.live link it prints.

If it stops and asks you to restart the session (it may need to switch Transformers to a compatible version), choose Runtime → Restart session and run the cell again.

Show the Gradio demo script (one Colab cell, ~200 lines)

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Inferno-Exp-1.6 — simple Gradio inference (loads like any Hub user, no pushing)
==============================================================================
Loads Inferno-Exp-1.6 with plain Transformers (AutoModelForCausalLM +
trust_remote_code=True) and serves a small Gradio app: open prompt box,
sampling controls, streaming output with a Stop button, token/speed readout.

Works with either source:
  • the Hub repo:          MODEL_SOURCE = "AlgoDriveAI/Inferno-Exp-1.6"
  • the local export folder the push script builds (test before uploading):
                           MODEL_SOURCE = "/content/hf_export_inferno_exp_1_0"

Private repo? Add a Colab secret named HF_TOKEN (🔑 icon in the left sidebar, turn on
notebook access), set the env var HF_TOKEN, or paste a READ token into HF_TOKEN below.

It is a base model: it continues text, it does not chat. Write the start of the text
you want (e.g. a word problem followed by "Answer:").
Hardcoded constants, no argparse. Paste into one Colab cell and run; open the share link.
"""
# =============================================================================
# SETTINGS — edit these
# =============================================================================
MODEL_SOURCE = "AlgoDriveAI/Inferno-Exp-1.6"   # Hub repo id (or a local folder with the same files)
HF_TOKEN = ""                                  # only for private repos; "" → Colab secret / env var
MODEL_DISPLAY_NAME = "Inferno-Exp-1.6"

# Default slider values (all adjustable in the UI)
DEF_MAX_NEW_TOKENS = 256
DEF_TEMPERATURE = 0.8
DEF_TOP_K = 40
DEF_TOP_P = 0.95
DEF_REPETITION_PENALTY = 1.1
DEF_SEED = -1             # -1 = different every run
DTYPE = "auto"            # "auto" (bf16 on newer GPUs, fp32 on CPU) / "float32" / "bfloat16" / "float16"

GRADIO_SHARE = True       # public *.gradio.live link (needed in Colab)
GRADIO_PORT = 7860
INSTALL_DEPENDENCIES = True

# =============================================================================
# RUN (no edits needed below)
# =============================================================================
import os, sys, time, threading, subprocess

try:
    import google.colab  # noqa: F401
    IN_COLAB = True
except ImportError:
    IN_COLAB = False
if IN_COLAB and INSTALL_DEPENDENCIES:
    import importlib.metadata as _md

    def _version(name):
        try:
            return _md.version(name)
        except _md.PackageNotFoundError:
            return None

    _before = _version("transformers")
    subprocess.check_call([sys.executable, "-m", "pip", "install", "--quiet",
                           "transformers>=4.45,<5", "safetensors", "huggingface_hub>=0.25", "gradio"])
    if _before != _version("transformers") and "transformers" in sys.modules:
        raise SystemExit("Transformers was just changed to a compatible version, but the old one is still loaded. "
                         "Restart the session (Runtime → Restart session) and run this cell again.")

import torch
import gradio as gr
from transformers import (AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList,
                          TextIteratorStreamer)

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
STOP_EVENT = threading.Event()


def get_token():
    if HF_TOKEN.strip():
        return HF_TOKEN.strip()
    if IN_COLAB:
        try:
            from google.colab import userdata
            value = userdata.get("HF_TOKEN")
            if value:
                return value
        except Exception:
            pass
    return os.environ.get("HF_TOKEN") or None


def pick_dtype():
    if DTYPE != "auto":
        return {"float32": torch.float32, "bfloat16": torch.bfloat16, "float16": torch.float16}[DTYPE]
    if DEVICE == "cuda":
        return torch.bfloat16 if torch.cuda.get_device_capability()[0] >= 8 else torch.float16
    return torch.float32


class StopOnEvent(StoppingCriteria):
    """Ends generate() as soon as the Stop button sets STOP_EVENT."""
    def __call__(self, input_ids, scores, **kwargs):
        return STOP_EVENT.is_set()


def load():
    token = None if os.path.isdir(MODEL_SOURCE) else get_token()
    dtype = pick_dtype()
    tok = AutoTokenizer.from_pretrained(MODEL_SOURCE, token=token)
    model = AutoModelForCausalLM.from_pretrained(MODEL_SOURCE, token=token, trust_remote_code=True,
                                                 torch_dtype=dtype).to(DEVICE).eval()
    return model, tok, dtype


MODEL, TOK, MODEL_DTYPE = load()
CFG = MODEL.config
N_PARAMS = sum(p.numel() for p in MODEL.parameters())
CONTEXT = CFG.max_position_embeddings
print(f"Loaded {MODEL_SOURCE}: {N_PARAMS / 1e6:.1f}M params · {DEVICE} · {MODEL_DTYPE}")


def generate(prompt, max_new, temperature, top_k, top_p, rep_penalty, seed):
    STOP_EVENT.clear()
    if not prompt:
        yield "", "Type a prompt first."
        return
    ids = TOK(prompt, return_tensors="pt", add_special_tokens=False).to(DEVICE)
    room = CONTEXT - ids["input_ids"].shape[1]
    if room <= 0:
        yield prompt, f"Prompt is longer than the model's {CONTEXT}-token context; shorten it."
        return
    seed = int(seed)
    if seed >= 0:
        torch.manual_seed(seed)
    greedy = temperature <= 0
    sampling = {} if greedy else dict(temperature=float(temperature), top_k=int(top_k), top_p=float(top_p))
    streamer = TextIteratorStreamer(TOK, skip_prompt=True, skip_special_tokens=True)
    result = {}

    def worker():
        with torch.no_grad():
            result["ids"] = MODEL.generate(
                **ids, max_new_tokens=min(int(max_new), room), do_sample=not greedy,
                repetition_penalty=float(rep_penalty), eos_token_id=TOK.eos_token_id, pad_token_id=TOK.pad_token_id,
                use_cache=False, streamer=streamer, stopping_criteria=StoppingCriteriaList([StopOnEvent()]), **sampling)

    started = time.time()
    thread = threading.Thread(target=worker, daemon=True)
    thread.start()
    text = ""
    for piece in streamer:
        text += piece
        yield prompt + text, f"generating… {time.time() - started:.1f}s"
    thread.join()
    new = result["ids"].shape[1] - ids["input_ids"].shape[1] if "ids" in result else 0
    took = time.time() - started
    status = "stopped" if STOP_EVENT.is_set() else "done"
    yield prompt + text, (f"**{status}** · {new} tokens · {took:.1f}s · {new / max(took, 1e-6):.1f} tok/s · "
                          f"{'greedy' if greedy else f'T={temperature} · top-k {int(top_k)} · top-p {top_p}'} · seed {seed}")


def output_box():
    kwargs = dict(label="Output (prompt + continuation)", lines=16, max_lines=30, interactive=False)
    for extra in (dict(show_copy_button=True), dict(buttons=["copy"]), {}):
        try:
            return gr.Textbox(**kwargs, **extra)
        except TypeError:
            continue


def build_ui():
    gr_major = int(gr.__version__.split(".")[0])
    theme = gr.themes.Soft(primary_hue="indigo", secondary_hue="blue")
    blocks_kwargs = {} if gr_major >= 6 else dict(theme=theme)     # Gradio 6 moved theme to launch()
    launch_kwargs = dict(theme=theme) if gr_major >= 6 else {}
    with gr.Blocks(title=MODEL_DISPLAY_NAME, **blocks_kwargs) as demo:
        gr.Markdown(f"# 🔥 {MODEL_DISPLAY_NAME}\n"
                    f"{N_PARAMS / 1e6:.1f}M-parameter math-first base model · `{CFG.num_hidden_layers}L × {CFG.hidden_size}d` · "
                    f"{CFG.num_attention_heads}Q/{CFG.num_key_value_heads}KV heads · context {CONTEXT} · "
                    f"continues text (not a chat model)")
        with gr.Row():
            with gr.Column(scale=3):
                prompt = gr.Textbox(label="Prompt", lines=6, placeholder="Write the beginning of the text to continue…")
                with gr.Row():
                    run = gr.Button("Generate", variant="primary")
                    stop = gr.Button("Stop", variant="stop")
                output = output_box()
                status = gr.Markdown("")
            with gr.Column(scale=1):
                max_new = gr.Slider(16, 1024, value=DEF_MAX_NEW_TOKENS, step=16, label="Max new tokens")
                temperature = gr.Slider(0.0, 1.5, value=DEF_TEMPERATURE, step=0.05, label="Temperature (0 = greedy)")
                top_k = gr.Slider(0, 200, value=DEF_TOP_K, step=1, label="Top-k (0 = off)")
                top_p = gr.Slider(0.05, 1.0, value=DEF_TOP_P, step=0.01, label="Top-p")
                rep = gr.Slider(1.0, 1.5, value=DEF_REPETITION_PENALTY, step=0.01, label="Repetition penalty")
                seed = gr.Number(value=DEF_SEED, precision=0, label="Seed (-1 = random)")
                with gr.Accordion("Model info", open=False):
                    gr.Markdown(f"- **Source:** `{MODEL_SOURCE}`\n- **Parameters:** `{N_PARAMS:,}`\n"
                                f"- **Device / dtype:** `{DEVICE}` / `{str(MODEL_DTYPE).replace('torch.', '')}`\n"
                                f"- **Vocab:** `{CFG.vocab_size:,}` · EOS `{TOK.eos_token}`")
        inputs = [prompt, max_new, temperature, top_k, top_p, rep, seed]
        run.click(generate, inputs=inputs, outputs=[output, status])
        prompt.submit(generate, inputs=inputs, outputs=[output, status])
        stop.click(lambda: STOP_EVENT.set(), queue=False)
    return demo, launch_kwargs


if __name__ == "__main__":
    demo, launch_kwargs = build_ui()
    demo.queue().launch(share=GRADIO_SHARE, server_port=GRADIO_PORT, server_name="0.0.0.0", **launch_kwargs)

Tips: it is a base model, so write the beginning of the text you want rather than a question to a chatbot, for example Sam had 47 marbles. His sister gave him 28 more. Now Sam has. Lower the temperature (or set it to 0) for arithmetic; raise it for stories.

Evaluation

All scores are zero-shot on the checkpoint in this repo. The ArithMark benchmarks were scored with AxiomicLabs' official scripts, unmodified; the language benchmarks with the lm-evaluation-harness.

Benchmark	Metric	Score	Standard error	Chance
ArithMark 3.0 (word problems)	acc_norm	83.10%	±1.19	25%
HellaSwag	acc_norm	29.80%	±0.46	25%
ARC-Easy	acc_norm	47.01%	±1.02	25%
ARC-Challenge	acc_norm	30.12%	±1.34	25%
PIQA	acc_norm	60.45%	±1.14	50%

On the Open SLM Leaderboard's Intelligence Index formula these scores give 26.23.

Inferno-Exp-1.6 is math-first, and the scores show it: it is strong on arithmetic word problems but modest on general commonsense and reading benchmarks, where its narrower training data gives it less to draw on. Upcoming versions keep the math foundation and broaden the training mix to lift these language and reasoning scores.

ArithMark 3.0

Scored with bencharithmark-3.py on ArithMark 3.0 (1,000 four-choice continuation word problems, grades 1–5) at commit 6f6e59dd: float32, batch size 32, max context 1,024, no BOS token, length-normalized log-likelihood as the primary metric.

Metric	Score
Length-normalized accuracy (acc_norm, primary)	83.10%
Raw continuation accuracy	83.50%
Standard error (acc_norm)	±1.19 points
Random-choice baseline	25.00%

Per-category results (17 categories)

Category	Items	Raw	acc_norm
Addition (grades 1–2)	128	91.41%	92.19%
Comparison	44	100.00%	100.00%
Comparison: difference	48	93.75%	93.75%
Data	43	100.00%	100.00%
Division	54	88.89%	88.89%
Fractions: counting	50	56.00%	56.00%
Geometry: area	52	90.38%	88.46%
Geometry: perimeter	45	71.11%	71.11%
Measurement	76	98.68%	98.68%
Money	64	96.88%	96.88%
Multiplication	74	91.89%	87.84%
Patterns	53	64.15%	64.15%
Subtraction (grades 1–2)	117	79.49%	78.63%
Time	55	100.00%	100.00%
Two-step add/subtract	46	34.78%	34.78%
Two-step addition	19	78.95%	78.95%
Two-step subtraction	32	40.62%	40.62%

Language understanding (lm-evaluation-harness)

Zero-shot, acc_norm, float32, batch size 16:

lm_eval --model hf \
  --model_args "pretrained=AlgoDriveAI/Inferno-Exp-1.6,trust_remote_code=True,dtype=float32" \
  --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0 --batch_size 16

Task	Items	acc_norm
HellaSwag	10,042	29.80% ± 0.46
ARC-Easy	2,376	47.01% ± 1.02
ARC-Challenge	1,172	30.12% ± 1.34
PIQA	1,838	60.45% ± 1.14

Notes

• Generation re-reads the full context each step (no KV cache); fine at this size.
• The tokenizer does not add a BOS token; its bos_token is set to <|endoftext|>, the document boundary the model saw during training, for tools that prepend one.