Blame for README.md

---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- small-language-model
- gqa
- custom_code
- arithmark
model-index:
- name: Inferno-Exp-1.6
  results:
  - task:
      type: text-generation
      name: Arithmetic word problems (continuation log-likelihood)
    dataset:
      name: ArithMark 3.0
      type: AxiomicLabs/Arithmark-3.0
      revision: 6f6e59dd9b7e2c63455f7af7f838f9ecc3d0a746
    metrics:
    - type: acc_norm
      value: 83.10
      name: Length-normalized accuracy
    - type: acc
      value: 83.50
      name: Raw continuation accuracy
    source:
      name: AxiomicLabs bencharithmark-3.py (official script, float32)
      url: https://huggingface.co/datasets/AxiomicLabs/Arithmark-3.0
  - task:
      type: text-generation
      name: Integer arithmetic (continuation log-likelihood)
    metrics:
    - type: acc
      value: 56.88
      name: Accuracy
    source:
      name: AxiomicLabs benchmark_arithmark-2.0.py (official script, bfloat16)
      url: https://huggingface.co/datasets/AxiomicLabs/ArithMark-2.0
  - task:
      type: text-generation
      name: Commonsense sentence completion
    dataset:
      name: HellaSwag
      type: Rowan/hellaswag
      split: validation
    metrics:
    - type: acc_norm
      value: 29.80
      name: Normalized accuracy (0-shot)
    source:
      name: lm-evaluation-harness (0-shot, float32)
      url: https://github.com/EleutherAI/lm-evaluation-harness
  - task:
      type: text-generation
      name: Science question answering
    dataset:
      name: ARC-Easy
      type: allenai/ai2_arc
      config: ARC-Easy
      split: test
    metrics:
    - type: acc_norm
      value: 47.01
      name: Normalized accuracy (0-shot)
    source:
      name: lm-evaluation-harness (0-shot, float32)
      url: https://github.com/EleutherAI/lm-evaluation-harness
  - task:
      type: text-generation
      name: Science question answering
    dataset:
      name: ARC-Challenge
      type: allenai/ai2_arc
      config: ARC-Challenge
      split: test
    metrics:
    - type: acc_norm
      value: 30.12
      name: Normalized accuracy (0-shot)
    source:
      name: lm-evaluation-harness (0-shot, float32)
      url: https://github.com/EleutherAI/lm-evaluation-harness
  - task:
      type: text-generation
      name: Physical commonsense reasoning
    dataset:
      name: PIQA
      type: ybisk/piqa
      split: validation
    metrics:
    - type: acc_norm
      value: 60.45
      name: Normalized accuracy (0-shot)
    source:
      name: lm-evaluation-harness (0-shot, float32)
      url: https://github.com/EleutherAI/lm-evaluation-harness
---

# Inferno-Exp-1.6

**Inferno-Exp-1.6** is a 393.6M-parameter decoder-only base language model trained from scratch by AlgoDriveAI
on about 4B tokens. It is a math-first model: its training emphasizes arithmetic and math word problems,
and it is geared toward the AxiomicLabs small-language-model benchmarks used in SLM challenges, ArithMark 3.0. Future versions will build on this math foundation with improvements aimed
at broader language understanding and logical reasoning.

| | |
|---|---|
| Parameters | **393,610,240 (393.6M)** |
| Layers × width | 32 × 1024 (deep and narrow) |
| Attention | grouped-query, 16 query / 4 KV heads, head dim 64, full-head RoPE |
| Feed-forward | plain SwiGLU, inner width 2816 |
| Context | 2048 tokens |
| Tokenizer | Mistral-7B-v0.1 (padded to 32,064); `<|endoftext|>` separates documents |
| Embeddings | input and output tied |
| Training | from scratch on AlgoDriveAI/SLM_v3 (expanded), ~4B tokens (one pass); AdamW, WSD schedule |

**Parameter accounting:** 32,833,536 tied token-embedding parameters + 360,775,680
parameters across 32 transformer blocks + 1,024 final normalization parameters
= **393,610,240 unique trainable parameters**. The tied LM head is not counted twice.

This is a **base model** (no instruction tuning). The model code ships in the repo, so load it with
`trust_remote_code=True`.

## Compared with Inferno-Exp-1.5 (76M)

Same architecture family, about 5.17× the parameters and 32 layers instead of 14.

| Benchmark | Inferno-Exp-1.5 | **Inferno-Exp-1.6** | Change |
|---|---|---|---|
| ArithMark 3.0 (acc_norm) | 71.00% | **83.10%** | +12.10 |
| HellaSwag | 27.85% | **29.80%** | +1.95 |
| ARC-Easy | 43.39% | **47.01%** | +3.62 |
| ARC-Challenge | 26.62% | **30.12%** | +3.50 |
| PIQA | 58.49% | **60.45%** | +1.96 |
| Open SLM Leaderboard Intelligence Index | 20.27 | **26.23** | +5.96 |

Word-problem arithmetic and every language benchmark improve; bare-equation arithmetic (ArithMark 2.0)
is lower than in Inferno-Exp-1.5.

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("AlgoDriveAI/Inferno-Exp-1.6")
model = AutoModelForCausalLM.from_pretrained("AlgoDriveAI/Inferno-Exp-1.6", trust_remote_code=True)

ids = tok("Once upon a time, a little fox", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=80, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))
```

## Try it: interactive demo (Gradio)

The easiest way to play with the model is a small web app with a prompt box, sampling sliders,
live streaming, and a Stop button:

1. Open a new notebook at [colab.research.google.com](https://colab.research.google.com) and choose a GPU
   (Runtime → Change runtime type → T4 GPU). It also runs on CPU, just slower.
2. Paste the script below into one cell and run it.
3. Click the `*.gradio.live` link it prints.

If it stops and asks you to restart the session (it may need to switch Transformers to a compatible
version), choose Runtime → Restart session and run the cell again.

<details>
<summary><b>Show the Gradio demo script</b> (one Colab cell, ~200 lines)</summary>

```python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Inferno-Exp-1.6 — simple Gradio inference (loads like any Hub user, no pushing)
==============================================================================
Loads Inferno-Exp-1.6 with plain Transformers (AutoModelForCausalLM +
trust_remote_code=True) and serves a small Gradio app: open prompt box,
sampling controls, streaming output with a Stop button, token/speed readout.

Works with either source:
  • the Hub repo:          MODEL_SOURCE = "AlgoDriveAI/Inferno-Exp-1.6"
  • the local export folder the push script builds (test before uploading):
                           MODEL_SOURCE = "/content/hf_export_inferno_exp_1_0"

Private repo? Add a Colab secret named HF_TOKEN (🔑 icon in the left sidebar, turn on
notebook access), set the env var HF_TOKEN, or paste a READ token into HF_TOKEN below.

It is a base model: it continues text, it does not chat. Write the start of the text
you want (e.g. a word problem followed by "Answer:").
Hardcoded constants, no argparse. Paste into one Colab cell and run; open the share link.
"""
# =============================================================================
# SETTINGS — edit these
# =============================================================================
MODEL_SOURCE = "AlgoDriveAI/Inferno-Exp-1.6"   # Hub repo id (or a local folder with the same files)
HF_TOKEN = ""                                  # only for private repos; "" → Colab secret / env var
MODEL_DISPLAY_NAME = "Inferno-Exp-1.6"

# Default slider values (all adjustable in the UI)
DEF_MAX_NEW_TOKENS = 256
DEF_TEMPERATURE = 0.8
DEF_TOP_K = 40
DEF_TOP_P = 0.95
DEF_REPETITION_PENALTY = 1.1
DEF_SEED = -1             # -1 = different every run
DTYPE = "auto"            # "auto" (bf16 on newer GPUs, fp32 on CPU) / "float32" / "bfloat16" / "float16"

GRADIO_SHARE = True       # public *.gradio.live link (needed in Colab)
GRADIO_PORT = 7860
INSTALL_DEPENDENCIES = True

# =============================================================================
# RUN (no edits needed below)
# =============================================================================
import os, sys, time, threading, subprocess

try:
    import google.colab  # noqa: F401
    IN_COLAB = True
except ImportError:
    IN_COLAB = False
if IN_COLAB and INSTALL_DEPENDENCIES:
    import importlib.metadata as _md

    def _version(name):
        try:
            return _md.version(name)
        except _md.PackageNotFoundError:
            return None

    _before = _version("transformers")
    subprocess.check_call([sys.executable, "-m", "pip", "install", "--quiet",
                           "transformers>=4.45,<5", "safetensors", "huggingface_hub>=0.25", "gradio"])
    if _before != _version("transformers") and "transformers" in sys.modules:
        raise SystemExit("Transformers was just changed to a compatible version, but the old one is still loaded. "
                         "Restart the session (Runtime → Restart session) and run this cell again.")

import torch
import gradio as gr
from transformers import (AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList,
                          TextIteratorStreamer)

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
STOP_EVENT = threading.Event()


def get_token():
    if HF_TOKEN.strip():
        return HF_TOKEN.strip()
    if IN_COLAB:
        try:
            from google.colab import userdata
            value = userdata.get("HF_TOKEN")
            if value:
                return value
        except Exception:
            pass
    return os.environ.get("HF_TOKEN") or None


def pick_dtype():
    if DTYPE != "auto":
        return {"float32": torch.float32, "bfloat16": torch.bfloat16, "float16": torch.float16}[DTYPE]
    if DEVICE == "cuda":
        return torch.bfloat16 if torch.cuda.get_device_capability()[0] >= 8 else torch.float16
    return torch.float32


class StopOnEvent(StoppingCriteria):
    """Ends generate() as soon as the Stop button sets STOP_EVENT."""
    def __call__(self, input_ids, scores, **kwargs):
        return STOP_EVENT.is_set()


def load():
    token = None if os.path.isdir(MODEL_SOURCE) else get_token()
    dtype = pick_dtype()
    tok = AutoTokenizer.from_pretrained(MODEL_SOURCE, token=token)
    model = AutoModelForCausalLM.from_pretrained(MODEL_SOURCE, token=token, trust_remote_code=True,
                                                 torch_dtype=dtype).to(DEVICE).eval()
    return model, tok, dtype


MODEL, TOK, MODEL_DTYPE = load()
CFG = MODEL.config
N_PARAMS = sum(p.numel() for p in MODEL.parameters())
CONTEXT = CFG.max_position_embeddings
print(f"Loaded {MODEL_SOURCE}: {N_PARAMS / 1e6:.1f}M params · {DEVICE} · {MODEL_DTYPE}")


def generate(prompt, max_new, temperature, top_k, top_p, rep_penalty, seed):
    STOP_EVENT.clear()
    if not prompt:
        yield "", "Type a prompt first."
        return
    ids = TOK(prompt, return_tensors="pt", add_special_tokens=False).to(DEVICE)
    room = CONTEXT - ids["input_ids"].shape[1]
    if room <= 0:
        yield prompt, f"Prompt is longer than the model's {CONTEXT}-token context; shorten it."
        return
    seed = int(seed)
    if seed >= 0:
        torch.manual_seed(seed)
    greedy = temperature <= 0
    sampling = {} if greedy else dict(temperature=float(temperature), top_k=int(top_k), top_p=float(top_p))
    streamer = TextIteratorStreamer(TOK, skip_prompt=True, skip_special_tokens=True)
    result = {}

    def worker():
        with torch.no_grad():
            result["ids"] = MODEL.generate(
                **ids, max_new_tokens=min(int(max_new), room), do_sample=not greedy,
                repetition_penalty=float(rep_penalty), eos_token_id=TOK.eos_token_id, pad_token_id=TOK.pad_token_id,
                use_cache=False, streamer=streamer, stopping_criteria=StoppingCriteriaList([StopOnEvent()]), **sampling)

    started = time.time()
    thread = threading.Thread(target=worker, daemon=True)
    thread.start()
    text = ""
    for piece in streamer:
        text += piece
        yield prompt + text, f"generating… {time.time() - started:.1f}s"
    thread.join()
    new = result["ids"].shape[1] - ids["input_ids"].shape[1] if "ids" in result else 0
    took = time.time() - started
    status = "stopped" if STOP_EVENT.is_set() else "done"
    yield prompt + text, (f"**{status}** · {new} tokens · {took:.1f}s · {new / max(took, 1e-6):.1f} tok/s · "
                          f"{'greedy' if greedy else f'T={temperature} · top-k {int(top_k)} · top-p {top_p}'} · seed {seed}")


def output_box():
    kwargs = dict(label="Output (prompt + continuation)", lines=16, max_lines=30, interactive=False)
    for extra in (dict(show_copy_button=True), dict(buttons=["copy"]), {}):
        try:
            return gr.Textbox(**kwargs, **extra)
        except TypeError:
            continue


def build_ui():
    gr_major = int(gr.__version__.split(".")[0])
    theme = gr.themes.Soft(primary_hue="indigo", secondary_hue="blue")
    blocks_kwargs = {} if gr_major >= 6 else dict(theme=theme)     # Gradio 6 moved theme to launch()
    launch_kwargs = dict(theme=theme) if gr_major >= 6 else {}
    with gr.Blocks(title=MODEL_DISPLAY_NAME, **blocks_kwargs) as demo:
        gr.Markdown(f"# 🔥 {MODEL_DISPLAY_NAME}\n"
                    f"{N_PARAMS / 1e6:.1f}M-parameter math-first base model · `{CFG.num_hidden_layers}L × {CFG.hidden_size}d` · "
                    f"{CFG.num_attention_heads}Q/{CFG.num_key_value_heads}KV heads · context {CONTEXT} · "
                    f"continues text (not a chat model)")
        with gr.Row():
            with gr.Column(scale=3):
                prompt = gr.Textbox(label="Prompt", lines=6, placeholder="Write the beginning of the text to continue…")
                with gr.Row():
                    run = gr.Button("Generate", variant="primary")
                    stop = gr.Button("Stop", variant="stop")
                output = output_box()
                status = gr.Markdown("")
            with gr.Column(scale=1):
                max_new = gr.Slider(16, 1024, value=DEF_MAX_NEW_TOKENS, step=16, label="Max new tokens")
                temperature = gr.Slider(0.0, 1.5, value=DEF_TEMPERATURE, step=0.05, label="Temperature (0 = greedy)")
                top_k = gr.Slider(0, 200, value=DEF_TOP_K, step=1, label="Top-k (0 = off)")
                top_p = gr.Slider(0.05, 1.0, value=DEF_TOP_P, step=0.01, label="Top-p")
                rep = gr.Slider(1.0, 1.5, value=DEF_REPETITION_PENALTY, step=0.01, label="Repetition penalty")
                seed = gr.Number(value=DEF_SEED, precision=0, label="Seed (-1 = random)")
                with gr.Accordion("Model info", open=False):
                    gr.Markdown(f"- **Source:** `{MODEL_SOURCE}`\n- **Parameters:** `{N_PARAMS:,}`\n"
                                f"- **Device / dtype:** `{DEVICE}` / `{str(MODEL_DTYPE).replace('torch.', '')}`\n"
                                f"- **Vocab:** `{CFG.vocab_size:,}` · EOS `{TOK.eos_token}`")
        inputs = [prompt, max_new, temperature, top_k, top_p, rep, seed]
        run.click(generate, inputs=inputs, outputs=[output, status])
        prompt.submit(generate, inputs=inputs, outputs=[output, status])
        stop.click(lambda: STOP_EVENT.set(), queue=False)
    return demo, launch_kwargs


if __name__ == "__main__":
    demo, launch_kwargs = build_ui()
    demo.queue().launch(share=GRADIO_SHARE, server_port=GRADIO_PORT, server_name="0.0.0.0", **launch_kwargs)
```

</details>

Tips: it is a base model, so write the beginning of the text you want rather than a question to a
chatbot, for example `Sam had 47 marbles. His sister gave him 28 more. Now Sam has`. Lower the
temperature (or set it to 0) for arithmetic; raise it for stories.

## Evaluation

All scores are zero-shot on the checkpoint in this repo. The ArithMark benchmarks were scored with
AxiomicLabs' official scripts, unmodified; the language benchmarks with the
[lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness).

| Benchmark | Metric | Score | Standard error | Chance |
|---|---|---|---|---|
| **ArithMark 3.0** (word problems) | acc_norm | **83.10%** | ±1.19 | 25% |
| HellaSwag | acc_norm | 29.80% | ±0.46 | 25% |
| ARC-Easy | acc_norm | 47.01% | ±1.02 | 25% |
| ARC-Challenge | acc_norm | 30.12% | ±1.34 | 25% |
| PIQA | acc_norm | 60.45% | ±1.14 | 50% |

On the [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard)'s
Intelligence Index formula these scores give **26.23**.

Inferno-Exp-1.6 is math-first, and the scores show it: it is strong on arithmetic word problems but modest on general
commonsense and reading benchmarks, where its narrower training data gives it less to draw on. Upcoming
versions keep the math foundation and broaden the training mix to lift these language and reasoning scores.

### ArithMark 3.0

Scored with `bencharithmark-3.py` on
[ArithMark 3.0](https://huggingface.co/datasets/AxiomicLabs/Arithmark-3.0) (1,000 four-choice
continuation word problems, grades 1–5) at commit `6f6e59dd`: float32, batch size 32,
max context 1,024, no BOS token, length-normalized log-likelihood as the primary metric.

| Metric | Score |
|---|---|
| **Length-normalized accuracy (acc_norm, primary)** | **83.10%** |
| Raw continuation accuracy | 83.50% |
| Standard error (acc_norm) | ±1.19 points |
| Random-choice baseline | 25.00% |

<details>
<summary><b>Per-category results</b> (17 categories)</summary>

| Category | Items | Raw | acc_norm |
|---|---|---|---|
| Addition (grades 1–2) | 128 | 91.41% | 92.19% |
| Comparison | 44 | 100.00% | 100.00% |
| Comparison: difference | 48 | 93.75% | 93.75% |
| Data | 43 | 100.00% | 100.00% |
| Division | 54 | 88.89% | 88.89% |
| Fractions: counting | 50 | 56.00% | 56.00% |
| Geometry: area | 52 | 90.38% | 88.46% |
| Geometry: perimeter | 45 | 71.11% | 71.11% |
| Measurement | 76 | 98.68% | 98.68% |
| Money | 64 | 96.88% | 96.88% |
| Multiplication | 74 | 91.89% | 87.84% |
| Patterns | 53 | 64.15% | 64.15% |
| Subtraction (grades 1–2) | 117 | 79.49% | 78.63% |
| Time | 55 | 100.00% | 100.00% |
| Two-step add/subtract | 46 | 34.78% | 34.78% |
| Two-step addition | 19 | 78.95% | 78.95% |
| Two-step subtraction | 32 | 40.62% | 40.62% |

</details>

### Language understanding (lm-evaluation-harness)

Zero-shot, `acc_norm`, float32, batch size 16:

```bash
lm_eval --model hf \
  --model_args "pretrained=AlgoDriveAI/Inferno-Exp-1.6,trust_remote_code=True,dtype=float32" \
  --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0 --batch_size 16
```

| Task | Items | acc_norm |
|---|---|---|
| HellaSwag | 10,042 | 29.80% ± 0.46 |
| ARC-Easy | 2,376 | 47.01% ± 1.02 |
| ARC-Challenge | 1,172 | 30.12% ± 1.34 |
| PIQA | 1,838 | 60.45% ± 1.14 |

## Notes

- Generation re-reads the full context each step (no KV cache); fine at this size.
- The tokenizer does not add a BOS token; its `bos_token` is set to `<|endoftext|>`, the document
  boundary the model saw during training, for tools that prepend one.