Blame for README.md

---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- inferno
- small-language-model
- gqa
- custom_code
- arithmark
datasets:
- AlgoDriveAI/SLM_v6
model-index:
- name: Inferno-Exp-1.7
  results:
  - task:
      type: text-generation
      name: Arithmetic word-problem continuation
    dataset:
      type: AxiomicLabs/Arithmark-3.0
      name: ArithMark 3.0
      split: test
      revision: 6f6e59dd9b7e2c63455f7af7f838f9ecc3d0a746
    metrics:
    - type: acc_norm
      name: Length-normalized accuracy
      value: 85.7
    - type: acc
      name: Raw continuation accuracy
      value: 85.6
  - task:
      type: text-generation
      name: Zero-shot continuation scoring
    dataset:
      type: Rowan/hellaswag
      name: HellaSwag
      split: validation
    metrics:
    - type: acc_norm
      name: Normalized accuracy (0-shot)
      value: 33.68
  - task:
      type: text-generation
      name: Zero-shot continuation scoring
    dataset:
      type: allenai/ai2_arc
      name: ARC-Easy
      split: test
    metrics:
    - type: acc_norm
      name: Normalized accuracy (0-shot)
      value: 51.6
  - task:
      type: text-generation
      name: Zero-shot continuation scoring
    dataset:
      type: allenai/ai2_arc
      name: ARC-Challenge
      split: test
    metrics:
    - type: acc_norm
      name: Normalized accuracy (0-shot)
      value: 34.04
  - task:
      type: text-generation
      name: Zero-shot continuation scoring
    dataset:
      type: ybisk/piqa
      name: PIQA
      split: validation
    metrics:
    - type: acc_norm
      name: Normalized accuracy (0-shot)
      value: 64.47
---

# Inferno-Exp-1.7

**Inferno-Exp-1.7** is AlgoDriveAI's 393.6M-parameter, math-first decoder-only base language model.
This release publishes the selected `32_indx.pt` snapshot at **continuation step 275,000**,
with **4,096-token context** and **85.70% ArithMark 3.0 normalized accuracy**.
It continues the Deep400M training line represented by [Inferno-Exp-1.6](https://huggingface.co/AlgoDriveAI/Inferno-Exp-1.6).
It is a base text-completion model; it has no chat template or instruction tuning.

| Property | Value |
|---|---|
| Unique parameters | 393,610,240 |
| Layers × width | 32 × 1024 |
| Attention | GQA: 16 query / 4 KV heads, head dimension 64 |
| Position encoding | Full-head, half-split RoPE; base 10000 |
| Feed-forward | Canonical same-token SwiGLU; inner width 2816 |
| Context | 4,096 tokens |
| Vocabulary | 32,064; Mistral tokenizer with training additions and padding |
| Embeddings | Input/output weights tied |
| Released weights | Float32 Safetensors |
| Source checkpoint | `32_indx.pt`, step 275,000, saved epoch index 1 |

Parameter accounting: 32,833,536 tied token-embedding parameters + 360,775,680 parameters
across 32 transformer blocks + 1,024 final normalization parameters = **393,610,240**.
The LM head is not counted twice.

## Compared with Inferno-Exp-1.6

| Benchmark | Inferno-Exp-1.6 | Inferno-Exp-1.7 | Change (points) |
|---|---:|---:|---:|
| ArithMark 3.0 (acc_norm) | 83.10% | **85.70%** | +2.60 |
| HellaSwag (acc_norm) | 29.80% | **33.68%** | +3.88 |
| ARC-Easy (acc_norm) | 47.01% | **51.60%** | +4.59 |
| ARC-Challenge (acc_norm) | 30.12% | **34.04%** | +3.92 |
| PIQA (acc_norm) | 60.45% | **64.47%** | +4.02 |

The ArithMark result was verified during this export. The four language scores are the full-set
training evaluations recorded at the same step, 275,000. They are not a new post-export harness run.
The previous release reports language evaluation batch size 16; the training evaluations here use 8.

Using these rounded scores and the training script's Open SLM Intelligence Index formula gives
**32.02**. This is a locally calculated summary, not an official leaderboard submission or rank.

## Usage

```python
# pip install "transformers==4.57.6" torch sentencepiece protobuf safetensors
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "AlgoDriveAI/Inferno-Exp-1.7"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True,
                                            torch_dtype=torch.float32)
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device).eval()
inputs = tok("Sam had 47 marbles. His sister gave him 28 more. Now Sam has",
             return_tensors="pt", add_special_tokens=False).to(device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=80, do_sample=False,
                            use_cache=False, pad_token_id=tok.pad_token_id)
print(tok.decode(output[0], skip_special_tokens=True))
```

The custom Transformers implementation is reused from the pinned 1.6 release, with this
checkpoint's 4,096-token configuration and new weights. The export checks strict tensor loading,
compares its logits with the supplied trainer, reloads the saved Safetensors, and scores the
exported model on ArithMark. Generation re-reads the context each step; no KV cache is implemented.

## Try it: interactive demo (Gradio)

Run a small web app with a prompt box, sampling sliders, live streaming, a **Stop** button,
and token/speed readouts:

1. Open a new notebook at [Google Colab](https://colab.research.google.com/) and choose a GPU
   (**Runtime → Change runtime type → T4 GPU**). CPU also works, but generation is slower.
2. Expand the script below, paste it into one code cell, and run it.
3. Open the `*.gradio.live` link printed by the cell.

The script loads the published Inferno-Exp-1.7 checkpoint and reads its **4,096-token context**
from the model configuration. The public model needs no Hugging Face token.
If the cell asks for a restart after changing Transformers, use **Runtime → Restart session**
and run that cell again.

<details>
<summary>Show the Gradio demo script (one Colab cell)</summary>

```python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Inferno-Exp-1.7 — simple Gradio inference (loads like any Hub user, no pushing)
==============================================================================
Loads Inferno-Exp-1.7 with plain Transformers (AutoModelForCausalLM +
trust_remote_code=True) and serves a small Gradio app: open prompt box,
sampling controls, streaming output with a Stop button, token/speed readout.

Works with either source:
  • the Hub repo:          MODEL_SOURCE = "AlgoDriveAI/Inferno-Exp-1.7"
  • the local export folder the push script builds (test before uploading):
                           MODEL_SOURCE = "/content/Inferno-Exp-1.7-release"

The public 1.7 model needs no token. For a private alternative, use an existing
Hub login, set the HF_TOKEN environment variable, or set HF_TOKEN below.

It is a base model: it continues text, it does not chat. Write the start of the text
you want (e.g. a word problem followed by "Answer:").
Hardcoded constants, no argparse. Paste into one Colab cell and run; open the share link.
"""
# =============================================================================
# SETTINGS — edit these
# =============================================================================
MODEL_SOURCE = "AlgoDriveAI/Inferno-Exp-1.7"   # Hub repo id (or a local folder with the same files)
MODEL_REVISION = "a24c67451dc7df3b3069e9ce54724ca815d96fda"  # verified 1.7 weights
HF_TOKEN = ""                                  # optional; otherwise use env var / existing Hub login
MODEL_DISPLAY_NAME = "Inferno-Exp-1.7"

# Default slider values (all adjustable in the UI)
DEF_MAX_NEW_TOKENS = 256
DEF_TEMPERATURE = 0.8
DEF_TOP_K = 40
DEF_TOP_P = 0.95
DEF_REPETITION_PENALTY = 1.1
DEF_SEED = -1             # -1 = different every run
DTYPE = "auto"            # "auto" (bf16 on newer GPUs, fp32 on CPU) / "float32" / "bfloat16" / "float16"

GRADIO_SHARE = True       # public *.gradio.live link (needed in Colab)
GRADIO_PORT = 7860
INSTALL_DEPENDENCIES = True

# =============================================================================
# RUN (no edits needed below)
# =============================================================================
import os, sys, time, threading, subprocess, queue
os.environ.setdefault("USE_TF", "0")
os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")

try:
    import google.colab  # noqa: F401
    IN_COLAB = True
except ImportError:
    IN_COLAB = False
if IN_COLAB and INSTALL_DEPENDENCIES:
    import importlib.metadata as _md

    def _version(name):
        try:
            return _md.version(name)
        except _md.PackageNotFoundError:
            return None

    _before = _version("transformers")
    subprocess.check_call([sys.executable, "-m", "pip", "install", "--quiet",
                           "transformers==4.57.6", "safetensors", "huggingface_hub>=0.25,<1",
                           "gradio>=4.44,<7", "sentencepiece", "protobuf"])
    if _before != _version("transformers") and "transformers" in sys.modules:
        raise SystemExit("Transformers was just changed to a compatible version, but the old one is still loaded. "
                         "Restart the session (Runtime → Restart session) and run this cell again.")

import torch
import gradio as gr
from transformers import (AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList,
                          TextIteratorStreamer)

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
STOP_EVENT = threading.Event()


def get_token():
    # token=None lets the Hub use an existing login; public downloads work without one.
    return HF_TOKEN.strip() or os.environ.get("HF_TOKEN") or None


def pick_dtype():
    if DTYPE != "auto":
        return {"float32": torch.float32, "bfloat16": torch.bfloat16, "float16": torch.float16}[DTYPE]
    if DEVICE == "cuda":
        return torch.bfloat16 if torch.cuda.get_device_capability()[0] >= 8 else torch.float16
    return torch.float32


class StopOnEvent(StoppingCriteria):
    """Ends generate() as soon as the Stop button sets STOP_EVENT."""
    def __call__(self, input_ids, scores, **kwargs):
        return STOP_EVENT.is_set()


def load():
    token = None if os.path.isdir(MODEL_SOURCE) else get_token()
    dtype = pick_dtype()
    revision = None if os.path.isdir(MODEL_SOURCE) else MODEL_REVISION
    tok = AutoTokenizer.from_pretrained(MODEL_SOURCE, revision=revision, token=token, use_fast=True)
    if tok.pad_token_id is None:
        tok.pad_token = tok.eos_token
    model = AutoModelForCausalLM.from_pretrained(MODEL_SOURCE, revision=revision, token=token, trust_remote_code=True,
                                                 torch_dtype=dtype).to(DEVICE).eval()
    return model, tok, dtype


MODEL, TOK, MODEL_DTYPE = load()
CFG = MODEL.config
N_PARAMS = sum(p.numel() for p in MODEL.parameters())
CONTEXT = CFG.max_position_embeddings
print(f"Loaded {MODEL_SOURCE}: {N_PARAMS / 1e6:.1f}M params · {DEVICE} · {MODEL_DTYPE}")


def generate(prompt, max_new, temperature, top_k, top_p, rep_penalty, seed):
    STOP_EVENT.clear()
    if not prompt:
        yield "", "Type a prompt first."
        return
    ids = TOK(prompt, return_tensors="pt", add_special_tokens=False).to(DEVICE)
    room = CONTEXT - ids["input_ids"].shape[1]
    if room <= 0:
        yield prompt, f"Prompt is longer than the model's {CONTEXT}-token context; shorten it."
        return
    seed = int(seed)
    if seed >= 0:
        torch.manual_seed(seed)
    greedy = temperature <= 0
    sampling = {} if greedy else dict(temperature=float(temperature), top_k=int(top_k), top_p=float(top_p))
    streamer = TextIteratorStreamer(TOK, skip_prompt=False, skip_special_tokens=True, timeout=1.0)
    result = {}

    def worker():
        try:
            with torch.inference_mode():
                result["ids"] = MODEL.generate(
                    **ids, max_new_tokens=min(int(max_new), room), do_sample=not greedy,
                    repetition_penalty=float(rep_penalty), eos_token_id=TOK.eos_token_id, pad_token_id=TOK.pad_token_id,
                    use_cache=False, streamer=streamer, stopping_criteria=StoppingCriteriaList([StopOnEvent()]), **sampling)
        except Exception as exc:
            result["error"] = exc
            streamer.end()  # unblock the UI if the generation worker fails


    started = time.time()
    thread = threading.Thread(target=worker, daemon=True)
    thread.start()
    text = ""
    try:
        while True:
            try:
                piece = next(streamer)
            except queue.Empty:
                if not thread.is_alive():
                    break
                continue
            except StopIteration:
                break
            text += piece
            yield text, f"generating… {time.time() - started:.1f}s"
        thread.join()
    finally:
        if thread.is_alive():
            STOP_EVENT.set()
    if "error" in result:
        exc = result["error"]
        yield text or prompt, f"Generation failed: {type(exc).__name__}: {exc}"
        return
    new = result["ids"].shape[1] - ids["input_ids"].shape[1] if "ids" in result else 0
    took = time.time() - started
    status = "stopped" if STOP_EVENT.is_set() else "done"
    yield text or prompt, (f"**{status}** · {new} tokens · {took:.1f}s · {new / max(took, 1e-6):.1f} tok/s · "
                          f"{'greedy' if greedy else f'T={temperature} · top-k {int(top_k)} · top-p {top_p}'} · seed {seed}")


def output_box():
    kwargs = dict(label="Output (prompt + continuation)", lines=16, max_lines=30, interactive=False)
    for extra in (dict(show_copy_button=True), dict(buttons=["copy"]), {}):
        try:
            return gr.Textbox(**kwargs, **extra)
        except TypeError:
            continue


def build_ui():
    gr_major = int(gr.__version__.split(".")[0])
    theme = gr.themes.Soft(primary_hue="indigo", secondary_hue="blue")
    blocks_kwargs = {} if gr_major >= 6 else dict(theme=theme)     # Gradio 6 moved theme to launch()
    launch_kwargs = dict(theme=theme) if gr_major >= 6 else {}
    with gr.Blocks(title=MODEL_DISPLAY_NAME, **blocks_kwargs) as demo:
        gr.Markdown(f"# 🔥 {MODEL_DISPLAY_NAME}\n"
                    f"{N_PARAMS / 1e6:.1f}M-parameter math-first base model · `{CFG.num_hidden_layers}L × {CFG.hidden_size}d` · "
                    f"{CFG.num_attention_heads}Q/{CFG.num_key_value_heads}KV heads · context {CONTEXT} · "
                    f"continues text (not a chat model)")
        with gr.Row():
            with gr.Column(scale=3):
                prompt = gr.Textbox(label="Prompt", lines=6, placeholder="Write the beginning of the text to continue…")
                with gr.Row():
                    run = gr.Button("Generate", variant="primary")
                    stop = gr.Button("Stop", variant="stop")
                output = output_box()
                status = gr.Markdown("")
            with gr.Column(scale=1):
                max_new = gr.Slider(16, CONTEXT, value=min(DEF_MAX_NEW_TOKENS, CONTEXT), step=16, label="Max new tokens")
                temperature = gr.Slider(0.0, 1.5, value=DEF_TEMPERATURE, step=0.05, label="Temperature (0 = greedy)")
                top_k = gr.Slider(0, 200, value=DEF_TOP_K, step=1, label="Top-k (0 = off)")
                top_p = gr.Slider(0.05, 1.0, value=DEF_TOP_P, step=0.01, label="Top-p")
                rep = gr.Slider(1.0, 1.5, value=DEF_REPETITION_PENALTY, step=0.01, label="Repetition penalty")
                seed = gr.Number(value=DEF_SEED, precision=0, label="Seed (-1 = random)")
                with gr.Accordion("Model info", open=False):
                    gr.Markdown(f"- **Source:** `{MODEL_SOURCE}`\n- **Parameters:** `{N_PARAMS:,}`\n"
                                f"- **Device / dtype:** `{DEVICE}` / `{str(MODEL_DTYPE).replace('torch.', '')}`\n"
                                f"- **Vocab:** `{CFG.vocab_size:,}` · EOS `{TOK.eos_token}`")
        inputs = [prompt, max_new, temperature, top_k, top_p, rep, seed]
        events = dict(inputs=inputs, outputs=[output, status], concurrency_limit=1, concurrency_id="inferno-generation")
        run.click(generate, **events)
        prompt.submit(generate, **events)
        stop.click(lambda: STOP_EVENT.set(), queue=False)
    return demo, launch_kwargs


if __name__ == "__main__":
    demo, launch_kwargs = build_ui()
    demo.queue().launch(share=GRADIO_SHARE, server_port=GRADIO_PORT, server_name="0.0.0.0", **launch_kwargs)
```

</details>

[Download the same script](inference/gradio_demo.py).

For this base model, write the beginning of the text you want continued, for example:
`Sam had 47 marbles. His sister gave him 28 more. Now Sam has`.
Set temperature to **0** for greedy decoding, or increase it for more varied stories.

## Evaluation

All reported accuracies are percentages. Chance is 25% for ArithMark, HellaSwag and ARC;
50% for PIQA.

### ArithMark 3.0

The official, unmodified [AxiomicLabs scorer](https://huggingface.co/datasets/AxiomicLabs/Arithmark-3.0/blob/6f6e59dd9b7e2c63455f7af7f838f9ecc3d0a746/bencharithmark-3.py)
at commit `6f6e59dd9b7e2c63455f7af7f838f9ecc3d0a746` scores 1,000 four-choice continuation word problems.
Settings: float32 weights, TF32 enabled as in the official entry point, fast training tokenizer,
no added BOS token, batch size 8 examples (32 choices), max context 1,024,
raw summed and per-token length-normalized log-likelihood. No few-shot examples.

| Metric | Score |
|---|---:|
| Normalized accuracy (primary) | **85.70%** (857/1000) |
| Raw accuracy | **85.60%** (856/1000) |
| Binomial standard error (normalized) | ±1.11 points |
| Random-choice baseline | 25.00% |

Dataset SHA-256: `bf8ab1a5193d52cdf0e05ff0b3ca226bdfcf416cb6e75562dcbe72e7e4559435`.
Full scores, predictions, category data and runtime settings are in
[`evaluation/arithmark-3_results.json`](evaluation/arithmark-3_results.json).

#### Per-category results

| Category | N | Raw | Normalized |
|---|---:|---:|---:|
| elementary_school_math_continuation::addition::grades_1_2::easy | 128 | 90.62% | 90.62% |
| elementary_school_math_continuation::comparison::grades_2_3::medium | 44 | 100.00% | 100.00% |
| elementary_school_math_continuation::comparison_difference::grades_2_3::medium | 48 | 100.00% | 100.00% |
| elementary_school_math_continuation::data::grades_2_3::easy | 43 | 90.70% | 90.70% |
| elementary_school_math_continuation::division::grades_3_4::medium | 54 | 85.19% | 85.19% |
| elementary_school_math_continuation::fractions_counting::grades_3_4::medium | 50 | 58.00% | 58.00% |
| elementary_school_math_continuation::geometry_area::grades_4_5::medium | 52 | 100.00% | 100.00% |
| elementary_school_math_continuation::geometry_perimeter::grades_4_5::medium | 45 | 80.00% | 80.00% |
| elementary_school_math_continuation::measurement::grades_2_3::easy | 76 | 96.05% | 96.05% |
| elementary_school_math_continuation::money::grades_3_4::medium | 64 | 100.00% | 100.00% |
| elementary_school_math_continuation::multiplication::grades_3_4::medium | 74 | 100.00% | 100.00% |
| elementary_school_math_continuation::patterns::grades_3_4::medium | 53 | 84.91% | 84.91% |
| elementary_school_math_continuation::subtraction::grades_1_2::easy | 117 | 77.78% | 77.78% |
| elementary_school_math_continuation::time::grades_2_3::easy | 55 | 100.00% | 100.00% |
| elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium | 46 | 34.78% | 34.78% |
| elementary_school_math_continuation::two_step_addition::grades_2_3::medium | 19 | 94.74% | 94.74% |
| elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium | 32 | 31.25% | 34.38% |

### Language understanding

Source: [`evaluation/language_step_275000.json`](evaluation/language_step_275000.json),
selected from the trainer's `lmeval_history.json` at the released checkpoint's step.
`lm-evaluation-harness`: zero-shot, full sets (`limit=None`), `acc_norm`, float32,
batch size 8, max length 4096. Item counts below are the full task split sizes.
Per-example predictions and harness standard errors were not retained by that training logger.

| Task | Items | acc_norm |
|---|---:|---:|
| HellaSwag (validation) | 10,042 | 33.68% |
| ARC-Easy (test) | 2,376 | 51.60% |
| ARC-Challenge (test) | 1,172 | 34.04% |
| PIQA (validation) | 1,838 | 64.47% |

To run a fresh evaluation of the published model:

```bash
lm_eval --model hf \
  --model_args "pretrained=AlgoDriveAI/Inferno-Exp-1.7,trust_remote_code=True,dtype=float32" \
  --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0 --batch_size 8
```

### Tokenizer reproducibility

The primary ArithMark result uses `LlamaTokenizerFast` from the exact saved training tokenizer.
The earlier notebook runner described its requested tokenizer as “slow”; that label did not
establish the actual tokenizer backend in its newer Transformers environment. During release
verification under pinned Transformers 4.57.6, the fast tokenizer reproduced the reported
85.70% normalized / 85.60% raw result, including all 17 category scores. A genuine
`LlamaTokenizer` slow-tokenizer check with the same weights gave 85.80% normalized / 86.00% raw.
These are different tokenization paths, not different weights. The slow-tokenizer reference is
retained in `evaluation/arithmark-3_slow-tokenizer-reference.json` and is not mixed into the
primary result. In this release, the actual tokenizer class is recorded explicitly.

- [`evaluation/evaluate_arithmark3.py`](evaluation/evaluate_arithmark3.py): standalone evaluation runner
  configured for this exact snapshot.
- [`release_manifest.json`](release_manifest.json): SHA-256 and size for every release artifact.

Exact supplied training source SHA-256: `ce792e3792a02edc3f31584294f4dfc68282d2f07f945168ec7ec905abf305f9`.
The source is supplied for reproducibility; its historical default settings are not rewritten to
pretend they were the saved checkpoint configuration. The released inference weights omit
optimizer/RNG state.

The root tokenizer uses the same vocabulary/EOS as training, with automatic BOS insertion
disabled and BOS set to the training document EOS for tools that need a start-token ID,
following the 1.6 export.

## Limitations

This math-first base model can produce incorrect arithmetic and unreliable text. Its general
language results are modest, and it is not trained to behave as a chat assistant.
ArithMark was monitored during training and checkpoint selection, so its score is not an
untouched holdout estimate. The four language tasks were also monitored during training.