Hugging Face's logo

AlgoDriveAI
/
Inferno-Exp-1.0

Text Generation
Transformers
Safetensors
English
inferno
small-language-model
gqa
custom_code
arithmark
Eval Results (legacy)
Model card Files
xet
 Community
Inferno-Exp-1.0 / README.md
AlgoDriveAI's picture
AlgoDriveAI
Update README.md
47139d5 verified
preview code
|
Raw History Blame Contribute Delete
18.3 kB
metadata 
license: apache-2.0
language:
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - small-language-model
  - gqa
  - custom_code
  - inferno
  - arithmark
model-index:
  - name: Inferno-Exp-1.0
    results:
      - task:
          type: text-generation
          name: Arithmetic word problems (continuation log-likelihood)
        dataset:
          name: ArithMark 3.0
          type: AxiomicLabs/Arithmark-3.0
          revision: 6f6e59dd9b7e2c63455f7af7f838f9ecc3d0a746
        metrics:
          - type: acc_norm
            value: 54.2
            name: Length-normalized accuracy
          - type: acc
            value: 54.2
            name: Raw continuation accuracy
        source:
          name: AxiomicLabs bencharithmark-3.py (official script, float32)
          url: https://huggingface.co/datasets/AxiomicLabs/Arithmark-3.0
      - task:
          type: text-generation
          name: Integer arithmetic (continuation log-likelihood)
        dataset:
          name: ArithMark 2.0
          type: AxiomicLabs/ArithMark-2.0
          revision: 5bd986c7305dcccec88432d630aae1ac9280a71f
        metrics:
          - type: acc
            value: 67.32
            name: Accuracy (overall)
          - type: acc
            value: 95.04
            name: Accuracy (1 operator)
          - type: acc
            value: 46.93
            name: Accuracy (2 operators)
          - type: acc
            value: 28.6
            name: Accuracy (3 operators)
        source:
          name: AxiomicLabs benchmark_arithmark-2.0.py (official script, bfloat16)
          url: https://huggingface.co/datasets/AxiomicLabs/ArithMark-2.0
      - task:
          type: text-generation
          name: Commonsense sentence completion
        dataset:
          name: HellaSwag
          type: Rowan/hellaswag
          split: validation
        metrics:
          - type: acc_norm
            value: 27.46
            name: Normalized accuracy (0-shot)
        source:
          name: lm-evaluation-harness (0-shot, float32)
          url: https://github.com/EleutherAI/lm-evaluation-harness
      - task:
          type: text-generation
          name: Science question answering
        dataset:
          name: ARC-Easy
          type: allenai/ai2_arc
          config: ARC-Easy
          split: test
        metrics:
          - type: acc_norm
            value: 40.24
            name: Normalized accuracy (0-shot)
        source:
          name: lm-evaluation-harness (0-shot, float32)
          url: https://github.com/EleutherAI/lm-evaluation-harness
      - task:
          type: text-generation
          name: Science question answering
        dataset:
          name: ARC-Challenge
          type: allenai/ai2_arc
          config: ARC-Challenge
          split: test
        metrics:
          - type: acc_norm
            value: 24.66
            name: Normalized accuracy (0-shot)
        source:
          name: lm-evaluation-harness (0-shot, float32)
          url: https://github.com/EleutherAI/lm-evaluation-harness
      - task:
          type: text-generation
          name: Physical commonsense reasoning
        dataset:
          name: PIQA
          type: ybisk/piqa
          split: validation
        metrics:
          - type: acc_norm
            value: 56.58
            name: Normalized accuracy (0-shot)
        source:
          name: lm-evaluation-harness (0-shot, float32)
          url: https://github.com/EleutherAI/lm-evaluation-harness

Inferno-Exp-1.0

Inferno-Exp-1.0 is a 76.2M-parameter decoder-only base language model trained from scratch by AlgoDriveAI. It is a math-first model: its training data centers on arithmetic and math word problems, and it is geared toward the AxiomicLabs small-language-model benchmarks used in SLM challenges, ArithMark 3.0 and ArithMark 2.0 (results below). Future versions will build on this math foundation with improvements aimed at broader language understanding and logical reasoning.

Parameters 76,163,200
Layers × width 14 × 640
Attention grouped-query, 8 query / 2 KV heads, head dim 80, full-head RoPE
Feed-forward plain SwiGLU, inner width 1536
Context 2048 tokens
Tokenizer Mistral-7B-v0.1 (padded to 32,064); `<
Embeddings input and output tied
Training AlgoDriveAI/SLM_v3, 2 epochs; AdamW, WSD schedule, label smoothing 0.05
Checkpoint step 290,000 (vanilla_gqa_epochs_r1)

This is a base model (no instruction tuning). The model code ships in the repo, so load it with trust_remote_code=True.

Usage 

from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("AlgoDriveAI/Inferno-Exp-1.0")
model = AutoModelForCausalLM.from_pretrained("AlgoDriveAI/Inferno-Exp-1.0", trust_remote_code=True)

ids = tok("Once upon a time, a little fox", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=80, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

Try it: interactive demo (Gradio)

The easiest way to play with the model is a small web app with a prompt box, sampling sliders, live streaming, and a Stop button:

  1. Open a new notebook at colab.research.google.com and choose a GPU (Runtime → Change runtime type → T4 GPU). It also runs on CPU, just slower.
  2. Paste the script below into one cell and run it.
  3. Click the *.gradio.live link it prints.

If it stops and asks you to restart the session (it may need to switch Transformers to a compatible version), choose Runtime → Restart session and run the cell again.

Show the Gradio demo script (one Colab cell, ~200 lines) 
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
INFERNO-EXP-1.0 — simple Gradio inference (loads like any Hub user, no pushing)
==============================================================================
Loads Inferno-Exp-1.0 with plain Transformers (AutoModelForCausalLM +
trust_remote_code=True) and serves a small Gradio app: open prompt box,
sampling controls, streaming output with a Stop button, token/speed readout.

Works with either source:
  • the Hub repo:          MODEL_SOURCE = "AlgoDriveAI/Inferno-Exp-1.0"
  • the local export folder the push script builds (test before uploading):
                           MODEL_SOURCE = "/content/hf_export_inferno_exp_1_0"

Private repo? Add a Colab secret named HF_TOKEN (🔑 icon in the left sidebar, turn on
notebook access), set the env var HF_TOKEN, or paste a READ token into HF_TOKEN below.

It is a base model: it continues text, it does not chat. Write the start of the text
you want (e.g. a word problem followed by "Answer:").
Hardcoded constants, no argparse. Paste into one Colab cell and run; open the share link.
"""
# =============================================================================
# SETTINGS — edit these
# =============================================================================
MODEL_SOURCE = "AlgoDriveAI/Inferno-Exp-1.0"   # Hub repo id (or a local folder with the same files)
HF_TOKEN = ""                                  # only for private repos; "" → Colab secret / env var
MODEL_DISPLAY_NAME = "Inferno-Exp-1.0"

# Default slider values (all adjustable in the UI)
DEF_MAX_NEW_TOKENS = 256
DEF_TEMPERATURE = 0.8
DEF_TOP_K = 40
DEF_TOP_P = 0.95
DEF_REPETITION_PENALTY = 1.0
DEF_SEED = 42             # -1 = different every run
DTYPE = "auto"            # "auto" (bf16 on newer GPUs, fp32 on CPU) / "float32" / "bfloat16" / "float16"

GRADIO_SHARE = True       # public *.gradio.live link (needed in Colab)
GRADIO_PORT = 7860
INSTALL_DEPENDENCIES = True

# =============================================================================
# RUN (no edits needed below)
# =============================================================================
import os, sys, time, threading, subprocess

try:
    import google.colab  # noqa: F401
    IN_COLAB = True
except ImportError:
    IN_COLAB = False
if IN_COLAB and INSTALL_DEPENDENCIES:
    import importlib.metadata as _md

    def _version(name):
        try:
            return _md.version(name)
        except _md.PackageNotFoundError:
            return None

    _before = _version("transformers")
    subprocess.check_call([sys.executable, "-m", "pip", "install", "--quiet",
                           "transformers>=4.45,<5", "safetensors", "huggingface_hub>=0.25", "gradio"])
    if _before != _version("transformers") and "transformers" in sys.modules:
        raise SystemExit("Transformers was just changed to a compatible version, but the old one is still loaded. "
                         "Restart the session (Runtime → Restart session) and run this cell again.")

import torch
import gradio as gr
from transformers import (AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList,
                          TextIteratorStreamer)

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
STOP_EVENT = threading.Event()


def get_token():
    if HF_TOKEN.strip():
        return HF_TOKEN.strip()
    if IN_COLAB:
        try:
            from google.colab import userdata
            value = userdata.get("HF_TOKEN")
            if value:
                return value
        except Exception:
            pass
    return os.environ.get("HF_TOKEN") or None


def pick_dtype():
    if DTYPE != "auto":
        return {"float32": torch.float32, "bfloat16": torch.bfloat16, "float16": torch.float16}[DTYPE]
    if DEVICE == "cuda":
        return torch.bfloat16 if torch.cuda.get_device_capability()[0] >= 8 else torch.float16
    return torch.float32


class StopOnEvent(StoppingCriteria):
    """Ends generate() as soon as the Stop button sets STOP_EVENT."""
    def __call__(self, input_ids, scores, **kwargs):
        return STOP_EVENT.is_set()


def load():
    token = None if os.path.isdir(MODEL_SOURCE) else get_token()
    dtype = pick_dtype()
    tok = AutoTokenizer.from_pretrained(MODEL_SOURCE, token=token)
    model = AutoModelForCausalLM.from_pretrained(MODEL_SOURCE, token=token, trust_remote_code=True,
                                                 torch_dtype=dtype).to(DEVICE).eval()
    return model, tok, dtype


MODEL, TOK, MODEL_DTYPE = load()
CFG = MODEL.config
N_PARAMS = sum(p.numel() for p in MODEL.parameters())
CONTEXT = CFG.max_position_embeddings
print(f"Loaded {MODEL_SOURCE}: {N_PARAMS / 1e6:.1f}M params · {DEVICE} · {MODEL_DTYPE}")


def generate(prompt, max_new, temperature, top_k, top_p, rep_penalty, seed):
    STOP_EVENT.clear()
    if not prompt:
        yield "", "Type a prompt first."
        return
    ids = TOK(prompt, return_tensors="pt", add_special_tokens=False).to(DEVICE)
    room = CONTEXT - ids["input_ids"].shape[1]
    if room <= 0:
        yield prompt, f"Prompt is longer than the model's {CONTEXT}-token context; shorten it."
        return
    seed = int(seed)
    if seed >= 0:
        torch.manual_seed(seed)
    greedy = temperature <= 0
    sampling = {} if greedy else dict(temperature=float(temperature), top_k=int(top_k), top_p=float(top_p))
    streamer = TextIteratorStreamer(TOK, skip_prompt=True, skip_special_tokens=True)
    result = {}

    def worker():
        with torch.no_grad():
            result["ids"] = MODEL.generate(
                **ids, max_new_tokens=min(int(max_new), room), do_sample=not greedy,
                repetition_penalty=float(rep_penalty), eos_token_id=TOK.eos_token_id, pad_token_id=TOK.pad_token_id,
                use_cache=False, streamer=streamer, stopping_criteria=StoppingCriteriaList([StopOnEvent()]), **sampling)

    started = time.time()
    thread = threading.Thread(target=worker, daemon=True)
    thread.start()
    text = ""
    for piece in streamer:
        text += piece
        yield prompt + text, f"generating… {time.time() - started:.1f}s"
    thread.join()
    new = result["ids"].shape[1] - ids["input_ids"].shape[1] if "ids" in result else 0
    took = time.time() - started
    status = "stopped" if STOP_EVENT.is_set() else "done"
    yield prompt + text, (f"**{status}** · {new} tokens · {took:.1f}s · {new / max(took, 1e-6):.1f} tok/s · "
                          f"{'greedy' if greedy else f'T={temperature} · top-k {int(top_k)} · top-p {top_p}'} · seed {seed}")


def output_box():
    kwargs = dict(label="Output (prompt + continuation)", lines=16, max_lines=30, interactive=False)
    for extra in (dict(show_copy_button=True), dict(buttons=["copy"]), {}):
        try:
            return gr.Textbox(**kwargs, **extra)
        except TypeError:
            continue


def build_ui():
    gr_major = int(gr.__version__.split(".")[0])
    theme = gr.themes.Soft(primary_hue="indigo", secondary_hue="blue")
    blocks_kwargs = {} if gr_major >= 6 else dict(theme=theme)     # Gradio 6 moved theme to launch()
    launch_kwargs = dict(theme=theme) if gr_major >= 6 else {}
    with gr.Blocks(title=MODEL_DISPLAY_NAME, **blocks_kwargs) as demo:
        gr.Markdown(f"# 🔥 {MODEL_DISPLAY_NAME}\n"
                    f"{N_PARAMS / 1e6:.1f}M-parameter math-first base model · `{CFG.num_hidden_layers}L × {CFG.hidden_size}d` · "
                    f"{CFG.num_attention_heads}Q/{CFG.num_key_value_heads}KV heads · context {CONTEXT} · "
                    f"continues text (not a chat model)")
        with gr.Row():
            with gr.Column(scale=3):
                prompt = gr.Textbox(label="Prompt", lines=6, placeholder="Write the beginning of the text to continue…")
                with gr.Row():
                    run = gr.Button("Generate", variant="primary")
                    stop = gr.Button("Stop", variant="stop")
                output = output_box()
                status = gr.Markdown("")
            with gr.Column(scale=1):
                max_new = gr.Slider(16, 1024, value=DEF_MAX_NEW_TOKENS, step=16, label="Max new tokens")
                temperature = gr.Slider(0.0, 1.5, value=DEF_TEMPERATURE, step=0.05, label="Temperature (0 = greedy)")
                top_k = gr.Slider(0, 200, value=DEF_TOP_K, step=1, label="Top-k (0 = off)")
                top_p = gr.Slider(0.05, 1.0, value=DEF_TOP_P, step=0.01, label="Top-p")
                rep = gr.Slider(1.0, 1.5, value=DEF_REPETITION_PENALTY, step=0.01, label="Repetition penalty")
                seed = gr.Number(value=DEF_SEED, precision=0, label="Seed (-1 = random)")
                with gr.Accordion("Model info", open=False):
                    gr.Markdown(f"- **Source:** `{MODEL_SOURCE}`\n- **Parameters:** `{N_PARAMS:,}`\n"
                                f"- **Device / dtype:** `{DEVICE}` / `{str(MODEL_DTYPE).replace('torch.', '')}`\n"
                                f"- **Vocab:** `{CFG.vocab_size:,}` · EOS `{TOK.eos_token}`")
        inputs = [prompt, max_new, temperature, top_k, top_p, rep, seed]
        run.click(generate, inputs=inputs, outputs=[output, status])
        prompt.submit(generate, inputs=inputs, outputs=[output, status])
        stop.click(lambda: STOP_EVENT.set(), queue=False)
    return demo, launch_kwargs


if __name__ == "__main__":
    demo, launch_kwargs = build_ui()
    demo.queue().launch(share=GRADIO_SHARE, server_port=GRADIO_PORT, server_name="0.0.0.0", **launch_kwargs)

Tips: it is a base model, so write the beginning of the text you want rather than a question to a chatbot, for example Sam had 47 marbles. His sister gave him 28 more. Now Sam has. Lower the temperature (or set it to 0) for arithmetic; raise it for stories.

Evaluation

All scores are zero-shot on the checkpoint in this repo. The ArithMark benchmarks were scored with AxiomicLabs' official scripts, unmodified; the language benchmarks with the lm-evaluation-harness.

Benchmark Metric Score Standard error Chance
ArithMark 3.0 (word problems) acc_norm 54.20% ±1.58 25%
ArithMark 2.0 (integer arithmetic) acc 67.32% ±0.94 25%
HellaSwag acc_norm 27.46% ±0.45 25%
ARC-Easy acc_norm 40.24% ±1.01 25%
ARC-Challenge acc_norm 24.66% ±1.26 25%
PIQA acc_norm 56.58% ±1.16 50%

On the Open SLM Leaderboard's Intelligence Index formula these scores give 14.16.

Inferno-Exp-1.0 is math-first, and the scores show it: it is strong on arithmetic but modest on general commonsense and reading benchmarks, where its narrower training data gives it less to draw on. Upcoming versions keep the math foundation and broaden the training mix to lift these language and reasoning scores.

ArithMark 3.0

Scored with bencharithmark-3.py on ArithMark 3.0 (1,000 four-choice continuation word problems, grades 1–5) at commit 6f6e59dd: float32, batch size 32, max context 1,024, no BOS token, length-normalized log-likelihood as the primary metric.

Metric Score
Length-normalized accuracy (acc_norm, primary) 54.20%
Raw continuation accuracy 54.20%
Standard error (acc_norm) ±1.58 points
Random-choice baseline 25.00%

ArithMark 2.0

Scored with benchmark_arithmark-2.0.py on ArithMark 2.0 (2,500 four-choice integer arithmetic continuations such as (16 / 4) + 44 =) at commit 5bd986c7: the script's standard GPU setting (bfloat16), batch size 16, max context 1,024, no BOS token, raw summed continuation log-likelihood (the benchmark's only metric).

Operators per expression Items Accuracy
1 1,250 95.04%
2 750 46.93%
3 500 28.60%
Overall 2,500 67.32% (1,683 / 2,500)

Standard error ±0.94 points; random-choice baseline 25.00%.

Language understanding (lm-evaluation-harness)

Zero-shot, acc_norm, float32, batch size 16:

lm_eval --model hf \
  --model_args "pretrained=AlgoDriveAI/Inferno-Exp-1.0,trust_remote_code=True,dtype=float32" \
  --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0 --batch_size 16
Task Items acc_norm
HellaSwag 10,042 27.46% ± 0.45
ARC-Easy 2,376 40.24% ± 1.01
ARC-Challenge 1,172 24.66% ± 1.26
PIQA 1,838 56.58% ± 1.16

Notes

  • Generation re-reads the full context each step (no KV cache); fine at this size.
  • The tokenizer does not add a BOS token; its bos_token is set to <|endoftext|>, the document boundary the model saw during training, for tools that prepend one.