| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - small-language-model |
| - gqa |
| - custom_code |
| - arithmark |
| model-index: |
| - name: Inferno-Exp-1.6 |
| results: |
| - task: |
| type: text-generation |
| name: Arithmetic word problems (continuation log-likelihood) |
| dataset: |
| name: ArithMark 3.0 |
| type: AxiomicLabs/Arithmark-3.0 |
| revision: 6f6e59dd9b7e2c63455f7af7f838f9ecc3d0a746 |
| metrics: |
| - type: acc_norm |
| value: 83.10 |
| name: Length-normalized accuracy |
| - type: acc |
| value: 83.50 |
| name: Raw continuation accuracy |
| source: |
| name: AxiomicLabs bencharithmark-3.py (official script, float32) |
| url: https://huggingface.co/datasets/AxiomicLabs/Arithmark-3.0 |
| - task: |
| type: text-generation |
| name: Integer arithmetic (continuation log-likelihood) |
| metrics: |
| - type: acc |
| value: 56.88 |
| name: Accuracy |
| source: |
| name: AxiomicLabs benchmark_arithmark-2.0.py (official script, bfloat16) |
| url: https://huggingface.co/datasets/AxiomicLabs/ArithMark-2.0 |
| - task: |
| type: text-generation |
| name: Commonsense sentence completion |
| dataset: |
| name: HellaSwag |
| type: Rowan/hellaswag |
| split: validation |
| metrics: |
| - type: acc_norm |
| value: 29.80 |
| name: Normalized accuracy (0-shot) |
| source: |
| name: lm-evaluation-harness (0-shot, float32) |
| url: https://github.com/EleutherAI/lm-evaluation-harness |
| - task: |
| type: text-generation |
| name: Science question answering |
| dataset: |
| name: ARC-Easy |
| type: allenai/ai2_arc |
| config: ARC-Easy |
| split: test |
| metrics: |
| - type: acc_norm |
| value: 47.01 |
| name: Normalized accuracy (0-shot) |
| source: |
| name: lm-evaluation-harness (0-shot, float32) |
| url: https://github.com/EleutherAI/lm-evaluation-harness |
| - task: |
| type: text-generation |
| name: Science question answering |
| dataset: |
| name: ARC-Challenge |
| type: allenai/ai2_arc |
| config: ARC-Challenge |
| split: test |
| metrics: |
| - type: acc_norm |
| value: 30.12 |
| name: Normalized accuracy (0-shot) |
| source: |
| name: lm-evaluation-harness (0-shot, float32) |
| url: https://github.com/EleutherAI/lm-evaluation-harness |
| - task: |
| type: text-generation |
| name: Physical commonsense reasoning |
| dataset: |
| name: PIQA |
| type: ybisk/piqa |
| split: validation |
| metrics: |
| - type: acc_norm |
| value: 60.45 |
| name: Normalized accuracy (0-shot) |
| source: |
| name: lm-evaluation-harness (0-shot, float32) |
| url: https://github.com/EleutherAI/lm-evaluation-harness |
| --- |
| |
| # Inferno-Exp-1.6 |
|
|
| **Inferno-Exp-1.6** is a 393.6M-parameter decoder-only base language model trained from scratch by AlgoDriveAI |
| on about 4B tokens. It is a math-first model: its training emphasizes arithmetic and math word problems, |
| and it is geared toward the AxiomicLabs small-language-model benchmarks used in SLM challenges, ArithMark 3.0. Future versions will build on this math foundation with improvements aimed |
| at broader language understanding and logical reasoning. |
|
|
| | | | |
| |---|---| |
| | Parameters | **393,610,240 (393.6M)** | |
| | Layers × width | 32 × 1024 (deep and narrow) | |
| | Attention | grouped-query, 16 query / 4 KV heads, head dim 64, full-head RoPE | |
| | Feed-forward | plain SwiGLU, inner width 2816 | |
| | Context | 2048 tokens | |
| | Tokenizer | Mistral-7B-v0.1 (padded to 32,064); `<|endoftext|>` separates documents | |
| | Embeddings | input and output tied | |
| | Training | from scratch on AlgoDriveAI/SLM_v3 (expanded), ~4B tokens (one pass); AdamW, WSD schedule | |
| |
| **Parameter accounting:** 32,833,536 tied token-embedding parameters + 360,775,680 |
| parameters across 32 transformer blocks + 1,024 final normalization parameters |
| = **393,610,240 unique trainable parameters**. The tied LM head is not counted twice. |
| |
| This is a **base model** (no instruction tuning). The model code ships in the repo, so load it with |
| `trust_remote_code=True`. |
| |
| ## Compared with Inferno-Exp-1.5 (76M) |
| |
| Same architecture family, about 5.17× the parameters and 32 layers instead of 14. |
| |
| | Benchmark | Inferno-Exp-1.5 | **Inferno-Exp-1.6** | Change | |
| |---|---|---|---| |
| | ArithMark 3.0 (acc_norm) | 71.00% | **83.10%** | +12.10 | |
| | HellaSwag | 27.85% | **29.80%** | +1.95 | |
| | ARC-Easy | 43.39% | **47.01%** | +3.62 | |
| | ARC-Challenge | 26.62% | **30.12%** | +3.50 | |
| | PIQA | 58.49% | **60.45%** | +1.96 | |
| | Open SLM Leaderboard Intelligence Index | 20.27 | **26.23** | +5.96 | |
|
|
| Word-problem arithmetic and every language benchmark improve; bare-equation arithmetic (ArithMark 2.0) |
| is lower than in Inferno-Exp-1.5. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| tok = AutoTokenizer.from_pretrained("AlgoDriveAI/Inferno-Exp-1.6") |
| model = AutoModelForCausalLM.from_pretrained("AlgoDriveAI/Inferno-Exp-1.6", trust_remote_code=True) |
| |
| ids = tok("Once upon a time, a little fox", return_tensors="pt") |
| out = model.generate(**ids, max_new_tokens=80, do_sample=True, temperature=0.8, top_p=0.95) |
| print(tok.decode(out[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Try it: interactive demo (Gradio) |
|
|
| The easiest way to play with the model is a small web app with a prompt box, sampling sliders, |
| live streaming, and a Stop button: |
|
|
| 1. Open a new notebook at [colab.research.google.com](https://colab.research.google.com) and choose a GPU |
| (Runtime → Change runtime type → T4 GPU). It also runs on CPU, just slower. |
| 2. Paste the script below into one cell and run it. |
| 3. Click the `*.gradio.live` link it prints. |
|
|
| If it stops and asks you to restart the session (it may need to switch Transformers to a compatible |
| version), choose Runtime → Restart session and run the cell again. |
|
|
| <details> |
| <summary><b>Show the Gradio demo script</b> (one Colab cell, ~200 lines)</summary> |
|
|
| ```python |
| #!/usr/bin/env python3 |
| # -*- coding: utf-8 -*- |
| """ |
| Inferno-Exp-1.6 — simple Gradio inference (loads like any Hub user, no pushing) |
| ============================================================================== |
| Loads Inferno-Exp-1.6 with plain Transformers (AutoModelForCausalLM + |
| trust_remote_code=True) and serves a small Gradio app: open prompt box, |
| sampling controls, streaming output with a Stop button, token/speed readout. |
| |
| Works with either source: |
| • the Hub repo: MODEL_SOURCE = "AlgoDriveAI/Inferno-Exp-1.6" |
| • the local export folder the push script builds (test before uploading): |
| MODEL_SOURCE = "/content/hf_export_inferno_exp_1_0" |
| |
| Private repo? Add a Colab secret named HF_TOKEN (🔑 icon in the left sidebar, turn on |
| notebook access), set the env var HF_TOKEN, or paste a READ token into HF_TOKEN below. |
| |
| It is a base model: it continues text, it does not chat. Write the start of the text |
| you want (e.g. a word problem followed by "Answer:"). |
| Hardcoded constants, no argparse. Paste into one Colab cell and run; open the share link. |
| """ |
| # ============================================================================= |
| # SETTINGS — edit these |
| # ============================================================================= |
| MODEL_SOURCE = "AlgoDriveAI/Inferno-Exp-1.6" # Hub repo id (or a local folder with the same files) |
| HF_TOKEN = "" # only for private repos; "" → Colab secret / env var |
| MODEL_DISPLAY_NAME = "Inferno-Exp-1.6" |
| |
| # Default slider values (all adjustable in the UI) |
| DEF_MAX_NEW_TOKENS = 256 |
| DEF_TEMPERATURE = 0.8 |
| DEF_TOP_K = 40 |
| DEF_TOP_P = 0.95 |
| DEF_REPETITION_PENALTY = 1.1 |
| DEF_SEED = -1 # -1 = different every run |
| DTYPE = "auto" # "auto" (bf16 on newer GPUs, fp32 on CPU) / "float32" / "bfloat16" / "float16" |
| |
| GRADIO_SHARE = True # public *.gradio.live link (needed in Colab) |
| GRADIO_PORT = 7860 |
| INSTALL_DEPENDENCIES = True |
| |
| # ============================================================================= |
| # RUN (no edits needed below) |
| # ============================================================================= |
| import os, sys, time, threading, subprocess |
| |
| try: |
| import google.colab # noqa: F401 |
| IN_COLAB = True |
| except ImportError: |
| IN_COLAB = False |
| if IN_COLAB and INSTALL_DEPENDENCIES: |
| import importlib.metadata as _md |
| |
| def _version(name): |
| try: |
| return _md.version(name) |
| except _md.PackageNotFoundError: |
| return None |
| |
| _before = _version("transformers") |
| subprocess.check_call([sys.executable, "-m", "pip", "install", "--quiet", |
| "transformers>=4.45,<5", "safetensors", "huggingface_hub>=0.25", "gradio"]) |
| if _before != _version("transformers") and "transformers" in sys.modules: |
| raise SystemExit("Transformers was just changed to a compatible version, but the old one is still loaded. " |
| "Restart the session (Runtime → Restart session) and run this cell again.") |
| |
| import torch |
| import gradio as gr |
| from transformers import (AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList, |
| TextIteratorStreamer) |
| |
| DEVICE = "cuda" if torch.cuda.is_available() else "cpu" |
| STOP_EVENT = threading.Event() |
| |
| |
| def get_token(): |
| if HF_TOKEN.strip(): |
| return HF_TOKEN.strip() |
| if IN_COLAB: |
| try: |
| from google.colab import userdata |
| value = userdata.get("HF_TOKEN") |
| if value: |
| return value |
| except Exception: |
| pass |
| return os.environ.get("HF_TOKEN") or None |
| |
| |
| def pick_dtype(): |
| if DTYPE != "auto": |
| return {"float32": torch.float32, "bfloat16": torch.bfloat16, "float16": torch.float16}[DTYPE] |
| if DEVICE == "cuda": |
| return torch.bfloat16 if torch.cuda.get_device_capability()[0] >= 8 else torch.float16 |
| return torch.float32 |
| |
| |
| class StopOnEvent(StoppingCriteria): |
| """Ends generate() as soon as the Stop button sets STOP_EVENT.""" |
| def __call__(self, input_ids, scores, **kwargs): |
| return STOP_EVENT.is_set() |
| |
| |
| def load(): |
| token = None if os.path.isdir(MODEL_SOURCE) else get_token() |
| dtype = pick_dtype() |
| tok = AutoTokenizer.from_pretrained(MODEL_SOURCE, token=token) |
| model = AutoModelForCausalLM.from_pretrained(MODEL_SOURCE, token=token, trust_remote_code=True, |
| torch_dtype=dtype).to(DEVICE).eval() |
| return model, tok, dtype |
| |
| |
| MODEL, TOK, MODEL_DTYPE = load() |
| CFG = MODEL.config |
| N_PARAMS = sum(p.numel() for p in MODEL.parameters()) |
| CONTEXT = CFG.max_position_embeddings |
| print(f"Loaded {MODEL_SOURCE}: {N_PARAMS / 1e6:.1f}M params · {DEVICE} · {MODEL_DTYPE}") |
| |
| |
| def generate(prompt, max_new, temperature, top_k, top_p, rep_penalty, seed): |
| STOP_EVENT.clear() |
| if not prompt: |
| yield "", "Type a prompt first." |
| return |
| ids = TOK(prompt, return_tensors="pt", add_special_tokens=False).to(DEVICE) |
| room = CONTEXT - ids["input_ids"].shape[1] |
| if room <= 0: |
| yield prompt, f"Prompt is longer than the model's {CONTEXT}-token context; shorten it." |
| return |
| seed = int(seed) |
| if seed >= 0: |
| torch.manual_seed(seed) |
| greedy = temperature <= 0 |
| sampling = {} if greedy else dict(temperature=float(temperature), top_k=int(top_k), top_p=float(top_p)) |
| streamer = TextIteratorStreamer(TOK, skip_prompt=True, skip_special_tokens=True) |
| result = {} |
| |
| def worker(): |
| with torch.no_grad(): |
| result["ids"] = MODEL.generate( |
| **ids, max_new_tokens=min(int(max_new), room), do_sample=not greedy, |
| repetition_penalty=float(rep_penalty), eos_token_id=TOK.eos_token_id, pad_token_id=TOK.pad_token_id, |
| use_cache=False, streamer=streamer, stopping_criteria=StoppingCriteriaList([StopOnEvent()]), **sampling) |
| |
| started = time.time() |
| thread = threading.Thread(target=worker, daemon=True) |
| thread.start() |
| text = "" |
| for piece in streamer: |
| text += piece |
| yield prompt + text, f"generating… {time.time() - started:.1f}s" |
| thread.join() |
| new = result["ids"].shape[1] - ids["input_ids"].shape[1] if "ids" in result else 0 |
| took = time.time() - started |
| status = "stopped" if STOP_EVENT.is_set() else "done" |
| yield prompt + text, (f"**{status}** · {new} tokens · {took:.1f}s · {new / max(took, 1e-6):.1f} tok/s · " |
| f"{'greedy' if greedy else f'T={temperature} · top-k {int(top_k)} · top-p {top_p}'} · seed {seed}") |
| |
| |
| def output_box(): |
| kwargs = dict(label="Output (prompt + continuation)", lines=16, max_lines=30, interactive=False) |
| for extra in (dict(show_copy_button=True), dict(buttons=["copy"]), {}): |
| try: |
| return gr.Textbox(**kwargs, **extra) |
| except TypeError: |
| continue |
| |
| |
| def build_ui(): |
| gr_major = int(gr.__version__.split(".")[0]) |
| theme = gr.themes.Soft(primary_hue="indigo", secondary_hue="blue") |
| blocks_kwargs = {} if gr_major >= 6 else dict(theme=theme) # Gradio 6 moved theme to launch() |
| launch_kwargs = dict(theme=theme) if gr_major >= 6 else {} |
| with gr.Blocks(title=MODEL_DISPLAY_NAME, **blocks_kwargs) as demo: |
| gr.Markdown(f"# 🔥 {MODEL_DISPLAY_NAME}\n" |
| f"{N_PARAMS / 1e6:.1f}M-parameter math-first base model · `{CFG.num_hidden_layers}L × {CFG.hidden_size}d` · " |
| f"{CFG.num_attention_heads}Q/{CFG.num_key_value_heads}KV heads · context {CONTEXT} · " |
| f"continues text (not a chat model)") |
| with gr.Row(): |
| with gr.Column(scale=3): |
| prompt = gr.Textbox(label="Prompt", lines=6, placeholder="Write the beginning of the text to continue…") |
| with gr.Row(): |
| run = gr.Button("Generate", variant="primary") |
| stop = gr.Button("Stop", variant="stop") |
| output = output_box() |
| status = gr.Markdown("") |
| with gr.Column(scale=1): |
| max_new = gr.Slider(16, 1024, value=DEF_MAX_NEW_TOKENS, step=16, label="Max new tokens") |
| temperature = gr.Slider(0.0, 1.5, value=DEF_TEMPERATURE, step=0.05, label="Temperature (0 = greedy)") |
| top_k = gr.Slider(0, 200, value=DEF_TOP_K, step=1, label="Top-k (0 = off)") |
| top_p = gr.Slider(0.05, 1.0, value=DEF_TOP_P, step=0.01, label="Top-p") |
| rep = gr.Slider(1.0, 1.5, value=DEF_REPETITION_PENALTY, step=0.01, label="Repetition penalty") |
| seed = gr.Number(value=DEF_SEED, precision=0, label="Seed (-1 = random)") |
| with gr.Accordion("Model info", open=False): |
| gr.Markdown(f"- **Source:** `{MODEL_SOURCE}`\n- **Parameters:** `{N_PARAMS:,}`\n" |
| f"- **Device / dtype:** `{DEVICE}` / `{str(MODEL_DTYPE).replace('torch.', '')}`\n" |
| f"- **Vocab:** `{CFG.vocab_size:,}` · EOS `{TOK.eos_token}`") |
| inputs = [prompt, max_new, temperature, top_k, top_p, rep, seed] |
| run.click(generate, inputs=inputs, outputs=[output, status]) |
| prompt.submit(generate, inputs=inputs, outputs=[output, status]) |
| stop.click(lambda: STOP_EVENT.set(), queue=False) |
| return demo, launch_kwargs |
| |
| |
| if __name__ == "__main__": |
| demo, launch_kwargs = build_ui() |
| demo.queue().launch(share=GRADIO_SHARE, server_port=GRADIO_PORT, server_name="0.0.0.0", **launch_kwargs) |
| ``` |
|
|
| </details> |
|
|
| Tips: it is a base model, so write the beginning of the text you want rather than a question to a |
| chatbot, for example `Sam had 47 marbles. His sister gave him 28 more. Now Sam has`. Lower the |
| temperature (or set it to 0) for arithmetic; raise it for stories. |
|
|
| ## Evaluation |
|
|
| All scores are zero-shot on the checkpoint in this repo. The ArithMark benchmarks were scored with |
| AxiomicLabs' official scripts, unmodified; the language benchmarks with the |
| [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness). |
|
|
| | Benchmark | Metric | Score | Standard error | Chance | |
| |---|---|---|---|---| |
| | **ArithMark 3.0** (word problems) | acc_norm | **83.10%** | ±1.19 | 25% | |
| | HellaSwag | acc_norm | 29.80% | ±0.46 | 25% | |
| | ARC-Easy | acc_norm | 47.01% | ±1.02 | 25% | |
| | ARC-Challenge | acc_norm | 30.12% | ±1.34 | 25% | |
| | PIQA | acc_norm | 60.45% | ±1.14 | 50% | |
| |
| On the [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard)'s |
| Intelligence Index formula these scores give **26.23**. |
| |
| Inferno-Exp-1.6 is math-first, and the scores show it: it is strong on arithmetic word problems but modest on general |
| commonsense and reading benchmarks, where its narrower training data gives it less to draw on. Upcoming |
| versions keep the math foundation and broaden the training mix to lift these language and reasoning scores. |
| |
| ### ArithMark 3.0 |
| |
| Scored with `bencharithmark-3.py` on |
| [ArithMark 3.0](https://huggingface.co/datasets/AxiomicLabs/Arithmark-3.0) (1,000 four-choice |
| continuation word problems, grades 1–5) at commit `6f6e59dd`: float32, batch size 32, |
| max context 1,024, no BOS token, length-normalized log-likelihood as the primary metric. |
| |
| | Metric | Score | |
| |---|---| |
| | **Length-normalized accuracy (acc_norm, primary)** | **83.10%** | |
| | Raw continuation accuracy | 83.50% | |
| | Standard error (acc_norm) | ±1.19 points | |
| | Random-choice baseline | 25.00% | |
|
|
| <details> |
| <summary><b>Per-category results</b> (17 categories)</summary> |
|
|
| | Category | Items | Raw | acc_norm | |
| |---|---|---|---| |
| | Addition (grades 1–2) | 128 | 91.41% | 92.19% | |
| | Comparison | 44 | 100.00% | 100.00% | |
| | Comparison: difference | 48 | 93.75% | 93.75% | |
| | Data | 43 | 100.00% | 100.00% | |
| | Division | 54 | 88.89% | 88.89% | |
| | Fractions: counting | 50 | 56.00% | 56.00% | |
| | Geometry: area | 52 | 90.38% | 88.46% | |
| | Geometry: perimeter | 45 | 71.11% | 71.11% | |
| | Measurement | 76 | 98.68% | 98.68% | |
| | Money | 64 | 96.88% | 96.88% | |
| | Multiplication | 74 | 91.89% | 87.84% | |
| | Patterns | 53 | 64.15% | 64.15% | |
| | Subtraction (grades 1–2) | 117 | 79.49% | 78.63% | |
| | Time | 55 | 100.00% | 100.00% | |
| | Two-step add/subtract | 46 | 34.78% | 34.78% | |
| | Two-step addition | 19 | 78.95% | 78.95% | |
| | Two-step subtraction | 32 | 40.62% | 40.62% | |
| |
| </details> |
| |
| ### Language understanding (lm-evaluation-harness) |
| |
| Zero-shot, `acc_norm`, float32, batch size 16: |
|
|
| ```bash |
| lm_eval --model hf \ |
| --model_args "pretrained=AlgoDriveAI/Inferno-Exp-1.6,trust_remote_code=True,dtype=float32" \ |
| --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0 --batch_size 16 |
| ``` |
|
|
| | Task | Items | acc_norm | |
| |---|---|---| |
| | HellaSwag | 10,042 | 29.80% ± 0.46 | |
| | ARC-Easy | 2,376 | 47.01% ± 1.02 | |
| | ARC-Challenge | 1,172 | 30.12% ± 1.34 | |
| | PIQA | 1,838 | 60.45% ± 1.14 | |
| |
| ## Notes |
| |
| - Generation re-reads the full context each step (no KV cache); fine at this size. |
| - The tokenizer does not add a BOS token; its `bos_token` is set to `<|endoftext|>`, the document |
| boundary the model saw during training, for tools that prepend one. |
|
|
