# maram-125m — model card (draft) **Author:** Vignesh Angurajan. **Status:** research prototype, published in the Hugging Face repository `Timegravity/maram-350m` as an ablation model. ## What it is `maram-125m` is a 125M-parameter decoder-only language model trained to produce K=3 parallel reasoning streams inside one sequence (block-interleaved every N=64 tokens) and to judge them. A judge head reads the hidden state at periodic `` tokens and predicts, per stream, P(correct) and the remaining fraction of the trace. At inference ("native" decoding) the lowest-scoring unfinished stream is killed at each checkpoint and re-forked from the best stream with a different strategy token (``). The attention mask guarantees streams never see each other; only `` positions see all streams. | field | value | |---|---| | architecture | pre-norm transformer, RMSNorm, RoPE (θ 10000), SwiGLU, bias-free, untied embeddings | | d_model / layers / heads / FFN | 768 / 12 / 12 / 2048 | | parameters | 111 301 634 (embed 12.6 M + lm_head 12.6 M, judge MLP 1.2 M) | | vocab | 16 384 byte-level BPE; ids 0–255 raw bytes (digits single tokens), 256–268 specials, 269+ merges | | context | 2048 | | judge head | concat(h_chk, h_stream_last) → 1536→768→2 (correct logit, remaining fraction) | | K / N | 3 streams / 64-token blocks (locked) | | precision | trained bf16 autocast, weights stored fp32 (`model.safetensors`, 445 MB) | ## Training data - General: FineWeb-Edu `sample-10BT`, 2.2 B packed tokens (1 epoch). - Math: traces generated by gpt-oss-20b (vLLM, reasoning effort low, visible channel only, 9 samples per problem over 3 strategy prompts) for GSM8K train, 120 k synthetic arithmetic problems and 60 k Countdown instances; labelled right/wrong against gold; packed as sibling triplets (`math_v2`, 248.6 M tokens, 3 epochs). - Curriculum: tokens 0–60 % general only, 60–90 % half/half, 90–100 % math only. 3.5 B tokens total. - Optimiser: AdamW (β 0.9/0.95, wd 0.1), peak LR 6e-4 → 6e-5 cosine, 2000 warm-up steps, 0.5 M tokens/step, grad-clip 1.0; loss = LM + 0.5·judge BCE (1.0 at the final checkpoint) + 0.1·remaining-fraction MSE. ## Evaluation All numbers, each linked to its log, are in `RESULTS.md` of the repository; the table below is copied from there and must not be edited independently. | metric (step 6675, held-out) | value | source | |---|---|---| | general LM loss / judge AUROC | 3.4502 / 0.9922 | `runs/maram-125m/eval.jsonl` | | math LM loss / judge AUROC (mixed-problem set) | 0.7008 / 0.9154 | `runs/maram-125m/eval.jsonl` | | leakage test on the final checkpoint | 3 passed, bit-exact | `logs/p5_leakage_final.log` | | judge AUROC on held-out self-generated streams, v1 / v2 (used for native rows) / v3 | 0.8097 / 0.8937 / 0.8815 | `eval/judge_eval.json`, `logs/p7_judge_eval_v*.log` | | decoding ablations (greedy1 by strategy, best-of-K, native K/N/judge-mode, anchored native, hard tier) | see `RESULTS.md` Phase 7 | `eval/*.json`, `logs/p7_*.log`, `logs/p7j_*.log`, `logs/p7x_*.log` | | flat causal baseline (greedy, beam, best-of-K) | see `RESULTS.md` Phase 8 | `eval/flat_*.json`, `logs/p8_*.log`, `logs/p8b_*.log` | | round 5: gated pruning sweep, judge-as-selector, never-kill-anchor, MARAM-trunk beam 3, judge AUROC on diverged pairs (95 % Wilson CIs) | see `RESULTS.md` Round 5 | `eval/native_gated_*.json`, `eval/native_select_*.json`, `eval/native_immune_*.json`, `eval/maram_beam3_decompose.json`, `eval/diverged_auroc.json`, `logs/p7y_*.log` | ## Intended use and limitations Research artefact for studying judge-arbitrated multi-stream decoding at small scale. Not a general assistant. Accuracy on GSM8K-style problems is low at 125M / 3.5 B tokens; the object of study is the judge's discrimination and the token-efficiency of native decoding versus best-of-K and a flat baseline. See the Limitations section of the repository README (single generator for traces, narrow "wrong" distribution, one seed, strategy diversity did not help, judge cannot separate same-strategy streams, flat-baseline data asymmetry). Headline finding: at this scale a single greedy `decompose` stream matches or beats every multi-stream configuration; the released `maram-125m-judge-v2` is the judge head fine-tuned on self-generated streams. Round 5 (RESULTS.md) tested the pruning rule itself: confidence-gated pruning (kill at a `` only if the judge's P(correct) is below τ and the best stream leads by δ) removes the anchor-killing failure (anchor killed in 7–14 % of problems instead of 43 %) without changing accuracy (0.722–0.725 on synth, inside the CI of the kill-always rule); using the judge only as a final selector gives 0.7290 [0.709, 0.748], below greedy decompose (0.7470) and below a value vote on the same streams (0.7540). The cause is measured directly: on (``, stream) pairs whose token prefixes have diverged, judge-v2's AUROC is 0.471 (synth) / 0.498 (synth-hard), i.e. it ranks strategies, not same-strategy streams. At matched tokens, beam search on this trunk (beam 3, decompose) reaches 0.7510 on synth and 0.4870 on synth-hard; the flat baseline's beam 3 reaches 0.8035 on synth. The inference package exposes `--prune {always,gated,none}` with `--tau`/`--delta` and defaults to gated (τ 0.35, δ 0.25). ## Files `tokenizer.json`, `maram-125m/{model.safetensors,config.json,train_config.json,export_info.json,CHECKSUMS.txt}`, `maram-125m-judge-v2/` (same trunk, judge head fine-tuned on self-generated streams; use for native decoding), `maram-flat-125m/` (flat causal baseline, same layout), `maram_inference/` (standalone decoder, no training code). ## Citation and DOI Paper: Angurajan, V. (2026). *MARAM: a free in-weights verifier for parallel reasoning streams — selection beats voting, matches beam at equal tokens, and pruning doesn't help.* Version 1, Zenodo. DOI [10.5281/zenodo.23857054](https://doi.org/10.5281/zenodo.23857054). Weights: https://huggingface.co/Timegravity/maram-350m. ## Licence Released under the Timegravity Research-Only Licence v1.0 (`LICENSE` in the release; SPDX `LicenseRef-Timegravity-Research-Only-1.0`). The paper is CC BY 4.0.