Download MODEL_CARD.md from Timegravity/maram-350m: direct link, hf CLI and curl.
- Browser
- Download file 6.21 kB
-
https://huggingface.co/Timegravity/maram-350m/resolve/main/MODEL_CARD.md
- Command line
-
hf download hf://Timegravity/maram-350m/MODEL_CARD.md
-
curl -L -o MODEL_CARD.md https://huggingface.co/Timegravity/maram-350m/resolve/main/MODEL_CARD.md
maram-125m — model card (draft)
Author: Vignesh Angurajan. Status: research prototype, published in the Hugging Face repository Timegravity/maram-350m as an ablation model.
What it is
maram-125m is a 125M-parameter decoder-only language model trained to produce K=3 parallel reasoning
streams inside one sequence (block-interleaved every N=64 tokens) and to judge them. A judge head reads the
hidden state at periodic <chk> tokens and predicts, per stream, P(correct) and the remaining fraction of the
trace. At inference ("native" decoding) the lowest-scoring unfinished stream is killed at each checkpoint and
re-forked from the best stream with a different strategy token (<strat:direct|decompose|check>).
The attention mask guarantees streams never see each other; only <chk> positions see all streams.
| field | value |
|---|---|
| architecture | pre-norm transformer, RMSNorm, RoPE (θ 10000), SwiGLU, bias-free, untied embeddings |
| d_model / layers / heads / FFN | 768 / 12 / 12 / 2048 |
| parameters | 111 301 634 (embed 12.6 M + lm_head 12.6 M, judge MLP 1.2 M) |
| vocab | 16 384 byte-level BPE; ids 0–255 raw bytes (digits single tokens), 256–268 specials, 269+ merges |
| context | 2048 |
| judge head | concat(h_chk, h_stream_last) → 1536→768→2 (correct logit, remaining fraction) |
| K / N | 3 streams / 64-token blocks (locked) |
| precision | trained bf16 autocast, weights stored fp32 (model.safetensors, 445 MB) |
Training data
- General: FineWeb-Edu
sample-10BT, 2.2 B packed tokens (1 epoch). - Math: traces generated by gpt-oss-20b (vLLM, reasoning effort low, visible channel only, 9 samples per
problem over 3 strategy prompts) for GSM8K train, 120 k synthetic arithmetic problems and 60 k Countdown
instances; labelled right/wrong against gold; packed as sibling triplets (
math_v2, 248.6 M tokens, 3 epochs). - Curriculum: tokens 0–60 % general only, 60–90 % half/half, 90–100 % math only. 3.5 B tokens total.
- Optimiser: AdamW (β 0.9/0.95, wd 0.1), peak LR 6e-4 → 6e-5 cosine, 2000 warm-up steps, 0.5 M tokens/step, grad-clip 1.0; loss = LM + 0.5·judge BCE (1.0 at the final checkpoint) + 0.1·remaining-fraction MSE.
Evaluation
All numbers, each linked to its log, are in RESULTS.md of the repository; the table below is copied from
there and must not be edited independently.
| metric (step 6675, held-out) | value | source |
|---|---|---|
| general LM loss / judge AUROC | 3.4502 / 0.9922 | runs/maram-125m/eval.jsonl |
| math LM loss / judge AUROC (mixed-problem set) | 0.7008 / 0.9154 | runs/maram-125m/eval.jsonl |
| leakage test on the final checkpoint | 3 passed, bit-exact | logs/p5_leakage_final.log |
| judge AUROC on held-out self-generated streams, v1 / v2 (used for native rows) / v3 | 0.8097 / 0.8937 / 0.8815 | eval/judge_eval.json, logs/p7_judge_eval_v*.log |
| decoding ablations (greedy1 by strategy, best-of-K, native K/N/judge-mode, anchored native, hard tier) | see RESULTS.md Phase 7 |
eval/*.json, logs/p7_*.log, logs/p7j_*.log, logs/p7x_*.log |
| flat causal baseline (greedy, beam, best-of-K) | see RESULTS.md Phase 8 |
eval/flat_*.json, logs/p8_*.log, logs/p8b_*.log |
| round 5: gated pruning sweep, judge-as-selector, never-kill-anchor, MARAM-trunk beam 3, judge AUROC on diverged pairs (95 % Wilson CIs) | see RESULTS.md Round 5 |
eval/native_gated_*.json, eval/native_select_*.json, eval/native_immune_*.json, eval/maram_beam3_decompose.json, eval/diverged_auroc.json, logs/p7y_*.log |
Intended use and limitations
Research artefact for studying judge-arbitrated multi-stream decoding at small scale. Not a general assistant.
Accuracy on GSM8K-style problems is low at 125M / 3.5 B tokens; the object of study is the judge's
discrimination and the token-efficiency of native decoding versus best-of-K and a flat baseline.
See the Limitations section of the repository README (single generator for traces, narrow "wrong"
distribution, one seed, strategy diversity did not help, judge cannot separate same-strategy streams, flat-baseline
data asymmetry). Headline finding: at this scale a single greedy decompose stream matches or beats every
multi-stream configuration; the released maram-125m-judge-v2 is the judge head fine-tuned on self-generated streams.
Round 5 (RESULTS.md) tested the pruning rule itself: confidence-gated pruning (kill at a <chk> only if the judge's
P(correct) is below τ and the best stream leads by δ) removes the anchor-killing failure (anchor killed in 7–14 % of
problems instead of 43 %) without changing accuracy (0.722–0.725 on synth, inside the CI of the kill-always rule);
using the judge only as a final selector gives 0.7290 [0.709, 0.748], below greedy decompose (0.7470) and below a value
vote on the same streams (0.7540). The cause is measured directly: on (<chk>, stream) pairs whose token prefixes have
diverged, judge-v2's AUROC is 0.471 (synth) / 0.498 (synth-hard), i.e. it ranks strategies, not same-strategy streams.
At matched tokens, beam search on this trunk (beam 3, decompose) reaches 0.7510 on synth and 0.4870 on synth-hard; the
flat baseline's beam 3 reaches 0.8035 on synth. The inference package exposes --prune {always,gated,none} with
--tau/--delta and defaults to gated (τ 0.35, δ 0.25).
Files
tokenizer.json, maram-125m/{model.safetensors,config.json,train_config.json,export_info.json,CHECKSUMS.txt},
maram-125m-judge-v2/ (same trunk, judge head fine-tuned on self-generated streams; use for native decoding),
maram-flat-125m/ (flat causal baseline, same layout), maram_inference/ (standalone decoder, no training code).
Citation and DOI
Paper: Angurajan, V. (2026). MARAM: a free in-weights verifier for parallel reasoning streams — selection beats voting, matches beam at equal tokens, and pruning doesn't help. Version 1, Zenodo. DOI 10.5281/zenodo.23857054. Weights: https://huggingface.co/Timegravity/maram-350m.
Licence
Released under the Timegravity Research-Only Licence v1.0 (LICENSE in the release; SPDX LicenseRef-Timegravity-Research-Only-1.0). The paper is CC BY 4.0.