maram-350m / MODEL_CARD.md
vignesh1234's picture
MARAM release: maram-350m-v2 trunk, judge-v2 head, 125M ablation models, tokenizer, inference package, held-out files, model cards, licence
7b41231 verified
|
Raw History Blame Contribute Delete
6.21 kB

maram-125m — model card (draft)

Author: Vignesh Angurajan. Status: research prototype, published in the Hugging Face repository Timegravity/maram-350m as an ablation model.

What it is

maram-125m is a 125M-parameter decoder-only language model trained to produce K=3 parallel reasoning streams inside one sequence (block-interleaved every N=64 tokens) and to judge them. A judge head reads the hidden state at periodic <chk> tokens and predicts, per stream, P(correct) and the remaining fraction of the trace. At inference ("native" decoding) the lowest-scoring unfinished stream is killed at each checkpoint and re-forked from the best stream with a different strategy token (<strat:direct|decompose|check>). The attention mask guarantees streams never see each other; only <chk> positions see all streams.

field value
architecture pre-norm transformer, RMSNorm, RoPE (θ 10000), SwiGLU, bias-free, untied embeddings
d_model / layers / heads / FFN 768 / 12 / 12 / 2048
parameters 111 301 634 (embed 12.6 M + lm_head 12.6 M, judge MLP 1.2 M)
vocab 16 384 byte-level BPE; ids 0–255 raw bytes (digits single tokens), 256–268 specials, 269+ merges
context 2048
judge head concat(h_chk, h_stream_last) → 1536→768→2 (correct logit, remaining fraction)
K / N 3 streams / 64-token blocks (locked)
precision trained bf16 autocast, weights stored fp32 (model.safetensors, 445 MB)

Training data

  • General: FineWeb-Edu sample-10BT, 2.2 B packed tokens (1 epoch).
  • Math: traces generated by gpt-oss-20b (vLLM, reasoning effort low, visible channel only, 9 samples per problem over 3 strategy prompts) for GSM8K train, 120 k synthetic arithmetic problems and 60 k Countdown instances; labelled right/wrong against gold; packed as sibling triplets (math_v2, 248.6 M tokens, 3 epochs).
  • Curriculum: tokens 0–60 % general only, 60–90 % half/half, 90–100 % math only. 3.5 B tokens total.
  • Optimiser: AdamW (β 0.9/0.95, wd 0.1), peak LR 6e-4 → 6e-5 cosine, 2000 warm-up steps, 0.5 M tokens/step, grad-clip 1.0; loss = LM + 0.5·judge BCE (1.0 at the final checkpoint) + 0.1·remaining-fraction MSE.

Evaluation

All numbers, each linked to its log, are in RESULTS.md of the repository; the table below is copied from there and must not be edited independently.

metric (step 6675, held-out) value source
general LM loss / judge AUROC 3.4502 / 0.9922 runs/maram-125m/eval.jsonl
math LM loss / judge AUROC (mixed-problem set) 0.7008 / 0.9154 runs/maram-125m/eval.jsonl
leakage test on the final checkpoint 3 passed, bit-exact logs/p5_leakage_final.log
judge AUROC on held-out self-generated streams, v1 / v2 (used for native rows) / v3 0.8097 / 0.8937 / 0.8815 eval/judge_eval.json, logs/p7_judge_eval_v*.log
decoding ablations (greedy1 by strategy, best-of-K, native K/N/judge-mode, anchored native, hard tier) see RESULTS.md Phase 7 eval/*.json, logs/p7_*.log, logs/p7j_*.log, logs/p7x_*.log
flat causal baseline (greedy, beam, best-of-K) see RESULTS.md Phase 8 eval/flat_*.json, logs/p8_*.log, logs/p8b_*.log
round 5: gated pruning sweep, judge-as-selector, never-kill-anchor, MARAM-trunk beam 3, judge AUROC on diverged pairs (95 % Wilson CIs) see RESULTS.md Round 5 eval/native_gated_*.json, eval/native_select_*.json, eval/native_immune_*.json, eval/maram_beam3_decompose.json, eval/diverged_auroc.json, logs/p7y_*.log

Intended use and limitations

Research artefact for studying judge-arbitrated multi-stream decoding at small scale. Not a general assistant. Accuracy on GSM8K-style problems is low at 125M / 3.5 B tokens; the object of study is the judge's discrimination and the token-efficiency of native decoding versus best-of-K and a flat baseline. See the Limitations section of the repository README (single generator for traces, narrow "wrong" distribution, one seed, strategy diversity did not help, judge cannot separate same-strategy streams, flat-baseline data asymmetry). Headline finding: at this scale a single greedy decompose stream matches or beats every multi-stream configuration; the released maram-125m-judge-v2 is the judge head fine-tuned on self-generated streams. Round 5 (RESULTS.md) tested the pruning rule itself: confidence-gated pruning (kill at a <chk> only if the judge's P(correct) is below τ and the best stream leads by δ) removes the anchor-killing failure (anchor killed in 7–14 % of problems instead of 43 %) without changing accuracy (0.722–0.725 on synth, inside the CI of the kill-always rule); using the judge only as a final selector gives 0.7290 [0.709, 0.748], below greedy decompose (0.7470) and below a value vote on the same streams (0.7540). The cause is measured directly: on (<chk>, stream) pairs whose token prefixes have diverged, judge-v2's AUROC is 0.471 (synth) / 0.498 (synth-hard), i.e. it ranks strategies, not same-strategy streams. At matched tokens, beam search on this trunk (beam 3, decompose) reaches 0.7510 on synth and 0.4870 on synth-hard; the flat baseline's beam 3 reaches 0.8035 on synth. The inference package exposes --prune {always,gated,none} with --tau/--delta and defaults to gated (τ 0.35, δ 0.25).

Files

tokenizer.json, maram-125m/{model.safetensors,config.json,train_config.json,export_info.json,CHECKSUMS.txt}, maram-125m-judge-v2/ (same trunk, judge head fine-tuned on self-generated streams; use for native decoding), maram-flat-125m/ (flat causal baseline, same layout), maram_inference/ (standalone decoder, no training code).

Citation and DOI

Paper: Angurajan, V. (2026). MARAM: a free in-weights verifier for parallel reasoning streams — selection beats voting, matches beam at equal tokens, and pruning doesn't help. Version 1, Zenodo. DOI 10.5281/zenodo.23857054. Weights: https://huggingface.co/Timegravity/maram-350m.

Licence

Released under the Timegravity Research-Only Licence v1.0 (LICENSE in the release; SPDX LicenseRef-Timegravity-Research-Only-1.0). The paper is CC BY 4.0.