maram-350m / MODEL_CARD.md
vignesh1234's picture
MARAM release: maram-350m-v2 trunk, judge-v2 head, 125M ablation models, tokenizer, inference package, held-out files, model cards, licence
7b41231 verified
|
Raw History Blame Contribute Delete
6.21 kB
# maram-125m — model card (draft)
**Author:** Vignesh Angurajan. **Status:** research prototype, published in the Hugging Face repository `Timegravity/maram-350m` as an ablation model.
## What it is
`maram-125m` is a 125M-parameter decoder-only language model trained to produce K=3 parallel reasoning
streams inside one sequence (block-interleaved every N=64 tokens) and to judge them. A judge head reads the
hidden state at periodic `<chk>` tokens and predicts, per stream, P(correct) and the remaining fraction of the
trace. At inference ("native" decoding) the lowest-scoring unfinished stream is killed at each checkpoint and
re-forked from the best stream with a different strategy token (`<strat:direct|decompose|check>`).
The attention mask guarantees streams never see each other; only `<chk>` positions see all streams.
| field | value |
|---|---|
| architecture | pre-norm transformer, RMSNorm, RoPE (θ 10000), SwiGLU, bias-free, untied embeddings |
| d_model / layers / heads / FFN | 768 / 12 / 12 / 2048 |
| parameters | 111 301 634 (embed 12.6 M + lm_head 12.6 M, judge MLP 1.2 M) |
| vocab | 16 384 byte-level BPE; ids 0–255 raw bytes (digits single tokens), 256–268 specials, 269+ merges |
| context | 2048 |
| judge head | concat(h_chk, h_stream_last) → 1536→768→2 (correct logit, remaining fraction) |
| K / N | 3 streams / 64-token blocks (locked) |
| precision | trained bf16 autocast, weights stored fp32 (`model.safetensors`, 445 MB) |
## Training data
- General: FineWeb-Edu `sample-10BT`, 2.2 B packed tokens (1 epoch).
- Math: traces generated by gpt-oss-20b (vLLM, reasoning effort low, visible channel only, 9 samples per
problem over 3 strategy prompts) for GSM8K train, 120 k synthetic arithmetic problems and 60 k Countdown
instances; labelled right/wrong against gold; packed as sibling triplets (`math_v2`, 248.6 M tokens, 3 epochs).
- Curriculum: tokens 0–60 % general only, 60–90 % half/half, 90–100 % math only. 3.5 B tokens total.
- Optimiser: AdamW (β 0.9/0.95, wd 0.1), peak LR 6e-4 → 6e-5 cosine, 2000 warm-up steps, 0.5 M tokens/step,
grad-clip 1.0; loss = LM + 0.5·judge BCE (1.0 at the final checkpoint) + 0.1·remaining-fraction MSE.
## Evaluation
All numbers, each linked to its log, are in `RESULTS.md` of the repository; the table below is copied from
there and must not be edited independently.
| metric (step 6675, held-out) | value | source |
|---|---|---|
| general LM loss / judge AUROC | 3.4502 / 0.9922 | `runs/maram-125m/eval.jsonl` |
| math LM loss / judge AUROC (mixed-problem set) | 0.7008 / 0.9154 | `runs/maram-125m/eval.jsonl` |
| leakage test on the final checkpoint | 3 passed, bit-exact | `logs/p5_leakage_final.log` |
| judge AUROC on held-out self-generated streams, v1 / v2 (used for native rows) / v3 | 0.8097 / 0.8937 / 0.8815 | `eval/judge_eval.json`, `logs/p7_judge_eval_v*.log` |
| decoding ablations (greedy1 by strategy, best-of-K, native K/N/judge-mode, anchored native, hard tier) | see `RESULTS.md` Phase 7 | `eval/*.json`, `logs/p7_*.log`, `logs/p7j_*.log`, `logs/p7x_*.log` |
| flat causal baseline (greedy, beam, best-of-K) | see `RESULTS.md` Phase 8 | `eval/flat_*.json`, `logs/p8_*.log`, `logs/p8b_*.log` |
| round 5: gated pruning sweep, judge-as-selector, never-kill-anchor, MARAM-trunk beam 3, judge AUROC on diverged pairs (95 % Wilson CIs) | see `RESULTS.md` Round 5 | `eval/native_gated_*.json`, `eval/native_select_*.json`, `eval/native_immune_*.json`, `eval/maram_beam3_decompose.json`, `eval/diverged_auroc.json`, `logs/p7y_*.log` |
## Intended use and limitations
Research artefact for studying judge-arbitrated multi-stream decoding at small scale. Not a general assistant.
Accuracy on GSM8K-style problems is low at 125M / 3.5 B tokens; the object of study is the judge's
discrimination and the token-efficiency of native decoding versus best-of-K and a flat baseline.
See the Limitations section of the repository README (single generator for traces, narrow "wrong"
distribution, one seed, strategy diversity did not help, judge cannot separate same-strategy streams, flat-baseline
data asymmetry). Headline finding: at this scale a single greedy `decompose` stream matches or beats every
multi-stream configuration; the released `maram-125m-judge-v2` is the judge head fine-tuned on self-generated streams.
Round 5 (RESULTS.md) tested the pruning rule itself: confidence-gated pruning (kill at a `<chk>` only if the judge's
P(correct) is below τ and the best stream leads by δ) removes the anchor-killing failure (anchor killed in 7–14 % of
problems instead of 43 %) without changing accuracy (0.722–0.725 on synth, inside the CI of the kill-always rule);
using the judge only as a final selector gives 0.7290 [0.709, 0.748], below greedy decompose (0.7470) and below a value
vote on the same streams (0.7540). The cause is measured directly: on (`<chk>`, stream) pairs whose token prefixes have
diverged, judge-v2's AUROC is 0.471 (synth) / 0.498 (synth-hard), i.e. it ranks strategies, not same-strategy streams.
At matched tokens, beam search on this trunk (beam 3, decompose) reaches 0.7510 on synth and 0.4870 on synth-hard; the
flat baseline's beam 3 reaches 0.8035 on synth. The inference package exposes `--prune {always,gated,none}` with
`--tau`/`--delta` and defaults to gated (τ 0.35, δ 0.25).
## Files
`tokenizer.json`, `maram-125m/{model.safetensors,config.json,train_config.json,export_info.json,CHECKSUMS.txt}`,
`maram-125m-judge-v2/` (same trunk, judge head fine-tuned on self-generated streams; use for native decoding),
`maram-flat-125m/` (flat causal baseline, same layout), `maram_inference/` (standalone decoder, no training code).
## Citation and DOI
Paper: Angurajan, V. (2026). *MARAM: a free in-weights verifier for parallel reasoning streams — selection beats voting, matches beam at equal tokens, and pruning doesn't help.* Version 1, Zenodo. DOI [10.5281/zenodo.23857054](https://doi.org/10.5281/zenodo.23857054). Weights: https://huggingface.co/Timegravity/maram-350m.
## Licence
Released under the Timegravity Research-Only Licence v1.0 (`LICENSE` in the release; SPDX `LicenseRef-Timegravity-Research-Only-1.0`). The paper is CC BY 4.0.