MARAM release: maram-350m-v2 trunk, judge-v2 head, 125M ablation models, tokenizer, inference package, held-out files, model cards, licence
7b41231 verified |
Download MODEL_CARD.md from Timegravity/maram-350m: direct link, hf CLI and curl.
- Browser
- Download file 6.21 kB
-
https://huggingface.co/Timegravity/maram-350m/resolve/main/MODEL_CARD.md
- Command line
-
hf download hf://Timegravity/maram-350m/MODEL_CARD.md
-
curl -L -o MODEL_CARD.md https://huggingface.co/Timegravity/maram-350m/resolve/main/MODEL_CARD.md
6.21 kB
| # maram-125m — model card (draft) | |
| **Author:** Vignesh Angurajan. **Status:** research prototype, published in the Hugging Face repository `Timegravity/maram-350m` as an ablation model. | |
| ## What it is | |
| `maram-125m` is a 125M-parameter decoder-only language model trained to produce K=3 parallel reasoning | |
| streams inside one sequence (block-interleaved every N=64 tokens) and to judge them. A judge head reads the | |
| hidden state at periodic `<chk>` tokens and predicts, per stream, P(correct) and the remaining fraction of the | |
| trace. At inference ("native" decoding) the lowest-scoring unfinished stream is killed at each checkpoint and | |
| re-forked from the best stream with a different strategy token (`<strat:direct|decompose|check>`). | |
| The attention mask guarantees streams never see each other; only `<chk>` positions see all streams. | |
| | field | value | | |
| |---|---| | |
| | architecture | pre-norm transformer, RMSNorm, RoPE (θ 10000), SwiGLU, bias-free, untied embeddings | | |
| | d_model / layers / heads / FFN | 768 / 12 / 12 / 2048 | | |
| | parameters | 111 301 634 (embed 12.6 M + lm_head 12.6 M, judge MLP 1.2 M) | | |
| | vocab | 16 384 byte-level BPE; ids 0–255 raw bytes (digits single tokens), 256–268 specials, 269+ merges | | |
| | context | 2048 | | |
| | judge head | concat(h_chk, h_stream_last) → 1536→768→2 (correct logit, remaining fraction) | | |
| | K / N | 3 streams / 64-token blocks (locked) | | |
| | precision | trained bf16 autocast, weights stored fp32 (`model.safetensors`, 445 MB) | | |
| ## Training data | |
| - General: FineWeb-Edu `sample-10BT`, 2.2 B packed tokens (1 epoch). | |
| - Math: traces generated by gpt-oss-20b (vLLM, reasoning effort low, visible channel only, 9 samples per | |
| problem over 3 strategy prompts) for GSM8K train, 120 k synthetic arithmetic problems and 60 k Countdown | |
| instances; labelled right/wrong against gold; packed as sibling triplets (`math_v2`, 248.6 M tokens, 3 epochs). | |
| - Curriculum: tokens 0–60 % general only, 60–90 % half/half, 90–100 % math only. 3.5 B tokens total. | |
| - Optimiser: AdamW (β 0.9/0.95, wd 0.1), peak LR 6e-4 → 6e-5 cosine, 2000 warm-up steps, 0.5 M tokens/step, | |
| grad-clip 1.0; loss = LM + 0.5·judge BCE (1.0 at the final checkpoint) + 0.1·remaining-fraction MSE. | |
| ## Evaluation | |
| All numbers, each linked to its log, are in `RESULTS.md` of the repository; the table below is copied from | |
| there and must not be edited independently. | |
| | metric (step 6675, held-out) | value | source | | |
| |---|---|---| | |
| | general LM loss / judge AUROC | 3.4502 / 0.9922 | `runs/maram-125m/eval.jsonl` | | |
| | math LM loss / judge AUROC (mixed-problem set) | 0.7008 / 0.9154 | `runs/maram-125m/eval.jsonl` | | |
| | leakage test on the final checkpoint | 3 passed, bit-exact | `logs/p5_leakage_final.log` | | |
| | judge AUROC on held-out self-generated streams, v1 / v2 (used for native rows) / v3 | 0.8097 / 0.8937 / 0.8815 | `eval/judge_eval.json`, `logs/p7_judge_eval_v*.log` | | |
| | decoding ablations (greedy1 by strategy, best-of-K, native K/N/judge-mode, anchored native, hard tier) | see `RESULTS.md` Phase 7 | `eval/*.json`, `logs/p7_*.log`, `logs/p7j_*.log`, `logs/p7x_*.log` | | |
| | flat causal baseline (greedy, beam, best-of-K) | see `RESULTS.md` Phase 8 | `eval/flat_*.json`, `logs/p8_*.log`, `logs/p8b_*.log` | | |
| | round 5: gated pruning sweep, judge-as-selector, never-kill-anchor, MARAM-trunk beam 3, judge AUROC on diverged pairs (95 % Wilson CIs) | see `RESULTS.md` Round 5 | `eval/native_gated_*.json`, `eval/native_select_*.json`, `eval/native_immune_*.json`, `eval/maram_beam3_decompose.json`, `eval/diverged_auroc.json`, `logs/p7y_*.log` | | |
| ## Intended use and limitations | |
| Research artefact for studying judge-arbitrated multi-stream decoding at small scale. Not a general assistant. | |
| Accuracy on GSM8K-style problems is low at 125M / 3.5 B tokens; the object of study is the judge's | |
| discrimination and the token-efficiency of native decoding versus best-of-K and a flat baseline. | |
| See the Limitations section of the repository README (single generator for traces, narrow "wrong" | |
| distribution, one seed, strategy diversity did not help, judge cannot separate same-strategy streams, flat-baseline | |
| data asymmetry). Headline finding: at this scale a single greedy `decompose` stream matches or beats every | |
| multi-stream configuration; the released `maram-125m-judge-v2` is the judge head fine-tuned on self-generated streams. | |
| Round 5 (RESULTS.md) tested the pruning rule itself: confidence-gated pruning (kill at a `<chk>` only if the judge's | |
| P(correct) is below τ and the best stream leads by δ) removes the anchor-killing failure (anchor killed in 7–14 % of | |
| problems instead of 43 %) without changing accuracy (0.722–0.725 on synth, inside the CI of the kill-always rule); | |
| using the judge only as a final selector gives 0.7290 [0.709, 0.748], below greedy decompose (0.7470) and below a value | |
| vote on the same streams (0.7540). The cause is measured directly: on (`<chk>`, stream) pairs whose token prefixes have | |
| diverged, judge-v2's AUROC is 0.471 (synth) / 0.498 (synth-hard), i.e. it ranks strategies, not same-strategy streams. | |
| At matched tokens, beam search on this trunk (beam 3, decompose) reaches 0.7510 on synth and 0.4870 on synth-hard; the | |
| flat baseline's beam 3 reaches 0.8035 on synth. The inference package exposes `--prune {always,gated,none}` with | |
| `--tau`/`--delta` and defaults to gated (τ 0.35, δ 0.25). | |
| ## Files | |
| `tokenizer.json`, `maram-125m/{model.safetensors,config.json,train_config.json,export_info.json,CHECKSUMS.txt}`, | |
| `maram-125m-judge-v2/` (same trunk, judge head fine-tuned on self-generated streams; use for native decoding), | |
| `maram-flat-125m/` (flat causal baseline, same layout), `maram_inference/` (standalone decoder, no training code). | |
| ## Citation and DOI | |
| Paper: Angurajan, V. (2026). *MARAM: a free in-weights verifier for parallel reasoning streams — selection beats voting, matches beam at equal tokens, and pruning doesn't help.* Version 1, Zenodo. DOI [10.5281/zenodo.23857054](https://doi.org/10.5281/zenodo.23857054). Weights: https://huggingface.co/Timegravity/maram-350m. | |
| ## Licence | |
| Released under the Timegravity Research-Only Licence v1.0 (`LICENSE` in the release; SPDX `LicenseRef-Timegravity-Research-Only-1.0`). The paper is CC BY 4.0. | |