Expand BestTerm model card with benchmark comparisons and reproducibility

Contrast NextTerm controls, the selected SLERP checkpoint and DGX Spark reproduction; add benchmark figure, source summaries, training context and usage examples.

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/benchmark_comparison.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,32 +1,177 @@
 ---
 library_name: transformers
+pipeline_tag: text-generation
 base_model: N8Programs/NextTerm-440M
+datasets:
+- N8Programs/oeis-massive
 tags:
 - qwen3
 - oeis
+- integer-sequences
 - sequence-modeling
 - slerp
+- mlx
 ---
 
-# BestTerm-440M Checkpoint
+# BestTerm-440M
 
-Tentative checkpoint for `BestTerm-440M`.
+**BestTerm-440M improves short-sequence puzzle completion over NextTerm-440M while retaining similar OEIS and polynomial continuation performance.** It is a 440.5M-parameter causal transformer specialized for comma-separated integer sequences, produced by continued pretraining on OEIS b-files and a weight-space merge with the released base model.
 
-This is a global parameter-vector SLERP between the released `N8Programs/NextTerm-440M` base model and the hot b-file-only continued-pretraining checkpoint at 500M tokens, with interpolation `t=0.80`.
+On the matched Ryskina & Knight comparison, greedy accuracy rises from **30/57 (52.63%) to 38/57 (66.67%)**. Beam-4 rises from **32/57 (56.14%) to 40/57 (70.18%)**. The improvements on this small puzzle benchmark come with much smaller changes on the broader evaluations below.
 
-## Quick Scores
+![Benchmark comparison: Ryskina, OEIS-Eval-Neo, M1 forecasting, and polynomial continuation](assets/benchmark_comparison.png)
 
-Ryskina & Knight sequence completion, exact next-term accuracy:
+This repository contains the selected **SLERP `t=0.80` checkpoint**. The original July 2026 results and a separate DGX Spark reproduction are reported independently. The documentation and supporting summaries were recovered and checked in September 2026; the scores are from those historical runs.
 
-- Greedy, old PyTorch/Transformers sanity path: `38/57` (`66.67%`).
-- Beam search, `num_beams=4`, comma stop and EOS/PAD suppressed: `40/57` (`70.18%`).
+## Direct comparison with NextTerm-440M
 
-Other preservation checks from the same sweep:
+### Sequence completion and forecasting
 
-- OEIS-Eval-Neo: `6532/19034` (`34.318%`).
-- M1 Competition 111 macro MAPE: `17.582548`.
-- Polynomial continuation: arithmetic `94.5625%`, quadratic `86.3043%`, cubic `74.5682%`, quartic `67.9524%`.
+| Benchmark | NextTerm-440M control | BestTerm-440M: original evaluation | BestTerm-440M: DGX Spark reproduction |
+| :--- | ---: | ---: | ---: |
+| Ryskina & Knight, greedy ↑ | 30/57 · 52.63% | **38/57 · 66.67%** | **38/57 · 66.67%** |
+| Ryskina & Knight, beam-4 ↑ | 32/57 · 56.14% | **40/57 · 70.18%** | — |
+| OEIS-Eval-Neo ↑ | **6,555/19,034 · 34.438%** | 6,532/19,034 · 34.318% | 6,536/19,034 · 34.339% |
+| M1 Competition 111, macro MAPE ↓ | 17.623927 | **17.582548** | 17.813680 |
 
-## Notes
+**Interpretation:** the matched greedy Ryskina gain is **8 questions / 14.04 percentage points**. OEIS accuracy remains within 0.13 percentage points of the base control. The original M1 gain is only 0.0414 MAPE points and does **not** persist in the Spark reproduction. Naive2 scores **17.798674** macro MAPE in the saved M1 evaluation, slightly better than the reproduced BestTerm result.
 
-This checkpoint is tentative and was selected as the aggressive Ryskina/M1 point on the base-to-hot500 SLERP sweep. The more conservative preservation point was `t=0.60`.
+The Ryskina controls use the matched Transformers greedy/beam runs. The initial MLX merge sweep instead scored the base at 29/57, with BestTerm still at 38/57; the two base results should not be interchanged. A [separate later decoding sweep](benchmarks/teacher_decoding.json) also reached 40/57 with beam-4 and beam-8, but is outside the main reproduction summary.
+
+Sources: [matched Ryskina results](benchmarks/original_ryskina.json), [original merge sweep](benchmarks/original_slerp_sweep.json), [base OEIS](benchmarks/original_base_oeis.json), [base M1](benchmarks/original_base_m1.json), and [Spark reproduction](benchmarks/alice_reproduction.json).
+
+### Polynomial continuation
+
+| Sequence family | NextTerm-440M control | BestTerm: original evaluation | BestTerm: Spark reproduction |
+| :--- | ---: | ---: | ---: |
+| Arithmetic | 94.2917% | 94.5625% | 94.5417% · 4,538/4,800 |
+| Quadratic | 86.3696% | 86.3043% | 86.2174% · 3,966/4,600 |
+| Cubic | 74.8409% | 74.5682% | 74.5682% · 3,281/4,400 |
+| Quartic | 68.1190% | 67.9524% | 68.1905% · 2,864/4,200 |
+
+These results show preservation of the base model's polynomial continuation ability, with small gains and losses across families. They aggregate over prefix lengths; they are not scores at one fixed prompt length. The recovered experiment controls differ slightly from the older published NextTerm-440M model-card values.
+
+Sources: [base polynomial results](benchmarks/original_base_polynomials.json), [original merge sweep](benchmarks/original_slerp_sweep.json), and [Spark polynomial results by prefix length](benchmarks/alice_polynomials.json).
+
+## Context among other models
+
+The following historical comparison comes from the [NextTerm-440M model card](https://huggingface.co/N8Programs/NextTerm-440M). BestTerm's row uses its **Spark greedy reproduction**. The other rows are previously reported results, not fresh matched reruns; the direct control comparisons above are preferable for measuring this intervention.
+
+| Model | Ryskina & Knight, greedy ↑ | OEIS-Eval-Neo ↑ | M1 macro MAPE ↓ |
+| :--- | ---: | ---: | ---: |
+| **BestTerm-440M** | **66.67%** | **34.34%** | **17.8137** |
+| NextTerm-440M, published | 52.63% | 34.43% | 17.6239 |
+| NextTerm-47M | 70.18% | 29.49% | 18.7621 |
+| Qwen3-0.6B | 33.33% | 18.44% | 22.7984 |
+| Qwen3-1.7B | 49.12% | 20.77% | 22.2411 |
+| Qwen3-4B | 63.16% | 23.74% | 19.1731 |
+| Qwen3-8B | 57.89% | 24.62% | 18.4027 |
+| Qwen3-14B | 59.65% | 26.00% | 17.9837 |
+
+NextTerm-47M remains ahead on greedy Ryskina. BestTerm reaches the same 70.18% score with **beam-4**, which uses additional decoding compute. OEIS-Eval-Neo is also distinct from the older OEIS-Eval benchmark; their scores should not be mixed.
+
+## How BestTerm was made
+
+1. **Base pretraining:** [NextTerm-440M](https://huggingface.co/N8Programs/NextTerm-440M) was trained for **13,999,999,995 tokens** on extended OEIS sequences plus synthetic augmentation, using preserved sequence prefixes and a 4,096-token training cap.
+2. **B-file-only continued pretraining:** the synthetic augmentation was removed for this phase, using `oeis_train_bfile_prefix4096.packed`. OEIS b-files provide extended lists of sequence terms. The selected intermediate checkpoint had seen **500,020,611 additional tokens** over **20,486 steps** ([training metadata](benchmarks/hot500_training.json)), from a planned 754,629,901-token run. The internal name **hot500** refers to the higher learning-rate schedule and approximately 500M tokens, not sampling temperature. The recipe used a Muon/AdamW hybrid with peak learning rates 0.01/0.0001, length-bucketed batches, and BF16 live weights with FP32 master weights.
+3. **Global SLERP merge:** the base and hot500 parameter vectors were combined at **`t=0.80`**, toward hot500. The recorded coefficients are **0.2010891797 × base + 0.8016321071 × hot500**. These coefficients come from spherical interpolation, so this is not an exact 20/80 linear average. See [merge metadata](slerp_meta.json).
+
+### Why this merge?
+
+| Selected candidate | Ryskina greedy | OEIS-Eval-Neo | M1 macro MAPE ↓ |
+| :--- | ---: | ---: | ---: |
+| `t=0.45` | 35/57 | 34.517% | 18.469475 |
+| `t=0.60` | 36/57 | **34.543%** | 17.775277 |
+| `t=0.70` | 37/57 | 34.428% | 17.821592 |
+| **`t=0.80` — this repository** | **38/57** | 34.318% | **17.582548** |
+| `t=0.90` | **38/57** | 34.254% | 17.851500 |
+
+`t=0.80` was selected for its Ryskina/M1 tradeoff; `t=0.60` was the more conservative OEIS-preservation point. **These benchmarks were used during checkpoint selection.** The sweep therefore does not constitute an untouched evaluation of a preselected model. All rows in this table are original sweep results.
+
+## Model and input format
+
+| Property | Value |
+| :--- | :--- |
+| Parameters | 440,500,224 |
+| Architecture | Qwen3-style causal LM |
+| Layers / hidden size / FFN size | 28 / 1,024 / 3,072 |
+| Attention heads / KV heads | 16 / 8 |
+| Vocabulary | 16 tokens |
+| Training sequence cap | 4,096 tokens |
+| Export dtype | BF16 |
+| Weight file | 881,035,528 bytes, approximately 840 MiB |
+
+Use comma-separated integers, for example `1,2,4,8,` or `1,-2,3,-4,`. Decimal digits are tokenized individually; large integers consume more tokens. The tokenizer ignores characters outside digits, comma, and minus. Natural-language instructions and chat templates are inappropriate for this vocabulary. Leading-zero forms such as `01,02,03,` were outside the training format.
+
+The exported configuration permits 40,960 positions, but this is **not evidence of training or evaluation at that context length**.
+
+## Usage
+
+### Transformers: predict one integer
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo = "N8Programs/BestTerm-440M-Checkpts"
+tokenizer = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForCausalLM.from_pretrained(
+    repo,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+).eval()
+
+inputs = tokenizer("1,2,4,8,", return_tensors="pt", return_token_type_ids=False)
+inputs = {name: value.to(model.device) for name, value in inputs.items()}
+comma_id = tokenizer.convert_tokens_to_ids(",")
+
+with torch.inference_mode():
+    output = model.generate(
+        **inputs,
+        do_sample=False,
+        num_beams=1,
+        max_new_tokens=64,
+        eos_token_id=[comma_id, tokenizer.eos_token_id],
+        pad_token_id=tokenizer.pad_token_id,
+    )
+
+completion = tokenizer.decode(
+    output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True
+)
+print(completion.split(",", 1)[0].strip())
+```
+
+For the reported **beam-4** decoding convention, use `num_beams=4`, `eos_token_id=comma_id`, and `suppress_tokens=[tokenizer.eos_token_id, tokenizer.pad_token_id]`. This stops at the next comma while suppressing premature EOS/PAD. The snippet is a single-prompt usage example; the reported benchmarks use the protocols below.
+
+### MLX: generate a continuation
+
+```bash
+mlx_lm.generate \
+  --model N8Programs/BestTerm-440M-Checkpts \
+  --prompt "1,2,4,8," \
+  --temp 0 \
+  --max-tokens 64
+```
+
+## Evaluation protocols and reproducibility
+
+- **Ryskina & Knight:** 57 short sequence-completion puzzles, exact next-term accuracy. Spark greedy evaluation stops at comma/EOS/PAD with a 20-token generation cap. The original beam-4 comparison uses comma stopping and suppresses EOS/PAD.
+- **OEIS-Eval-Neo:** 19,034 held-out next-term examples, described as decontaminated in the base model release. The Spark run uses a 4,096-token context cap and 196 new-token cap; all examples were evaluated and parsed, with none skipped. The recovered predictions were checked against their saved answers, confirming 6,536 exact matches. Decontamination was not independently re-audited during this documentation recovery.
+- **Polynomials:** 200 generated sequences per prefix length, seed 0; lengths 2–25 for arithmetic, 3–25 for quadratic, 4–25 for cubic and 5–25 for quartic. Exact next-term match gives 4,800 / 4,600 / 4,400 / 4,200 examples. See the base release's [evaluation script](https://huggingface.co/N8Programs/NextTerm-440M/blob/main/arithmetic_eval.py) for coefficient sampling.
+- **M1 Competition 111:** 68 monthly series with horizon 18, 23 quarterly with horizon 8, and 20 yearly with horizon 6. Macro MAPE weights each series equally. The reproduced score is **17.813680**; point-weighted MAPE is **18.409850**. All 111 series parsed fully, with no missing terms or malformed series.
+
+The Spark reproduction used **MLX 0.31.1, mlx-cuda-13 0.31.1 and mlx-lm 0.31.2**, with CUDA graphs disabled. OEIS batch/queue sizes were 32/512, M1 queue size was 16, and polynomial queue size was 64. The original evaluation had different execution settings. Small numerical and generation differences are visible across runs; their exact cause was not isolated.
+
+The [benchmark summaries](benchmarks/), [comparison CSV](benchmarks/comparison.csv), and [provenance manifest](benchmarks/provenance.json) are included here. Absolute machine-specific paths in copied summaries were reduced to filenames; metrics and settings were preserved. Benchmark datasets and original evaluator implementations are available in the [NextTerm-440M repository](https://huggingface.co/N8Programs/NextTerm-440M/tree/main).
+
+Weight SHA-256, matching the recovered and published checkpoint:
+
+```text
+a6a4244107d621c7138060d37168345b9e83a2e8036ccf4ccbcd465b048068f2
+```
+
+## Limitations and attribution
+
+This is a specialized integer-sequence model. A finite prefix can admit multiple plausible rules, so a generated continuation need not be the intended one. Improvements on 57 selected puzzles do not establish universal rule induction, and the merge does not improve every benchmark. The reported M1 advantage is sensitive to the evaluation run.
+
+The checkpoint inherits training on the [On-Line Encyclopedia of Integer Sequences](https://oeis.org/) through NextTerm-440M and its b-file-only continuation phase. See the [base release](https://huggingface.co/N8Programs/NextTerm-440M) for dataset attribution and licensing information.

assets/plot_benchmarks.py

@@ -0,0 +1,28 @@
+from pathlib import Path
+import json
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+out=Path(__file__).resolve().parent.parent
+base=json.loads((out/"benchmarks/original_base_polynomials.json").read_text())["tasks"]
+orig=next(x for x in json.loads((out/"benchmarks/original_slerp_sweep.json").read_text())["rows"] if x["tag"]=="t080")
+rep=json.loads((out/"benchmarks/alice_reproduction.json").read_text())
+plt.rcParams.update({'font.family':'DejaVu Sans','font.size':11,'axes.spines.top':False,'axes.spines.right':False,'axes.spines.left':False,'axes.titleweight':'bold','axes.labelcolor':'#475569','xtick.color':'#334155','ytick.color':'#475569','figure.facecolor':'white','axes.facecolor':'white','savefig.facecolor':'white'})
+colors=['#64748b','#2563eb','#0f9d8a'];fig,axs=plt.subplots(2,2,figsize=(13,8.6));fig.subplots_adjust(top=.84,bottom=.14,left=.07,right=.98,hspace=.55,wspace=.25)
+fig.suptitle('BestTerm-440M vs. NextTerm-440M',x=.07,y=.97,ha='left',fontsize=23,fontweight='bold',color='#0f172a')
+fig.text(.07,.918,'Stronger short-sequence completion; broadly preserved OEIS and polynomial accuracy.',fontsize=12,color='#475569')
+for ax in axs.flat:ax.set_axisbelow(True);ax.grid(axis='y',color='#e2e8f0');ax.tick_params(axis='both',length=0)
+ax=axs[0,0];x=np.arange(2)
+for offset,vals,color,label in [(-.18,[30/57*100,32/57*100],colors[0],'NextTerm-440M'),(.18,[38/57*100,40/57*100],colors[1],'BestTerm-440M')]:
+ bars=ax.bar(x+offset,vals,.32,color=color,label=label);ax.bar_label(bars,labels=[f'{v:.2f}%' for v in vals],padding=4,fontsize=10)
+ax.set_xticks(x,['Greedy','Beam-4']);ax.set_ylim(0,85);ax.set_title('Ryskina & Knight · higher is better',loc='left',pad=12);ax.set_ylabel('Exact next-term accuracy (%)');ax.legend(frameon=False,fontsize=9,loc='upper left',ncol=2)
+labels=['Base','BestTerm\noriginal','BestTerm\nSpark']
+ax=axs[0,1];vals=[6555/19034*100,6532/19034*100,6536/19034*100];bars=ax.bar(range(3),vals,.6,color=colors);ax.bar_label(bars,labels=[f'{v:.3f}%' for v in vals],padding=4);ax.set_xticks(range(3),labels);ax.set_ylim(0,42);ax.set_title('OEIS-Eval-Neo · higher is better',loc='left',pad=12);ax.set_ylabel('Exact next-term accuracy (%)')
+ax=axs[1,0];vals=[17.623926615,17.582547764,17.813680040];bars=ax.bar(range(3),vals,.6,color=colors);ax.bar_label(bars,labels=[f'{v:.4f}' for v in vals],padding=4);ax.set_xticks(range(3),labels);ax.set_ylim(0,22);ax.set_title('M1 Competition 111 · lower is better',loc='left',pad=12);ax.set_ylabel('Macro MAPE');ax.axhline(17.798673617,color='#b45309',linestyle='--',linewidth=1.2);ax.text(2.4,20.1,'Naive2: 17.7987',ha='right',color='#92400e',fontsize=10)
+ax=axs[1,1];tasks=['arithmetic','quadratic','cubic','quartic'];x=np.arange(4)
+for offset,vals,color,label in [(-.24,[100*base[k]['accuracy'] for k in tasks],colors[0],'Base'),(0,[100*orig['poly_'+k] for k in tasks],colors[1],'BestTerm original'),(.24,[100*rep['poly_'+k] for k in tasks],colors[2],'BestTerm Spark')]:ax.bar(x+offset,vals,.22,color=color,label=label)
+ax.set_xticks(x,['Arithmetic','Quadratic','Cubic','Quartic']);ax.set_ylim(0,110);ax.set_title('Polynomial continuation · higher is better',loc='left',pad=12);ax.set_ylabel('Exact next-term accuracy (%)');ax.legend(frameon=False,fontsize=8,ncol=3,loc='upper center')
+fig.text(.07,.06,'Ryskina: matched Transformers comparison. Other panels: original experiment controls and saved Spark reproduction.\nHistorical July 2026 results; recovered September 2026. Selection used these benchmarks. See model card for protocols.',fontsize=10,color='#64748b',linespacing=1.6)
+fig.savefig(out/'assets/benchmark_comparison.png',dpi=180);plt.close(fig)
+print("Saved assets/benchmark_comparison.png", flush=True)

benchmarks/alice_m1.json

@@ -0,0 +1,55 @@
+{
+  "data_path": "m1_competition_111.jsonl",
+  "model": "BestTerm-440M",
+  "output": "bestterm_m1_competition111_mlx_b64_clean.jsonl",
+  "series_loaded": 111,
+  "series_evaluated": 111,
+  "skipped_long": 0,
+  "max_context_tokens": 2048,
+  "max_new_tokens": 1024,
+  "batch_size": 64,
+  "queue_chunk_size": 16,
+  "missing_policy": "zero",
+  "suppress_eos_until_horizon": true,
+  "seconds": 13.734958076995099,
+  "series_count": 111,
+  "horizon_counts": {
+    "6": 20,
+    "8": 23,
+    "18": 68
+  },
+  "model_macro_mape": 17.813680039670427,
+  "model_median_series_mape": 12.933409374944766,
+  "model_point_mape": 18.409849536595008,
+  "naive2_macro_mape": 17.798673616724457,
+  "naive2_point_mape": 17.63296459764299,
+  "parsed_full_series": 111,
+  "total_missing_terms": 0,
+  "malformed_series": 0,
+  "by_period": {
+    "MONTHLY": {
+      "series_count": 68,
+      "model_macro_mape": 18.89995402978854,
+      "model_point_mape": 18.89995402978854,
+      "naive2_macro_mape": 17.387066649846865,
+      "naive2_point_mape": 17.387066649846865,
+      "parsed_full_series": 68
+    },
+    "QUARTERLY": {
+      "series_count": 23,
+      "model_macro_mape": 18.347319504113486,
+      "model_point_mape": 18.34731950411349,
+      "naive2_macro_mape": 19.59580196054251,
+      "naive2_point_mape": 19.59580196054251,
+      "parsed_full_series": 23
+    },
+    "YEARLY": {
+      "series_count": 20,
+      "model_macro_mape": 13.50666308915932,
+      "model_point_mape": 13.50666308915932,
+      "naive2_macro_mape": 17.131439708717515,
+      "naive2_point_mape": 17.131439708717515,
+      "parsed_full_series": 20
+    }
+  }
+}

benchmarks/alice_oeis.json

@@ -0,0 +1,20 @@
+{
+  "data_path": "oeis_val_neo.jsonl",
+  "model": "BestTerm-440M",
+  "output": "bestterm_oeis_eval_neo_mlx_b32.jsonl",
+  "documents": 19034,
+  "evaluated": 19034,
+  "expected_evaluated": 19034,
+  "skipped_long": 0,
+  "parsed": 19034,
+  "correct": 6536,
+  "accuracy": 0.3433855206472628,
+  "max_new_tokens": 196,
+  "max_context_tokens": 4096,
+  "batch_size": 32,
+  "queue_chunk_size": 512,
+  "restrict_digit_comma_eos": false,
+  "restrict_integer_comma_eos": false,
+  "allowed_token_ids": null,
+  "seconds": 1071.5684291959915
+}

benchmarks/alice_polynomials.json

@@ -0,0 +1,676 @@
+{
+  "csv_path": "bestterm_polynomial_continuation_mlx_b128.csv",
+  "queue_chunk_size": 64,
+  "tasks": {
+    "arithmetic": {
+      "degree": 1,
+      "parsed": 4800,
+      "total": 4800,
+      "correct": 4538,
+      "accuracy": 0.9454166666666667,
+      "parsed_fraction": 1.0,
+      "by_k": [
+        {
+          "k": 2,
+          "parsed": 200,
+          "total": 200,
+          "correct": 16,
+          "accuracy": 0.08
+        },
+        {
+          "k": 3,
+          "parsed": 200,
+          "total": 200,
+          "correct": 155,
+          "accuracy": 0.775
+        },
+        {
+          "k": 4,
+          "parsed": 200,
+          "total": 200,
+          "correct": 188,
+          "accuracy": 0.94
+        },
+        {
+          "k": 5,
+          "parsed": 200,
+          "total": 200,
+          "correct": 197,
+          "accuracy": 0.985
+        },
+        {
+          "k": 6,
+          "parsed": 200,
+          "total": 200,
+          "correct": 197,
+          "accuracy": 0.985
+        },
+        {
+          "k": 7,
+          "parsed": 200,
+          "total": 200,
+          "correct": 195,
+          "accuracy": 0.975
+        },
+        {
+          "k": 8,
+          "parsed": 200,
+          "total": 200,
+          "correct": 199,
+          "accuracy": 0.995
+        },
+        {
+          "k": 9,
+          "parsed": 200,
+          "total": 200,
+          "correct": 197,
+          "accuracy": 0.985
+        },
+        {
+          "k": 10,
+          "parsed": 200,
+          "total": 200,
+          "correct": 198,
+          "accuracy": 0.99
+        },
+        {
+          "k": 11,
+          "parsed": 200,
+          "total": 200,
+          "correct": 198,
+          "accuracy": 0.99
+        },
+        {
+          "k": 12,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 13,
+          "parsed": 200,
+          "total": 200,
+          "correct": 199,
+          "accuracy": 0.995
+        },
+        {
+          "k": 14,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 15,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 16,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 17,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 18,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 19,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 20,
+          "parsed": 200,
+          "total": 200,
+          "correct": 199,
+          "accuracy": 0.995
+        },
+        {
+          "k": 21,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 22,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 23,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 24,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 25,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        }
+      ]
+    },
+    "quadratic": {
+      "degree": 2,
+      "parsed": 4600,
+      "total": 4600,
+      "correct": 3966,
+      "accuracy": 0.8621739130434782,
+      "parsed_fraction": 1.0,
+      "by_k": [
+        {
+          "k": 3,
+          "parsed": 200,
+          "total": 200,
+          "correct": 7,
+          "accuracy": 0.035
+        },
+        {
+          "k": 4,
+          "parsed": 200,
+          "total": 200,
+          "correct": 57,
+          "accuracy": 0.285
+        },
+        {
+          "k": 5,
+          "parsed": 200,
+          "total": 200,
+          "correct": 107,
+          "accuracy": 0.535
+        },
+        {
+          "k": 6,
+          "parsed": 200,
+          "total": 200,
+          "correct": 153,
+          "accuracy": 0.765
+        },
+        {
+          "k": 7,
+          "parsed": 200,
+          "total": 200,
+          "correct": 174,
+          "accuracy": 0.87
+        },
+        {
+          "k": 8,
+          "parsed": 200,
+          "total": 200,
+          "correct": 184,
+          "accuracy": 0.92
+        },
+        {
+          "k": 9,
+          "parsed": 200,
+          "total": 200,
+          "correct": 185,
+          "accuracy": 0.925
+        },
+        {
+          "k": 10,
+          "parsed": 200,
+          "total": 200,
+          "correct": 185,
+          "accuracy": 0.925
+        },
+        {
+          "k": 11,
+          "parsed": 200,
+          "total": 200,
+          "correct": 189,
+          "accuracy": 0.945
+        },
+        {
+          "k": 12,
+          "parsed": 200,
+          "total": 200,
+          "correct": 192,
+          "accuracy": 0.96
+        },
+        {
+          "k": 13,
+          "parsed": 200,
+          "total": 200,
+          "correct": 196,
+          "accuracy": 0.98
+        },
+        {
+          "k": 14,
+          "parsed": 200,
+          "total": 200,
+          "correct": 194,
+          "accuracy": 0.97
+        },
+        {
+          "k": 15,
+          "parsed": 200,
+          "total": 200,
+          "correct": 196,
+          "accuracy": 0.98
+        },
+        {
+          "k": 16,
+          "parsed": 200,
+          "total": 200,
+          "correct": 193,
+          "accuracy": 0.965
+        },
+        {
+          "k": 17,
+          "parsed": 200,
+          "total": 200,
+          "correct": 199,
+          "accuracy": 0.995
+        },
+        {
+          "k": 18,
+          "parsed": 200,
+          "total": 200,
+          "correct": 190,
+          "accuracy": 0.95
+        },
+        {
+          "k": 19,
+          "parsed": 200,
+          "total": 200,
+          "correct": 192,
+          "accuracy": 0.96
+        },
+        {
+          "k": 20,
+          "parsed": 200,
+          "total": 200,
+          "correct": 196,
+          "accuracy": 0.98
+        },
+        {
+          "k": 21,
+          "parsed": 200,
+          "total": 200,
+          "correct": 194,
+          "accuracy": 0.97
+        },
+        {
+          "k": 22,
+          "parsed": 200,
+          "total": 200,
+          "correct": 194,
+          "accuracy": 0.97
+        },
+        {
+          "k": 23,
+          "parsed": 200,
+          "total": 200,
+          "correct": 197,
+          "accuracy": 0.985
+        },
+        {
+          "k": 24,
+          "parsed": 200,
+          "total": 200,
+          "correct": 195,
+          "accuracy": 0.975
+        },
+        {
+          "k": 25,
+          "parsed": 200,
+          "total": 200,
+          "correct": 197,
+          "accuracy": 0.985
+        }
+      ]
+    },
+    "cubic": {
+      "degree": 3,
+      "parsed": 4400,
+      "total": 4400,
+      "correct": 3281,
+      "accuracy": 0.7456818181818182,
+      "parsed_fraction": 1.0,
+      "by_k": [
+        {
+          "k": 4,
+          "parsed": 200,
+          "total": 200,
+          "correct": 6,
+          "accuracy": 0.03
+        },
+        {
+          "k": 5,
+          "parsed": 200,
+          "total": 200,
+          "correct": 12,
+          "accuracy": 0.06
+        },
+        {
+          "k": 6,
+          "parsed": 200,
+          "total": 200,
+          "correct": 30,
+          "accuracy": 0.15
+        },
+        {
+          "k": 7,
+          "parsed": 200,
+          "total": 200,
+          "correct": 69,
+          "accuracy": 0.345
+        },
+        {
+          "k": 8,
+          "parsed": 200,
+          "total": 200,
+          "correct": 94,
+          "accuracy": 0.47
+        },
+        {
+          "k": 9,
+          "parsed": 200,
+          "total": 200,
+          "correct": 138,
+          "accuracy": 0.69
+        },
+        {
+          "k": 10,
+          "parsed": 200,
+          "total": 200,
+          "correct": 153,
+          "accuracy": 0.765
+        },
+        {
+          "k": 11,
+          "parsed": 200,
+          "total": 200,
+          "correct": 175,
+          "accuracy": 0.875
+        },
+        {
+          "k": 12,
+          "parsed": 200,
+          "total": 200,
+          "correct": 185,
+          "accuracy": 0.925
+        },
+        {
+          "k": 13,
+          "parsed": 200,
+          "total": 200,
+          "correct": 177,
+          "accuracy": 0.885
+        },
+        {
+          "k": 14,
+          "parsed": 200,
+          "total": 200,
+          "correct": 184,
+          "accuracy": 0.92
+        },
+        {
+          "k": 15,
+          "parsed": 200,
+          "total": 200,
+          "correct": 186,
+          "accuracy": 0.93
+        },
+        {
+          "k": 16,
+          "parsed": 200,
+          "total": 200,
+          "correct": 192,
+          "accuracy": 0.96
+        },
+        {
+          "k": 17,
+          "parsed": 200,
+          "total": 200,
+          "correct": 183,
+          "accuracy": 0.915
+        },
+        {
+          "k": 18,
+          "parsed": 200,
+          "total": 200,
+          "correct": 190,
+          "accuracy": 0.95
+        },
+        {
+          "k": 19,
+          "parsed": 200,
+          "total": 200,
+          "correct": 186,
+          "accuracy": 0.93
+        },
+        {
+          "k": 20,
+          "parsed": 200,
+          "total": 200,
+          "correct": 187,
+          "accuracy": 0.935
+        },
+        {
+          "k": 21,
+          "parsed": 200,
+          "total": 200,
+          "correct": 184,
+          "accuracy": 0.92
+        },
+        {
+          "k": 22,
+          "parsed": 200,
+          "total": 200,
+          "correct": 190,
+          "accuracy": 0.95
+        },
+        {
+          "k": 23,
+          "parsed": 200,
+          "total": 200,
+          "correct": 190,
+          "accuracy": 0.95
+        },
+        {
+          "k": 24,
+          "parsed": 200,
+          "total": 200,
+          "correct": 182,
+          "accuracy": 0.91
+        },
+        {
+          "k": 25,
+          "parsed": 200,
+          "total": 200,
+          "correct": 188,
+          "accuracy": 0.94
+        }
+      ]
+    },
+    "quartic": {
+      "degree": 4,
+      "parsed": 4200,
+      "total": 4200,
+      "correct": 2864,
+      "accuracy": 0.6819047619047619,
+      "parsed_fraction": 1.0,
+      "by_k": [
+        {
+          "k": 5,
+          "parsed": 200,
+          "total": 200,
+          "correct": 8,
+          "accuracy": 0.04
+        },
+        {
+          "k": 6,
+          "parsed": 200,
+          "total": 200,
+          "correct": 16,
+          "accuracy": 0.08
+        },
+        {
+          "k": 7,
+          "parsed": 200,
+          "total": 200,
+          "correct": 27,
+          "accuracy": 0.135
+        },
+        {
+          "k": 8,
+          "parsed": 200,
+          "total": 200,
+          "correct": 45,
+          "accuracy": 0.225
+        },
+        {
+          "k": 9,
+          "parsed": 200,
+          "total": 200,
+          "correct": 80,
+          "accuracy": 0.4
+        },
+        {
+          "k": 10,
+          "parsed": 200,
+          "total": 200,
+          "correct": 121,
+          "accuracy": 0.605
+        },
+        {
+          "k": 11,
+          "parsed": 200,
+          "total": 200,
+          "correct": 138,
+          "accuracy": 0.69
+        },
+        {
+          "k": 12,
+          "parsed": 200,
+          "total": 200,
+          "correct": 148,
+          "accuracy": 0.74
+        },
+        {
+          "k": 13,
+          "parsed": 200,
+          "total": 200,
+          "correct": 162,
+          "accuracy": 0.81
+        },
+        {
+          "k": 14,
+          "parsed": 200,
+          "total": 200,
+          "correct": 161,
+          "accuracy": 0.805
+        },
+        {
+          "k": 15,
+          "parsed": 200,
+          "total": 200,
+          "correct": 173,
+          "accuracy": 0.865
+        },
+        {
+          "k": 16,
+          "parsed": 200,
+          "total": 200,
+          "correct": 170,
+          "accuracy": 0.85
+        },
+        {
+          "k": 17,
+          "parsed": 200,
+          "total": 200,
+          "correct": 176,
+          "accuracy": 0.88
+        },
+        {
+          "k": 18,
+          "parsed": 200,
+          "total": 200,
+          "correct": 172,
+          "accuracy": 0.86
+        },
+        {
+          "k": 19,
+          "parsed": 200,
+          "total": 200,
+          "correct": 176,
+          "accuracy": 0.88
+        },
+        {
+          "k": 20,
+          "parsed": 200,
+          "total": 200,
+          "correct": 175,
+          "accuracy": 0.875
+        },
+        {
+          "k": 21,
+          "parsed": 200,
+          "total": 200,
+          "correct": 175,
+          "accuracy": 0.875
+        },
+        {
+          "k": 22,
+          "parsed": 200,
+          "total": 200,
+          "correct": 183,
+          "accuracy": 0.915
+        },
+        {
+          "k": 23,
+          "parsed": 200,
+          "total": 200,
+          "correct": 190,
+          "accuracy": 0.95
+        },
+        {
+          "k": 24,
+          "parsed": 200,
+          "total": 200,
+          "correct": 184,
+          "accuracy": 0.92
+        },
+        {
+          "k": 25,
+          "parsed": 200,
+          "total": 200,
+          "correct": 184,
+          "accuracy": 0.92
+        }
+      ]
+    }
+  }
+}

benchmarks/alice_reproduction.json

@@ -0,0 +1,32 @@
+{
+  "model": "BestTerm-440M",
+  "env": "nextterm-mlx",
+  "mlx": "0.31.1",
+  "mlx_cuda_13": "0.31.1",
+  "mlx_lm": "0.31.2",
+  "mlx_use_cuda_graphs": "0",
+  "ryskina_correct": 38,
+  "ryskina_total": 57,
+  "ryskina_accuracy": 0.6666666666666666,
+  "oeis_correct": 6536,
+  "oeis_total": 19034,
+  "oeis_accuracy": 0.3433855206472628,
+  "oeis_batch_size": 32,
+  "oeis_queue_chunk_size": 512,
+  "m1_macro_mape": 17.813680039670427,
+  "m1_point_mape": 18.409849536595008,
+  "m1_queue_chunk_size": 16,
+  "poly_arithmetic_correct": 4538,
+  "poly_arithmetic_total": 4800,
+  "poly_arithmetic": 0.9454166666666667,
+  "poly_quadratic_correct": 3966,
+  "poly_quadratic_total": 4600,
+  "poly_quadratic": 0.8621739130434782,
+  "poly_cubic_correct": 3281,
+  "poly_cubic_total": 4400,
+  "poly_cubic": 0.7456818181818182,
+  "poly_quartic_correct": 2864,
+  "poly_quartic_total": 4200,
+  "poly_quartic": 0.6819047619047619,
+  "poly_queue_chunk_size": 64
+}

benchmarks/comparison.csv

@@ -0,0 +1,9 @@
+Benchmark,NextTerm-440M control,BestTerm original sweep,BestTerm saved Alice reproduction
+Ryskina greedy,30/57 (52.63%),38/57 (66.67%),38/57 (66.67%)
+Ryskina beam-4,32/57 (56.14%),40/57 (70.18%),"40/57 (70.18%), separate teacher sweep"
+OEIS-Eval-Neo,6555/19034 (34.4384%),6532/19034 (34.3175%),6536/19034 (34.3386%)
+M1 macro MAPE (lower better),17.623927,17.582548,17.813680
+Polynomial arithmetic,94.2917%,94.5625%,94.5417% (4538/4800)
+Polynomial quadratic,86.3696%,86.3043%,86.2174% (3966/4600)
+Polynomial cubic,74.8409%,74.5682%,74.5682% (3281/4400)
+Polynomial quartic,68.1190%,67.9524%,68.1905% (2864/4200)

benchmarks/hot500_training.json

@@ -0,0 +1,114 @@
+{
+  "checkpoint": "checkpoint_tokens_000500020611.pt",
+  "trainer_state": {
+    "completed_steps": 20486,
+    "train_tokens_seen": 500020611,
+    "last_loss": 0.759232759475708
+  },
+  "checkpoint_args": {
+    "data": "oeis_train_bfile_prefix4096.packed",
+    "model_backend": "custom",
+    "param_dtype": "fp32",
+    "weight_update_mode": "bf16_live_fp32_master",
+    "batch_size": 1,
+    "seq_len": 4096,
+    "steps": 5,
+    "warmup_steps": 2,
+    "target_tokens": 754629901,
+    "max_steps": 0,
+    "batch_mode": "bucketed",
+    "pad_to_seq_len": false,
+    "index_dir": "oeis_train_bfile_prefix4096.packed.idx",
+    "bucket_tokens_per_batch": 16384,
+    "bucket_token_budget_spec": "512:32768,*:24576",
+    "bucket_max_batch_size": 512,
+    "bucket_pad_multiple": 8,
+    "bucket_pad_to_upper": true,
+    "bucket_replacement": false,
+    "bucket_repeat_epochs": true,
+    "bucket_sampling": "token_mass",
+    "vocab_mode": "oeis",
+    "hidden_size": 1024,
+    "intermediate_size": 3072,
+    "num_hidden_layers": 28,
+    "num_attention_heads": 16,
+    "num_key_value_heads": 8,
+    "head_dim": 0,
+    "compile": true,
+    "compile_mode": "reduce-overhead",
+    "compile_dynamic": true,
+    "compile_skip_dynamic_cudagraphs": true,
+    "prewarm_bucket_shapes": true,
+    "prewarm_bucket_passes": 2,
+    "prewarm_restore_state": true,
+    "prewarm_verify_restore": false,
+    "prewarm_update_optimizer": false,
+    "prewarm_materialize_optimizer_state": true,
+    "gradient_checkpointing": false,
+    "native_gqa": true,
+    "amp": true,
+    "lr": 0.0003,
+    "optimizer_mode": "torch_muon_hybrid",
+    "adamw_lr": 0.0001,
+    "muon_lr": 0.01,
+    "body_lr_mult": 1.0,
+    "adamw_weight_decay": 0.01,
+    "muon_weight_decay": 0.0,
+    "muon_momentum": 0.95,
+    "muon_ns_steps": 5,
+    "muon_adjust_lr_fn": "",
+    "no_muon_nesterov": false,
+    "lr_schedule": "warmup_cosine_cooldown",
+    "lr_total_tokens": 754629901,
+    "lr_warmup_tokens": 0,
+    "lr_warmup_fraction": 0.005,
+    "lr_decay_end_fraction": 0.95,
+    "lr_peak_factor": 1.0,
+    "lr_min_factor": 0.1,
+    "lr_final_factor": 0.0,
+    "checkpoint_dir": "checkpoints",
+    "checkpoint_every_tokens": 250000000,
+    "checkpoint_every_steps": 0,
+    "keep_last_checkpoints": 4,
+    "resume": "",
+    "init_from_hf_dir": "base_model",
+    "save_final": true,
+    "trim_final_batch": true,
+    "allow_token_overshoot": false,
+    "val_data": "oeis_val_decontam.jsonl",
+    "val_format": "jsonl",
+    "val_index_dir": "",
+    "val_every_tokens": 250000000,
+    "val_every_steps": 0,
+    "val_batches": 128,
+    "val_batch_size": 32,
+    "val_max_examples": 0,
+    "val_max_context_tokens": 4096,
+    "oeis_eval_data": "",
+    "oeis_eval_every_tokens": 0,
+    "oeis_eval_every_steps": 0,
+    "oeis_eval_batch_size": 64,
+    "oeis_eval_max_examples": 0,
+    "oeis_eval_max_new_tokens": 20,
+    "oeis_eval_max_context_tokens": 0,
+    "oeis_eval_collect_examples": 3,
+    "oeis_eval_generation_backend": "legacy",
+    "expected_loss_tokens": 754629901,
+    "safe_preflight": true,
+    "allow_synthetic_only": false,
+    "allow_replacement_sampling": false,
+    "preflight_only": false,
+    "wandb": true,
+    "wandb_project": "oeis-massive",
+    "wandb_entity": "n8programs",
+    "wandb_run_name": "nextterm-440m-bfile-only-1epoch-20260701-gqa-lrpeak100",
+    "wandb_id": "nextterm440m-bfile-only-1epoch-20260701-gqa-lrpeak100",
+    "wandb_resume": "allow",
+    "wandb_mode": "online",
+    "wandb_anonymous": "",
+    "wandb_tags": "bfile-only,continued-pretraining,440m,no-synth,invlen-repair,lrpeak100",
+    "log_every_steps": 10,
+    "seed": 0,
+    "report_json": "report.json"
+  }
+}

benchmarks/original_base_m1.json

@@ -0,0 +1,54 @@
+{
+  "data_path": "m1_competition_111.jsonl",
+  "model": "base_model",
+  "output": "base_m1_competition111_mlx_b64.jsonl",
+  "series_loaded": 111,
+  "series_evaluated": 111,
+  "skipped_long": 0,
+  "max_context_tokens": 2048,
+  "max_new_tokens": 1024,
+  "batch_size": 64,
+  "missing_policy": "zero",
+  "suppress_eos_until_horizon": true,
+  "seconds": 4.492863438004861,
+  "series_count": 111,
+  "horizon_counts": {
+    "6": 20,
+    "8": 23,
+    "18": 68
+  },
+  "model_macro_mape": 17.623926615357572,
+  "model_median_series_mape": 13.829597418172249,
+  "model_point_mape": 18.049557899117335,
+  "naive2_macro_mape": 17.798673616724457,
+  "naive2_point_mape": 17.63296459764299,
+  "parsed_full_series": 111,
+  "total_missing_terms": 0,
+  "malformed_series": 0,
+  "by_period": {
+    "MONTHLY": {
+      "series_count": 68,
+      "model_macro_mape": 18.340690204852887,
+      "model_point_mape": 18.340690204852887,
+      "naive2_macro_mape": 17.387066649846865,
+      "naive2_point_mape": 17.387066649846865,
+      "parsed_full_series": 68
+    },
+    "QUARTERLY": {
+      "series_count": 23,
+      "model_macro_mape": 19.047524714417133,
+      "model_point_mape": 19.047524714417133,
+      "naive2_macro_mape": 19.59580196054251,
+      "naive2_point_mape": 19.59580196054251,
+      "parsed_full_series": 23
+    },
+    "YEARLY": {
+      "series_count": 20,
+      "model_macro_mape": 13.549792597154996,
+      "model_point_mape": 13.549792597154996,
+      "naive2_macro_mape": 17.131439708717515,
+      "naive2_point_mape": 17.131439708717515,
+      "parsed_full_series": 20
+    }
+  }
+}

benchmarks/original_base_oeis.json

@@ -0,0 +1,19 @@
+{
+  "data_path": "oeis_val_neo.jsonl",
+  "model": "base_model",
+  "output": "base_oeis_eval_neo_mlx_full_b512.jsonl",
+  "documents": 19034,
+  "evaluated": 19034,
+  "expected_evaluated": 19034,
+  "skipped_long": 0,
+  "parsed": 19033,
+  "correct": 6555,
+  "accuracy": 0.3443837343700746,
+  "max_new_tokens": 196,
+  "max_context_tokens": 4096,
+  "batch_size": 128,
+  "restrict_digit_comma_eos": false,
+  "restrict_integer_comma_eos": false,
+  "allowed_token_ids": null,
+  "seconds": 23.02982224902371
+}

benchmarks/original_base_polynomials.json

@@ -0,0 +1,675 @@
+{
+  "csv_path": "base_polynomial_continuation_mlx_b128.csv",
+  "tasks": {
+    "arithmetic": {
+      "degree": 1,
+      "parsed": 4800,
+      "total": 4800,
+      "correct": 4526,
+      "accuracy": 0.9429166666666666,
+      "parsed_fraction": 1.0,
+      "by_k": [
+        {
+          "k": 2,
+          "parsed": 200,
+          "total": 200,
+          "correct": 9,
+          "accuracy": 0.045
+        },
+        {
+          "k": 3,
+          "parsed": 200,
+          "total": 200,
+          "correct": 153,
+          "accuracy": 0.765
+        },
+        {
+          "k": 4,
+          "parsed": 200,
+          "total": 200,
+          "correct": 187,
+          "accuracy": 0.935
+        },
+        {
+          "k": 5,
+          "parsed": 200,
+          "total": 200,
+          "correct": 197,
+          "accuracy": 0.985
+        },
+        {
+          "k": 6,
+          "parsed": 200,
+          "total": 200,
+          "correct": 195,
+          "accuracy": 0.975
+        },
+        {
+          "k": 7,
+          "parsed": 200,
+          "total": 200,
+          "correct": 196,
+          "accuracy": 0.98
+        },
+        {
+          "k": 8,
+          "parsed": 200,
+          "total": 200,
+          "correct": 197,
+          "accuracy": 0.985
+        },
+        {
+          "k": 9,
+          "parsed": 200,
+          "total": 200,
+          "correct": 198,
+          "accuracy": 0.99
+        },
+        {
+          "k": 10,
+          "parsed": 200,
+          "total": 200,
+          "correct": 198,
+          "accuracy": 0.99
+        },
+        {
+          "k": 11,
+          "parsed": 200,
+          "total": 200,
+          "correct": 199,
+          "accuracy": 0.995
+        },
+        {
+          "k": 12,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 13,
+          "parsed": 200,
+          "total": 200,
+          "correct": 199,
+          "accuracy": 0.995
+        },
+        {
+          "k": 14,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 15,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 16,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 17,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 18,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 19,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 20,
+          "parsed": 200,
+          "total": 200,
+          "correct": 199,
+          "accuracy": 0.995
+        },
+        {
+          "k": 21,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 22,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 23,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 24,
+          "parsed": 200,
+          "total": 200,
+          "correct": 200,
+          "accuracy": 1.0
+        },
+        {
+          "k": 25,
+          "parsed": 200,
+          "total": 200,
+          "correct": 199,
+          "accuracy": 0.995
+        }
+      ]
+    },
+    "quadratic": {
+      "degree": 2,
+      "parsed": 4600,
+      "total": 4600,
+      "correct": 3973,
+      "accuracy": 0.8636956521739131,
+      "parsed_fraction": 1.0,
+      "by_k": [
+        {
+          "k": 3,
+          "parsed": 200,
+          "total": 200,
+          "correct": 1,
+          "accuracy": 0.005
+        },
+        {
+          "k": 4,
+          "parsed": 200,
+          "total": 200,
+          "correct": 63,
+          "accuracy": 0.315
+        },
+        {
+          "k": 5,
+          "parsed": 200,
+          "total": 200,
+          "correct": 99,
+          "accuracy": 0.495
+        },
+        {
+          "k": 6,
+          "parsed": 200,
+          "total": 200,
+          "correct": 144,
+          "accuracy": 0.72
+        },
+        {
+          "k": 7,
+          "parsed": 200,
+          "total": 200,
+          "correct": 182,
+          "accuracy": 0.91
+        },
+        {
+          "k": 8,
+          "parsed": 200,
+          "total": 200,
+          "correct": 184,
+          "accuracy": 0.92
+        },
+        {
+          "k": 9,
+          "parsed": 200,
+          "total": 200,
+          "correct": 187,
+          "accuracy": 0.935
+        },
+        {
+          "k": 10,
+          "parsed": 200,
+          "total": 200,
+          "correct": 191,
+          "accuracy": 0.955
+        },
+        {
+          "k": 11,
+          "parsed": 200,
+          "total": 200,
+          "correct": 190,
+          "accuracy": 0.95
+        },
+        {
+          "k": 12,
+          "parsed": 200,
+          "total": 200,
+          "correct": 192,
+          "accuracy": 0.96
+        },
+        {
+          "k": 13,
+          "parsed": 200,
+          "total": 200,
+          "correct": 195,
+          "accuracy": 0.975
+        },
+        {
+          "k": 14,
+          "parsed": 200,
+          "total": 200,
+          "correct": 196,
+          "accuracy": 0.98
+        },
+        {
+          "k": 15,
+          "parsed": 200,
+          "total": 200,
+          "correct": 194,
+          "accuracy": 0.97
+        },
+        {
+          "k": 16,
+          "parsed": 200,
+          "total": 200,
+          "correct": 196,
+          "accuracy": 0.98
+        },
+        {
+          "k": 17,
+          "parsed": 200,
+          "total": 200,
+          "correct": 198,
+          "accuracy": 0.99
+        },
+        {
+          "k": 18,
+          "parsed": 200,
+          "total": 200,
+          "correct": 192,
+          "accuracy": 0.96
+        },
+        {
+          "k": 19,
+          "parsed": 200,
+          "total": 200,
+          "correct": 194,
+          "accuracy": 0.97
+        },
+        {
+          "k": 20,
+          "parsed": 200,
+          "total": 200,
+          "correct": 198,
+          "accuracy": 0.99
+        },
+        {
+          "k": 21,
+          "parsed": 200,
+          "total": 200,
+          "correct": 195,
+          "accuracy": 0.975
+        },
+        {
+          "k": 22,
+          "parsed": 200,
+          "total": 200,
+          "correct": 195,
+          "accuracy": 0.975
+        },
+        {
+          "k": 23,
+          "parsed": 200,
+          "total": 200,
+          "correct": 194,
+          "accuracy": 0.97
+        },
+        {
+          "k": 24,
+          "parsed": 200,
+          "total": 200,
+          "correct": 197,
+          "accuracy": 0.985
+        },
+        {
+          "k": 25,
+          "parsed": 200,
+          "total": 200,
+          "correct": 196,
+          "accuracy": 0.98
+        }
+      ]
+    },
+    "cubic": {
+      "degree": 3,
+      "parsed": 4400,
+      "total": 4400,
+      "correct": 3293,
+      "accuracy": 0.7484090909090909,
+      "parsed_fraction": 1.0,
+      "by_k": [
+        {
+          "k": 4,
+          "parsed": 200,
+          "total": 200,
+          "correct": 3,
+          "accuracy": 0.015
+        },
+        {
+          "k": 5,
+          "parsed": 200,
+          "total": 200,
+          "correct": 12,
+          "accuracy": 0.06
+        },
+        {
+          "k": 6,
+          "parsed": 200,
+          "total": 200,
+          "correct": 26,
+          "accuracy": 0.13
+        },
+        {
+          "k": 7,
+          "parsed": 200,
+          "total": 200,
+          "correct": 62,
+          "accuracy": 0.31
+        },
+        {
+          "k": 8,
+          "parsed": 200,
+          "total": 200,
+          "correct": 103,
+          "accuracy": 0.515
+        },
+        {
+          "k": 9,
+          "parsed": 200,
+          "total": 200,
+          "correct": 143,
+          "accuracy": 0.715
+        },
+        {
+          "k": 10,
+          "parsed": 200,
+          "total": 200,
+          "correct": 160,
+          "accuracy": 0.8
+        },
+        {
+          "k": 11,
+          "parsed": 200,
+          "total": 200,
+          "correct": 172,
+          "accuracy": 0.86
+        },
+        {
+          "k": 12,
+          "parsed": 200,
+          "total": 200,
+          "correct": 181,
+          "accuracy": 0.905
+        },
+        {
+          "k": 13,
+          "parsed": 200,
+          "total": 200,
+          "correct": 179,
+          "accuracy": 0.895
+        },
+        {
+          "k": 14,
+          "parsed": 200,
+          "total": 200,
+          "correct": 183,
+          "accuracy": 0.915
+        },
+        {
+          "k": 15,
+          "parsed": 200,
+          "total": 200,
+          "correct": 185,
+          "accuracy": 0.925
+        },
+        {
+          "k": 16,
+          "parsed": 200,
+          "total": 200,
+          "correct": 189,
+          "accuracy": 0.945
+        },
+        {
+          "k": 17,
+          "parsed": 200,
+          "total": 200,
+          "correct": 186,
+          "accuracy": 0.93
+        },
+        {
+          "k": 18,
+          "parsed": 200,
+          "total": 200,
+          "correct": 192,
+          "accuracy": 0.96
+        },
+        {
+          "k": 19,
+          "parsed": 200,
+          "total": 200,
+          "correct": 188,
+          "accuracy": 0.94
+        },
+        {
+          "k": 20,
+          "parsed": 200,
+          "total": 200,
+          "correct": 190,
+          "accuracy": 0.95
+        },
+        {
+          "k": 21,
+          "parsed": 200,
+          "total": 200,
+          "correct": 187,
+          "accuracy": 0.935
+        },
+        {
+          "k": 22,
+          "parsed": 200,
+          "total": 200,
+          "correct": 187,
+          "accuracy": 0.935
+        },
+        {
+          "k": 23,
+          "parsed": 200,
+          "total": 200,
+          "correct": 187,
+          "accuracy": 0.935
+        },
+        {
+          "k": 24,
+          "parsed": 200,
+          "total": 200,
+          "correct": 189,
+          "accuracy": 0.945
+        },
+        {
+          "k": 25,
+          "parsed": 200,
+          "total": 200,
+          "correct": 189,
+          "accuracy": 0.945
+        }
+      ]
+    },
+    "quartic": {
+      "degree": 4,
+      "parsed": 4200,
+      "total": 4200,
+      "correct": 2861,
+      "accuracy": 0.6811904761904762,
+      "parsed_fraction": 1.0,
+      "by_k": [
+        {
+          "k": 5,
+          "parsed": 200,
+          "total": 200,
+          "correct": 5,
+          "accuracy": 0.025
+        },
+        {
+          "k": 6,
+          "parsed": 200,
+          "total": 200,
+          "correct": 10,
+          "accuracy": 0.05
+        },
+        {
+          "k": 7,
+          "parsed": 200,
+          "total": 200,
+          "correct": 25,
+          "accuracy": 0.125
+        },
+        {
+          "k": 8,
+          "parsed": 200,
+          "total": 200,
+          "correct": 39,
+          "accuracy": 0.195
+        },
+        {
+          "k": 9,
+          "parsed": 200,
+          "total": 200,
+          "correct": 79,
+          "accuracy": 0.395
+        },
+        {
+          "k": 10,
+          "parsed": 200,
+          "total": 200,
+          "correct": 114,
+          "accuracy": 0.57
+        },
+        {
+          "k": 11,
+          "parsed": 200,
+          "total": 200,
+          "correct": 139,
+          "accuracy": 0.695
+        },
+        {
+          "k": 12,
+          "parsed": 200,
+          "total": 200,
+          "correct": 156,
+          "accuracy": 0.78
+        },
+        {
+          "k": 13,
+          "parsed": 200,
+          "total": 200,
+          "correct": 155,
+          "accuracy": 0.775
+        },
+        {
+          "k": 14,
+          "parsed": 200,
+          "total": 200,
+          "correct": 160,
+          "accuracy": 0.8
+        },
+        {
+          "k": 15,
+          "parsed": 200,
+          "total": 200,
+          "correct": 169,
+          "accuracy": 0.845
+        },
+        {
+          "k": 16,
+          "parsed": 200,
+          "total": 200,
+          "correct": 173,
+          "accuracy": 0.865
+        },
+        {
+          "k": 17,
+          "parsed": 200,
+          "total": 200,
+          "correct": 177,
+          "accuracy": 0.885
+        },
+        {
+          "k": 18,
+          "parsed": 200,
+          "total": 200,
+          "correct": 175,
+          "accuracy": 0.875
+        },
+        {
+          "k": 19,
+          "parsed": 200,
+          "total": 200,
+          "correct": 176,
+          "accuracy": 0.88
+        },
+        {
+          "k": 20,
+          "parsed": 200,
+          "total": 200,
+          "correct": 178,
+          "accuracy": 0.89
+        },
+        {
+          "k": 21,
+          "parsed": 200,
+          "total": 200,
+          "correct": 184,
+          "accuracy": 0.92
+        },
+        {
+          "k": 22,
+          "parsed": 200,
+          "total": 200,
+          "correct": 182,
+          "accuracy": 0.91
+        },
+        {
+          "k": 23,
+          "parsed": 200,
+          "total": 200,
+          "correct": 189,
+          "accuracy": 0.945
+        },
+        {
+          "k": 24,
+          "parsed": 200,
+          "total": 200,
+          "correct": 189,
+          "accuracy": 0.945
+        },
+        {
+          "k": 25,
+          "parsed": 200,
+          "total": 200,
+          "correct": 187,
+          "accuracy": 0.935
+        }
+      ]
+    }
+  }
+}

benchmarks/original_ryskina.json

@@ -0,0 +1,34 @@
+[
+  {
+    "label": "base",
+    "beams": 1,
+    "correct": 30,
+    "documents": 57,
+    "accuracy_percent": 52.63157894736842,
+    "output": "base_ryskina_knight_eval_hf_transformers_generate_beam1.json"
+  },
+  {
+    "label": "base",
+    "beams": 4,
+    "correct": 32,
+    "documents": 57,
+    "accuracy_percent": 56.14035087719298,
+    "output": "base_ryskina_knight_eval_hf_transformers_generate_beam4.json"
+  },
+  {
+    "label": "slerp_t080",
+    "beams": 1,
+    "correct": 38,
+    "documents": 57,
+    "accuracy_percent": 66.66666666666667,
+    "output": "slerp_t080_ryskina_knight_eval_hf_transformers_generate_beam1.json"
+  },
+  {
+    "label": "slerp_t080",
+    "beams": 4,
+    "correct": 40,
+    "documents": 57,
+    "accuracy_percent": 70.17543859649123,
+    "output": "slerp_t080_ryskina_knight_eval_hf_transformers_generate_beam4.json"
+  }
+]

benchmarks/original_slerp_sweep.json

@@ -0,0 +1,69 @@
+{
+  "rows": [
+    {
+      "tag": "t045",
+      "t": 0.45,
+      "ryskina_correct": 35,
+      "ryskina_accuracy": 0.6140350877192983,
+      "oeis_correct": 6570,
+      "oeis_accuracy": 0.3451717978354523,
+      "m1_macro_mape": 18.469474667112838,
+      "poly_arithmetic": 0.9460416666666667,
+      "poly_quadratic": 0.8673913043478261,
+      "poly_cubic": 0.7495454545454545,
+      "poly_quartic": 0.6930952380952381
+    },
+    {
+      "tag": "t060",
+      "t": 0.6,
+      "ryskina_correct": 36,
+      "ryskina_accuracy": 0.631578947368421,
+      "oeis_correct": 6575,
+      "oeis_accuracy": 0.34543448565724494,
+      "m1_macro_mape": 17.775277211811733,
+      "poly_arithmetic": 0.9454166666666667,
+      "poly_quadratic": 0.8652173913043478,
+      "poly_cubic": 0.7488636363636364,
+      "poly_quartic": 0.6885714285714286
+    },
+    {
+      "tag": "t070",
+      "t": 0.7,
+      "ryskina_correct": 37,
+      "ryskina_accuracy": 0.6491228070175439,
+      "oeis_correct": 6553,
+      "oeis_accuracy": 0.3442786592413576,
+      "m1_macro_mape": 17.82159170400584,
+      "poly_arithmetic": 0.9460416666666667,
+      "poly_quadratic": 0.8641304347826086,
+      "poly_cubic": 0.7495454545454545,
+      "poly_quartic": 0.6857142857142857
+    },
+    {
+      "tag": "t080",
+      "t": 0.8,
+      "ryskina_correct": 38,
+      "ryskina_accuracy": 0.6666666666666666,
+      "oeis_correct": 6532,
+      "oeis_accuracy": 0.3431753703898287,
+      "m1_macro_mape": 17.582547763955844,
+      "poly_arithmetic": 0.945625,
+      "poly_quadratic": 0.8630434782608696,
+      "poly_cubic": 0.7456818181818182,
+      "poly_quartic": 0.6795238095238095
+    },
+    {
+      "tag": "t090",
+      "t": 0.9,
+      "ryskina_correct": 38,
+      "ryskina_accuracy": 0.6666666666666666,
+      "oeis_correct": 6520,
+      "oeis_accuracy": 0.34254491961752653,
+      "m1_macro_mape": 17.851500398729986,
+      "poly_arithmetic": 0.945,
+      "poly_quadratic": 0.8602173913043478,
+      "poly_cubic": 0.7452272727272727,
+      "poly_quartic": 0.6780952380952381
+    }
+  ]
+}

benchmarks/provenance.json

@@ -0,0 +1,53 @@
+{
+  "recovered": "2026-09-25",
+  "experiments": "July 2026",
+  "normalization": "Absolute machine-specific paths in copied JSON summaries replaced by their basenames. Metrics and evaluation settings preserved.",
+  "weights_sha256": "a6a4244107d621c7138060d37168345b9e83a2e8036ccf4ccbcd465b048068f2",
+  "sources": [
+    {
+      "published_file": "original_ryskina.json",
+      "source_filename": "ryskina_hf_transformers_generate_beam_summary.json",
+      "original_source_sha256": "98bfad6b14b801f85f810af9f5a43b1bb3548a8a79ec074ec850ed8933ade53e"
+    },
+    {
+      "published_file": "original_slerp_sweep.json",
+      "source_filename": "full_candidate_sweep_summary.json",
+      "original_source_sha256": "6b42dca5944cb49f52193b328340d4723235292574f8a08cb838afc7cca01c73"
+    },
+    {
+      "published_file": "original_base_oeis.json",
+      "source_filename": "base_oeis_eval_neo_mlx_full_b512_summary.json",
+      "original_source_sha256": "93524fa7150d062f1eeb0003e0beb358955824aa0190a7440bea3822716e9e72"
+    },
+    {
+      "published_file": "original_base_m1.json",
+      "source_filename": "base_m1_competition111_mlx_b64_summary.json",
+      "original_source_sha256": "c8ae6aa834dc1a7855aaaa197ae9d7a3e00bdf4f8b7ae88e84414313ef88aa02"
+    },
+    {
+      "published_file": "original_base_polynomials.json",
+      "source_filename": "base_polynomial_continuation_mlx_b128_summary.json",
+      "original_source_sha256": "ec060c99c7468919bb862ee48153e9b7497d7b92abb7270fb891bc47b54c685e"
+    },
+    {
+      "published_file": "alice_reproduction.json",
+      "source_filename": "bestterm_repro_summary.json",
+      "original_source_sha256": "d3336fde463c4aedbef6772a6d0f7537ad951e6bbee7bf8875aff3079a24b072"
+    },
+    {
+      "published_file": "alice_m1.json",
+      "source_filename": "bestterm_m1_competition111_mlx_b64_clean_summary.json",
+      "original_source_sha256": "bde3be1a84083623cbff0fbdd8ac7cc17ea0cd5a5dc87484b5e629a2756aa2dc"
+    },
+    {
+      "published_file": "alice_oeis.json",
+      "source_filename": "bestterm_oeis_eval_neo_mlx_b32_summary.json",
+      "original_source_sha256": "3bff5e3eab2100139a8a903732f255fdc32c7841802805e5e13eb5b71d531905"
+    },
+    {
+      "published_file": "alice_polynomials.json",
+      "source_filename": "bestterm_polynomial_continuation_mlx_b128_summary.json",
+      "original_source_sha256": "c389efedefc295d3d13c5c1473a435b558742173806982ecb03034279073cf58"
+    }
+  ]
+}

benchmarks/teacher_decoding.json

@@ -0,0 +1,36 @@
+{
+  "model": "N8Programs/BestTerm-440M-Checkpts",
+  "table": {
+    "greedy": "38/57",
+    "greedy_restricted": "38/57",
+    "beam4_t64": "40/57",
+    "beam4_t160": "40/57",
+    "beam8_t64": "40/57",
+    "beam8_t160": "40/57",
+    "beam16_t64": "ERROR",
+    "beam16_t160": "ERROR",
+    "vote32_T0.7": "31/57",
+    "rerank32_T0.7_h1_total": "38/57",
+    "rerank32_T0.7_h1_mean": "39/57",
+    "rerank32_T0.7_h2_total": "39/57",
+    "rerank32_T0.7_h2_mean": "39/57",
+    "rerank32_T0.7_h4_total": "39/57",
+    "rerank32_T0.7_h4_mean": "39/57",
+    "rerank32_T0.7_h8_total": "38/57",
+    "rerank32_T0.7_h8_mean": "40/57",
+    "rerank32_T0.7_h16_total": "37/57",
+    "rerank32_T0.7_h16_mean": "38/57",
+    "vote32_T1.0": "35/57",
+    "rerank32_T1.0_h1_total": "38/57",
+    "rerank32_T1.0_h1_mean": "39/57",
+    "rerank32_T1.0_h2_total": "39/57",
+    "rerank32_T1.0_h2_mean": "39/57",
+    "rerank32_T1.0_h4_total": "38/57",
+    "rerank32_T1.0_h4_mean": "38/57",
+    "rerank32_T1.0_h8_total": "40/57",
+    "rerank32_T1.0_h8_mean": "40/57",
+    "rerank32_T1.0_h16_total": "39/57",
+    "rerank32_T1.0_h16_mean": "40/57"
+  },
+  "num_samples": 32
+}