Add sourced SmolLM2 scores and highlight benchmark leaders

Use the independently published SmolLM2 result set consistently, bold the highest reported metric values, and remove redundant evaluation graphics. Model weights and inference runtime are unchanged.

FILE_MANIFEST.json

@@ -22,8 +22,8 @@
       "sha256": "0e6865f1a81cfd69ca48e6987658854ce0c0d23e91d72ae545393cd5fa032375"
     },
     "README.md": {
-      "bytes": 13030,
-      "sha256": "cf0dc751a9b2091d161c76e1c815ce4ad6dca9f5eeca90a07bae7542b9cc2161"
+      "bytes": 12573,
+      "sha256": "650ef6dc4b02ec96ebdcd879f014b99e7caf75aadbd143e3accd4c4924460672"
     },
     "REPOSITORIES.json": {
       "bytes": 716,
@@ -130,8 +130,8 @@
       "sha256": "e0b0c730321360c40f20e53a22890c52cd5294bad2a5777a0a891096557a2f0a"
     },
     "docs/EVALUATION.md": {
-      "bytes": 8582,
-      "sha256": "e890892eb870896a8f5d585bde6ab748820f91b2ff49e8d70434bc14140767a7"
+      "bytes": 8336,
+      "sha256": "b78247452825d577713115632bfd9900decf64ff991b098156184dec7a41f415"
     },
     "docs/EXAMPLES.md": {
       "bytes": 1117,
@@ -397,6 +397,10 @@
       "bytes": 2471,
       "sha256": "2cafe36ca3f2fe3dd00cf9aa63314eac64a73de2c58e2de92cdeee47306bd8ae"
     },
+    "results/baselines/smollm2_independent.json": {
+      "bytes": 3080,
+      "sha256": "99d582c9ed4e90d87f16835e814beb4604687519734c7a74972367a61f8c10c9"
+    },
     "results/package_inference_verification.json": {
       "bytes": 26015,
       "sha256": "fed55313b61008e8f379a7b4bc49535cc501a31a942c7560687de4a4bd1cd47f"

README.md

@@ -57,12 +57,12 @@ Zero-shot likelihood scoring on the complete ARC Easy, ARC Challenge, PIQA and H
 
 | Benchmark | Solstice-1 acc / acc_norm | OpenSML-150M acc / acc_norm | SmolLM2-135M Instruct acc / acc_norm |
 | --- | --- | --- | --- |
-| ARC Easy | 62.58 / 62.33 | 56.65 / 55.43 | — / — |
-| ARC Challenge | 29.35 / 30.89 | 26.02 / 29.52 | — / — |
-| PIQA | 66.49 / 66.21 | 64.53 / 64.09 | — / 66.3 |
-| HellaSwag | 32.85 / 39.40 | 30.55 / 33.94 | — / 40.9 |
+| ARC Easy | **62.58** / **62.33** | 56.65 / 55.43 | 54.00 / 48.82 |
+| ARC Challenge | **29.35** / **30.89** | 26.02 / 29.52 | 25.94 / 27.73 |
+| PIQA | 66.49 / 66.21 | 64.53 / 64.09 | **66.70** / **66.76** |
+| HellaSwag | 32.85 / 39.40 | 30.55 / 33.94 | **35.02** / **42.90** |
 
-All values are percentages, shown as **acc / acc_norm**. SmolLM2 publishes ARC only as a combined average (**37.3% acc_norm**); separate Easy/Challenge scores are unavailable in its card.
+Values are percentages, shown as **acc / acc_norm**. Bold marks the highest reported value for each metric. SmolLM2 scores are from an independent published evaluation; protocol differences are noted below.
 
 ### Instruction following
 
@@ -70,19 +70,16 @@ IFEval checks whether generated responses follow explicit instructions. It measu
 
 | IFEval accuracy | Solstice-1 | OpenSML-150M | SmolLM2-135M Instruct |
 | --- | --- | --- | --- |
-| Strict prompts | 17.93% | 15.16% | — |
-| Strict instructions | 30.10% | 25.30% | — |
-| Loose prompts | 19.22% | 15.71% | — |
-| Loose instructions | 32.49% | 25.78% | — |
-| Publisher prompt/instruction average | — | — | 29.9% |
+| Strict prompts | 17.93% | 15.16% | **21.63%** |
+| Strict instructions | 30.10% | 25.30% | **35.85%** |
+| Loose prompts | 19.22% | 15.71% | **22.55%** |
+| Loose instructions | 32.49% | 25.78% | **37.29%** |
 
-![IFEval accuracy compared with OpenSML-150M and the earlier checkpoint 167](docs/figures/ifeval-comparison.png)
+Solstice uses greedy generation, no system message or repetition penalty, and a 1,024-token output cap. Of 541 prompts, 130 reach that cap; they remain in the scores. The 150M release and the independent SmolLM2 evaluation used a 1,280-token cap and different chat formats.
 
-Solstice uses greedy generation, no system message or repetition penalty, and a 1,024-token output cap. Of 541 prompts, 130 reach that cap; they remain in the scores. The 150M release used a 1,280-token cap and a different chat format.
+Sources: [OpenSML-150M model card](https://huggingface.co/OpenSML/OpenSML-150M) and [PetitGPT’s independent SmolLM2 evaluation](https://github.com/yangqi0/petitgpt/blob/a129f3b709d44ad6a53aef0d44a7054281099df2/README.md#results-at-a-glance).
 
-Baseline scores come from the published [OpenSML-150M card](https://huggingface.co/OpenSML/OpenSML-150M) and [SmolLM2-135M-Instruct card](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct), not fresh reruns. “—” means the required breakdown is not published in those cards. SmolLM2 reports ARC and IFEval as averages; no missing sub-scores are inferred.
-
-Solstice uses native MLX FP32 scoring. SmolLM2 uses LightEval; its [evaluation code](https://github.com/huggingface/smollm/tree/438be18dae658a638f3a6976fdf5190bb6d68619/text/evaluation/smollm2) specifies character-normalized choice scores with a leading space excluded and instruct runs with a chat template. OpenSML-150M uses an older native pipeline. Prompt formatting, precision and generation budgets differ, so these are reference comparisons, not a controlled leaderboard. [Exact sources and metric definitions](results/baselines/published_models.json).
+All three use FP32 choice scoring, but implementations and chat formats differ. Bold marks the highest reported score; these differences have not been tested for statistical significance. [SmolLM2 results and protocol](results/baselines/smollm2_independent.json) · [150M and official-card references](results/baselines/published_models.json).
 
 ### Conversation checks
 
@@ -94,8 +91,6 @@ A separate authored chat panel measures short-answer matching and response behav
 | Natural stopping, all responses | 99.48% |
 | Repetition flags, all responses | 1.56% |
 
-![Answer accuracy by chat category](docs/figures/chat-diagnostics.png)
-
 The panel contains 180 checked answers and 12 open-ended responses. Repetition flags were reviewed: two were clear loops and one was a milder repeated idea. The full panel and scorer are included in [the saved results](results/public/chat/results.json).
 
 On the separate reserved authored test, Solstice reaches **51.9%**, versus **20.0%** after SFT 1 and **25.9%** after SFT 2. On an older, reused panel, the corresponding scores are **53.84%, 52.13% and 54.55%**: the last stage does not improve every measure. The [evaluation report](docs/EVALUATION.md) explains selection, uncertainty and test exposure.

SHA256SUMS

@@ -3,7 +3,7 @@ f09ed498cc42b482dae88d9fb5026bf74e1d8b9187f9ea301aeda0db76d6c859  BACKUP_AND_CLE
 af646a190d135ec25a6ee2c76d1594c3f9ba8005cd3ca6df908606a15023273e  CITATION.cff
 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  LICENSE
 0e6865f1a81cfd69ca48e6987658854ce0c0d23e91d72ae545393cd5fa032375  PACKAGING_VERIFICATION.json
-cf0dc751a9b2091d161c76e1c815ce4ad6dca9f5eeca90a07bae7542b9cc2161  README.md
+650ef6dc4b02ec96ebdcd879f014b99e7caf75aadbd143e3accd4c4924460672  README.md
 152a87d0098752350d54fdd3cde3861a77eb497c53dc973ee1a8370d12a13ff6  REPOSITORIES.json
 fd05e1595bcdc52942e9425b6efb2c7e89553e6b0ad7d5f93d9075969cd75dba  RUN_GUIDE.md
 87fa33719fe1182d467a05ab417be2a362f50eea59f0ac6496d40e677544e27c  SOURCE_NOTICE.md
@@ -30,7 +30,7 @@ fb204be1d9e72eb1cae140bfdb03dbcadc1843c47b1894a731396ceeedb9a22f  configs/sft1/p
 51d66108416988bad1015d7b52cf63db0571f87f202b2b18c7e98e67a29ec62b  configs/sft2/plan.json
 ae708eca3c09d5a27eab3fddbe4106d0658787beebc9393de57f75234480a284  configs/sft3/plan.json
 e0b0c730321360c40f20e53a22890c52cd5294bad2a5777a0a891096557a2f0a  docs/DATA.md
-e890892eb870896a8f5d585bde6ab748820f91b2ff49e8d70434bc14140767a7  docs/EVALUATION.md
+b78247452825d577713115632bfd9900decf64ff991b098156184dec7a41f415  docs/EVALUATION.md
 7a818c365f1c047b349479829396ba5cd841ca8d2f04fa14fc1489083a1fd4b3  docs/EXAMPLES.md
 f98d45b39b9595510fa80d495a3758b211d8c770adc739d381763d3cfb2a2a0b  docs/TECHNICAL_REPORT.md
 37438bfe291e2cc5b7513ec912cab9eaf7ce3d6352543f8e63d44da3a4f7d5b3  docs/figures/benchmark-comparison.png
@@ -97,6 +97,7 @@ f76237736d1aab8f04517ad9e3336925ca9773399a9ad888f07c15d3ceaa4f87  results/baseli
 a5d49e3dc60a6c4dda4799090881cc2aab033b922ac0b5dafd52c0c6c7e2e6d8  results/baselines/attempt5.json
 2b59f728ebc9bef6deaf378e6908f7b55723b368f60da83f9af5b2b1731d0346  results/baselines/parent2000.json
 2cafe36ca3f2fe3dd00cf9aa63314eac64a73de2c58e2de92cdeee47306bd8ae  results/baselines/published_models.json
+99d582c9ed4e90d87f16835e814beb4604687519734c7a74972367a61f8c10c9  results/baselines/smollm2_independent.json
 fed55313b61008e8f379a7b4bc49535cc501a31a942c7560687de4a4bd1cd47f  results/package_inference_verification.json
 ba2cfb7c8b446dede8e7f33bb78885855e74bd75f861b0c201d6e01e88bc987f  results/pretraining_steps.csv
 c4e3210780814051e3ad1bd99a9c143b9c5dbe910224239a134fddc546e9c9a8  results/pretraining_validation.csv

docs/EVALUATION.md

@@ -6,12 +6,12 @@ All results identify the same release weight hash, `7b2d8777040f05d8e0849acac4e4
 
 | Benchmark | Solstice-1 acc / acc_norm | OpenSML-150M acc / acc_norm | SmolLM2-135M Instruct acc / acc_norm |
 | --- | --- | --- | --- |
-| ARC Easy | 62.58 / 62.33 | 56.65 / 55.43 | — / — |
-| ARC Challenge | 29.35 / 30.89 | 26.02 / 29.52 | — / — |
-| PIQA | 66.49 / 66.21 | 64.53 / 64.09 | — / 66.3 |
-| HellaSwag | 32.85 / 39.40 | 30.55 / 33.94 | — / 40.9 |
+| ARC Easy | **62.58** / **62.33** | 56.65 / 55.43 | 54.00 / 48.82 |
+| ARC Challenge | **29.35** / **30.89** | 26.02 / 29.52 | 25.94 / 27.73 |
+| PIQA | 66.49 / 66.21 | 64.53 / 64.09 | **66.70** / **66.76** |
+| HellaSwag | 32.85 / 39.40 | 30.55 / 33.94 | **35.02** / **42.90** |
 
-All values are percentages, shown as **acc / acc_norm**. SmolLM2 publishes ARC only as a combined average (**37.3% acc_norm**); separate Easy/Challenge scores are unavailable in its card.
+Values are percentages, shown as **acc / acc_norm**. Bold marks the highest reported value for each metric. SmolLM2 scores are from an independent published evaluation; protocol differences are noted below.
 
 Native MLX evaluates complete ARC Easy/Challenge test splits, PIQA validation and HellaSwag validation. It uses zero-shot raw completion prompts, no chat template or inserted BOS/EOS, FP32 weights, vanilla attention, batch size four and chunked scoring. Raw accuracy ranks summed continuation log-likelihood; normalized accuracy divides by choice Unicode-character count. No generated explanation or LLM judge determines the answer. No choices were context-truncated. Dataset revisions and scorer settings are preserved in [the result record](../results/public/choices/results.json).
 
@@ -19,38 +19,33 @@ Native MLX evaluates complete ARC Easy/Challenge test splits, PIQA validation an
 
 | IFEval accuracy | Solstice-1 | OpenSML-150M | SmolLM2-135M Instruct |
 | --- | --- | --- | --- |
-| Strict prompts | 17.93% | 15.16% | — |
-| Strict instructions | 30.10% | 25.30% | — |
-| Loose prompts | 19.22% | 15.71% | — |
-| Loose instructions | 32.49% | 25.78% | — |
-| Publisher prompt/instruction average | — | — | 29.9% |
+| Strict prompts | 17.93% | 15.16% | **21.63%** |
+| Strict instructions | 30.10% | 25.30% | **35.85%** |
+| Loose prompts | 19.22% | 15.71% | **22.55%** |
+| Loose instructions | 32.49% | 25.78% | **37.29%** |
 
 Google IFEval is pinned to dataset revision `966cd89545d6b6acfd7638bc708b98261ca58e84` and google-research commit `83f1e7b59e762648c953de751d31f75720683622`. All 541 prompts and 834 instructions use the recorded strict/loose programmatic verifiers. Generation is greedy, native-chat formatted, FP32, without a system message or repetition penalty. Output cap: 1,024 tokens. Token-cap hits: **130/541**; context-cap hits: **0**. All generated responses are scored, including cap hits. The official dataset's split is named `train`; it is used as evaluation input here, not SFT training material.
 
 ## Published small-model references
 
-Baseline scores come from the published [OpenSML-150M card](https://huggingface.co/OpenSML/OpenSML-150M) and [SmolLM2-135M-Instruct card](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct), not fresh reruns. “—” means the required breakdown is not published in those cards. SmolLM2 reports ARC and IFEval as averages; no missing sub-scores are inferred.
+Sources: [OpenSML-150M model card](https://huggingface.co/OpenSML/OpenSML-150M) and [PetitGPT’s independent SmolLM2 evaluation](https://github.com/yangqi0/petitgpt/blob/a129f3b709d44ad6a53aef0d44a7054281099df2/README.md#results-at-a-glance).
 
-Solstice uses native MLX FP32 scoring. SmolLM2 uses LightEval; its [evaluation code](https://github.com/huggingface/smollm/tree/438be18dae658a638f3a6976fdf5190bb6d68619/text/evaluation/smollm2) specifies character-normalized choice scores with a leading space excluded and instruct runs with a chat template. OpenSML-150M uses an older native pipeline. Prompt formatting, precision and generation budgets differ, so these are reference comparisons, not a controlled leaderboard. [Exact sources and metric definitions](../results/baselines/published_models.json).
+All three use FP32 choice scoring, but implementations and chat formats differ. Bold marks the highest reported score; these differences have not been tested for statistical significance. [SmolLM2 results and protocol](../results/baselines/smollm2_independent.json) · [150M and official-card references](../results/baselines/published_models.json).
 
-![Published normalized-accuracy comparison](figures/benchmark-comparison.png)
-
-![IFEval comparison](figures/ifeval-comparison.png)
-
-Figure data is read directly from the saved JSON records by [plot_results.py](../scripts/plot_results.py). Run that script with matplotlib 3.10.8 to regenerate PNG and SVG figures. The SmolLM2 card does not identify how strict versus loose IFEval metrics contribute to its published average, so that average is not assigned to any one of Solstice's four IFEval metrics.
+The independent evaluation pins SmolLM2 to revision `12fd25f77366fa6b3b4b768ec3050bf629380bac`. Choice scoring uses complete splits, FP32, batch size one, raw completion prompts and character-length normalization. IFEval uses the model's native template, including its default system text, with 233/541 responses reaching the 1,280-token cap. These are reported measurements from another project, not a rerun in the Solstice release environment. The official Hugging Face card's aggregate scores are retained separately in the source record and are not mixed into these tables.
 
 ## Historical comparison
 
 | Metric | Published 150M | 167 | Parent 2000 | Prior attempt 5 | Solstice-1 |
 | --- | --- | --- | --- | --- | --- |
-| ARC Easy acc_norm | 55.43% | 61.49% | 62.33% | 62.33% | 62.33% |
-| ARC Challenge acc_norm | 29.52% | 30.46% | 30.63% | 31.23% | 30.89% |
-| PIQA acc_norm | 64.09% | 65.67% | 66.43% | 66.43% | 66.21% |
-| HellaSwag acc_norm | 33.94% | 38.60% | 39.51% | 39.44% | 39.40% |
-| IFEval strict prompt | 15.16% | 16.45% | 14.97% | 15.53% | 17.93% |
-| IFEval strict instruction | 25.30% | 29.14% | 26.86% | 28.18% | 30.10% |
-| IFEval loose prompt | 15.71% | 17.93% | 16.82% | 16.45% | 19.22% |
-| IFEval loose instruction | 25.78% | 31.29% | 29.62% | 30.34% | 32.49% |
+| ARC Easy acc_norm | 55.43% | 61.49% | **62.33%** | **62.33%** | **62.33%** |
+| ARC Challenge acc_norm | 29.52% | 30.46% | 30.63% | **31.23%** | 30.89% |
+| PIQA acc_norm | 64.09% | 65.67% | **66.43%** | **66.43%** | 66.21% |
+| HellaSwag acc_norm | 33.94% | 38.60% | **39.51%** | 39.44% | 39.40% |
+| IFEval strict prompt | 15.16% | 16.45% | 14.97% | 15.53% | **17.93%** |
+| IFEval strict instruction | 25.30% | 29.14% | 26.86% | 28.18% | **30.10%** |
+| IFEval loose prompt | 15.71% | 17.93% | 16.82% | 16.45% | **19.22%** |
+| IFEval loose instruction | 25.78% | 31.29% | 29.62% | 30.34% | **32.49%** |
 
 Solstice, parent 2000, prior attempt 5 and 167 share the recorded native FP32 protocol. The 150M numbers are from its [published model card](https://huggingface.co/OpenSML/OpenSML-150M), not a fresh release-bundle rerun. Its IFEval uses plain `User:/Assistant:` formatting, EOS ID 1 and a 1,280-token cap; newer models use structural chat IDs and 1,024 tokens. The earlier choice run uses batch size one. These differences limit controlled head-to-head claims. No external-model leaderboard rank or statistical significance of every small benchmark difference is asserted.
 
@@ -77,8 +72,6 @@ The new panel flags unnecessary abstention on 111/200 answerable extraction case
 
 Manual inspection of the three chat repetition flags found two clear loops and one milder repeated sentence/idea. Keep the original three-flag result for consistent comparisons; it does not mean all three are equally severe. All 180 objective chat responses stop naturally; the qualitative cases account for the single token-cap hit and the repetition flags.
 
-![Chat category diagnostics](figures/chat-diagnostics.png)
-
 ## Selection and test exposure
 
 The final five-trial search selected one checkpoint using the 300-case new development panel plus historical development, stopping, repetition and prose-retention guards. Finalist identity was frozen before the new 1,000-case test. Small tolerated regressions were part of the declared guards; passing them does not mean every metric improved. The complete comparison and selection record includes the thresholds and reasons.

results/baselines/smollm2_independent.json

@@ -0,0 +1,62 @@
+{
+  "model": "HuggingFaceTB/SmolLM2-135M-Instruct",
+  "model_revision": "12fd25f77366fa6b3b4b768ec3050bf629380bac",
+  "evaluation_author": "PetitGPT project (Yang Qi)",
+  "type": "independent published evaluation; not the official Hugging Face benchmark run",
+  "checked_on": "2026-11-09",
+  "source_revision": "a129f3b709d44ad6a53aef0d44a7054281099df2",
+  "choice_source": "https://github.com/yangqi0/petitgpt/blob/a129f3b709d44ad6a53aef0d44a7054281099df2/docs/petitgpt-v1/tables/PUBLIC_BENCHMARK_RESULTS.csv",
+  "ifeval_source": "https://github.com/yangqi0/petitgpt/blob/a129f3b709d44ad6a53aef0d44a7054281099df2/docs/petitgpt-v1/tables/IFEVAL_RESULTS.csv",
+  "model_identity_source": "https://github.com/yangqi0/petitgpt/blob/a129f3b709d44ad6a53aef0d44a7054281099df2/configs/research-v1/benchmark_models.json",
+  "protocol_source": "https://github.com/yangqi0/petitgpt/blob/a129f3b709d44ad6a53aef0d44a7054281099df2/README.md",
+  "choices": {
+    "arc_easy": {
+      "acc": 0.539983164983165,
+      "acc_norm": 0.4882154882154882,
+      "evaluated": 2376,
+      "dataset": "allenai/ai2_arc",
+      "dataset_revision": "210d026faf9955653af8916fad021475a3f00453",
+      "result_version": "FP32_V2"
+    },
+    "piqa": {
+      "acc": 0.6670293797606094,
+      "acc_norm": 0.6675734494015234,
+      "evaluated": 1838,
+      "dataset": "baber/piqa",
+      "dataset_revision": "142f6d7367fd9877f0fb3b5734ea6a545f54cdd1",
+      "result_version": "FP32_V2"
+    },
+    "arc_challenge": {
+      "acc": 0.2593856655290102,
+      "acc_norm": 0.2773037542662116,
+      "evaluated": 1172,
+      "dataset": "allenai/ai2_arc",
+      "dataset_revision": "210d026faf9955653af8916fad021475a3f00453",
+      "result_version": "BENCHMARK_EXTENSION_V1"
+    },
+    "hellaswag": {
+      "acc": 0.35022903804023103,
+      "acc_norm": 0.4289982075283808,
+      "evaluated": 10042,
+      "dataset": "Rowan/hellaswag",
+      "dataset_revision": "218ec52e09a7e7462a5400043bb9a69a41d06b76",
+      "result_version": "BENCHMARK_EXTENSION_V1"
+    }
+  },
+  "ifeval": {
+    "strict": {
+      "prompt_accuracy": 0.21626617375231053,
+      "instruction_accuracy": 0.35851318944844124
+    },
+    "loose": {
+      "prompt_accuracy": 0.2255083179297597,
+      "instruction_accuracy": 0.37290167865707435
+    }
+  },
+  "protocol": {
+    "choices": "Full splits, zero-shot completion likelihood; FP32 parameters/forward, no chat template or BOS/EOS insertion, batch size 1, character-length normalization excluding leading delimiter; native PyTorch evaluator, not installed lm-eval.",
+    "ifeval": "541 prompts, 834 instructions; greedy native chat template including SmolLM2's default system text; 1280 generated-token cap, 233 cap hits; official strict/loose verifiers.",
+    "ifeval_dataset_revision": "966cd89545d6b6acfd7638bc708b98261ca58e84",
+    "solstice_comparison": "Reference comparison, not a new controlled rerun: native MLX vs PyTorch, different tokenizers/templates, Solstice 1024-token IFEval cap. Values from the official SmolLM2 card are retained separately and are not mixed into this result set."
+  }
+}