Compare Solstice-1 with OpenSML-150M in model card

FILE_MANIFEST.json

@@ -22,8 +22,8 @@
       "sha256": "0e6865f1a81cfd69ca48e6987658854ce0c0d23e91d72ae545393cd5fa032375"
     },
     "README.md": {
-      "bytes": 13125,
-      "sha256": "fb8c647b3adb0d7fdbd8585efac0db74137179c8fcbe293eed09e881d5a2cba3"
+      "bytes": 12413,
+      "sha256": "df6c6d55774e6b788d24a47fdaa57695b05b0d4a7e84fa8f03923e2e830897a7"
     },
     "REPOSITORIES.json": {
       "bytes": 723,

README.md

@@ -57,32 +57,32 @@ Zero-shot likelihood scoring on the complete ARC Easy, ARC Challenge, PIQA and H
 
 <table>
 <tbody>
-<tr><td role="columnheader" align="center"><strong>Benchmark</strong></td><td role="columnheader" align="center"><strong>Solstice-1</strong><br><sub><em>acc / acc_norm</em></sub></td><td role="columnheader" align="center"><strong>OpenSML-150M</strong><br><sub><em>acc / acc_norm</em></sub></td><td role="columnheader" align="center"><strong>SmolLM2-135M</strong><br><sub><em>acc / acc_norm</em></sub></td></tr>
-<tr><td>ARC Easy</td><td><strong>62.58</strong> / <strong>62.33</strong></td><td>56.65 / 55.43</td><td>54.00 / 48.82</td></tr>
-<tr><td>ARC Challenge</td><td><strong>29.35</strong> / <strong>30.89</strong></td><td>26.02 / 29.52</td><td>25.94 / 27.73</td></tr>
-<tr><td>PIQA</td><td>66.49 / 66.21</td><td>64.53 / 64.09</td><td><strong>66.70</strong> / <strong>66.76</strong></td></tr>
-<tr><td>HellaSwag</td><td>32.85 / 39.40</td><td>30.55 / 33.94</td><td><strong>35.02</strong> / <strong>42.90</strong></td></tr>
+<tr><td role="columnheader" align="center"><strong>Benchmark</strong></td><td role="columnheader" align="center"><strong>Solstice-1</strong><br><sub><em>acc / acc_norm</em></sub></td><td role="columnheader" align="center"><strong>OpenSML-150M</strong><br><sub><em>acc / acc_norm</em></sub></td></tr>
+<tr><td>ARC Easy</td><td><strong>62.58</strong> / <strong>62.33</strong></td><td>56.65 / 55.43</td></tr>
+<tr><td>ARC Challenge</td><td><strong>29.35</strong> / <strong>30.89</strong></td><td>26.02 / 29.52</td></tr>
+<tr><td>PIQA</td><td><strong>66.49</strong> / <strong>66.21</strong></td><td>64.53 / 64.09</td></tr>
+<tr><td>HellaSwag</td><td><strong>32.85</strong> / <strong>39.40</strong></td><td>30.55 / 33.94</td></tr>
 </tbody>
 </table>
 
-Values are percentages, shown as **acc / acc_norm**. Bold marks the highest reported value for each metric. SmolLM2 scores are from an independent published evaluation; protocol differences are noted below.
+Values are percentages, shown as **acc / acc_norm**. Bold marks the higher score for each metric in this comparison.
 
 ### Instruction following
 
 IFEval checks whether generated responses follow explicit instructions. It measures accuracy, with separate strict and loose scoring at the prompt and instruction levels. **It has no `acc_norm` metric.**
 
-| IFEval accuracy | Solstice-1 | OpenSML-150M | SmolLM2-135M |
-| --- | --- | --- | --- |
-| Strict prompts | 17.93% | 15.16% | **21.63%** |
-| Strict instructions | 30.10% | 25.30% | **35.85%** |
-| Loose prompts | 19.22% | 15.71% | **22.55%** |
-| Loose instructions | 32.49% | 25.78% | **37.29%** |
+| IFEval accuracy | Solstice-1 | OpenSML-150M |
+| --- | --- | --- |
+| Strict prompts | **17.93%** | 15.16% |
+| Strict instructions | **30.10%** | 25.30% |
+| Loose prompts | **19.22%** | 15.71% |
+| Loose instructions | **32.49%** | 25.78% |
 
-Solstice uses greedy generation, no system message or repetition penalty, and a 1,024-token output cap. Of 541 prompts, 130 reach that cap; they remain in the scores. The 150M release and the independent SmolLM2 evaluation used a 1,280-token cap and different chat formats.
+Solstice uses greedy generation, no system message or repetition penalty, and a 1,024-token output cap. Of 541 prompts, 130 reach that cap; they remain in the scores. The 150M release used a 1,280-token cap and a different chat format.
 
-Sources: [OpenSML-150M model card](https://huggingface.co/OpenSML/OpenSML-150M) and [PetitGPT’s independent SmolLM2 evaluation](https://github.com/yangqi0/petitgpt/blob/a129f3b709d44ad6a53aef0d44a7054281099df2/README.md#results-at-a-glance).
+Comparison source: [OpenSML-150M model card](https://huggingface.co/OpenSML/OpenSML-150M).
 
-The SmolLM2 column refers to **SmolLM2-135M-Instruct**. All three use FP32 choice scoring, but implementations and chat formats differ. Bold marks the highest reported score; these differences have not been tested for statistical significance. [SmolLM2 results and protocol](results/baselines/smollm2_independent.json) · [150M and official-card references](results/baselines/published_models.json).
+Both models use FP32 choice scoring, but implementations and chat formats differ. The score differences have not been tested for statistical significance. [150M results and protocol](results/baselines/published_models.json).
 
 ### Conversation checks
 

SHA256SUMS

@@ -3,7 +3,7 @@ f09ed498cc42b482dae88d9fb5026bf74e1d8b9187f9ea301aeda0db76d6c859  BACKUP_AND_CLE
 af646a190d135ec25a6ee2c76d1594c3f9ba8005cd3ca6df908606a15023273e  CITATION.cff
 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  LICENSE
 0e6865f1a81cfd69ca48e6987658854ce0c0d23e91d72ae545393cd5fa032375  PACKAGING_VERIFICATION.json
-fb8c647b3adb0d7fdbd8585efac0db74137179c8fcbe293eed09e881d5a2cba3  README.md
+df6c6d55774e6b788d24a47fdaa57695b05b0d4a7e84fa8f03923e2e830897a7  README.md
 291c0f7dc01cd1c0aeb5f49bb8822e0ab32617c72a6886615673ab08e9431891  REPOSITORIES.json
 cf237498a98efb229977939cfceaf78a112f25658f32559cad41aab3b8880202  RUN_GUIDE.md
 87fa33719fe1182d467a05ab417be2a362f50eea59f0ac6496d40e677544e27c  SOURCE_NOTICE.md
@@ -473,4 +473,4 @@ ff056da7218dfbcae9d5c30e1a7770739683866bff398eebc05f3cfd686b1957  training/sourc
 ff84dc0dfacceb506e54b6af73c1a76d65fcbf9cf75c7d96fc4489b8ab4f79e6  training/source_blobs/ff84dc0dfacceb506e54b6af73c1a76d65fcbf9cf75c7d96fc4489b8ab4f79e6.py
 ff9cd7e99a7b66c901137a7f61c97c9ecbd3e5effec879cb82a5c68e9ced35f7  training/source_blobs/ff9cd7e99a7b66c901137a7f61c97c9ecbd3e5effec879cb82a5c68e9ced35f7.py
 ffc3d395e98ab0f55c9d38060d65ff8dcdae4b954a25aafb40c14f513d8067ba  training/source_blobs/ffc3d395e98ab0f55c9d38060d65ff8dcdae4b954a25aafb40c14f513d8067ba.py
-b5db6ca19833c36e95806b520265c1a27dd10485405c2cd1912dcc21c7a3949c  FILE_MANIFEST.json
+4831e3256d28b7a04a99b60dc1fcceb8f1e0f29225b97931b843e30985f37f6c  FILE_MANIFEST.json