aethertp commited on
Commit
6650df0
·
verified ·
1 Parent(s): 80dcc21

Synchronize unified cross-generation benchmark table

Browse files
Files changed (1) hide show
  1. README.md +13 -12
README.md CHANGED
@@ -45,22 +45,23 @@ PicoLM-V3 Standard is specialized for **natural dialogue, commonsense reasoning,
45
 
46
  ---
47
 
48
- ## Evaluation & Benchmark Comparisons
49
 
50
- All models evaluated strictly using the **EleutherAI LM-Evaluation-Harness** standard (character-length normalized log-likelihood scoring for multiple-choice tasks, exact match continuation):
51
 
52
- | Model | Parameters | Training Tokens | ARC-Easy (`acc_norm`) | ARC-Challenge (`acc_norm`) | PIQA (`acc_norm`) | HellaSwag (`acc_norm`) |
53
- | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
54
- | **PicoLM-V3 (Standard)** | **82.2M** | **758M** | **39.32%** | **23.63%** | **58.49%** | **36.00%** |
55
- | PicoLM-V2.1-Instruct | 81.9M | ~380M | ~28.0% | ~22.5% | ~56.5% | ~32.0% |
56
- | GPT-2 (OpenAI) | 124M | ~10B | 31.40% | 22.10% | 62.80% | 31.50% |
57
- | MobileLLM-125M (Meta) | 125M | 1.0T | 43.90% | 27.10% | 65.30% | 38.90% |
58
- | SmolLM-135M (Hugging Face) | 135M | 600B | 43.20% | 28.50% | 67.20% | 41.20% |
59
- | SmolLM2-135M (Hugging Face)| 135M | 2.0T | 43.90% | 30.20% | 68.40% | 42.10% |
60
 
61
- > **Compute-Efficiency Note:** While competitors such as SmolLM2 and MobileLLM were trained on **600 Billion to 2 Trillion tokens** (>1,000× more compute), PicoLM-V3 achieves competitive commonsense reasoning (HellaSwag 36.00%, PIQA 58.49%) using only **758 Million tokens** on community compute.
 
 
 
 
 
 
 
 
 
 
62
 
63
- ---
64
 
65
  ## Pre-Training & SFT Pipeline
66
 
 
45
 
46
  ---
47
 
 
48
 
49
+ ## Standardized Evaluation & Generational Progress
50
 
51
+ All models evaluated strictly using the **EleutherAI LM-Evaluation-Harness** standard:
 
 
 
 
 
 
 
52
 
53
+ | Model | Parameters | Tokens | Context | Legacy Scale (250 Q, Raw) | Official ARC-Easy (`acc_norm`) | PIQA (`acc_norm`) | HellaSwag (`acc_norm`) | ARC-Challenge (`acc_norm`) |
54
+ | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
55
+ | **PicoLM-V3 (Standard)** | **82.2M** | **758M** | **2,048** | **44.80%** | **39.32%** | **58.49%** | **36.00%** | **23.63%** |
56
+ | PicoLM-V3-Pro (Flagship)| 82.2M | 1.62B | 4,096 | 46.00% | 43.54% (Base) / 39.10% | 57.89% | 34.95% | 24.06% |
57
+ | PicoLM-V2.1-Instruct | 81.9M | ~380M | 2,048 | 42.00%* | *(legacy 250Q only)* | ~56.5% | ~32.0% | ~22.5% |
58
+ | PicoLM-V2-Instruct | 81.9M | ~380M | 2,048 | 42.00%* | *(legacy 250Q only)* | ~56.5% | ~32.0% | ~22.5% |
59
+ | GPT-2 (OpenAI) | 124M | ~10B | 1,024 | - | 31.40% | 62.80% | 31.50% | 22.10% |
60
+ | MobileLLM-125M (Meta) | 125M | 1.0T | 2,048 | - | 43.90% | 65.30% | 38.90% | 27.10% |
61
+ | SmolLM2-135M (HF) | 135M | 2.0T | 8,192 | - | 43.90% | 68.40% | 42.10% | 30.20% |
62
+
63
+ > *\*PicoLM-V2 and V2.1 were evaluated exclusively on the legacy unnormalized 250-question sample. On the exact same 250-question scale, PicoLM-V3 outperforms V2 by +2.8 points (44.80% vs 42.00%).*
64
 
 
65
 
66
  ## Pre-Training & SFT Pipeline
67