yiqichen01 commited on
Commit
2c325e2
·
verified ·
1 Parent(s): 2095415
Files changed (4) hide show
  1. .gitattributes +1 -0
  2. README.md +50 -0
  3. README_CN.md +50 -0
  4. pretrained-model-bpb.png +3 -0
.gitattributes CHANGED
@@ -37,3 +37,4 @@ assets/benchmark.png filter=lfs diff=lfs merge=lfs -text
37
  assets/dfly2.png filter=lfs diff=lfs merge=lfs -text
38
  assets/performance-cost-open-benchmarks.png filter=lfs diff=lfs merge=lfs -text
39
  assets/performance-cost-workindex.png filter=lfs diff=lfs merge=lfs -text
 
 
37
  assets/dfly2.png filter=lfs diff=lfs merge=lfs -text
38
  assets/performance-cost-open-benchmarks.png filter=lfs diff=lfs merge=lfs -text
39
  assets/performance-cost-workindex.png filter=lfs diff=lfs merge=lfs -text
40
+ pretrained-model-bpb.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -46,6 +46,7 @@ tags:
46
  ## Table of Contents
47
 
48
  - [Model Introduction](#model-introduction)
 
49
  - [Practicality and Cost Efficiency](#practicality-and-cost-efficiency)
50
  - [Optimization Approach](#optimization-approach)
51
  - [Task Performance and Cost](#task-performance-and-cost)
@@ -131,6 +132,55 @@ Compared with Hy4, Hy4 Flash introduces three notable architectural changes:
131
  | Vision Position Embedding Size | 128 × 128 |
132
  | Visual Feature Merging | 2 x 2 |
133
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
134
  ## Practicality and Cost Efficiency
135
 
136
  ### Optimization Approach
 
46
  ## Table of Contents
47
 
48
  - [Model Introduction](#model-introduction)
49
+ - [Pretrained model capability](#pretrained-model-capability)
50
  - [Practicality and Cost Efficiency](#practicality-and-cost-efficiency)
51
  - [Optimization Approach](#optimization-approach)
52
  - [Task Performance and Cost](#task-performance-and-cost)
 
132
  | Vision Position Embedding Size | 128 × 128 |
133
  | Visual Feature Merging | 2 x 2 |
134
 
135
+ ## Pretrained model capability
136
+
137
+ Bits-per-byte (BPB) on internal code & agent validation, lower is better:
138
+
139
+ <p align="center">
140
+ <a href="pretrained-model-bpb.png"><img src="pretrained-model-bpb.png" alt="Bits-per-byte on internal code and agent validation versus active parameters; lower is better" width="100%"/></a>
141
+ </p>
142
+
143
+ Few-shot pretrain benchmark evaluation:
144
+
145
+ | Category | Benchmark | #Shots | Hy4 flash Base | Hy4 Base | Hy3 preview Base | DeepSeek-V4-flash-Base | DeepSeek-V4-Pro-Base |
146
+ | --- | --- | :---: | ---: | ---: | ---: | ---: | ---: |
147
+ | **Knowledge** | MMLU† | 5 | 88.7 | **91.6** | 87.5 | 89.7 | 91.1 |
148
+ | | MMLU-Pro† | 5 | 71.1 | **73.9** | 65.8 | 68.6 | 73.4 |
149
+ | | MMLU-Redux† | 5 | 87.6 | **90.6** | 87.0 | 88.8 | 90.2 |
150
+ | | SuperGPQA† | 5 | 57.8 | **62.7** | 52.3 | 52.5 | 54.8 |
151
+ | | C-Eval† | 5 | 92.0 | **95.0** | 90.1 | 91.9 | 93.4 |
152
+ | | CMMLU† | 5 | 91.6 | **93.1** | 89.6 | 89.8 | 90.7 |
153
+ | | SimpleQA | 5 | 37.5 | **57.2** | 26.4 | 31.3 | 52.1 |
154
+ | | Chinese-SimpleQA† | 5 | 74.6 | **83.9** | 69.6 | 71.9 | 76.6 |
155
+ | | AGIEval (English)† | 3 | 75.4 | **79.3** | 76.0 | 76.3 | 78.2 |
156
+ | | AGIEval (Chinese)† | 3 | 90.6 | **91.2** | 87.8 | 88.5 | 89.8 |
157
+ | **Reasoning** | HellaSwag (3,000-item subset) | 10 | 85.2 | **87.1** | 85.7 | 85.1 | 86.8 |
158
+ | | PIQA | 4 | 84.4 | **85.3** | 84.6 | 83.8 | 84.9 |
159
+ | | ARC-Challenge | 0 | 95.0 | **97.3** | 96.0 | **97.3** | 97.0 |
160
+ | | Social IQa | 0 | 80.1 | **82.7** | 80.5 | 72.5 | 73.1 |
161
+ | | KOR-Bench | 3 | 54.7 | **57.6** | 54.6 | 53.4 | 55.4 |
162
+ | | C³ | 0 | 77.3 | **82.4** | 75.9 | 74.1 | 78.3 |
163
+ | **Math** | U-MATH | 5 | 69.8 | **71.6** | 60.9 | 57.4 | 64.8 |
164
+ | | OCW Courses | 5 | 64.0 | 66.7 | 62.6 | 63.3 | **68.2** |
165
+ | **Code** | MBPP-plus | 3 | 73.7 | 74.0 | 77.4 | 76.8 | **79.1** |
166
+ | | BigCodeBench (Full) | 0 | 58.8 | 47.9 | **59.6** | 58.0 | 57.5 |
167
+ | **Multilingual** | Belebele (5-language subset)† | 5 | 93.8 | 95.0 | 94.2 | 94.4 | **95.2** |
168
+ | | INCLUDE† | 5 | 82.8 | **88.1** | 84.3 | 84.0 | 84.3 |
169
+ | **Multimodal** | MMMU-Pro (Standard, 10 opts) | 4 | 52.0 | **59.6** | — | — | — |
170
+ | | SimpleVQA | 4 | 48.5 | **54.5** | — | — | — |
171
+ | | CVBench | 4 | 83.4 | **86.7** | — | — | — |
172
+ | | RealWorldQA | 4 | 71.1 | **72.3** | — | — | — |
173
+ | | TextVQA | 4 | 73.4 | **78.2** | — | — | — |
174
+ | | CountBenchQA | 4 | 86.6 | **90.6** | — | — | — |
175
+
176
+ † Macro-averaged accuracy for the marked text benchmarks. Other text rows use the accuracy metric reported by our evaluation suite. Bold marks the highest available numerical score in each row; it does not indicate statistical significance.
177
+
178
+ Multimodal results are reported only for Hy4 flash Base and Hy4 Base.
179
+
180
+ ARC-Challenge and HellaSwag use 299-item and 3,000-item subsets, respectively. INCLUDE uses a 5,000-example Lite subset covering 44 languages. Belebele covers German, Korean, Vietnamese, Spanish, and Russian (4,475 examples in total).
181
+
182
+ **Disclaimer: Because our base model was exposed to a large amount of synthetic instruction data during mid-training, it occasionally fails to follow the expected answer format in few-shot evaluations, which may distort the scores.**
183
+
184
  ## Practicality and Cost Efficiency
185
 
186
  ### Optimization Approach
README_CN.md CHANGED
@@ -30,6 +30,7 @@
30
  ## 目录
31
 
32
  - [模型介绍](#模型介绍)
 
33
  - [实用性与成本效率](#实用性与成本效率)
34
  - [优化思路](#优化思路)
35
  - [任务效果与成本](#任务效果与成本)
@@ -115,6 +116,55 @@
115
  | Vision Position Embedding Size | 128 × 128 |
116
  | Visual Feature Merging | 2 x 2 |
117
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
118
  ## 实用性与成本效率
119
 
120
  ### 优化思路
 
30
  ## 目录
31
 
32
  - [模型介绍](#模型介绍)
33
+ - [预训练模型能力](#预训练模型能力)
34
  - [实用性与成本效率](#实用性与成本效率)
35
  - [优化思路](#优化思路)
36
  - [任务效果与成本](#任务效果与成本)
 
116
  | Vision Position Embedding Size | 128 × 128 |
117
  | Visual Feature Merging | 2 x 2 |
118
 
119
+ ## 预训练模型能力
120
+
121
+ 内部代码与 Agent 验证集上的 BPB(Bits-per-byte,每字节比特数),数值越低越好:
122
+
123
+ <p align="center">
124
+ <a href="pretrained-model-bpb.png"><img src="pretrained-model-bpb.png" alt="内部代码与 Agent 验证集上的 BPB 与激活参数量对比,BPB 越低越好" width="100%"/></a>
125
+ </p>
126
+
127
+ 预训练模型的 few-shot 基准评测结果:
128
+
129
+ | 类别 | 评测基准 | #Shots | Hy4 flash Base | Hy4 Base | Hy3 preview Base | DeepSeek-V4-flash-Base | DeepSeek-V4-Pro-Base |
130
+ | --- | --- | :---: | ---: | ---: | ---: | ---: | ---: |
131
+ | **知识** | MMLU† | 5 | 88.7 | **91.6** | 87.5 | 89.7 | 91.1 |
132
+ | | MMLU-Pro† | 5 | 71.1 | **73.9** | 65.8 | 68.6 | 73.4 |
133
+ | | MMLU-Redux† | 5 | 87.6 | **90.6** | 87.0 | 88.8 | 90.2 |
134
+ | | SuperGPQA† | 5 | 57.8 | **62.7** | 52.3 | 52.5 | 54.8 |
135
+ | | C-Eval† | 5 | 92.0 | **95.0** | 90.1 | 91.9 | 93.4 |
136
+ | | CMMLU† | 5 | 91.6 | **93.1** | 89.6 | 89.8 | 90.7 |
137
+ | | SimpleQA | 5 | 37.5 | **57.2** | 26.4 | 31.3 | 52.1 |
138
+ | | Chinese-SimpleQA† | 5 | 74.6 | **83.9** | 69.6 | 71.9 | 76.6 |
139
+ | | AGIEval (English)† | 3 | 75.4 | **79.3** | 76.0 | 76.3 | 78.2 |
140
+ | | AGIEval (Chinese)† | 3 | 90.6 | **91.2** | 87.8 | 88.5 | 89.8 |
141
+ | **推理** | HellaSwag (3,000-item subset) | 10 | 85.2 | **87.1** | 85.7 | 85.1 | 86.8 |
142
+ | | PIQA | 4 | 84.4 | **85.3** | 84.6 | 83.8 | 84.9 |
143
+ | | ARC-Challenge | 0 | 95.0 | **97.3** | 96.0 | **97.3** | 97.0 |
144
+ | | Social IQa | 0 | 80.1 | **82.7** | 80.5 | 72.5 | 73.1 |
145
+ | | KOR-Bench | 3 | 54.7 | **57.6** | 54.6 | 53.4 | 55.4 |
146
+ | | C³ | 0 | 77.3 | **82.4** | 75.9 | 74.1 | 78.3 |
147
+ | **数学** | U-MATH | 5 | 69.8 | **71.6** | 60.9 | 57.4 | 64.8 |
148
+ | | OCW Courses | 5 | 64.0 | 66.7 | 62.6 | 63.3 | **68.2** |
149
+ | **代码** | MBPP-plus | 3 | 73.7 | 74.0 | 77.4 | 76.8 | **79.1** |
150
+ | | BigCodeBench (Full) | 0 | 58.8 | 47.9 | **59.6** | 58.0 | 57.5 |
151
+ | **多语言** | Belebele (5-language subset)† | 5 | 93.8 | 95.0 | 94.2 | 94.4 | **95.2** |
152
+ | | INCLUDE† | 5 | 82.8 | **88.1** | 84.3 | 84.0 | 84.3 |
153
+ | **多模态** | MMMU-Pro (Standard, 10 opts) | 4 | 52.0 | **59.6** | — | — | — |
154
+ | | SimpleVQA | 4 | 48.5 | **54.5** | — | — | — |
155
+ | | CVBench | 4 | 83.4 | **86.7** | — | — | — |
156
+ | | RealWorldQA | 4 | 71.1 | **72.3** | — | — | — |
157
+ | | TextVQA | 4 | 73.4 | **78.2** | — | — | — |
158
+ | | CountBenchQA | 4 | 86.6 | **90.6** | — | — | — |
159
+
160
+ † 标注的文本评测采用宏平均准确率,其余文本评测采用本评测框架报告的准确率。每行最高分以粗体标示;加粗不代表差异具有统计显著性。
161
+
162
+ 多模态结果仅列出 Hy4 flash Base 和 Hy4 Base。
163
+
164
+ ARC-Challenge 和 HellaSwag 分别使用包含 299 道和 3,000 道题目的子集。INCLUDE 使用覆盖 44 种语言、包含 5,000 个样本的 Lite 子集。Belebele 覆盖德语、韩语、越南语、西班牙语和俄语,共 4,475 个样本。
165
+
166
+ **免责声明:我们的基座模型在中期训练中使用了大量合成指令数据,因此在 few-shot 评测中偶尔无法遵循预期的答案格式,可能导致得分失真。**
167
+
168
  ## 实用性与成本效率
169
 
170
  ### 优化思路
pretrained-model-bpb.png ADDED

Git LFS Details

  • SHA256: 1ad7aada7fcac0f34a88b629789bb6dce512b403b310faa5f7201dfc013b9640
  • Pointer size: 131 Bytes
  • Size of remote file: 400 kB