Update README.md

README.md

@@ -1,3 +1,477 @@
 ---
 license: mit
+pipeline_tag: text-generation
 ---
+
+<p align="center">
+    <img src="https://mdn.alipayobjects.com/huamei_qa8qxu/afts/img/A*4QxcQrBlTiAAAAAAQXAAAAgAemJ7AQ/original" width="100" alt="inclusionAI" />
+</p>
+<p align="center">🤗 <a href="https://huggingface.co/inclusionAI">Hugging Face</a>&nbsp;&nbsp; | &nbsp;&nbsp;🤖 <a href="https://modelscope.cn/organization/inclusionAI">ModelScope</a></p>
+
+# Ling-3.1-Flash-Base
+
+## Introduction
+
+We introduce **Ling-3.1-Flash-Base**, the base checkpoint of Ling-3.1-Flash, released to support research and community-driven innovation. With **560B total parameters** and **25B active parameters per token**, the model provides a foundation for continued pretraining, domain adaptation, and post-training research.
+
+Ling-3.1-Flash-Base shares the same hybrid-linear Mixture-of-Experts (MoE) architecture as Ling-3.1-Flash, combining KDA with Gated MLA. As a base checkpoint, it is intended for further training and adaptation.
+
+## Model Overview
+
+### Key Features
+
+- **Sparse MoE architecture:** 512 routed experts, with 8 routed experts and 1 shared expert activated per token, for a total of 25B active parameters per token.
+- **Hybrid linear attention:** 63 KDA layers and 9 Gated MLA layers combine linear attention with Gated MLA at a 7:1 ratio.
+
+### Model Specifications
+
+| Specification | Value |
+| --- | --- |
+| Model Type | Base checkpoint |
+| Architecture | Hybrid-linear MoE |
+| Parameter Scale | Total 560B, Activated 25B |
+| Transformer Layers | 63 KDA + 9 Gated MLA (7:1) |
+| Number of Dense Layers | 4 |
+| Number of Routed Experts | 512 |
+| Number of Shared Experts | 1 |
+| Number of Activated Routed Experts | 8 |
+| Attention Heads | 64 |
+| Hidden Size | 4096 |
+| Expert Intermediate Size | 1280 |
+| Dense Intermediate Size | 10240 |
+| Vocabulary Size | 157184 |
+
+### Architecture Diagram
+
+![Ling-3.1-flash architecture](https://cdn-uploads.huggingface.co/production/uploads/678143d4ddb1f1ce16daca8e/qwmfVu_K08auI7R_ijbIX.png)
+
+## Base Model Evaluation
+
+We evaluate **Ling-3.1-Flash-Base** on 42 benchmarks across eight domains: language understanding, knowledge, code, mathematics, reasoning, multilingual capabilities, medical knowledge, and long-context understanding. The results below compare the model with hy3-preview-Base, DeepSeek-V4-Pro-Base, and MiMo-V2.5-Pro-Base under the shot configurations shown.
+
+**Bold** indicates the highest score in each row; <u>underlining</u> indicates the second-highest distinct score. Tied scores receive the same formatting. EM denotes exact match, and Acc denotes accuracy. Parameter scales are shown as total parameters and active parameters per token (A).
+
+<style>
+  .benchmark-wrap { width: 100%; overflow-x: auto; }
+  .benchmark-table { border-collapse: collapse; margin: 0 auto; }
+  .benchmark-table th, .benchmark-table td {
+    padding: 0.55rem 0.8rem;
+    text-align: center !important;
+    vertical-align: middle;
+  }
+  .benchmark-table .ling { background-color: #e7f6ff; }
+  .benchmark-table thead .ling { color: #1677ff; }
+  .benchmark-table thead tr:first-child .ling { border-radius: 28px 28px 0 0; }
+  .benchmark-table tbody tr:last-child .ling { border-radius: 0 0 28px 28px; }
+  .benchmark-table .section-start > * { border-top: 2px solid #d0d7de; }
+  .benchmark-table .table-footnote {
+    padding-top: 0.85rem;
+    text-align: left !important;
+    color: #57606a;
+    font-size: 0.9em;
+  }
+</style>
+<div class="benchmark-wrap">
+  <table class="benchmark-table" align="center">
+    <thead>
+      <tr>
+        <th rowspan="2" scope="col" align="center">Domain</th>
+        <th rowspan="2" scope="col" align="center">Benchmark</th>
+        <th rowspan="2" scope="col" align="center">Shot Config</th>
+        <th class="ling" scope="col" align="center">Ling-3.1-Flash-Base</th>
+        <th scope="col" align="center">hy3-preview-Base</th>
+        <th scope="col" align="center">DeepSeek-V4-Pro-Base</th>
+        <th scope="col" align="center">MiMo-V2.5-Pro-Base</th>
+      </tr>
+      <tr>
+        <th class="ling" align="center">560B A25B</th>
+        <th align="center">295B A21B</th>
+        <th align="center">1.6T A49B</th>
+        <th align="center">1.02T A42B</th>
+      </tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td class="domain" rowspan="5" align="center">Language Understanding</td>
+        <td class="benchmark" align="center">Race-High<sub>(EM)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>93.88</strong></td>
+        <td align="center">93.37</td>
+        <td align="center">93.37</td>
+        <td align="center"><u>93.60</u></td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">SquadV2<sub>(Acc)</sub></td>
+        <td class="shot" align="center">1-shot</td>
+        <td class="ling" align="center"><strong>92.79</strong></td>
+        <td align="center">88.84</td>
+        <td align="center">87.70</td>
+        <td align="center"><u>88.93</u></td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">Belebele<sub>(EM)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>94.67</strong></td>
+        <td align="center"><u>94.44</u></td>
+        <td align="center">93.83</td>
+        <td align="center">92.72</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">C3<sub>(EM)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>83.34</strong></td>
+        <td align="center">79.89</td>
+        <td align="center"><u>81.64</u></td>
+        <td align="center">81.53</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">CCPM<sub>(EM)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><u>91.88</u></td>
+        <td align="center">88.64</td>
+        <td align="center"><strong>93.31</strong></td>
+        <td align="center">90.18</td>
+      </tr>
+      <tr class="section-start">
+        <td class="domain" rowspan="10" align="center">Knowledge</td>
+        <td class="benchmark" align="center">ARC-c<sub>(EM)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>97.97</strong></td>
+        <td align="center">94.58</td>
+        <td align="center">96.61</td>
+        <td align="center"><u>96.95</u></td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">AGIEval<sub>(Acc)</sub></td>
+        <td class="shot" align="center">few-shot(0~5)</td>
+        <td class="ling" align="center"><strong>82.59</strong></td>
+        <td align="center">77.61</td>
+        <td align="center"><u>81.26</u></td>
+        <td align="center">79.90</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">SimpleQA-Verified<sub>(Acc)</sub></td>
+        <td class="shot" align="center">5-shot</td>
+        <td class="ling" align="center"><strong>52.40</strong></td>
+        <td align="center">24.20</td>
+        <td align="center"><u>52.10</u></td>
+        <td align="center">32.60</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">C-SimpleQA<sub>(Acc)</sub></td>
+        <td class="shot" align="center">5-shot</td>
+        <td class="ling" align="center"><u>77.43</u></td>
+        <td align="center">70.57</td>
+        <td align="center"><strong>77.60</strong></td>
+        <td align="center">70.93</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">AA-Omniscience<sub>(Acc)</sub></td>
+        <td class="shot" align="center">3-shot</td>
+        <td class="ling" align="center"><u>37.67</u></td>
+        <td align="center">33.00</td>
+        <td align="center"><strong>48.16</strong></td>
+        <td align="center">36.45</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">MMLU-Pro<sub>(EM)</sub></td>
+        <td class="shot" align="center">5-shot</td>
+        <td class="ling" align="center"><strong>76.71</strong></td>
+        <td align="center">66.44</td>
+        <td align="center"><u>75.28</u></td>
+        <td align="center">71.26</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">CEval<sub>(EM)</sub></td>
+        <td class="shot" align="center">5-shot</td>
+        <td class="ling" align="center"><u>92.36</u></td>
+        <td align="center">90.74</td>
+        <td align="center"><strong>93.76</strong></td>
+        <td align="center">92.08</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">CMMLU<sub>(EM)</sub></td>
+        <td class="shot" align="center">5-shot</td>
+        <td class="ling" align="center"><strong>91.60</strong></td>
+        <td align="center">89.16</td>
+        <td align="center"><u>90.52</u></td>
+        <td align="center">88.33</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">GPQA-Diamond<sub>(EM)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>56.57</strong></td>
+        <td align="center">44.95</td>
+        <td align="center">52.02</td>
+        <td align="center"><u>54.04</u></td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">SuperGPQA<sub>(EM)</sub></td>
+        <td class="shot" align="center">5-shot</td>
+        <td class="ling" align="center"><strong>55.84</strong></td>
+        <td align="center">52.06</td>
+        <td align="center"><u>53.19</u></td>
+        <td align="center">48.65</td>
+      </tr>
+      <tr class="section-start">
+        <td class="domain" rowspan="7" align="center">Code</td>
+        <td class="benchmark" align="center">HumanEval-Plus<sub>(Pass@1)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><u>79.88</u></td>
+        <td align="center">79.27</td>
+        <td align="center">57.93</td>
+        <td align="center"><strong>82.93</strong></td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">CruxEval<sub>(Pass@1)</sub></td>
+        <td class="shot" align="center">1-shot</td>
+        <td class="ling" align="center"><u>80.50</u></td>
+        <td align="center"><strong>82.94</strong></td>
+        <td align="center">69.06</td>
+        <td align="center">71.25</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">MultiPL-E<sub>(Pass@1)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>70.55</strong></td>
+        <td align="center">45.77</td>
+        <td align="center">55.94</td>
+        <td align="center"><u>64.74</u></td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">MBPP-Plus<sub>(Pass@1)</sub></td>
+        <td class="shot" align="center">3-shot</td>
+        <td class="ling" align="center"><u>76.19</u></td>
+        <td align="center"><u>76.19</u></td>
+        <td align="center">73.81</td>
+        <td align="center"><strong>77.51</strong></td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">BigCodeBench<sub>(Pass@1)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><u>53.60</u></td>
+        <td align="center">52.02</td>
+        <td align="center">51.49</td>
+        <td align="center"><strong>66.75</strong></td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">LCBench<sup>1</sup><sub>(Pass@1)</sub></td>
+        <td class="shot" align="center">3-shot</td>
+        <td class="ling" align="center"><strong>68.48</strong></td>
+        <td align="center">46.94</td>
+        <td align="center">59.14</td>
+        <td align="center"><u>67.58</u></td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">LCBench-cn<sup>1</sup><sub>(Pass@1)</sub></td>
+        <td class="shot" align="center">3-shot</td>
+        <td class="ling" align="center"><u>60.83</u></td>
+        <td align="center">48.76</td>
+        <td align="center">57.35</td>
+        <td align="center"><strong>63.36</strong></td>
+      </tr>
+      <tr class="section-start">
+        <td class="domain" rowspan="6" align="center">Math</td>
+        <td class="benchmark" align="center">MATH500<sub>(Acc)</sub></td>
+        <td class="shot" align="center">4-shot</td>
+        <td class="ling" align="center"><strong>79.40</strong></td>
+        <td align="center">60.60</td>
+        <td align="center"><u>68.20</u></td>
+        <td align="center">57.20</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">CollegeMath<sub>(Acc)</sub></td>
+        <td class="shot" align="center">4-shot</td>
+        <td class="ling" align="center"><u>73.41</u></td>
+        <td align="center"><strong>73.98</strong></td>
+        <td align="center">63.15</td>
+        <td align="center">66.61</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">FinanceReasoning<sub>(Acc)</sub></td>
+        <td class="shot" align="center">3-shot</td>
+        <td class="ling" align="center"><u>81.76</u></td>
+        <td align="center">80.14</td>
+        <td align="center"><strong>82.16</strong></td>
+        <td align="center">80.89</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">OlympiadBench<sub>(Acc)</sub></td>
+        <td class="shot" align="center">3-shot</td>
+        <td class="ling" align="center"><strong>60.24</strong></td>
+        <td align="center">41.27</td>
+        <td align="center">46.39</td>
+        <td align="center"><u>48.04</u></td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">TheoremQA<sub>(Acc)</sub></td>
+        <td class="shot" align="center">5-shot</td>
+        <td class="ling" align="center"><strong>63.56</strong></td>
+        <td align="center">60.96</td>
+        <td align="center"><u>61.92</u></td>
+        <td align="center">60.27</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">OmniMath<sub>(Acc)</sub></td>
+        <td class="shot" align="center">3-shot</td>
+        <td class="ling" align="center"><strong>65.31</strong></td>
+        <td align="center">33.33</td>
+        <td align="center">34.89</td>
+        <td align="center"><u>40.13</u></td>
+      </tr>
+      <tr class="section-start">
+        <td class="domain" rowspan="5" align="center">Reasoning</td>
+        <td class="benchmark" align="center">CommonSenseQA<sub>(EM)</sub></td>
+        <td class="shot" align="center">5-shot</td>
+        <td class="ling" align="center"><strong>92.22</strong></td>
+        <td align="center">86.98</td>
+        <td align="center">86.49</td>
+        <td align="center"><u>88.53</u></td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">BBH<sub>(EM)</sub></td>
+        <td class="shot" align="center">3-shot</td>
+        <td class="ling" align="center"><strong>90.25</strong></td>
+        <td align="center">76.00</td>
+        <td align="center"><u>89.98</u></td>
+        <td align="center">89.80</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">KorBench<sub>(Acc)</sub></td>
+        <td class="shot" align="center">3-shot</td>
+        <td class="ling" align="center"><strong>54.96</strong></td>
+        <td align="center">50.08</td>
+        <td align="center"><u>54.56</u></td>
+        <td align="center">53.20</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">AutoLogi<sub>(Acc)</sub></td>
+        <td class="shot" align="center">3-shot</td>
+        <td class="ling" align="center"><strong>67.13</strong></td>
+        <td align="center">61.00</td>
+        <td align="center">64.81</td>
+        <td align="center"><u>66.23</u></td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">ZebraLogic<sub>(Acc)</sub></td>
+        <td class="shot" align="center">1-shot</td>
+        <td class="ling" align="center"><u>30.80</u></td>
+        <td align="center"><strong>32.10</strong></td>
+        <td align="center">20.30</td>
+        <td align="center">28.90</td>
+      </tr>
+      <tr class="section-start">
+        <td class="domain" rowspan="4" align="center">Multilingual</td>
+        <td class="benchmark" align="center">mARC<sub>(EM)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>92.98</strong></td>
+        <td align="center"><u>89.45</u></td>
+        <td align="center">89.10</td>
+        <td align="center">88.55</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">MMMLU<sub>(EM)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>77.56</strong></td>
+        <td align="center"><u>69.67</u></td>
+        <td align="center">68.81</td>
+        <td align="center">67.15</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">MultiGSM<sub>(Acc)</sub></td>
+        <td class="shot" align="center">4-shot</td>
+        <td class="ling" align="center"><strong>87.66</strong></td>
+        <td align="center"><u>86.33</u></td>
+        <td align="center">85.87</td>
+        <td align="center"><u>86.33</u></td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">HumanEvalXL<sub>(Pass@1)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><u>83.25</u></td>
+        <td align="center">81.09</td>
+        <td align="center">70.31</td>
+        <td align="center"><strong>87.34</strong></td>
+      </tr>
+      <tr class="section-start">
+        <td class="domain" rowspan="3" align="center">Med</td>
+        <td class="benchmark" align="center">DiagnosisArena<sub>(EM)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>62.20</strong></td>
+        <td align="center">55.10</td>
+        <td align="center"><u>55.50</u></td>
+        <td align="center">52.70</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">medmcqa<sub>(EM)</sub></td>
+        <td class="shot" align="center">5-shot</td>
+        <td class="ling" align="center"><u>78.08</u></td>
+        <td align="center">75.90</td>
+        <td align="center"><strong>80.33</strong></td>
+        <td align="center">77.50</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">medqa<sub>(EM)</sub></td>
+        <td class="shot" align="center">5-shot</td>
+        <td class="ling" align="center"><strong>87.82</strong></td>
+        <td align="center">81.46</td>
+        <td align="center"><u>87.75</u></td>
+        <td align="center">85.55</td>
+      </tr>
+      <tr class="section-start">
+        <td class="domain" rowspan="2" align="center">Long-context</td>
+        <td class="benchmark" align="center">LongBench<sub>(Acc)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>51.90</strong></td>
+        <td align="center"><u>20.58</u></td>
+        <td align="center">17.70</td>
+        <td align="center">15.66</td>
+      </tr>
+      <tr>
+        <td class="benchmark" align="center">LEval<sub>(Acc)</sub></td>
+        <td class="shot" align="center">0-shot</td>
+        <td class="ling" align="center"><strong>85.09</strong></td>
+        <td align="center"><u>66.29</u></td>
+        <td align="center">58.34</td>
+        <td align="center">61.49</td>
+      </tr>
+    </tbody>
+    <tfoot>
+      <tr>
+        <td class="table-footnote" colspan="7" align="left">
+          <strong>Note:</strong><br>
+          <sup>1</sup> LCBench and LCBench-cn: January 2023–February 2025 (2301-2502).
+        </td>
+      </tr>
+    </tfoot>
+  </table>
+</div>
+
+## Intended Use
+
+Recommended use cases:
+
+- Continued pretraining.
+- Mid-training.
+- Supervised fine-tuning for domain adaptation.
+- Preference optimization and reinforcement-learning post-training.
+- Distillation research.
+- Long-context and MoE systems research.
+
+Not recommended as-is for:
+
+- Direct end-user chat deployment.
+- Safety-critical applications without additional alignment and evaluation.
+- Production use without post-training and task-specific validation.
+
+## Usage
+
+For fine-tuning examples, please refer to our [ling-cookbook](https://github.com/inclusionAI/ling-cookbook/).
+
+## FAQ
+
+For questions or feedback, please open a discussion in the model repository.
+
+## License
+
+This model is released under the **MIT License**.