123sg commited on
Commit
3841946
·
verified ·
1 Parent(s): 70d5182

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +64 -0
README.md ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: transformers
4
+ tags:
5
+ - coverage
6
+ - evaluation
7
+ ---
8
+ # MacroCoverageLM-1B
9
+ <!-- markdownlint-disable first-line-h1 -->
10
+ <!-- markdownlint-disable html -->
11
+ <!-- markdownlint-disable no-duplicate-header -->
12
+
13
+ <div align="center">
14
+ <img src="figures/fig1.png" width="60%" alt="MacroCoverageLM-1B" />
15
+ </div>
16
+ <hr>
17
+
18
+ <div align="center" style="line-height: 1;">
19
+ <img alt="Apache-2.0" src="figures/fig2.png" style="display: inline-block; vertical-align: middle;"/>
20
+ </div>
21
+
22
+ ## 1. Introduction
23
+
24
+ MacroCoverageLM-1B is an open checkpoint family trained for broad-coverage
25
+ reasoning. Every checkpoint in this release comes from the same pretraining
26
+ run and differs only in how many optimisation steps it received.
27
+
28
+ The release reports the run the process selected, with each benchmark suite
29
+ weighted equally.
30
+
31
+ <p align="center">
32
+ <img width="80%" src="figures/fig3.png">
33
+ </p>
34
+
35
+ ## 2. Evaluation Results
36
+
37
+ ### Comprehensive Benchmark Results
38
+
39
+ <div align="center">
40
+
41
+ | | Benchmark | Base-1B | Chat-1B | MacroCoverageLM-1B-Small | MacroCoverageLM-1B |
42
+ |---|---|---|---|---|---|
43
+ | **Core Reasoning Tasks** | Math Reasoning | 0.548 | 0.561 | 0.572 | 0.450 |
44
+ | | Logical Reasoning | 0.548 | 0.561 | 0.572 | 0.450 |
45
+ | | Common Sense | 0.548 | 0.561 | 0.572 | 0.700 |
46
+ | **Language Understanding** | Reading Comprehension | 0.548 | 0.561 | 0.572 | 0.700 |
47
+ | | Question Answering | 0.548 | 0.561 | 0.572 | 0.450 |
48
+ | | Text Classification | 0.548 | 0.561 | 0.572 | 0.700 |
49
+ | | Sentiment Analysis | 0.548 | 0.561 | 0.572 | 0.700 |
50
+ | **Generation Tasks** | Code Generation | 0.548 | 0.561 | 0.572 | 0.450 |
51
+ | | Creative Writing | 0.548 | 0.561 | 0.572 | 0.700 |
52
+ | | Dialogue Generation | 0.548 | 0.561 | 0.572 | 0.700 |
53
+ | | Summarization | 0.548 | 0.561 | 0.572 | 0.700 |
54
+ | **Specialized Capabilities** | Translation | 0.548 | 0.561 | 0.572 | 0.700 |
55
+ | | Knowledge Retrieval | 0.548 | 0.561 | 0.572 | 0.700 |
56
+ | | Instruction Following | 0.548 | 0.561 | 0.572 | 0.450 |
57
+ | | Safety Evaluation | 0.548 | 0.561 | 0.572 | 0.450 |
58
+
59
+ </div>
60
+
61
+ ### Overall Performance Summary
62
+
63
+ Scores are the equal-weight mean over the fifteen suites. Each cell is the
64
+ accuracy over the full held-out split.