123sg's picture
Upload README.md with huggingface_hub
3841946 verified
|
Raw History Blame Contribute Delete
2.19 kB
---
license: apache-2.0
library_name: transformers
tags:
- coverage
- evaluation
---
# MacroCoverageLM-1B
<!-- markdownlint-disable first-line-h1 -->
<!-- markdownlint-disable html -->
<!-- markdownlint-disable no-duplicate-header -->
<div align="center">
<img src="figures/fig1.png" width="60%" alt="MacroCoverageLM-1B" />
</div>
<hr>
<div align="center" style="line-height: 1;">
<img alt="Apache-2.0" src="figures/fig2.png" style="display: inline-block; vertical-align: middle;"/>
</div>
## 1. Introduction
MacroCoverageLM-1B is an open checkpoint family trained for broad-coverage
reasoning. Every checkpoint in this release comes from the same pretraining
run and differs only in how many optimisation steps it received.
The release reports the run the process selected, with each benchmark suite
weighted equally.
<p align="center">
<img width="80%" src="figures/fig3.png">
</p>
## 2. Evaluation Results
### Comprehensive Benchmark Results
<div align="center">
| | Benchmark | Base-1B | Chat-1B | MacroCoverageLM-1B-Small | MacroCoverageLM-1B |
|---|---|---|---|---|---|
| **Core Reasoning Tasks** | Math Reasoning | 0.548 | 0.561 | 0.572 | 0.450 |
| | Logical Reasoning | 0.548 | 0.561 | 0.572 | 0.450 |
| | Common Sense | 0.548 | 0.561 | 0.572 | 0.700 |
| **Language Understanding** | Reading Comprehension | 0.548 | 0.561 | 0.572 | 0.700 |
| | Question Answering | 0.548 | 0.561 | 0.572 | 0.450 |
| | Text Classification | 0.548 | 0.561 | 0.572 | 0.700 |
| | Sentiment Analysis | 0.548 | 0.561 | 0.572 | 0.700 |
| **Generation Tasks** | Code Generation | 0.548 | 0.561 | 0.572 | 0.450 |
| | Creative Writing | 0.548 | 0.561 | 0.572 | 0.700 |
| | Dialogue Generation | 0.548 | 0.561 | 0.572 | 0.700 |
| | Summarization | 0.548 | 0.561 | 0.572 | 0.700 |
| **Specialized Capabilities** | Translation | 0.548 | 0.561 | 0.572 | 0.700 |
| | Knowledge Retrieval | 0.548 | 0.561 | 0.572 | 0.700 |
| | Instruction Following | 0.548 | 0.561 | 0.572 | 0.450 |
| | Safety Evaluation | 0.548 | 0.561 | 0.572 | 0.450 |
</div>
### Overall Performance Summary
Scores are the equal-weight mean over the fifteen suites. Each cell is the
accuracy over the full held-out split.