Small updates
Browse files
README.md
CHANGED
|
@@ -1,3 +1,52 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# **Model Details**
|
| 2 |
+
|
| 3 |
+
The Mistral AI-7B-v0.1 Large Language Model (LLM) is a pretrained generative text model with 7 billion parameters. Mistral AI-7B-v0.1 outperforms Llama 2 13B on all benchmarks we tested.
|
| 4 |
+
|
| 5 |
+
**Model Developers** Mistral AI.
|
| 6 |
+
|
| 7 |
+
**Variations** None.
|
| 8 |
+
|
| 9 |
+
**Input** Text only.
|
| 10 |
+
|
| 11 |
+
**Output** Text only.
|
| 12 |
+
|
| 13 |
+
**Model Architecture** Mistral AI-7B-v0.1 is a transformer model, with the following architecture choices:
|
| 14 |
+
- Grouped-Query Attention
|
| 15 |
+
- Sliding-Window Attention
|
| 16 |
+
- Byte-fallback BPE tokenizer
|
| 17 |
+
|
| 18 |
+
**Model Dates** Mistral AI-7B-v0.1 was trained between June and September 2023.
|
| 19 |
+
|
| 20 |
+
**Status** This is a static model. Future models will have new version numbers.
|
| 21 |
+
|
| 22 |
+
**License** Apache 2.0 license.
|
| 23 |
+
|
| 24 |
+
**Research Paper** TODO: Coming soon.
|
| 25 |
+
|
| 26 |
+
**Where to send questions or comments about the model** TODO: How do people send comments?
|
| 27 |
+
|
| 28 |
+
# **Intended Use**
|
| 29 |
+
**Intended Use Cases** Mistral AI-7B-v0.1 is for commercial and research use. It can be adapted for a variety of natural language generation tasks.
|
| 30 |
+
|
| 31 |
+
# **Evaluation Results**
|
| 32 |
+
We report the standard benchmark results for Mistral AI-7B-v0.1. We use a custom evaluation library to produce the results.
|
| 33 |
+
|
| 34 |
+
| Model | Size | hellaswag | winogrande | piqa | boolq | arc_easy | arc_challenge | naturalqs | naturalqs_5shot | triviaqa_5shot | triviaqa | humaneval_pass@1 | mbpp_pass@1 | mmlu | math | gsm8k |
|
| 35 |
+
|-----------------|------|-----------|------------|--------|--------|----------|---------------|-----------|-----------------|----------------|----------|------------------|-------------|--------|--------|--------|
|
| 36 |
+
| Mistral-7B-v0.1 | 7B | 81.19% | 75.53% | 82.92% | 83.52% | 80.01% | 55.38% | 23.96% | 28.92% | 69.88% | 63.22% | 29.88% | 47.86% | 59.99% | 11.94% | 39.35% |
|
| 37 |
+
|
| 38 |
+
**Theme-based grouping**
|
| 39 |
+
- Commonsense Reasoning: 0-shot average of Hellaswag, Winogrande, PIQA, SIQA, OpenbookQA, ARC-Easy, ARC-Challenge, and CommonsenseQA.
|
| 40 |
+
|
| 41 |
+
- World Knowledge: 5-shot average of NaturalQuestions and TriviaQA.
|
| 42 |
+
|
| 43 |
+
- Reading Comprehension: 0-shot average of BoolQ and QuAC.
|
| 44 |
+
|
| 45 |
+
- Math: Average of 8-shot GSM8K with maj@8 and 4-shot MATH with maj@4
|
| 46 |
+
|
| 47 |
+
- Code: Average of 0-shot Humaneval and 3-shot MBPP
|
| 48 |
+
|
| 49 |
+
- Popular aggregated results: 5-shot MMLU, 3-shot BBH, and 3-5-shot AGI Eval (English multiple-choice questions only)
|
| 50 |
+
|
| 51 |
+
# **Ethical Considerations and Limitations**
|
| 52 |
+
TODO: what do we say here?
|