| --- |
| library_name: transformers |
| tags: |
| - text-generation |
| language: |
| - fr |
| - en |
| license: apache-2.0 |
| datasets: |
| - Locutusque/TM-DATA-V2 |
| - LLM360/TxT360 |
| - mlfoundations/dclm-baseline-1.0 |
| - Skylion007/openwebtext |
| - JeanKaddour/minipile |
| - eminorhan/gutenberg_en |
| model-index: |
| - name: TinyMistral-248M-v3 |
| results: |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: IFEval (0-Shot) |
| type: HuggingFaceH4/ifeval |
| args: |
| num_few_shot: 0 |
| metrics: |
| - type: inst_level_strict_acc and prompt_level_strict_acc |
| value: 16.39 |
| name: strict accuracy |
| source: |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=M4-ai/TinyMistral-248M-v3 |
| name: Open LLM Leaderboard |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: BBH (3-Shot) |
| type: BBH |
| args: |
| num_few_shot: 3 |
| metrics: |
| - type: acc_norm |
| value: 1.78 |
| name: normalized accuracy |
| source: |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=M4-ai/TinyMistral-248M-v3 |
| name: Open LLM Leaderboard |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: MATH Lvl 5 (4-Shot) |
| type: hendrycks/competition_math |
| args: |
| num_few_shot: 4 |
| metrics: |
| - type: exact_match |
| value: 0.0 |
| name: exact match |
| source: |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=M4-ai/TinyMistral-248M-v3 |
| name: Open LLM Leaderboard |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: GPQA (0-shot) |
| type: Idavidrein/gpqa |
| args: |
| num_few_shot: 0 |
| metrics: |
| - type: acc_norm |
| value: 0.0 |
| name: acc_norm |
| source: |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=M4-ai/TinyMistral-248M-v3 |
| name: Open LLM Leaderboard |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: MuSR (0-shot) |
| type: TAUR-Lab/MuSR |
| args: |
| num_few_shot: 0 |
| metrics: |
| - type: acc_norm |
| value: 5.15 |
| name: acc_norm |
| source: |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=M4-ai/TinyMistral-248M-v3 |
| name: Open LLM Leaderboard |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: MMLU-PRO (5-shot) |
| type: TIGER-Lab/MMLU-Pro |
| config: main |
| split: test |
| args: |
| num_few_shot: 5 |
| metrics: |
| - type: acc |
| value: 1.47 |
| name: accuracy |
| source: |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=M4-ai/TinyMistral-248M-v3 |
| name: Open LLM Leaderboard |
| --- |
| |
| still in training. Trained on about ~21 billion tokens so far. |
|
|
| | Tasks |Version| Filter |n-shot| Metric | | Value | |Stderr| |
| |----------------------------------------|-------|----------------|-----:|-----------|---|------:|---|-----:| |
| |Open LLM Leaderboard | N/A| | | | | | | | |
| | - arc_challenge | 1|none | 25|acc |↑ | 0.2005|± |0.0117| |
| | | |none | 25|acc_norm |↑ | 0.2406|± |0.0125| |
| | - gsm8k | 3|flexible-extract| 5|exact_match|↑ | 0.0083|± |0.0025| |
| | | |strict-match | 5|exact_match|↑ | 0.0000|± |0.0000| |
| | - hellaswag | 1|none | 10|acc |↑ | 0.2724|± |0.0044| |
| | | |none | 10|acc_norm |↑ | 0.2838|± |0.0045| |
| | - mmlu | 2|none | |acc |↑ | 0.2290|± |0.0035| |
| | - humanities | 2|none | |acc |↑ | 0.2380|± |0.0062| |
| | - formal_logic | 1|none | 5|acc |↑ | 0.2460|± |0.0385| |
| | - high_school_european_history | 1|none | 5|acc |↑ | 0.1818|± |0.0301| |
| | - high_school_us_history | 1|none | 5|acc |↑ | 0.2647|± |0.0310| |
| | - high_school_world_history | 1|none | 5|acc |↑ | 0.2911|± |0.0296| |
| | - international_law | 1|none | 5|acc |↑ | 0.2149|± |0.0375| |
| | - jurisprudence | 1|none | 5|acc |↑ | 0.2685|± |0.0428| |
| | - logical_fallacies | 1|none | 5|acc |↑ | 0.2209|± |0.0326| |
| | - moral_disputes | 1|none | 5|acc |↑ | 0.2457|± |0.0232| |
| | - moral_scenarios | 1|none | 5|acc |↑ | 0.2369|± |0.0142| |
| | - philosophy | 1|none | 5|acc |↑ | 0.1865|± |0.0221| |
| | - prehistory | 1|none | 5|acc |↑ | 0.1975|± |0.0222| |
| | - professional_law | 1|none | 5|acc |↑ | 0.2432|± |0.0110| |
| | - world_religions | 1|none | 5|acc |↑ | 0.3099|± |0.0355| |
| | - other | 2|none | |acc |↑ | 0.2375|± |0.0076| |
| | - business_ethics | 1|none | 5|acc |↑ | 0.3200|± |0.0469| |
| | - clinical_knowledge | 1|none | 5|acc |↑ | 0.2226|± |0.0256| |
| | - college_medicine | 1|none | 5|acc |↑ | 0.1965|± |0.0303| |
| | - global_facts | 1|none | 5|acc |↑ | 0.1800|± |0.0386| |
| | - human_aging | 1|none | 5|acc |↑ | 0.3004|± |0.0308| |
| | - management | 1|none | 5|acc |↑ | 0.1942|± |0.0392| |
| | - marketing | 1|none | 5|acc |↑ | 0.2735|± |0.0292| |
| | - medical_genetics | 1|none | 5|acc |↑ | 0.3000|± |0.0461| |
| | - miscellaneous | 1|none | 5|acc |↑ | 0.2478|± |0.0154| |
| | - nutrition | 1|none | 5|acc |↑ | 0.2222|± |0.0238| |
| | - professional_accounting | 1|none | 5|acc |↑ | 0.2021|± |0.0240| |
| | - professional_medicine | 1|none | 5|acc |↑ | 0.1912|± |0.0239| |
| | - virology | 1|none | 5|acc |↑ | 0.2590|± |0.0341| |
| | - social sciences | 2|none | |acc |↑ | 0.2203|± |0.0075| |
| | - econometrics | 1|none | 5|acc |↑ | 0.2368|± |0.0400| |
| | - high_school_geography | 1|none | 5|acc |↑ | 0.2020|± |0.0286| |
| | - high_school_government_and_politics| 1|none | 5|acc |↑ | 0.1865|± |0.0281| |
| | - high_school_macroeconomics | 1|none | 5|acc |↑ | 0.2205|± |0.0210| |
| | - high_school_microeconomics | 1|none | 5|acc |↑ | 0.2143|± |0.0267| |
| | - high_school_psychology | 1|none | 5|acc |↑ | 0.1908|± |0.0168| |
| | - human_sexuality | 1|none | 5|acc |↑ | 0.2672|± |0.0388| |
| | - professional_psychology | 1|none | 5|acc |↑ | 0.2386|± |0.0172| |
| | - public_relations | 1|none | 5|acc |↑ | 0.1727|± |0.0362| |
| | - security_studies | 1|none | 5|acc |↑ | 0.2367|± |0.0272| |
| | - sociology | 1|none | 5|acc |↑ | 0.2488|± |0.0306| |
| | - us_foreign_policy | 1|none | 5|acc |↑ | 0.2600|± |0.0441| |
| | - stem | 2|none | |acc |↑ | 0.2157|± |0.0073| |
| | - abstract_algebra | 1|none | 5|acc |↑ | 0.2200|± |0.0416| |
| | - anatomy | 1|none | 5|acc |↑ | 0.1778|± |0.0330| |
| | - astronomy | 1|none | 5|acc |↑ | 0.1908|± |0.0320| |
| | - college_biology | 1|none | 5|acc |↑ | 0.2778|± |0.0375| |
| | - college_chemistry | 1|none | 5|acc |↑ | 0.2200|± |0.0416| |
| | - college_computer_science | 1|none | 5|acc |↑ | 0.2100|± |0.0409| |
| | - college_mathematics | 1|none | 5|acc |↑ | 0.2100|± |0.0409| |
| | - college_physics | 1|none | 5|acc |↑ | 0.2157|± |0.0409| |
| | - computer_security | 1|none | 5|acc |↑ | 0.2700|± |0.0446| |
| | - conceptual_physics | 1|none | 5|acc |↑ | 0.2638|± |0.0288| |
| | - electrical_engineering | 1|none | 5|acc |↑ | 0.2483|± |0.0360| |
| | - elementary_mathematics | 1|none | 5|acc |↑ | 0.2037|± |0.0207| |
| | - high_school_biology | 1|none | 5|acc |↑ | 0.1774|± |0.0217| |
| | - high_school_chemistry | 1|none | 5|acc |↑ | 0.2020|± |0.0282| |
| | - high_school_computer_science | 1|none | 5|acc |↑ | 0.2500|± |0.0435| |
| | - high_school_mathematics | 1|none | 5|acc |↑ | 0.2148|± |0.0250| |
| | - high_school_physics | 1|none | 5|acc |↑ | 0.2053|± |0.0330| |
| | - high_school_statistics | 1|none | 5|acc |↑ | 0.1481|± |0.0242| |
| | - machine_learning | 1|none | 5|acc |↑ | 0.3125|± |0.0440| |
| | - truthfulqa_gen | 3|none | 0|bleu_acc |↑ | 0.2362|± |0.0149| |
| | | |none | 0|bleu_diff |↑ |-1.0138|± |0.2569| |
| | | |none | 0|bleu_max |↑ | 7.9522|± |0.4088| |
| | | |none | 0|rouge1_acc |↑ | 0.2595|± |0.0153| |
| | | |none | 0|rouge1_diff|↑ |-1.9129|± |0.4349| |
| | | |none | 0|rouge1_max |↑ |21.7885|± |0.7307| |
| | | |none | 0|rouge2_acc |↑ | 0.1200|± |0.0114| |
| | | |none | 0|rouge2_diff|↑ |-1.9771|± |0.3475| |
| | | |none | 0|rouge2_max |↑ | 9.0199|± |0.5842| |
| | | |none | 0|rougeL_acc |↑ | 0.2570|± |0.0153| |
| | | |none | 0|rougeL_diff|↑ |-1.8812|± |0.4185| |
| | | |none | 0|rougeL_max |↑ |19.6284|± |0.6850| |
| | - truthfulqa_mc1 | 2|none | 0|acc |↑ | 0.1983|± |0.0140| |
| | - truthfulqa_mc2 | 2|none | 0|acc |↑ | 0.3861|± |0.0147| |
| | - winogrande | 1|none | 5|acc |↑ | 0.4972|± |0.0141| |
| |
| | Groups |Version|Filter|n-shot|Metric| |Value | |Stderr| |
| |-------------------|------:|------|------|------|---|-----:|---|-----:| |
| | - mmlu | 2|none | |acc |↑ |0.2290|± |0.0035| |
| | - humanities | 2|none | |acc |↑ |0.2380|± |0.0062| |
| | - other | 2|none | |acc |↑ |0.2375|± |0.0076| |
| | - social sciences| 2|none | |acc |↑ |0.2203|± |0.0075| |
| | - stem | 2|none | |acc |↑ |0.2157|± |0.0073| |
| |
| | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| |
| |---------------------------------|------:|------|-----:|--------|---|-----:|---|-----:| |
| |agieval_nous | 0|none | |acc_norm|↑ |0.2133|± |0.0081| |
| | - agieval_aqua_rat | 1|none | 0|acc |↑ |0.2047|± |0.0254| |
| | | |none | 0|acc_norm|↑ |0.1969|± |0.0250| |
| | - agieval_logiqa_en | 1|none | 0|acc |↑ |0.2043|± |0.0158| |
| | | |none | 0|acc_norm|↑ |0.2304|± |0.0165| |
| | - agieval_lsat_ar | 1|none | 0|acc |↑ |0.1739|± |0.0250| |
| | | |none | 0|acc_norm|↑ |0.1957|± |0.0262| |
| | - agieval_lsat_lr | 1|none | 0|acc |↑ |0.1549|± |0.0160| |
| | | |none | 0|acc_norm|↑ |0.1608|± |0.0163| |
| | - agieval_lsat_rc | 1|none | 0|acc |↑ |0.1636|± |0.0226| |
| | | |none | 0|acc_norm|↑ |0.2119|± |0.0250| |
| | - agieval_sat_en | 1|none | 0|acc |↑ |0.2670|± |0.0309| |
| | | |none | 0|acc_norm|↑ |0.2621|± |0.0307| |
| | - agieval_sat_en_without_passage| 1|none | 0|acc |↑ |0.2670|± |0.0309| |
| | | |none | 0|acc_norm|↑ |0.2621|± |0.0307| |
| | - agieval_sat_math | 1|none | 0|acc |↑ |0.2182|± |0.0279| |
| | | |none | 0|acc_norm|↑ |0.2318|± |0.0285| |
| |arc_challenge | 1|none | 0|acc |↑ |0.1945|± |0.0116| |
| | | |none | 0|acc_norm|↑ |0.2372|± |0.0124| |
| |truthfulqa_mc2 | 2|none | 0|acc |↑ |0.3861|± |0.0147| |
|
|
| | Groups |Version|Filter|n-shot| Metric | |Value | |Stderr| |
| |------------|------:|------|------|--------|---|-----:|---|-----:| |
| |agieval_nous| 0|none | |acc_norm|↑ |0.2133|± |0.0081| |
| # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) |
| Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_M4-ai__TinyMistral-248M-v3) |
|
|
| | Metric |Value| |
| |-------------------|----:| |
| |Avg. | 4.13| |
| |IFEval (0-Shot) |16.39| |
| |BBH (3-Shot) | 1.78| |
| |MATH Lvl 5 (4-Shot)| 0.00| |
| |GPQA (0-shot) | 0.00| |
| |MuSR (0-shot) | 5.15| |
| |MMLU-PRO (5-shot) | 1.47| |
|
|
|
|
