Add community-contributed multilingual eval (ES/CA/EU/GL vs Salamandra-2b-instruct)
ff295db verified | license: apache-2.0 | |
| language: | |
| - en | |
| - zh | |
| base_model: | |
| - openbmb/MiniCPM5-1B | |
| base_model_relation: quantized | |
| library_name: litert-lm | |
| pipeline_tag: text-generation | |
| tags: | |
| - litert-lm | |
| - minicpm | |
| - minicpm5 | |
| - on-device | |
| - edge-ai | |
| # MiniCPM5-1B — LiteRT-LM (on-device) | |
| Main Model Card: [openbmb/MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B) | |
| MiniCPM5-1B packaged in the `.litertlm` format so it runs fully **on-device** — Android, iOS, Desktop and IoT — with the [LiteRT-LM](https://ai.google.dev/edge/litert-lm) framework. No internet connection required at inference time. | |
| ## 👉 Not sure which file to pick? Download `MiniCPM5-1B.litertlm` | |
| That's the one and only file here, and it's meant to "just work". You don't need to understand quantization to use it. | |
| | File | What it is | Who it's for | | |
| | :--- | :--- | :--- | | |
| | **`MiniCPM5-1B.litertlm`** | The ready-to-run model (INT8, ~1.1 GB) | **Everyone — this is the default** | | |
| | `chat_template.jinja` | The chat format (auto-used by the runtime) | Advanced / reference | | |
| ## Try it in one command (Desktop) | |
| ```bash | |
| uv tool install litert-lm | |
| uvx litert-lm run --from-huggingface-repo=luispoveda93/MiniCPM5-1B MiniCPM5-1B.litertlm --prompt="What is the capital of France?" | |
| ``` | |
| ## Run it on your phone (Android) | |
| 1. Install the [Google AI Edge Gallery](https://play.google.com/store/apps/details?id=com.google.ai.edge.gallery) app. | |
| 2. Open **Model manager → “+” → Import from HF**, paste this repo's link, and download `MiniCPM5-1B.litertlm`. | |
| See the [Edge Gallery wiki](https://github.com/google-ai-edge/gallery/wiki) for details. | |
| ## About this model | |
| MiniCPM5-1B is a compact 1B decoder-only model from OpenBMB (standard `LlamaForCausalLM`), built for local assistants, coding agents, and tool use, with a hybrid `<think>` chat mode and long-context support. | |
| | Item | Value | | |
| | :--- | :--- | | |
| | Parameters | 1,080,632,832 (~1B) | | |
| | Layers | 24 | | |
| | Attention (GQA) | 16 query heads / 2 KV heads | | |
| | Context length | 131,072 | | |
| | Quantization | INT8 dynamic-range (weights), FP32 activations | | |
| | Signatures | `prefill_128`, `prefill_512`, `decode` (KV-cache 1280) | | |
| ### Recommended sampling (from the source model card) | |
| | Mode | Params | Enable | | |
| | :--- | :--- | :--- | | |
| | **Think** | `temperature=0.9, top_p=0.95` | `enable_thinking=True` | | |
| | **No Think** | `temperature=0.7, top_p=0.95` | `enable_thinking=False` | | |
| ## How this was built (transparency) | |
| Converted from the BF16 source ([openbmb/MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B)) with Google AI Edge tooling: | |
| - **Re-authoring + TFLite conversion**: `ai-edge-torch` Generative API. MiniCPM5-1B maps onto the standard decoder-only building blocks (hidden 1536, 24 layers, GQA 16/2, head_dim 128, RMSNorm 1e-6, RoPE θ=5e6, untied `lm_head`, vocab 130560). | |
| - **Quantization**: INT8 dynamic-range weight-only. | |
| - **Packaging**: `litert-lm-builder`, native `minicpm5` runtime type, source Hugging Face tokenizer, start token `<s>`, stop tokens `[1, 130073]` (`</s>`, `<|im_end|>`), `thought` reasoning channel, and an `ExecutorMetadata` section declaring all 48 KV-cache state buffers (required for on-device prefill). | |
| - **Verification**: the re-authored model was checked against the original `transformers` implementation — last-token logits matched with **cosine 1.0000**, identical argmax, 5/5 top-5 overlap; parameter count matches exactly. The packaged bundle was also run end-to-end with `litert-lm run` and generates correctly (reasoning channel parses as expected). | |
| > This is a personal community contribution, not an official OpenBMB or Google release. An INT4 (`wi4b32`) variant with a smaller footprint is available from the official [litert-community/MiniCPM5-1B](https://huggingface.co/litert-community/MiniCPM5-1B) repo. | |
| ## Community-Contributed Evaluation: Spanish, Catalan, Basque, Galician (0-shot) | |
| > **⚠️ Note:** This section is a **community addition** — it does **not** appear in the original [openbmb/MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B) model card. It was benchmarked independently by this repo's maintainer and is provided as supplementary information, not an official OpenBMB result. | |
| To understand how this compact 1B EN/ZH-pretrained model performs on Spain's co-official languages, we ran the exact task selection published on the [Salamandra-2b-instruct model card](https://huggingface.co/BSC-LT/salamandra-2b-instruct) — a 2.25B model whose pretraining specifically oversampled these languages 2x — using the same [LM Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) task suite (SpanishBench, CatalanBench, BasqueBench, GalicianBench), 0-shot, chat template applied. Evaluation was run on the BF16 source model (`openbmb/MiniCPM5-1B`), not the on-device `.litertlm` build in this repo. The Salamandra-2B column is transcribed verbatim from its published card and was not re-run. | |
| ### Spanish | |
| <table><thead> | |
| <tr> | |
| <th>Category</th> | |
| <th>Task</th> | |
| <th>Metric</th> | |
| <th>MiniCPM5-1B</th> | |
| <th>Salamandra-2B</th> | |
| </tr></thead> | |
| <tbody> | |
| <tr> | |
| <td>Commonsense Reasoning</td> | |
| <td>xstorycloze_es</td> | |
| <td>acc</td> | |
| <td>51.69</td> | |
| <td>61.95</td> | |
| </tr> | |
| <tr> | |
| <td rowspan="2">NLI</td> | |
| <td>wnli_es</td> | |
| <td>acc</td> | |
| <td>43.66</td> | |
| <td>49.30</td> | |
| </tr> | |
| <tr> | |
| <td>xnli_es</td> | |
| <td>acc</td> | |
| <td>42.49</td> | |
| <td>48.52</td> | |
| </tr> | |
| <tr> | |
| <td>Paraphrasing</td> | |
| <td>paws_es</td> | |
| <td>acc</td> | |
| <td><strong>57.30</strong></td> | |
| <td>57.10</td> | |
| </tr> | |
| <tr> | |
| <td>QA</td> | |
| <td>xquad_es</td> | |
| <td>f1<sup>†</sup></td> | |
| <td>0.00</td> | |
| <td>29.60</td> | |
| </tr> | |
| <tr> | |
| <td>Translation</td> | |
| <td>flores_es</td> | |
| <td>bleu</td> | |
| <td>0.00</td> | |
| <td>15.92</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <sup>†</sup> Salamandra's card lists this task under metric "acc"; the installed `xquad_es` task only computes `f1`/`exact_match`, so `f1` is reported here for MiniCPM5-1B. | |
| ### Catalan | |
| <table><thead> | |
| <tr> | |
| <th>Category</th> | |
| <th>Task</th> | |
| <th>Metric</th> | |
| <th>MiniCPM5-1B</th> | |
| <th>Salamandra-2B</th> | |
| </tr></thead> | |
| <tbody> | |
| <tr> | |
| <td rowspan="2">Commonsense Reasoning</td> | |
| <td>copa_ca</td> | |
| <td>acc</td> | |
| <td>50.80</td> | |
| <td>71.00</td> | |
| </tr> | |
| <tr> | |
| <td>xstorycloze_ca</td> | |
| <td>acc</td> | |
| <td>48.64</td> | |
| <td>61.61</td> | |
| </tr> | |
| <tr> | |
| <td rowspan="2">NLI</td> | |
| <td>wnli_ca</td> | |
| <td>acc</td> | |
| <td>54.93</td> | |
| <td>56.34</td> | |
| </tr> | |
| <tr> | |
| <td>xnli_ca</td> | |
| <td>acc</td> | |
| <td>37.83</td> | |
| <td>52.54</td> | |
| </tr> | |
| <tr> | |
| <td rowspan="2">Paraphrasing</td> | |
| <td>parafraseja</td> | |
| <td>acc</td> | |
| <td>56.75</td> | |
| <td>64.48</td> | |
| </tr> | |
| <tr> | |
| <td>paws_ca</td> | |
| <td>acc</td> | |
| <td>56.45</td> | |
| <td>60.30</td> | |
| </tr> | |
| <tr> | |
| <td rowspan="5">QA</td> | |
| <td>arc_ca_easy</td> | |
| <td>acc</td> | |
| <td>28.87</td> | |
| <td>52.15</td> | |
| </tr> | |
| <tr> | |
| <td>arc_ca_challenge</td> | |
| <td>acc</td> | |
| <td>20.22</td> | |
| <td>30.03</td> | |
| </tr> | |
| <tr> | |
| <td>openbookqa_ca</td> | |
| <td>acc</td> | |
| <td>21.80</td> | |
| <td>29.20</td> | |
| </tr> | |
| <tr> | |
| <td>piqa_ca</td> | |
| <td>acc</td> | |
| <td>53.32</td> | |
| <td>63.93</td> | |
| </tr> | |
| <tr> | |
| <td>siqa_ca</td> | |
| <td>acc</td> | |
| <td>33.76</td> | |
| <td>42.32</td> | |
| </tr> | |
| <tr> | |
| <td>Translation</td> | |
| <td>flores_ca</td> | |
| <td>bleu</td> | |
| <td>0.00</td> | |
| <td>20.04</td> | |
| </tr> | |
| </tbody></table> | |
| ### Basque | |
| <table><thead> | |
| <tr> | |
| <th>Category</th> | |
| <th>Task</th> | |
| <th>Metric</th> | |
| <th>MiniCPM5-1B</th> | |
| <th>Salamandra-2B</th> | |
| </tr></thead> | |
| <tbody> | |
| <tr> | |
| <td rowspan="2">Commonsense Reasoning</td> | |
| <td>xcopa_eu</td> | |
| <td>acc</td> | |
| <td>48.20</td> | |
| <td>55.20</td> | |
| </tr> | |
| <tr> | |
| <td>xstorycloze_eu</td> | |
| <td>acc</td> | |
| <td>49.77</td> | |
| <td>55.26</td> | |
| </tr> | |
| <tr> | |
| <td rowspan="2">NLI</td> | |
| <td>wnli_eu</td> | |
| <td>acc</td> | |
| <td><strong>52.11</strong></td> | |
| <td>52.11</td> | |
| </tr> | |
| <tr> | |
| <td>xnli_eu</td> | |
| <td>acc</td> | |
| <td>33.91</td> | |
| <td>45.89</td> | |
| </tr> | |
| <tr> | |
| <td rowspan="3">QA</td> | |
| <td>eus_exams<sup>‡</sup></td> | |
| <td>acc</td> | |
| <td>24.40</td> | |
| <td>24.94</td> | |
| </tr> | |
| <tr> | |
| <td>eus_proficiency</td> | |
| <td>acc</td> | |
| <td>24.28</td> | |
| <td>25.15</td> | |
| </tr> | |
| <tr> | |
| <td>eus_trivia</td> | |
| <td>acc</td> | |
| <td><strong>30.44</strong></td> | |
| <td>27.35</td> | |
| </tr> | |
| <tr> | |
| <td>Reading Comprehension</td> | |
| <td>eus_reading</td> | |
| <td>acc</td> | |
| <td>25.85</td> | |
| <td>28.69</td> | |
| </tr> | |
| <tr> | |
| <td>Translation</td> | |
| <td>flores_eu</td> | |
| <td>bleu</td> | |
| <td>0.00</td> | |
| <td>7.93</td> | |
| </tr> | |
| </tbody></table> | |
| <sup>‡</sup> `eus_exams` is a group of 29 exam sub-tasks; MiniCPM5-1B's score is the sample-count-weighted average across all sub-tasks. | |
| ### Galician | |
| <table><thead> | |
| <tr> | |
| <th>Category</th> | |
| <th>Task</th> | |
| <th>Metric</th> | |
| <th>MiniCPM5-1B</th> | |
| <th>Salamandra-2B</th> | |
| </tr></thead> | |
| <tbody> | |
| <tr> | |
| <td rowspan="2">Paraphrasing</td> | |
| <td>parafrases_gl</td> | |
| <td>acc</td> | |
| <td>52.72</td> | |
| <td>53.40</td> | |
| </tr> | |
| <tr> | |
| <td>paws_gl</td> | |
| <td>acc</td> | |
| <td>53.95</td> | |
| <td>56.85</td> | |
| </tr> | |
| <tr> | |
| <td>QA</td> | |
| <td>openbookqa_gl</td> | |
| <td>acc</td> | |
| <td>21.60</td> | |
| <td>26.20</td> | |
| </tr> | |
| <tr> | |
| <td>Translation</td> | |
| <td>flores_gl</td> | |
| <td>bleu</td> | |
| <td>0.00</td> | |
| <td>17.53</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| **Bold** values indicate MiniCPM5-1B matches or exceeds the Salamandra-2B-instruct published baseline despite the parameter and pretraining-data gap. MiniCPM5-1B has no targeted Spanish/Catalan/Basque/Galician pretraining data; the gaps on Basque and Galician (and all FLORES/BLEU translation tasks, which scored ≈0 across every language) are the expected result of that mismatch, not a defect in the on-device conversion. This comparison is offered as context for anyone considering MiniCPM5-1B for these languages — it is **not** a claim of official parity with Salamandra. | |
| ## License | |
| [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE), following the source model. | |