Instructions to use litert-community/MiniCPM5-1B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT
How to use litert-community/MiniCPM5-1B with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Update LlmMetadata for litert-lm 0.18.0: chat template with content parts, supports_thinking / supports_function_calling (weights unchanged)
#12
by mlboydaisuke - opened
MiniCPM5-1B_dynamic_wi8_afp32.litertlm
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 1105464896
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:13ec3c4372dd41720ecf62ba07557514d7d0fff92f5fb058295742b4b9c3786b
|
| 3 |
size 1105464896
|
README.md
CHANGED
|
@@ -24,9 +24,9 @@ This repository hosts the [**LiteRT-LM**](https://ai.google.dev/edge/litert-lm)
|
|
| 24 |
|
| 25 |
## Available Models
|
| 26 |
|
| 27 |
-
* **`minicpm_wi4b32_wi8_afp32.litertlm`**: This model features mixed INT4-block32(linear)/INT8(embed and lmhead) quantization (wi4b32_wi8) with FP32 activations (afp32).
|
| 28 |
-
* **`minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm`**: This model uses same quantization recipe with minicpm_wi4b32_wi8_afp32 but optimized for GPU execution.
|
| 29 |
-
* **`MiniCPM5-1B_dynamic_wi8_afp32.litertlm`**: This model features dynamic weight-only INT8 quantization (wi8) with FP32 activations (afp32). It is configured with static prefill memory allocation.
|
| 30 |
|
| 31 |
## Quantization Benchmark
|
| 32 |
|
|
|
|
| 24 |
|
| 25 |
## Available Models
|
| 26 |
|
| 27 |
+
* **`minicpm_wi4b32_wi8_afp32.litertlm`**: This model features mixed INT4-block32(linear)/INT8(embed and lmhead) quantization (wi4b32_wi8) with FP32 activations (afp32). Needs litert-lm 0.18.0 or newer (tested on 0.18.0) and carries LiteRT-LM's official minicpm5 configuration, so thinking is off unless `enable_thinking` is true, the reasoning arrives on the thought channel, and tool calls are parsed by the runtime.
|
| 28 |
+
* **`minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm`**: This model uses same quantization recipe with minicpm_wi4b32_wi8_afp32 but optimized for GPU execution. Needs litert-lm 0.18.0 or newer (tested on 0.18.0) and carries LiteRT-LM's official minicpm5 configuration, so thinking is off unless `enable_thinking` is true, the reasoning arrives on the thought channel, and tool calls are parsed by the runtime.
|
| 29 |
+
* **`MiniCPM5-1B_dynamic_wi8_afp32.litertlm`**: This model features dynamic weight-only INT8 quantization (wi8) with FP32 activations (afp32). It is configured with static prefill memory allocation. Needs litert-lm 0.18.0 or newer (tested on 0.18.0) and carries LiteRT-LM's official minicpm5 configuration, so thinking is off unless `enable_thinking` is true, the reasoning arrives on the thought channel, and tool calls are parsed by the runtime.
|
| 30 |
|
| 31 |
## Quantization Benchmark
|
| 32 |
|
minicpm_wi4b32_wi8_afp32.litertlm
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ee1b70697ef583c46d26df5772055946d4ca2a9a80b059149bb51f4ab9646863
|
| 3 |
+
size 792619648
|
minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:40ffba999ca45223c1713f130299e69b798b72fd561ea420300beb0adb49e191
|
| 3 |
+
size 793029040
|