| --- |
| license: apache-2.0 |
| base_model: |
| - Qwen/Qwen2.5-0.5B-Instruct |
| base_model_relation: quantized |
| library_name: executorch |
| pipeline_tag: text-generation |
| inference: false |
| tags: |
| - executorch |
| - webgpu |
| - on-device |
| - browser |
| --- |
| # Qwen2.5 0.5B Instruct for ExecuTorch WebGPU |
|
|
| This repository contains an [ExecuTorch](https://github.com/pytorch/executorch) conversion of |
| [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) for the ExecuTorch **WebGPU** backend, so the model runs entirely in a web browser on |
| the visitor's GPU. It powers the corresponding demo in the ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)). |
|
|
| - **Base model:** [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) |
| - **Format:** ExecuTorch `.pte` programs delegated to the WebGPU backend |
| - **Quantization:** 4-bit groupwise weights (q4gsw), fp16 KV cache |
| - **Total size:** 296.5 MB |
|
|
| ## About the original model |
|
|
| The 0.5B instruction-tuned model of the Qwen2.5 family. |
|
|
| - **Developed by:** Qwen Team, Alibaba Cloud |
| - **Paper:** [Qwen2.5 Technical Report](https://arxiv.org/abs/2412.15115) |
| - 0.49B parameters, 24 layers |
| - Context length 32,768 tokens; generation up to 8,192 tokens |
| - Supports over 29 languages |
|
|
| For training data, evaluation results and full details, see the original model card: [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct). |
|
|
| ## Intended use and limitations |
|
|
| Chat and instruction following in a small footprint. |
|
|
| Small language models can state false information confidently, repeat themselves, and reflect biases in their training data. Check important outputs. |
|
|
| ## What changed in this conversion |
|
|
| - Weights quantized to 4-bit groupwise (q4gsw); the KV cache is fp16 |
| - Context capped at 8,192 tokens (upstream: 32,768) |
|
|
| ## Inputs and outputs |
|
|
| - `forward(tokens int64 [1, 1..512], input_pos int64 [1]) -> logits float32 [1, 151936]` (last position only) |
| - The KV cache lives inside the program; call it with up to 512 new tokens at a time, then one token per step |
| - Tokenize with the files in `tokenizer/` (the upstream Qwen2.5 tokenizer) and the ChatML template |
|
|
| ## Files |
|
|
| | File | Size | SHA-256 | |
| |---|---|---| |
| | `qwen2_5_0_5b_instruct_q4gsw.pte` | 285.6 MB | `474274b934978f6c…` | |
| | `tokenizer/config.json` | 659 B | `18e18afcaccafade…` | |
| | `tokenizer/generation_config.json` | 242 B | `e558847a8b440261…` | |
| | `tokenizer/merges.txt` | 1.6 MB | `599bab5407508877…` | |
| | `tokenizer/tokenizer.json` | 6.7 MB | `c0382117ea329cdf…` | |
| | `tokenizer/tokenizer_config.json` | 7.1 KB | `5b5d4f65d0acd3b2…` | |
| | `tokenizer/vocab.json` | 2.6 MB | `ca10d7e9fb3ed185…` | |
|
|
| Full checksums are in `manifest.json`. The playground verifies them after download. |
|
|
| ## Running it |
|
|
| These programs were exported with the ExecuTorch WebGPU backend |
| ([backends/webgpu](https://github.com/pytorch/executorch/tree/main/backends/webgpu)) and validated with the |
| WebAssembly runtime build used by the ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)). That runtime build is not published yet; until it is, |
| the playground is the supported way to run them. It downloads these files once, caches them in the browser, |
| and runs everything locally. |
|
|
| Requirements: a browser with WebGPU enabled. The language models use the `shader-f16` GPU feature when available and fall back to fp32 math without it. |
|
|
| ## Provenance |
|
|
| - **Upstream:** `Qwen/Qwen2.5-0.5B-Instruct` at revision `7ae557604adf67be50417f59c2c2f167def9a775` (official) |
| - **Exported with:** executorch 1.5.1+cpu, torch 2.14.1+cpu, torchao 0.18.0+cpu, transformers 5.19.0 (executorch git `3b6068392324`) |
| - **Reproduce:** `python recipes/qwen2.5-0.5b-instruct.py --out out/qwen2.5-0.5b-instruct` (recipe in `export/`) |
| - Re-running the recipe produces byte-identical files. |
|
|
| ## Citation |
|
|
| If you use this model, please cite the original work: |
|
|
| ```bibtex |
| @misc{qwen2.5, |
| title={Qwen2.5: A Party of Foundation Models}, |
| url={https://qwenlm.github.io/blog/qwen2.5/}, |
| author={Qwen Team}, |
| month={September}, |
| year={2024} |
| } |
| ``` |
|
|
| ## License |
|
|
| This is a derivative of the upstream model and is distributed under the same terms |
| (`apache-2.0`); see `LICENSE`. |
| Modifications (conversion to the ExecuTorch format and quantization) by Meta Platforms, Inc. |
|
|
