Add ExecuTorch WebGPU program for smolvlm-500m-instruct

README.md

@@ -5,6 +5,7 @@ base_model:
 base_model_relation: quantized
 library_name: executorch
 pipeline_tag: image-text-to-text
+inference: false
 tags:
 - executorch
 - webgpu
@@ -15,14 +16,43 @@ tags:
 
 This repository contains an [ExecuTorch](https://github.com/pytorch/executorch) conversion of
 [HuggingFaceTB/SmolVLM-500M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct) for the ExecuTorch **WebGPU** backend, so the model runs entirely in a web browser on
-the visitor's GPU. It powers the corresponding demo in the
-ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)).
+the visitor's GPU. It powers the corresponding demo in the ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)).
 
 - **Base model:** [HuggingFaceTB/SmolVLM-500M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct)
 - **Format:** ExecuTorch `.pte` programs delegated to the WebGPU backend
 - **Quantization:** int8 vision encoder and token embeddings; decoder as exported
 - **Total size:** 893.9 MB
 
+## About the original model
+
+A compact vision-language model that answers questions about images, describes them and transcribes text in them. It pairs a 93M SigLIP vision encoder with the SmolLM2-360M language model.
+
+- **Developed by:** Hugging Face
+- **Paper:** [SmolVLM: Redefining small and efficient multimodal models](https://arxiv.org/abs/2504.05299)
+- 507M parameters
+- English
+- Based on the Idefics3 architecture
+
+For training data, evaluation results and full details, see the original model card: [HuggingFaceTB/SmolVLM-500M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct).
+
+## Intended use and limitations
+
+Image description, visual question answering and reading text in images.
+
+Not for high-stakes decisions about people (for example identity, employment or credit). It can describe images inaccurately or invent details; see the misuse and out-of-scope section of the upstream card.
+
+## What changed in this conversion
+
+- Vision encoder and token embeddings quantized to int8
+- Vision encoder, prefill and decoder exported as separate programs
+- Context capped at 512 tokens
+
+## Inputs and outputs
+
+- `vlm_vision_q8.pte`: one image to visual tokens
+- `vlm_prefill.pte` and `vlm_decoder.pte`: prompt prefill, then token-by-token decoding; embeddings come from `embed_tokens.i8.bin` with `embed_tokens.scale.f32.bin`
+- Tokenize with the files in `tokenizer/`; `vlm_meta.json` lists the special tokens and sizes
+
 ## Files
 
 | File | Size | SHA-256 |
@@ -41,16 +71,32 @@ Full checksums are in `manifest.json`. The playground verifies them after downlo
 
 ## Running it
 
-These programs need the ExecuTorch WebGPU runtime compiled to WebAssembly
-([backends/webgpu](https://github.com/pytorch/executorch/tree/main/backends/webgpu)). The easiest way to
-try the model is the ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)),
-which downloads these files once, caches them in the browser, and runs everything locally.
+These programs were exported with the ExecuTorch WebGPU backend
+([backends/webgpu](https://github.com/pytorch/executorch/tree/main/backends/webgpu)) and validated with the
+WebAssembly runtime build used by the ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)). That runtime build is not published yet; until it is,
+the playground is the supported way to run them. It downloads these files once, caches them in the browser,
+and runs everything locally.
+
+Requirements: a browser with WebGPU enabled.
 
 ## Provenance
 
 - Converted from the upstream checkpoint by the ExecuTorch team with the WebGPU export path.
 - A reproducible export recipe will be published alongside the open-source ExecuTorch WebGPU runtime.
 
+## Citation
+
+If you use this model, please cite the original work:
+
+```bibtex
+@article{marafioti2025smolvlm,
+  title={SmolVLM: Redefining small and efficient multimodal models},
+  author={Andrés Marafioti and Orr Zohar and Miquel Farré and Merve Noyan and Elie Bakouch and Pedro Cuenca and Cyril Zakka and Loubna Ben Allal and Anton Lozhkov and Nouamane Tazi and Vaibhav Srivastav and Joshua Lochner and Hugo Larcher and Mathieu Morlon and Lewis Tunstall and Leandro von Werra and Thomas Wolf},
+  journal={arXiv preprint arXiv:2504.05299},
+  year={2025}
+}
+```
+
 ## License
 
 This is a derivative of the upstream model and is distributed under the same terms