---
license: apache-2.0
base_model:
- HuggingFaceTB/SmolVLM-500M-Instruct
base_model_relation: quantized
library_name: executorch
pipeline_tag: image-text-to-text
inference: false
tags:
- executorch
- webgpu
- on-device
- browser
---
# SmolVLM 500M Instruct for ExecuTorch WebGPU

This repository contains an [ExecuTorch](https://github.com/pytorch/executorch) conversion of
[HuggingFaceTB/SmolVLM-500M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct) for the ExecuTorch **WebGPU** backend, so the model runs entirely in a web browser on
the visitor's GPU. It powers the corresponding demo in the [ExecuTorch WebGPU Playground](https://executorch.ai/playground/).

- **Base model:** [HuggingFaceTB/SmolVLM-500M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct)
- **Format:** ExecuTorch `.pte` programs delegated to the WebGPU backend
- **Quantization:** 8-bit linears in the vision encoder and decoder (int8 static activations in two passes, int8 per-channel GPTQ weights); int8 token embeddings
- **Total size:** 497.7 MB

## About the original model

A compact vision-language model that answers questions about images, describes them and transcribes text in them. It pairs a 93M SigLIP vision encoder with the SmolLM2-360M language model.

- **Developed by:** Hugging Face
- **Paper:** [SmolVLM: Redefining small and efficient multimodal models](https://arxiv.org/abs/2504.05299)
- 507M parameters
- English
- Based on the Idefics3 architecture

For training data, evaluation results and full details, see the original model card: [HuggingFaceTB/SmolVLM-500M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct).

## Intended use and limitations

Image description, visual question answering and reading text in images.

Not for high-stakes decisions about people (for example identity, employment or credit). It can describe images inaccurately or invent details; see the misuse and out-of-scope section of the upstream card.

## What changed in this conversion

- All linear layers of the vision encoder and decoder use 8-bit weights (per-channel, GPTQ) and 8-bit activations with one static scale per layer, applied in two passes (the second pass quantizes the first pass's rounding error)
- Token embeddings quantized to int8
- The key/value of the first prompt token (`<|im_start|>`) are built into the decoder, so prompts must start with it
- Context capped at 512 tokens; quantization changes outputs slightly compared with the original model

## Inputs and outputs

- `smolvlm_vision_q8.pte`: `pixels float32 [1, 3, 512, 512]` (scaled as `(x/255 - 0.5) / 0.5`) -> `image_embeds float32 [1, 64, 960]`
- `smolvlm_decoder_q8.pte`: `forward(embeds float32 [1, 4, 960], input_pos int64 [1]) -> logits float32 [1, 49280]` for the last of the 4 rows; feed windows of 4 consecutive positions, the KV cache lives inside the program
- Text embeddings come from `embed_tokens.i8.bin` with `embed_tokens.scale.f32.bin`; tokenize with the files in `tokenizer/`; `vlm_meta.json` lists the special tokens and sizes

## Files

| File | Size | SHA-256 |
|---|---|---|
| `embed_tokens.i8.bin` | 45.1 MB | `d3ea47247be0b070…` |
| `embed_tokens.scale.f32.bin` | 192.5 KB | `adefdadaae81d572…` |
| `smolvlm_decoder_q8.pte` | 349.9 MB | `a54d81fbafd289c3…` |
| `smolvlm_vision_q8.pte` | 99.0 MB | `a48d6919aa2067be…` |
| `tokenizer/special_tokens_map.json` | 1.0 KB | `aa0ff906077086df…` |
| `tokenizer/tokenizer.json` | 3.4 MB | `5ece781dc8d2b2f3…` |
| `tokenizer/tokenizer_config.json` | 27.6 KB | `36c6fd44d07d10fd…` |
| `vlm_meta.json` | 201 B | `b36fcf7ef7da8c2a…` |

Full checksums are in `manifest.json`. The playground verifies them after download.

## Running it

These programs were exported with the ExecuTorch WebGPU backend
([backends/webgpu](https://github.com/pytorch/executorch/tree/main/backends/webgpu)) and run in a browser (headless Chromium, WebGPU on SwiftShader) with the WebAssembly runtime build used by the [ExecuTorch WebGPU Playground](https://executorch.ai/playground/), where every generated token (78 of 78 over 4 turns) matched the same programs run on CPU. That runtime build is published in
[executorch-community/ExecuTorch-WebGPU-Runtime](https://huggingface.co/executorch-community/ExecuTorch-WebGPU-Runtime); the playground is the easiest way to run them. It downloads these files once, caches them in the browser,
and runs everything locally.

Requirements: a browser with WebGPU enabled.

## Provenance

- **Upstream:** `HuggingFaceTB/SmolVLM-500M-Instruct` at revision `a7da5b986cb59b408707209984f360a5f4ad7e47` (official)
- **Exported with:** executorch 1.5.1+cpu, torch 2.14.1+cpu, torchao 0.18.0+cpu, transformers 5.19.0 (executorch git `3b6068392324`)
- **Reproduce:** `python recipes/smolvlm-500m-instruct-q8.py --out out/smolvlm-500m-instruct-q8` (recipe in `export/`)
- Re-running the recipe produces byte-identical files.

## Citation

If you use this model, please cite the original work:

```bibtex
@article{marafioti2025smolvlm,
  title={SmolVLM: Redefining small and efficient multimodal models},
  author={Andrés Marafioti and Orr Zohar and Miquel Farré and Merve Noyan and Elie Bakouch and Pedro Cuenca and Cyril Zakka and Loubna Ben Allal and Anton Lozhkov and Nouamane Tazi and Vaibhav Srivastav and Joshua Lochner and Hugo Larcher and Mathieu Morlon and Lewis Tunstall and Leandro von Werra and Thomas Wolf},
  journal={arXiv preprint arXiv:2504.05299},
  year={2025}
}
```

## License

This is a derivative of the upstream model and is distributed under the same terms
(`apache-2.0`); see `LICENSE`.
Modifications (conversion to the ExecuTorch format and quantization) by Meta Platforms, Inc.