README.md

---
license: llama3.2
license_name: llama3.2
license_link: LICENSE
base_model:
- meta-llama/Llama-3.2-1B-Instruct
base_model_relation: quantized
library_name: executorch
pipeline_tag: text-generation
inference: false
tags:
- executorch
- webgpu
- on-device
- browser
---
# Llama 3.2 1B Instruct for ExecuTorch WebGPU

**Built with Llama.**

This repository contains an [ExecuTorch](https://github.com/pytorch/executorch) conversion of
[meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) for the ExecuTorch **WebGPU** backend, so the model runs entirely in a web browser on
the visitor's GPU. It powers the corresponding demo in the ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)).

- **Base model:** [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct)
- **Format:** ExecuTorch `.pte` programs delegated to the WebGPU backend
- **Quantization:** 4-bit groupwise weights (q4gsw, group size 64) for all linear layers and the tied token-embedding/output head; fp32 activations and KV cache
- **Total size:** 709.5 MB

## About the original model

The 1B instruction-tuned, text-only model of Llama 3.2, tuned with supervised fine-tuning and RLHF for helpfulness and safety.

- **Developed by:** Meta
- **Paper:** [The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)
- 1.23B parameters
- Pretrained on up to 9T tokens of publicly available data; knowledge cutoff December 2023
- Officially supported languages: English, German, French, Italian, Portuguese, Hindi, Spanish and Thai

For training data, evaluation results and full details, see the original model card: [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct).

## Intended use and limitations

Assistant-like chat, knowledge retrieval and summarization, writing assistance, and query rewriting, in the supported languages. Use must comply with the Llama 3.2 Community License and Acceptable Use Policy (`USE_POLICY.md`).

Out of scope: any use that violates applicable laws or the Acceptable Use Policy, and use in languages beyond the supported ones. Like all language models it can produce inaccurate, biased or objectionable responses; see the Ethical Considerations and Limitations section of the upstream card.

## What changed in this conversion

- Exported with ExecuTorch's `export_llm` (`model_class=llama3_2`) from the original Meta-format checkpoint (`original/consolidated.00.pth`, bf16), with Llama 3 scaled RoPE (factor 32)
- All linear layers quantized to 4-bit groupwise weights (q4gsw, group size 64, round to nearest); the token embedding is quantized the same way. Llama 3.2 ties the embedding and the output head, so they quantize to the same bytes and the program stores them once
- Activations and the KV cache buffers in the program are fp32
- Context capped at 8,960 tokens (upstream: 128K); at most 512 tokens per forward call
- 4-bit quantization changes the outputs: on 3 test prompts, greedy text from this program agreed with the fp32 model for the first 6, 8 (the whole answer) and 0 tokens, and gave fluent, on-topic answers in all three

## Inputs and outputs

- `forward(tokens int64 [1, 1..512], input_pos int64 [1]) -> logits float32 [1, 128256]` (last position only)
- The KV cache lives inside the program; call it with up to 512 new tokens at a time, then one token per step. Metadata methods: `get_max_seq_len` 513, `get_max_context_len` 8960, `get_n_layers` 16, `get_vocab_size` 128256
- Tokenize with the files in `tokenizer/` (the upstream Llama 3.2 tokenizer and chat template); stop on token 128001, 128008 or 128009

## Files

| File | Size | SHA-256 |
|---|---|---|
| `llama3_2_1b_instruct_q4gsw.pte` | 700.8 MB | `49de6eec3a54f1f4…` |
| `tokenizer/config.json` | 877 B | `2febf68cea25bf46…` |
| `tokenizer/generation_config.json` | 189 B | `88effbb63300dbbc…` |
| `tokenizer/special_tokens_map.json` | 296 B | `6f38c73729248f6c…` |
| `tokenizer/tokenizer.json` | 8.7 MB | `79e3e522635f3171…` |
| `tokenizer/tokenizer_config.json` | 53.2 KB | `9823dcfdc1121869…` |

Full checksums are in `manifest.json`. The playground verifies them after download.

## Running it

These programs were exported with the ExecuTorch WebGPU backend
([backends/webgpu](https://github.com/pytorch/executorch/tree/main/backends/webgpu)) and run in a browser (headless Chromium, WebGPU on SwiftShader) with a development build of the ExecuTorch
WebGPU WebAssembly runtime, where its greedy output matched the CPU reference token for token on 3 test prompts. That runtime build is not published yet; until it is,
the playground is the supported way to run them. It downloads these files once, caches them in the browser,
and runs everything locally.

Requirements: a browser with WebGPU enabled. The language models use the `shader-f16` GPU feature when available and fall back to fp32 math without it.

## Provenance

- **Upstream:** `meta-llama/Llama-3.2-1B-Instruct` at revision `9213176726f574b556790deb65791e0c5aa438b6` (official)
- **Weights source:** downloaded from the public mirror `LIF1014/ptdbench-Llama-3.2-1B-Instruct` @ `828e5f37c833a02415416260d5a93780fd41d10e`; the gated official repo was not downloaded. Each of the 9 files taken from the mirror (checkpoint `original/consolidated.00.pth`, `original/params.json`, the 5 tokenizer/config files, `LICENSE.txt`, `USE_POLICY.md`) was verified against the hash the official repo publishes for it at the pinned revision (LFS sha256 for the checkpoint, `fc17d497df5e4175b3a8acb4f5865b26f7fc1b009b25bef814b95fde10e8a1f3`; git blob SHA-1 for the small files), so the bytes used are the official ones. `export/llama-3.2-1b-instruct.py` repeats this check on every run
- **Exported with:** executorch 1.5.1+cpu, torch 2.14.1+cpu, torchao 0.18.0+cpu, transformers 5.19.0 (executorch git `3b6068392324`)
- **Reproduce:** `python recipes/llama-3.2-1b-instruct.py --out out/llama-3.2-1b-instruct` (recipe in `export/`)
- Re-running the recipe produces byte-identical files.

## Citation

If you use this model, please cite the original work:

```bibtex
@article{grattafiori2024llama3,
  title={The Llama 3 Herd of Models},
  author={Grattafiori, Aaron and others},
  journal={arXiv preprint arXiv:2407.21783},
  year={2024}
}
```

## License

This is a derivative of the upstream model and is distributed under the same terms
(`llama3.2`); see `LICENSE` and `USE_POLICY.md`.
Modifications (conversion to the ExecuTorch format and quantization) by Meta Platforms, Inc.