Instructions to use litert-community/vit-base-patch16-224 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT
How to use litert-community/vit-base-patch16-224 with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
File size: 12,662 Bytes
2c83b43 35989ed 2c83b43 35989ed 2c83b43 35989ed 2c83b43 35989ed 2c83b43 f10f70a 2c83b43 35989ed 2c83b43 35989ed 2c83b43 35989ed 2c83b43 35989ed 2c83b43 35989ed 2c83b43 35989ed 2c83b43 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 | ---
library_name: litert
pipeline_tag: image-classification
license: apache-2.0
base_model: google/vit-base-patch16-224
tags:
- vision
- image-classification
- vit
- tflite
datasets:
- imagenet-1k
- imagenet-21k
---
# Google ViT Base Patch16 224 — LiteRT
LiteRT/TFLite variants of
[`google/vit-base-patch16-224`](https://huggingface.co/google/vit-base-patch16-224),
pinned to source revision `3f49326eb077187dfe1c2a2bb15fbd74e6ab91e3`.
The package provides FP32, A16W8 and mixed A8W8 models for image classification.
All have FP32 input and output and require external image preprocessing.
The `.tflite` files are portable source artifacts for target-specific NPU
compilation; they do not contain device-specific compiled NPU programs.
## Model files
| File | Description |
| --- | --- |
| `model_fp32.tflite` | Floating-point reference with FP32 weights. |
| `model_a16w8.tflite` | INT16 transformer activations and INT8 fully connected weights; the primary quantized option for compatible NPUs. |
| `model_a16w8_qualcomm.tflite` | Dedicated Qualcomm NPU compilation input for A16W8 using UINT16 transformer activations and LayerNorm interfaces. |
| `model_a8w8.tflite` | Experimental mixed precision: INT8 fully connected layers with INT16 attention, residual and LayerNorm boundaries. |
| `config.json` | Original source configuration and class mapping. |
| `preprocessor_config.json` | Source image preprocessing configuration. |
| `labels.json` | Class index to label mapping. |
| `model_metadata.json` | Source identity, tensor interfaces, quantization scope and model hashes. |
| `run_image.py` | CPU image-classification example. |
| `requirements.txt` | Runtime and preprocessing package versions. |
## Choosing a variant
- **FP32** provides the floating-point reference and is the example's default.
- **A16W8** offers the best observed accuracy/speed balance on the tested NPUs.
Its fully connected layers use INT16 inputs and outputs (or UINT16 in
`model_a16w8_qualcomm.tflite` for Qualcomm HTP compilation), INT8 weights and
INT64 biases. CPU fallback can be slow, and the tested LiteRT GPU IR importer
does not support its 16-bit integer quantization.
- **Mixed A8W8** is retained as a reference that may suit devices limited to
32-bit integer accumulation in their fully connected kernels. It has a
substantial accuracy tradeoff and retains INT16 regions, so backend support
must be verified.
Extra notes: Apple ANE and QC NPUs support FP16 natively and the performances are
usually very good. Deloying a FP16 model is much easier since it does not require
calibration steps. Note that the A16W8 and A8W8 assets provided here were calibrated
from a particular dataset and should not used in production directly. A FP16 model
file is not provided here is because it is easy to be generated from the fp32 asset.
It is actually a top choice.
## TorchAO quantization recipe
Both variants use post-training static quantization, without QAT or fine-tuning.
The run used PyTorch `2.14.1+cu130`, TorchAO `0.18.0+cu130` and Transformers
`5.21.0`. To reproduce the quantization approach:
- Load the pinned source checkpoint in FP32 evaluation mode. Calibrate with
500 ImageNet-1k validation images, batch size one, using the included PIL
preprocessing configuration. Keep calibration separate from evaluation.
- Observe inputs to all 73 `torch.nn.Linear` layers, including the classifier,
with `AffineQuantizedMinMaxObserver`: asymmetric INT8 `[-128, 127]`,
`PerTensor()` granularity, FP32 scales and INT32 zero points.
- Quantize the original FP32 weights with symmetric `PerRow()` granularity
(one scale per output channel), zero point zero and the explicit narrow
range `[-127, 127]`. Use TorchAO's `choose_qparams_affine` and
`quantize_affine` with these bounds; this weight-range customization is
additional to the activation configuration below.
After calibrating each layer's input observer, its static configuration is:
```python
from torchao.quantization import Int8StaticActivationInt8WeightConfig
from torchao.quantization.granularity import PerRow, PerTensor
from torchao.quantization.quant_primitives import MappingType
scale, zero_point = observer.calculate_qparams()
config = Int8StaticActivationInt8WeightConfig(
act_quant_scale=scale,
act_quant_zero_point=zero_point,
granularity=[PerTensor(), PerRow()],
act_mapping_type=MappingType.ASYMMETRIC,
set_inductor_config=False,
)
```
The final TFLite export also needs calibrated linear-output and transformer
activation ranges, collected using the same 500 images. **A16W8** uses symmetric
INT16 activations with zero point zero and INT64 fully connected biases (and
`model_a16w8_qualcomm.tflite` provides the corresponding UINT16 export for
Qualcomm NPU compilation). **Mixed A8W8** keeps asymmetric INT8 fully connected
inputs/outputs and INT32 biases, with INT16 attention, residual and LayerNorm
boundaries. Both retain the same INT8 weight codes and scales; activation scales
are per-tensor.
The transformer graphs keep activations quantized across attention, residual,
reshape, transpose, softmax and GELU operations. LayerNorm uses composite
operators with quantized interfaces and floating-point decomposition subgraphs
for fallback execution. The patch embedding and public input/output remain
floating-point, so these models do not use integer arithmetic throughout.
A16W8 retains input-scale coarsening for fully connected layers that needed it
under a 32-bit accumulator profile, but omits explicit accumulator input
clipping. Validate accumulator behavior and accuracy on another backend.
TorchAO provides the linear quantization stage; recreating the complete TFLite
graph requires export support for the activation encodings above. The exact
calibration image list is not distributed, so this recipe does not guarantee
identical scales or files. See the
[TorchAO static quantization guide](https://docs.pytorch.org/ao/stable/eager_tutorials/static_quantization.html)
for the observer/calibration workflow.
## Accuracy evaluation
The results below use the same 5,000-image subset of ImageNet-1k validation:
five images per class, selected with seed `20261031`. The selection excludes
the 500 calibration images and 100 earlier diagnostic images. All runs use
the same input tensors, produced with the included Hugging Face PIL
preprocessing configuration.
The original PyTorch FP32 checkpoint scored **79.68% top-1** and **95.40%
top-5**, with TF32 and autocast disabled. The TFLite results below were
measured with AOT-compiled models fully delegated to each device's NPU; all
outputs were finite. FP32 denotes the source model, with FP16 execution
permitted by the NPU compiler.
| Model file | `SM-S937U1` top-1 (%) | `SM-S937U1` top-5 (%) | `LJ9` top-1 (%) | `LJ9` top-5 (%) |
| --- | ---: | ---: | ---: | ---: |
| `model_fp32.tflite` | 79.92 | 95.40 | 79.68 | 95.38 |
| `model_a16w8.tflite` | 79.72 | 95.32 | 79.66 | 95.32 |
| `model_a8w8.tflite` | 75.14 | 92.50 | 75.10 | 92.76 |
A16W8 stays close to the PyTorch baseline on this subset. Mixed A8W8 reduces
top-1 accuracy by 4.54 percentage points on `SM-S937U1` and 4.58 points on
`LJ9` relative to that baseline.
This subset was reused while comparing quantization recipes. It is not an
independent final test set or a full 50,000-image ImageNet validation run.
Small score differences should not be interpreted as general improvements
over the original checkpoint. Raw outputs, input images and diagnostic
reports are not included in this package.
The A16W8 scores were collected before unused tensor records were removed.
That cleanup preserved all live computation, tensor encodings and weights,
and left the Qualcomm compiled file identical. A fresh 100-image MTK check
reproduced the earlier logits exactly; the full 5,000-image MTK run was not
repeated after this metadata cleanup.
## NPU performance
These are representative measurements from a retail device to give a rough
sense of on-device runtime behavior, not a direct comparison between hardware
platforms. All latency numbers were collected with LiteRT's compiled-model
`benchmark_model` launched through `adb shell` on the connected device; they
are not app-level measurements from an integrated Android application.
Warm batch-one inference latency, rounded to the nearest tenth of a
millisecond. Each value is the median of three process means, with five warmup
invocations and 30 measured invocations per process. All configurations are
fully delegated in a single partition with no residual CPU operations (AOT
compiled for NPU, and pre-warmed OpenCL program cache for GPU).
| Model file | Backend / precision | `SM-S937U` (ms) |
| --- | --- | ---: |
| `model_fp32.tflite` | LiteRT GPU (MLDrift OpenCL, FP32) | 23.7 |
| `model_fp32.tflite` | LiteRT GPU (MLDrift OpenCL, FP16) | 12.8 |
| `model_fp32.tflite` | Qualcomm NPU (HTP, FP16) | 3.4 |
| `model_a16w8_qualcomm.tflite` | Qualcomm NPU (HTP, A16W8) | 2.5 |
On `SM-S937U`, Qualcomm NPU execution is **3.8× faster** than MLDrift OpenCL
GPU at matching FP16 precision (`3.4 ms` vs `12.8 ms`), and **5.0× faster**
with `A16W8` quantization (`2.5 ms` vs `12.8 ms`; **9.3× faster** than FP32
OpenCL GPU at `23.7 ms`).
FP32 identifies the source model, which Qualcomm compiles to FP16-based NPU
execution. The floating-source NPU timing therefore does not imply FP32
arithmetic throughout the NPU program. `model_a16w8_qualcomm.tflite` is the
dedicated UINT16 compilation input corresponding to `model_a16w8.tflite`. Mixed
A8W8 is omitted because a Qualcomm-specific UINT16 A8W8 compilation input is
not published in this package.
Measurements use LiteRT's compiled-model benchmark with persistent buffers
and deterministic random input (`clFinish()` synchronized per iteration on
OpenCL GPU). They exclude model initialization, shader/graph compilation, image
preprocessing, file I/O and explicit host input/output buffer transfers.
The `SM-S937U` ran Android 16 with Qualcomm QNN/QAIRT 2.52 in burst performance
mode (`O3` optimization) for NPU and LiteRT's MLDrift OpenCL backend on Adreno
for GPU. Power settings were not normalized, and energy consumption was not
measured. These results apply to the listed device/compiler/runtime
combinations and do not characterize JIT initialization latency.
**Hardware benchmark disclosure:** Results were measured by us on retail
devices. These results are not affiliated with, sponsored by, endorsed by, or
verified by Samsung, TECNO, Qualcomm, MediaTek, Google, MLCommons, or Hugging
Face. Results depend on device SKU, OS build, thermal state, battery mode,
backend, model quantization, runtime version, and benchmark settings.
## Input and output
- Input: FP32 image tensor, NCHW `[1, 3, 224, 224]`.
- Output: FP32 classifier logits, `[1, 1000]`, ordered by `labels.json`.
- Preprocessing: convert to RGB, resize to 224 × 224 with the included
Transformers PIL processor, rescale by 1/255, then normalize each channel
with mean and standard deviation 0.5. The example selects `backend="pil"`.
The models have static batch and spatial dimensions. All 25 LayerNorm
operations use TFLite `STABLEHLO_COMPOSITE` boundaries named `odml.group_norm`,
with FP32 decomposition subgraphs.
## Run a local image
Download the listed files into one directory and run the following from that
directory. The CPU example and dependencies were checked with Python 3.13 on
Linux:
```bash
uv venv --python 3.13
source .venv/bin/activate
uv pip install -r requirements.txt
python run_image.py --image image.jpg
python run_image.py --image image.jpg --model model_a8w8.tflite
python run_image.py --image image.jpg --model model_a16w8.tflite
```
The helper loads the included processor configuration and TFLite model, then
prints the top classes with their raw logits. A16W8 CPU execution may be slow;
the example is an interface check for that variant.
For NPU deployment, compile the selected `.tflite` for the target SoC using the
matching LiteRT compiler plugin, then load the compiled model with the matching
device dispatch runtime and select the NPU accelerator. Verify full delegation
on that device. The CPU example above does not select the NPU.
## Source and attribution
This artifact derives from the pinned Google checkpoint, whose model card
declares the Apache-2.0 license. Refer to that card for training details and
intended use. The architecture was introduced in
[An Image Is Worth 16x16 Words](https://arxiv.org/abs/2010.11929).
The accompanying `config.json` describes the original Transformers model;
inference here uses the `.tflite` file.
|