imagevolve-clip-vit-base-patch32

The CLIP ViT-B/32 files that imagevolve runs in your browser. imagevolve is a CLIP-guided evolutionary search over deterministic image transformations that bring your picture closer to your words.

How imagevolve uses CLIP

A variant is a short program of operations such as tile shuffles, swirls or recolourings. CLIP scores it by how far its image embedding moves from the original's along a text direction: from the embedding of "a photograph" to that of your words, minus the mean of such directions over 430 generic phrases. Nothing is optimized by gradient descent: CLIP only ranks the variants. On WebGPU, imagevolve scores about 1,100 variants a second on a $350 GPU with these files.

Files

They are Xenova's ONNX export of OpenAI's CLIP at 4-bit weights with 16-bit activations (q4f16), with two changes. The text model stores its token table in 8 bits, which makes it 47.3 MB instead of 72.5. A copy of the vision model reads 160-pixel pictures instead of 224, which halves its work per picture. imagevolve now runs an 8-bit copy of it, made from the fp32 export, which needs no 16-bit floats in shaders.

File	Source
onnx/vision_model_q8a4_160.onnx	Xenova's fp32 vision_model.onnx resized to 160 × 160 the same way, every weight MatMul in 8 bits (MatMulNBits, block 32, symmetric) computed in int8 (accuracy level 4)
onnx/vision_model_q4f16_160.onnx	Xenova's vision_model_q4f16.onnx for 160 × 160 input: the 7 × 7 grid of position embeddings resized to 5 × 5 (bilinear), the reshapes set to 26 tokens
onnx/vision_model_q4f16.onnx	Xenova's file, unchanged, which imagevolve used before the 160-pixel copy
onnx/text_model_q4f16_t8.onnx	Xenova's text_model_q4f16.onnx, token table (49,408 × 512) in int8 with an fp16 scale per row
tokenizer.json, tokenizer_config.json	Xenova's files, unchanged

The files run in ONNX Runtime Web on WebGPU or WebAssembly. imagevolve loads them at a pinned revision of this repository.

Fidelity

The token lookup becomes a Gather of int8 rows, a Cast to fp16 and a multiplication by the row's scale, all standard ONNX operators. For three test phrases, its text embeddings have a cosine of 0.9998 or more with the original's, in Node and in the browser on both backends. Over 3,600 transformed photographs and 16 phrases, imagevolve's direction scores follow the original's with a correlation of 0.995 at worst, and follow fp32's as closely as the original does.

The 8-bit vision model's direction scores follow the fp32 160-pixel model's with a correlation of 0.998 at the median and 0.994 at worst (80 transformed photographs, 3 phrases, Node), against 0.857 and 0.769 for the q4f16 copy, and each picture's score is independent of its batch. On WebGPU it takes 2.1 ms per picture in batches of 32 against 2.3 for the q4f16 copy (Intel Arc A770). On WebAssembly imagevolve rewrites it in the browser to int8 activations with one scale per token, 24 ms per picture against 44 (8 threads, Intel i5-14400F).

The 160-pixel model was not fine-tuned. Its direction scores follow the 224-pixel model's with a correlation of 0.79 at the median, yet from the same 1,200 candidates per photograph, a blind reviewer preferred its picks in 4 of 6 cases, with one tie. In the browser on WebAssembly it takes 45 ms per picture against 71 (Edge, Intel i5-14400F).

Licence

CLIP was released by OpenAI under the MIT licence (github.com/openai/CLIP).