File size: 2,545 Bytes
9381784 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | ---
license: mit
base_model: openai/clip-vit-base-patch32
pipeline_tag: feature-extraction
library_name: coreai
tags:
- core-ai
- ios
- clip
- embeddings
---
# CLIP ViT-B/32 encoders for Core AI
[`openai/clip-vit-base-patch32`](https://huggingface.co/openai/clip-vit-base-patch32) exported to Apple Core AI as one `.aimodel` with separate text and image encoders, so text and images embed on their own into the same space.
Apple's [`models/clip`](https://github.com/apple/coreai-models/tree/main/models/clip) recipe traces one function that needs an image and a fixed batch of three prompts on every call, so this export uses [`export.py`](./export.py), based on it (coreai-torch 0.4.3, torch 2.13):
```sh
uv run --script export.py --output-dir exports --dtype float16
```
| Folder | Contents | Size | Platforms |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | ---------- |
| `clip-vit-base-patch32/` | `clip-vit-base-patch32.aimodel` with `encode_text(input_ids, attention_mask) -> text_embeds` (`[batch, 77]` int32) and `encode_image(pixel_values) -> image_embeds` (`[batch, 3, 224, 224]` float16, CLIP mean and std), batch 1–64, 512-dim L2-normalized embeddings. `tokenizer/`: the CLIP tokenizer | 290 MB | iOS, macOS |
## Use
With [`@react-native-ai/core-ai`](https://github.com/callstackincubator/ai/tree/main/packages/core-ai), download `clip-vit-base-patch32/` into app storage, then embed text with the AI SDK and images with the native API:
```ts
import { coreAI, CoreAI } from '@react-native-ai/core-ai'
import { cosineSimilarity, embedMany } from 'ai'
const model = coreAI.embeddingModel({
id: 'clip-vit-base-patch32',
source: { type: 'file', uri: pathToFolder },
})
const { embeddings } = await embedMany({ model, values: ['a photo of a cat', 'a photo of a car'] })
const { embeddings: [image] } = await model.embedImages([imageUri])
cosineSimilarity(embeddings[0], image) // higher for the matching caption
```
## License
CLIP is MIT licensed. This repository only converts it to Core AI format.
|