| --- |
| license: mit |
| base_model: openai/clip-vit-base-patch32 |
| pipeline_tag: feature-extraction |
| library_name: coreai |
| tags: |
| - core-ai |
| - ios |
| - clip |
| - embeddings |
| --- |
| |
| # CLIP ViT-B/32 encoders for Core AI |
|
|
| [`openai/clip-vit-base-patch32`](https://huggingface.co/openai/clip-vit-base-patch32) exported to Apple Core AI as one `.aimodel` with separate text and image encoders, so text and images embed on their own into the same space. |
|
|
| Apple's [`models/clip`](https://github.com/apple/coreai-models/tree/main/models/clip) recipe traces one function that needs an image and a fixed batch of three prompts on every call, so this export uses [`export.py`](./export.py), based on it (coreai-torch 0.4.3, torch 2.13): |
|
|
| ```sh |
| uv run --script export.py --output-dir exports --dtype float16 |
| ``` |
|
|
| | Folder | Contents | Size | Platforms | |
| | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | ---------- | |
| | `clip-vit-base-patch32/` | `clip-vit-base-patch32.aimodel` with `encode_text(input_ids, attention_mask) -> text_embeds` (`[batch, 77]` int32) and `encode_image(pixel_values) -> image_embeds` (`[batch, 3, 224, 224]` float16, CLIP mean and std), batch 1–64, 512-dim L2-normalized embeddings. `tokenizer/`: the CLIP tokenizer | 290 MB | iOS, macOS | |
|
|
| ## Use |
|
|
| With [`@react-native-ai/core-ai`](https://github.com/callstackincubator/ai/tree/main/packages/core-ai), download `clip-vit-base-patch32/` into app storage, then embed text with the AI SDK and images with the native API: |
|
|
| ```ts |
| import { coreAI, CoreAI } from '@react-native-ai/core-ai' |
| import { cosineSimilarity, embedMany } from 'ai' |
| |
| const model = coreAI.embeddingModel({ |
| id: 'clip-vit-base-patch32', |
| source: { type: 'file', uri: pathToFolder }, |
| }) |
| const { embeddings } = await embedMany({ model, values: ['a photo of a cat', 'a photo of a car'] }) |
| const { embeddings: [image] } = await model.embedImages([imageUri]) |
| |
| cosineSimilarity(embeddings[0], image) // higher for the matching caption |
| ``` |
|
|
| ## License |
|
|
| CLIP is MIT licensed. This repository only converts it to Core AI format. |
|
|
