File size: 6,761 Bytes

---
pipeline_tag: feature-extraction
tags:
- eidora
- eidora-model-zoo
- onnx
- onnxruntime
- embeddings
- multimodal
- modality:multimodal
- clip
base_model: openai/clip-vit-base-patch32
---

# CLIP ViT-B/32 — Multimodal

This EIDORA package combines the CLIP ViT-B/32 image encoder and text encoder. When both inputs are selected, it produces one normalized 512-dimensional fused embedding using EIDORA’s normalized-mean rule.

## Training Data and Checkpoint Provenance

- **Architecture:** CLIP ViT-B/32 image encoder and CLIP text Transformer, with two separate ONNX components.
- **Original checkpoint:** [openai/clip-vit-base-patch32](https://huggingface.co/openai/clip-vit-base-patch32).
- **Pinned revision:** `b97b0100e55e367c057773c2a614676470b0d575`.
- **Training data:** Publicly available image-caption data collected from the internet; the exact training dataset was not released.
- **Training method:** Contrastive image–text pretraining; the complete training pairs were not released.
- **Source:** CLIP ViT-B/32 (original author attribution retained).

CLIP alignment relies on both encoders originating from the same upstream checkpoint and revision.

## Model Overview

- **Model:** CLIP ViT-B/32 — Multimodal
- **EIDORA identifier:** `EIDORA/CLIP_ViTB32_Multimodal`
- **Modality:** Multimodal
- **Embedding dimensions:** 512
- **Embedding normalization:** L2-normalized
- **Similarity measure:** Cosine similarity
- **Package schema:** Version 3
- **Model version:** `0.1.0`

## EIDORA Use and Supported Modalities

**Supported modality:** Images and text.

Researchers can use this package to:

- Represent items that have both an image and a text description.
- Explore clusters derived from combined visual and language information.
- Compare fused representations against the standalone Image and Text variants.

Available EIDORA variants of the same checkpoint:

- **Image:** [CLIP Image](https://huggingface.co/EIDORA/CLIP_ViTB32_Image) — image inputs only.
- **Text:** [CLIP Text](https://huggingface.co/EIDORA/CLIP_ViTB32_Text) — text inputs only.
- **Multimodal:** [CLIP Multimodal](https://huggingface.co/EIDORA/CLIP_ViTB32_Multimodal) — image and text inputs combined by EIDORA.

When image and text are selected together, EIDORA returns their combined embedding. Use the standalone encoders to calculate standard CLIP image–text cosine similarities.

## Input and Preprocessing

This package has two independent preprocessing contracts. Both are defined in `config.yaml`.

**Image component** (`image_encoder.onnx`):

1. Decode as RGB and resize the shorter edge to 224 pixels, then center-crop to 224 × 224.
2. Use **bicubic** interpolation with **truncate** resize rounding.
3. Rescale by `0.00392156862745098` and normalize by the channel mean `[0.48145466, 0.4578275, 0.40821073]` and standard deviation `[0.26862954, 0.26130258, 0.27577711]`.
4. Send a `float32` tensor in `NCHW` format (`[batch, 3, 224, 224]`).

**Text component** (`text_encoder.onnx`):

1. Read UTF-8 text and tokenize with the packaged CLIP tokenizer (`tokenizer/`).
2. Add special tokens; pad or truncate to **77 tokens** on the right.
3. Provide `input_ids` and `attention_mask` as `int64` tensors of shape `[batch, 77]`.

The two components are aligned to the same CLIP checkpoint and should not be substituted with an encoder from a different revision.

## Output Embeddings

The ONNX output is `float32` with shape `['batch', 512]`. Each vector has **512 dimensions**, is **L2-normalized**, and is intended for **cosine similarity** comparisons.

EIDORA runs the original aligned encoders separately, L2-normalizes their embeddings, averages the selected modalities and L2-normalizes the combined vector. The image and text graphs remain distinct components.

**Fusion:** Each component first produces a normalized CLIP vector. EIDORA combines selected component vectors by `normalized_mean` and normalizes the resulting vector. The fused vector is not CLIP’s standard pairwise image–text similarity score.

## Runtime and Compatibility

The package contains two ONNX graphs (`image_encoder.onnx` and `text_encoder.onnx`), a packaged CLIP tokenizer and per-component schema-v3 runtime declarations. The fusion contract is `normalized_mean`.

It declares **`CPUExecutionProvider`**. Actual execution availability depends on the installed ONNX Runtime and EIDORA Desktop environment. `config.yaml` is authoritative for graph inputs, preprocessing, provider and embedding dimensions.

## Limitations

- The normalized-mean fusion is an EIDORA adaptation, not CLIP’s original contrastive similarity score.
- A fused image-plus-text vector should not be mistaken for the original image-to-text matching probability.
- Bias and domain differences in both image and text inputs can influence results.

Results should be evaluated against the intended research data and questions.

## Research Context and References

### Main paper

**Learning Transferable Visual Models From Natural Language Supervision**

Paper: https://arxiv.org/abs/2103.00020

The publication describes the upstream learning method and original model. The embeddings returned by EIDORA additionally depend on the specific preprocessing and output adaptation documented here.

### Primary citation (BibTeX)

```bibtex
@inproceedings{radford2021learning,
  title={Learning Transferable Visual Models From Natural Language Supervision},
  author={Radford, Alec and Kim, Jong Wook and Hallacy, Chris and Ramesh, Aditya and Goh, Gabriel and Agarwal, Sandhini and Sastry, Girish and Askell, Amanda and Mishkin, Pamela and Clark, Jack and Krueger, Gretchen and Sutskever, Ilya},
  booktitle={ICML},
  year={2021}
}
```

## License and Attribution

The OpenAI CLIP source code is MIT-licensed. That statement alone does not determine licensing of pretrained CLIP weights. This package contains ONNX conversions of the image and text encoders and does not assert an independently verified checkpoint license.

No separate upstream checkpoint `LICENSE` file was supplied in the captured EIDORA repository; that absence should not be interpreted as permission or prohibition.

## Package Information

- **Model repository:** `EIDORA/CLIP_ViTB32_Multimodal`
- **Package version:** `0.1.0`
- **Package schema:** `3`
- **EIDORA configuration:** `config.yaml`
- **ONNX graph(s):** `image_encoder.onnx`, `text_encoder.onnx`
- **Supported modality:** Multimodal
- **License identifier:** `other`
- **Declared provider:** CPUExecutionProvider
- **Required supplementary files:** `tokenizer/` (packaged tokenizer)

The configuration defines the authoritative model-facing runtime contract. Other packaged assets, where present, do not introduce additional inference modalities.