Sky-Kim/com.sky.sentis.tipsv2-b14

sentis-tipsv2-b14

Google TIPSv2-B/14 (google/tipsv2-b14, Apache-2.0), a contrastive image-text model, converted to Unity Inference Engine (Sentis) with FP16 weights. Image crops and English text are embedded into the same 768-d space, so a text query can rank image regions (zero-shot search: "a person wearing a red jacket", "a person lying on the ground", "a white car").

Files

Models~/tipsv2_b14_vision_fp16.sentis   # pixels [1, 3, 224, 224] RGB 0-1 -> embedding [1, 768] (global / CLS token)
Models~/tipsv2_b14_text_fp16.sentis     # ids [1, L] int (1 <= L <= 64)   -> embedding [1, 768] (masked mean pool)
Models~/tokenizer.json                  # WebLI SentencePiece BPE (32k) as a Hugging Face tokenizer.json
Runtime/TipsV2B14.cs                    # self-contained Unity Sentis inference

Usage

Add the package (UPM, git URL or local folder), then:

using SentisModels;

var tips = new TipsV2B14(BackendType.GPUCompute);   // falls back to CPU without compute shaders
tips.Load(modelRoot);                                // folder holding the Models~ files
tips.Warmup();

float[] query = tips.EmbedText("a photo of a person wearing a red jacket");   // unit length, blocking, once per query
float[][] crops = await tips.EmbedCropsAsync(frameTexture, personBoxes);      // boxes in source pixels, top-left origin
float score = TipsV2B14.Dot(query, crops[0]);                                 // cosine similarity

Dim = 768, InputSize = 224. EmbedCropsAsync schedules one crop per frame, keeps each output with CopyOutput and reads them back with AsyncGPUReadback; LastCropsMs is the wall-clock time of the last call. Canvas is the last vision input, for inspection. Tokenize returns the token ids.

Preprocessing

• Vision: each box is widened by 15% of its width on each side and 5% of its height top and bottom, clamped to the texture, then letterboxed (aspect kept, centred) into 224 x 224 on mid-grey (0.5) on the GPU (sRGB render texture, top-left origin). In Linear colour-space projects the clear colour is converted (s_Pad.linear) so the pad still reaches the model as 0.5 (it would otherwise be sRGB-encoded to 0.737). TIPS takes plain RGB in [0, 1]: no mean / std normalisation.
• Text: lowercased, SentencePiece BPE without BOS / EOS, truncated to 64 tokens. The reference zero-pads to 64 and masks the padding out of attention, the MLP and the mean pool; the exported graph takes only the real tokens, which gives the same embedding (verified below) for less work. The tokenizer runs on Unity's built-in Unity.InferenceEngine.Tokenization HF parser.
• Embedding: image = the final-norm CLS token (cls_token in the model card, the global embedding used for zero-shot classification); text = the masked mean of the final tokens. Both are L2-normalised in C#.
• The model's learned temperature is 0.00507 (logit scale ~197) if you need softmax probabilities over labels.

Conversion notes

• Source: google/tipsv2-b14 (transformers remote code). Wrapped in PyTorch and exported with torch.onnx.export(opset 17, dynamo=False): the vision graph returns only cls_token[:, 0], with the (antialiased, not exportable) position-embedding resize baked for 224 x 224 (the checkpoint's native size is 448); the text graph re-implements the reference forward without the padding mask (dynamic token count). Imported into Sentis 2.6.1, weights quantized to FP16 with ModelQuantizer.QuantizeWeights(Float16), saved with ModelWriter.Save. The only import warning is the benign LayerNormalization axis one (graph axis is -1).
• tokenizer.model (SentencePiece, model_type: BPE, nmt_nfkc, byte fallback) is converted to tokenizer.json (BPE with merges ordered by piece score, Precompiled NFKC normalizer + strip + space collapse, Metaspace). Token ids are identical to SentencePiece on 310 test strings in Python (incl. emoji, Korean, full-width, tabs) and on 15 strings through TipsV2B14.Tokenize in Unity.
• Why 224 and not 448: on the crops below 224 ranks the probes as well as 448 (12/12 vs 11/12 correct on a 12-prompt probe), and person crops from a CCTV frame are small, while the vision cost grows ~4.5x at 448 (1025 tokens).

Parity (5 crops + 5 sentences, same input bytes everywhere)

Edge A, PyTorch reference (model.encode_image(...).cls_token, model.encode_text(...)) vs ONNX Runtime: vision min cosine 1.000000 (max abs 7.4e-6), text min cosine 1.000000 (max abs 2.9e-6).

Edge B, ONNX Runtime vs Sentis (and vs PyTorch), Apple M5 Max, Unity 6000.3.23f1:

tower	weights	backend	min cos vs ORT	max abs vs ORT	min cos vs PyTorch	warm ms (blocking)
vision	fp32	CPU	1.000000	8.23e-6	1.000000	198.8
vision	fp32	GPUCompute	1.000000	9.06e-6	1.000000	18.6
vision	fp16	CPU	1.000000	7.69e-4	1.000000	214.4
vision	fp16	GPUCompute	1.000000	7.69e-4	1.000000	19.5
text	fp32	CPU	1.000000	1.91e-6	1.000000	13.3
text	fp32	GPUCompute	1.000000	1.91e-6	1.000000	9.8
text	fp16	CPU	1.000000	5.00e-4	1.000000	36.9
text	fp16	GPUCompute	1.000000	5.00e-4	1.000000	12.1

Through the package class (FP16): EmbedText min cosine vs PyTorch 1.000000 on both backends; EmbedCropsAsync vs a PIL letterbox of the same widened boxes min cosine 0.9855 in a Gamma project and 0.9844 in a Linear project (GPU bilinear vs PIL resampling of tiny crops, and in Linear Unity filters sRGB textures in linear light; not a model difference).

Zero-shot sanity check (Unity, GPUCompute, FP16, Linear colour space)

Crops from the police demo recordings (Ch1_City, Ch2_Accident), cosine similarity:

sentence	red jacket person	lying person	white van	walking person	red car
a photo of a person wearing a red jacket	0.131	0.116	0.074	0.124	0.123
a photo of a person lying on the ground	0.135	0.151	0.098	0.138	0.104
a photo of a white car	0.078	0.103	0.143	0.083	0.122
a photo of a person walking	0.141	0.125	0.110	0.137	0.111
a photo of a red car	0.079	0.095	0.109	0.077	0.153

The three target sentences rank the matching crop first, on GPUCompute and CPU, in Linear and Gamma projects. "a person walking" picks the red-jacket person, who is also walking. As with other CLIP-style models, raw cosines are small (~0.1-0.2): rank crops, or compare against a threshold calibrated on your own footage, rather than using an absolute cut-off.

Warm timing (Apple M5 Max, Unity editor, FP16, GPUCompute)

• EmbedText: 11.5 ms per sentence (blocking, once per query).
• EmbedCropsAsync: ~19 ms GPU per crop (blocking schedule + readback, table above); 10 crops took 184 ms wall clock = 18.4 ms per crop, one crop per frame including the frame waits.
• CPU backend: 206 ms per crop, 35 ms per sentence, so use GPUCompute where available.

License & attribution

Model weights and tokenizer: Apache License 2.0, from Google's TIPSv2 (model card, code), modified as described in LICENSE.md. The wrapper code is MIT. If you use the model, cite:

@inproceedings{cao2026tipsv2,
  title     = {{TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment}},
  author    = {Cao, Bingyi and Chen, Koert and Maninis, Kevis-Kokitsi and Chen, Kaifeng and Karpur, Arjun and Xia, Ye and Dua, Sahil and Dabral, Tanmaya and Han, Guangxing and Han, Bohyung and Ainslie, Joshua and Bewley, Alex and Jacob, Mithun and Wagner, Rene and Ramos, Washington and Choromanski, Krzysztof and Seyedhosseini, Mojtaba and Zhou, Howard and Araujo, Andre},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2604.12012}
}