Model card (TheStage Apple SDK)

README.md

@@ -0,0 +1,224 @@
+---
+base_model:
+- google/embeddinggemma-2
+base_model_relation: quantized
+license: apache-2.0
+pipeline_tag: feature-extraction
+library_name: coreml
+tags:
+- embeddings
+- multimodal
+- sentence-similarity
+- retrieval
+- matryoshka
+- apple-silicon
+- on-device
+- coreml
+- ios
+- macos
+- thestage-apple-sdk
+---
+
+# EmbeddingGemma 2 (Apple Silicon)
+
+![On-Device Apple SDK](assets/on-device-apple-sdk-banner.png)
+
+
+[TheStage Apple SDK](https://github.com/TheStageAI/AppleSDK)
+
+**Original model:** [google/embeddinggemma-2](https://huggingface.co/google/embeddinggemma-2) (Google)
+
+On-device **multimodal embeddings** compressed and packaged by **TheStage AI** for Apple Silicon (CoreML / Neural Engine). Text, images, audio, video, and mixed inputs (an image with its caption) map into **one 768-d, L2-normalized space**, so a typed query can find a paragraph, a photo, a sound or a clip. Matryoshka-trained: the first 512, 256 or 128 values are themselves embeddings.
+
+| | |
+| --- | --- |
+| Runtime | TheStage Apple SDK (`EmbeddingPipeline`) |
+| Embedding dim | **768** (L2-normalized); Matryoshka **512 / 256 / 128** |
+| Inputs | text, image, audio, video, mixed |
+| Default device | **NPU** (Neural Engine) |
+| SDK | **1.7.0+** |
+| HF engines | `TheStageAI/embeddinggemma-2` (tag `v1.7`) |
+
+
+## Overview
+
+---
+
+This repo hosts **CoreML engine bundles** for the TheStage Apple SDK. Point `engines_path` at the repo id; the SDK downloads, caches, and loads the engines automatically. Task prompts, token limits and Matryoshka sizes all come from the bundle. Open only the inputs you use (`modalities:`) to shorten the first load.
+
+## System Requirements
+
+---
+
+| **Property** | **Value** |
+| --- | --- |
+| **Hardware** | Apple Silicon Mac, or physical iPhone / iPad |
+| **macOS** | 15.0+ |
+| **iOS** | 18.0+ |
+| **Xcode** | 16.0+ |
+| **Swift** | 6.0+ |
+| **Flutter** (optional) | 3.24+ |
+
+> Simulator is **not** supported — run on real Apple Silicon hardware.
+
+
+## TheStage AI Access Token
+
+---
+
+Create a token at [app.thestage.ai](https://app.thestage.ai) and pass it to the SDK:
+
+```swift
+try await TheStageAI.shared.initialize(api_token: "th_…")
+```
+
+```dart
+await TheStageFlutterSDK.initialize(api_token: 'th_…');
+```
+
+Token is checked online in `initialize` (once per app process when reachable). Inference runs fully on-device. Offline initialize fails — reconnect and call `initialize` again.
+
+
+## TheStage Apple SDK
+
+---
+
+Docs: [TheStage Apple SDK · Embeddings](https://docs.thestage.ai) · Repo: [TheStage Apple SDK](https://github.com/TheStageAI/AppleSDK)
+
+
+### Installation (SwiftPM)
+
+In Xcode: **File → Add Package Dependencies…**, paste `https://github.com/TheStageAI/AppleSDK.git`, and add the `TheStageSDK` product. Or in `Package.swift`:
+
+```swift
+.package(
+    url: "https://github.com/TheStageAI/AppleSDK.git",
+    exact: Version(1, 7, 0)
+)
+```
+
+### Installation (Flutter / iOS)
+
+```yaml
+# pubspec.yaml
+dependencies:
+  thestage_apple_sdk:
+    git:
+      url: https://github.com/TheStageAI/AppleSDK.git
+      path: plugin/thestage_apple_sdk
+      ref: 1.7.0
+```
+
+
+### Swift
+
+```swift
+import TheStageSDK
+
+try await TheStageAI.shared.initialize(api_token: "th_…")
+
+let embedder = try await EmbeddingPipeline(
+    engines_path: "TheStageAI/embeddinggemma-2",
+    modalities: [.TEXT, .IMAGE]          // nil = every input kind
+)
+
+// Search: queries and documents use different task prompts
+let query: EmbeddingResult = try await embedder.infer(text: "how to reset the router", task: .QUERY)
+let doc: EmbeddingResult = try await embedder.infer(text: doc_text, task: .DOCUMENT)
+let score: Float = try EmbeddingPipeline.similarity(query, doc)
+
+// Other inputs — same space
+let photo: EmbeddingResult = try await embedder.infer(image: cg_image)
+let product: EmbeddingResult = try await embedder.infer(inputs: [.image(cg_image), .text("red sneakers")])
+// with modalities [.AUDIO] / [.VIDEO]:
+//   try await embedder.infer(audio: samples, sample_rate: 44_100)
+//   try await embedder.infer(video: file_url, max_frames: 8)
+
+// Smaller vectors
+let small: EmbeddingResult = try await embedder.infer(text: "how to reset the router", task: .QUERY, dimension: 256)
+```
+
+### Flutter (iOS)
+
+```dart
+await TheStageFlutterSDK.start_model(
+  model_name: 'embed',
+  engines_path: 'TheStageAI/embeddinggemma-2',
+);
+
+final rows = await TheStageFlutterSDK.infer(
+  model_name: 'embed',
+  input_json: {'text': 'how to reset the router', 'task': 'query'},
+);
+final embedding = (rows[0]['embedding'] as List).cast<double>();
+
+// Images, audio, video: pass a file path
+final photo = await TheStageFlutterSDK.infer(
+  model_name: 'embed',
+  input_json: {'image_path': '/path/to/photo.jpg', 'dimension': 256},
+);
+```
+
+### Inputs / outputs
+
+| Direction | Swift | Flutter key | Notes |
+| --- | --- | --- | --- |
+| in | `infer(text:)` | `text` | Up to 2048 tokens; longer text is cut (`truncated`) |
+| in | `infer(image:)` | `image_path` | Any size; 256–280 tokens per image |
+| in | `infer(audio:sample_rate:)` | `audio_path` | Mono, any rate (resampled); 25 tokens/s, first 30 s (`truncated` past that) |
+| in | `infer(video:max_frames:)` | `video_path`, `max_frames` | 1 frame/s, up to 14 frames (default), up to 140 tokens per frame |
+| in | `infer(inputs:)` | several keys | One embedding for mixed parts (Flutter order: text, image, audio, video) |
+| in | `task:` | `task` | `query`, `document`, `STS`, `Classification`, `Clustering`, `CodeRetrieval`, `QuestionAnswering`, `FactChecking` |
+| in | `dimension:` | `dimension` | 768 (default), 512, 256, 128 |
+| out | `embedding` | `embedding` | Unit-length vector |
+| out | `dimension`, `embedding_space_id` | same | Compare only results with equal `embedding_space_id` |
+| out | `token_count`, `truncated` | same | Input length read; whether anything was cut |
+
+### Matryoshka sizes
+
+Reference quality of the base model (Google model card, bf16):
+
+| Size | MTEB eng v2 | MTEB Code | MIEB lite | MAEB | MMEB v2 |
+| --- | --- | --- | --- | --- | --- |
+| 768 | 68.46 | 78.68 | 65.67 | 49.87 | 59.01 |
+| 512 | 68.41 | 77.24 | 65.34 | 49.76 | 58.38 |
+| 256 | 67.78 | 76.18 | 64.16 | 49.13 | 56.24 |
+| 128 | 65.68 | 71.41 | 59.97 | 47.28 | 45.65 |
+
+256 is near-lossless for text; 128 costs noticeably more on images, audio and video. For large indexes, store 768-d vectors, shortlist on the first 256 values renormalized, and re-rank the shortlist on 768.
+
+### On-device latency
+
+Release build, Neural Engine, warmed, median of 5 (guidance, not an SLA):
+
+| Input | Tokens | Apple M-series Mac | iPhone 13 |
+| --- | --- | --- | --- |
+| Text query | 16 | **8.1 ms** | **12.6 ms** |
+| Text document | 368 | **47.6 ms** | **69.7 ms** |
+| Image 768x768 | 260 | **439.5 ms** | **538.0 ms** |
+| Audio 3 s | 79 | **137.7 ms** | **227.7 ms** |
+| Audio 30 s | 754 | **236.6 ms** | **382.7 ms** |
+| Video, 4 frames | 494 | **607.1 ms** | **757.8 ms** |
+| Image + caption | 271 | **439.8 ms** | **538.4 ms** |
+
+The first load on a device prepares every engine (about two minutes on a Mac, under four on an iPhone 13, with all four inputs); later loads take seconds (7.9 s on an iPhone 13). Memory stays under 200 MB once loaded on an iPhone 13. Always measure release builds on device.
+
+## Acknowledgments
+
+---
+
+This work builds on **EmbeddingGemma 2** by Google: [google/embeddinggemma-2](https://huggingface.co/google/embeddinggemma-2).
+
+The base model [google/embeddinggemma-2](https://huggingface.co/google/embeddinggemma-2) is released under Apache-2.0. This Apple Silicon package is produced by TheStage AI; it is not an official Google release.
+
+## Links
+
+---
+
+
+* __Original model__: [google/embeddinggemma-2](https://huggingface.co/google/embeddinggemma-2)
+* __Platform__: [app.thestage.ai](https://app.thestage.ai)
+* __TheStage Apple SDK__: [github.com/TheStageAI/AppleSDK](https://github.com/TheStageAI/AppleSDK)
+* __Docs__: [docs.thestage.ai](https://docs.thestage.ai)
+* __Subscribe for updates__: [TheStageAI X](https://x.com/TheStageAI)
+* __Contact email__: contact@thestage.ai