| --- |
| license: apache-2.0 |
| tags: |
| - multimodal |
| - decision-router |
| - agent-routing |
| - onnx |
| - gemma |
| - multilingual |
| pipeline_tag: feature-extraction |
| language: |
| - en |
| - fr |
| - de |
| - es |
| - zh |
| - ar |
| - hi |
| - ja |
| - ru |
| - pt |
| --- |
| |
| # Zesyde: Multimodal Decision & Routing Foundation Model |
|
|
| **Zesyde-2B Multimodal Decision Router** (`zoebchhatriwala/Zesyde-2B-Multimodal-Router`) is a fast, lightweight foundation model built for one specific job: **making calibrated decisions and routing tasks in 15–35 milliseconds on pure CPU**. |
|
|
| Instead of waiting seconds for a giant generative LLM to generate text, Zesyde evaluates all your options at once in a single forward pass. It supports **140+ languages, images, audio clips, and 2D/3D spatial coordinates** natively. |
|
|
| --- |
|
|
| ## Benchmark Highlights (150,000 Records) |
|
|
| * 🎯 **87.65% Real Validation Accuracy**: Evaluated on 22,500 unseen held-out samples across 7 real-world domains (Belebele QA, Cambridge VSR, ESC-50, SMS Fraud, PAWS-X, CodeAlpaca, XNLI). |
| * ⚡ **Ultra Fast (15–35ms on CPU)**: Powered by `model_int8.onnx` with direct streaming dynamic quantization. Zero GPU required at inference time. |
| * 📦 **Dual-Precision Artifacts**: Standard `model_fp32.onnx` (FP32 precision) and `model_int8.onnx` (INT8 linear quantization, ~1/4 size) committed alongside `model.safetensors`. |
| * 🛡️ **Autonomous Escalation**: Built-in safety check. If a question is too ambiguous, high-risk, or low-confidence, Zesyde automatically flags it to escalate to a human or a heavier frontier model (like GPT-4 or Claude). |
| * ☁️ **Serverless A100 Training**: Full production training and export pipeline run on Modal.com with a single command. |
|
|
| --- |
|
|
| ## Installation & Quickstart |
|
|
| ```bash |
| pip install zesyde |
| ``` |
|
|
| ### 2. Basic Text Decision (15–35ms on CPU) |
|
|
| ```python |
| from zesyde.router import ZesydeRouter |
| |
| # 1. Zero-config load (defaults to zoebchhatriwala/Zesyde-2B-Multimodal-Router on Hugging Face, |
| # or loads from local ./models if weights exist locally): |
| router = ZesydeRouter.from_pretrained() |
| |
| # Or explicitly select your precision: |
| # router = ZesydeRouter.from_pretrained(precision="int8") # Fast CPU mode (default: 15-35ms) |
| # router = ZesydeRouter.from_pretrained(precision="fp32") # High-fidelity unquantized mode |
| # router = ZesydeRouter.from_pretrained(precision="torch") # Pure PyTorch safetensors mode |
| |
| # Or point to any custom local weights directory: |
| # router = ZesydeRouter.from_pretrained("./my_local_model") |
| |
| # 2. Dynamic candidate options (evaluated simultaneously in 1 forward pass) |
| result = router.decide( |
| context="User: 'Can you show me the quarterly sales report for Q3?'", |
| question="Which agent tool should handle this request?", |
| options=["search_web", "query_database", "create_calendar_invite", "send_email"] |
| ) |
| |
| print(f"Choice: {result.selected_option}") # query_database |
| print(f"Confidence: {result.confidence:.1%}") # 94.2% |
| print(f"Latency: {result.latency_ms:.1f}ms") # ~18ms |
| print(f"Escalate: {result.should_escalate}") # False |
| ``` |
|
|
| --- |
|
|
| ### 3. Multimodal Decision (Vision & Audio) |
|
|
| Zesyde accepts image and audio signals directly alongside text: |
|
|
| ```python |
| # Vision decision: Road hazard assessment |
| result = router.decide( |
| context="Front-facing vehicle camera feed.", |
| image="dashcam_frame_042.jpg", |
| question="What action should the autonomous navigation system take?", |
| options=["Maintain speed", "Slow down and yield", "Emergency brake", "Overtake"] |
| ) |
| print(result.selected_option) # Slow down and yield |
| |
| # Audio decision: Acoustic security event detection |
| result = router.decide( |
| context="Smart home microphone stream.", |
| audio="living_room_mic.wav", |
| question="Identify the detected acoustic sound event:", |
| options=["Speech / Conversation", "Glass break / Alarm", "Dog barking", "Background HVAC"] |
| ) |
| print(result.selected_option) # Glass break / Alarm |
| ``` |
|
|
| --- |
|
|
| ### 4. Spatial Awareness (2D & 3D Grounding) |
|
|
| Pass bounding boxes or metric 3D coordinates `[x, y, z]` to let Zesyde route spatial and robotic tasks: |
|
|
| ```python |
| # Spatial tabletop manipulation |
| result = router.decide( |
| context="Robotic arm workspace. Gripper origin at [0.0, 0.0, 0.0].", |
| question="Which target object should the robot pick up next?", |
| options=[ |
| "Red Mug at [0.15, 0.20, 0.40]", |
| "Laptop at [0.85, 1.10, 0.90]", |
| "Desk Lamp at [1.40, 0.30, 0.25]" |
| ], |
| question_type="spatial" |
| ) |
| |
| print(result.selected_option) # Red Mug at [0.15, 0.20, 0.40] |
| ``` |
|
|
| --- |
|
|
| ### 5. Multilingual Across 1,800+ Languages |
|
|
| Zesyde natively understands global languages without translation layers: |
|
|
| ```python |
| # Spanish |
| result = router.decide( |
| context="El cliente solicitó cancelar su suscripción anual inmediatamente.", |
| question="¿Cuál es la intención del usuario?", |
| options=["Soporte técnico", "Cancelación de cuenta", "Consulta de facturación"] |
| ) |
| print(result.selected_option) # Cancelación de cuenta |
| |
| # Hindi |
| result = router.decide( |
| context="उपयोगकर्ता ने अपना पासवर्ड रीसेट करने का अनुरोध किया है।", |
| question="यह किस प्रकार का अनुरोध है?", |
| options=["पासवर्ड रीसेट", "खाता हटाना", "भुगतान समस्या"] |
| ) |
| print(result.selected_option) # पासवर्ड रीसेट |
| ``` |
|
|
| --- |
|
|
| ## Training on Modal.com |
|
|
| All training, parameter-efficient fine-tuning (LoRA), cross-candidate attention training, temperature calibration, and INT8 ONNX dynamic quantization are contained in **a single file**: `scripts/train_modal.py`. |
|
|
| To launch serverless GPU training on Modal: |
|
|
| ```bash |
| modal run scripts/train_modal.py |
| ``` |
|
|
| Optional parameters (e.g. smoke testing or custom dataset scale): |
|
|
| ```bash |
| modal run scripts/train_modal.py --train-limit 150000 |
| ``` |
|
|
| The resulting optimized ONNX models and calibration parameters are automatically saved to your Modal persistent volume (`zesyde-models`): |
| - `model_fp32.onnx` & `model_fp32.onnx.data`: FP32 full-precision ONNX model |
| - `model_int8.onnx` & `model_int8.onnx.data`: INT8 dynamic quantized model (~1/4 size) |
| - `model.safetensors` & `pytorch_model.bin`: Standard production PyTorch weights |
| - `calibration.json`: Task-specific temperature scaling parameters |
| - `config.json` & tokenizer vocabulary |
|
|
| --- |
|
|
| ## Architecture at a Glance |
|
|
| * **Backbone**: `google/gemma-4-E2B` native omni-modal foundation model (140+ languages, vision, audio, spatial). |
| * **Accuracy Head**: `SemanticTransformerHead` using cross-candidate self-attention so candidate options compete directly against each other before scoring. |
| * **Perception Channels**: Native text tokens, visual patch tokens (`[IMAGE]`), acoustic tokens (`[AUDIO]`), and normalized coordinate tokens (`[SPATIAL]`). |
| * **Uncertainty & Escalation Head**: Binary logit head assessing epistemic ambiguity. If confidence is low or risk is high, queries gracefully fall back to frontier models or humans. |
| * **Serving**: Dynamic INT8 ONNX graphs (`model_int8.onnx`) runnable anywhere via pure Python with zero heavy GPU dependencies. |
|
|
| --- |
|
|
| ## License |
|
|
| MIT License. Free for commercial and research use. |
|
|
