--- license: apache-2.0 tags: - multimodal - decision-router - agent-routing - onnx - gemma - multilingual pipeline_tag: feature-extraction language: - en - fr - de - es - zh - ar - hi - ja - ru - pt --- # Zesyde: Multimodal Decision & Routing Foundation Model **Zesyde-2B Multimodal Decision Router** (`zoebchhatriwala/Zesyde-2B-Multimodal-Router`) is a fast, lightweight foundation model built for one specific job: **making calibrated decisions and routing tasks in 15–35 milliseconds on pure CPU**. Instead of waiting seconds for a giant generative LLM to generate text, Zesyde evaluates all your options at once in a single forward pass. It supports **140+ languages, images, audio clips, and 2D/3D spatial coordinates** natively. --- ## Benchmark Highlights (150,000 Records) * 🎯 **87.65% Real Validation Accuracy**: Evaluated on 22,500 unseen held-out samples across 7 real-world domains (Belebele QA, Cambridge VSR, ESC-50, SMS Fraud, PAWS-X, CodeAlpaca, XNLI). * ⚡ **Ultra Fast (15–35ms on CPU)**: Powered by `model_int8.onnx` with direct streaming dynamic quantization. Zero GPU required at inference time. * 📦 **Dual-Precision Artifacts**: Standard `model_fp32.onnx` (FP32 precision) and `model_int8.onnx` (INT8 linear quantization, ~1/4 size) committed alongside `model.safetensors`. * 🛡️ **Autonomous Escalation**: Built-in safety check. If a question is too ambiguous, high-risk, or low-confidence, Zesyde automatically flags it to escalate to a human or a heavier frontier model (like GPT-4 or Claude). * ☁️ **Serverless A100 Training**: Full production training and export pipeline run on Modal.com with a single command. --- ## Installation & Quickstart ```bash pip install zesyde ``` ### 2. Basic Text Decision (15–35ms on CPU) ```python from zesyde.router import ZesydeRouter # 1. Zero-config load (defaults to zoebchhatriwala/Zesyde-2B-Multimodal-Router on Hugging Face, # or loads from local ./models if weights exist locally): router = ZesydeRouter.from_pretrained() # Or explicitly select your precision: # router = ZesydeRouter.from_pretrained(precision="int8") # Fast CPU mode (default: 15-35ms) # router = ZesydeRouter.from_pretrained(precision="fp32") # High-fidelity unquantized mode # router = ZesydeRouter.from_pretrained(precision="torch") # Pure PyTorch safetensors mode # Or point to any custom local weights directory: # router = ZesydeRouter.from_pretrained("./my_local_model") # 2. Dynamic candidate options (evaluated simultaneously in 1 forward pass) result = router.decide( context="User: 'Can you show me the quarterly sales report for Q3?'", question="Which agent tool should handle this request?", options=["search_web", "query_database", "create_calendar_invite", "send_email"] ) print(f"Choice: {result.selected_option}") # query_database print(f"Confidence: {result.confidence:.1%}") # 94.2% print(f"Latency: {result.latency_ms:.1f}ms") # ~18ms print(f"Escalate: {result.should_escalate}") # False ``` --- ### 3. Multimodal Decision (Vision & Audio) Zesyde accepts image and audio signals directly alongside text: ```python # Vision decision: Road hazard assessment result = router.decide( context="Front-facing vehicle camera feed.", image="dashcam_frame_042.jpg", question="What action should the autonomous navigation system take?", options=["Maintain speed", "Slow down and yield", "Emergency brake", "Overtake"] ) print(result.selected_option) # Slow down and yield # Audio decision: Acoustic security event detection result = router.decide( context="Smart home microphone stream.", audio="living_room_mic.wav", question="Identify the detected acoustic sound event:", options=["Speech / Conversation", "Glass break / Alarm", "Dog barking", "Background HVAC"] ) print(result.selected_option) # Glass break / Alarm ``` --- ### 4. Spatial Awareness (2D & 3D Grounding) Pass bounding boxes or metric 3D coordinates `[x, y, z]` to let Zesyde route spatial and robotic tasks: ```python # Spatial tabletop manipulation result = router.decide( context="Robotic arm workspace. Gripper origin at [0.0, 0.0, 0.0].", question="Which target object should the robot pick up next?", options=[ "Red Mug at [0.15, 0.20, 0.40]", "Laptop at [0.85, 1.10, 0.90]", "Desk Lamp at [1.40, 0.30, 0.25]" ], question_type="spatial" ) print(result.selected_option) # Red Mug at [0.15, 0.20, 0.40] ``` --- ### 5. Multilingual Across 1,800+ Languages Zesyde natively understands global languages without translation layers: ```python # Spanish result = router.decide( context="El cliente solicitó cancelar su suscripción anual inmediatamente.", question="¿Cuál es la intención del usuario?", options=["Soporte técnico", "Cancelación de cuenta", "Consulta de facturación"] ) print(result.selected_option) # Cancelación de cuenta # Hindi result = router.decide( context="उपयोगकर्ता ने अपना पासवर्ड रीसेट करने का अनुरोध किया है।", question="यह किस प्रकार का अनुरोध है?", options=["पासवर्ड रीसेट", "खाता हटाना", "भुगतान समस्या"] ) print(result.selected_option) # पासवर्ड रीसेट ``` --- ## Training on Modal.com All training, parameter-efficient fine-tuning (LoRA), cross-candidate attention training, temperature calibration, and INT8 ONNX dynamic quantization are contained in **a single file**: `scripts/train_modal.py`. To launch serverless GPU training on Modal: ```bash modal run scripts/train_modal.py ``` Optional parameters (e.g. smoke testing or custom dataset scale): ```bash modal run scripts/train_modal.py --train-limit 150000 ``` The resulting optimized ONNX models and calibration parameters are automatically saved to your Modal persistent volume (`zesyde-models`): - `model_fp32.onnx` & `model_fp32.onnx.data`: FP32 full-precision ONNX model - `model_int8.onnx` & `model_int8.onnx.data`: INT8 dynamic quantized model (~1/4 size) - `model.safetensors` & `pytorch_model.bin`: Standard production PyTorch weights - `calibration.json`: Task-specific temperature scaling parameters - `config.json` & tokenizer vocabulary --- ## Architecture at a Glance * **Backbone**: `google/gemma-4-E2B` native omni-modal foundation model (140+ languages, vision, audio, spatial). * **Accuracy Head**: `SemanticTransformerHead` using cross-candidate self-attention so candidate options compete directly against each other before scoring. * **Perception Channels**: Native text tokens, visual patch tokens (`[IMAGE]`), acoustic tokens (`[AUDIO]`), and normalized coordinate tokens (`[SPATIAL]`). * **Uncertainty & Escalation Head**: Binary logit head assessing epistemic ambiguity. If confidence is low or risk is high, queries gracefully fall back to frontier models or humans. * **Serving**: Dynamic INT8 ONNX graphs (`model_int8.onnx`) runnable anywhere via pure Python with zero heavy GPU dependencies. --- ## License MIT License. Free for commercial and research use.