zoebchhatriwala/Zesyde-2B-Multimodal-Router

Zesyde: Multimodal Decision & Routing Foundation Model

Zesyde-2B Multimodal Decision Router (zoebchhatriwala/Zesyde-2B-Multimodal-Router) is a fast, lightweight foundation model built for one specific job: making calibrated decisions and routing tasks in 15–35 milliseconds on pure CPU.

Instead of waiting seconds for a giant generative LLM to generate text, Zesyde evaluates all your options at once in a single forward pass. It supports 140+ languages, images, audio clips, and 2D/3D spatial coordinates natively.

---

Benchmark Highlights (150,000 Records)

• 🎯 87.65% Real Validation Accuracy: Evaluated on 22,500 unseen held-out samples across 7 real-world domains (Belebele QA, Cambridge VSR, ESC-50, SMS Fraud, PAWS-X, CodeAlpaca, XNLI).
• ⚡ Ultra Fast (15–35ms on CPU): Powered by model_int8.onnx with direct streaming dynamic quantization. Zero GPU required at inference time.
• 📦 Dual-Precision Artifacts: Standard model_fp32.onnx (FP32 precision) and model_int8.onnx (INT8 linear quantization, ~1/4 size) committed alongside model.safetensors.
• 🛡️ Autonomous Escalation: Built-in safety check. If a question is too ambiguous, high-risk, or low-confidence, Zesyde automatically flags it to escalate to a human or a heavier frontier model (like GPT-4 or Claude).
• ☁️ Serverless A100 Training: Full production training and export pipeline run on Modal.com with a single command.

---

Installation & Quickstart

pip install zesyde

2. Basic Text Decision (15–35ms on CPU)

from zesyde.router import ZesydeRouter

# 1. Zero-config load (defaults to zoebchhatriwala/Zesyde-2B-Multimodal-Router on Hugging Face,
# or loads from local ./models if weights exist locally):
router = ZesydeRouter.from_pretrained()

# Or explicitly select your precision:
# router = ZesydeRouter.from_pretrained(precision="int8")   # Fast CPU mode (default: 15-35ms)
# router = ZesydeRouter.from_pretrained(precision="fp32")   # High-fidelity unquantized mode
# router = ZesydeRouter.from_pretrained(precision="torch")  # Pure PyTorch safetensors mode

# Or point to any custom local weights directory:
# router = ZesydeRouter.from_pretrained("./my_local_model")

# 2. Dynamic candidate options (evaluated simultaneously in 1 forward pass)
result = router.decide(
    context="User: 'Can you show me the quarterly sales report for Q3?'",
    question="Which agent tool should handle this request?",
    options=["search_web", "query_database", "create_calendar_invite", "send_email"]
)

print(f"Choice: {result.selected_option}")       # query_database
print(f"Confidence: {result.confidence:.1%}")     # 94.2%
print(f"Latency: {result.latency_ms:.1f}ms")       # ~18ms
print(f"Escalate: {result.should_escalate}")     # False

---

3. Multimodal Decision (Vision & Audio)

Zesyde accepts image and audio signals directly alongside text:

# Vision decision: Road hazard assessment
result = router.decide(
    context="Front-facing vehicle camera feed.",
    image="dashcam_frame_042.jpg",
    question="What action should the autonomous navigation system take?",
    options=["Maintain speed", "Slow down and yield", "Emergency brake", "Overtake"]
)
print(result.selected_option)  # Slow down and yield

# Audio decision: Acoustic security event detection
result = router.decide(
    context="Smart home microphone stream.",
    audio="living_room_mic.wav",
    question="Identify the detected acoustic sound event:",
    options=["Speech / Conversation", "Glass break / Alarm", "Dog barking", "Background HVAC"]
)
print(result.selected_option)  # Glass break / Alarm

---

4. Spatial Awareness (2D & 3D Grounding)

Pass bounding boxes or metric 3D coordinates [x, y, z] to let Zesyde route spatial and robotic tasks:

# Spatial tabletop manipulation
result = router.decide(
    context="Robotic arm workspace. Gripper origin at [0.0, 0.0, 0.0].",
    question="Which target object should the robot pick up next?",
    options=[
        "Red Mug at [0.15, 0.20, 0.40]",
        "Laptop at [0.85, 1.10, 0.90]",
        "Desk Lamp at [1.40, 0.30, 0.25]"
    ],
    question_type="spatial"
)

print(result.selected_option)  # Red Mug at [0.15, 0.20, 0.40]

---

5. Multilingual Across 1,800+ Languages

Zesyde natively understands global languages without translation layers:

# Spanish
result = router.decide(
    context="El cliente solicitó cancelar su suscripción anual inmediatamente.",
    question="¿Cuál es la intención del usuario?",
    options=["Soporte técnico", "Cancelación de cuenta", "Consulta de facturación"]
)
print(result.selected_option)  # Cancelación de cuenta

# Hindi
result = router.decide(
    context="उपयोगकर्ता ने अपना पासवर्ड रीसेट करने का अनुरोध किया है।",
    question="यह किस प्रकार का अनुरोध है?",
    options=["पासवर्ड रीसेट", "खाता हटाना", "भुगतान समस्या"]
)
print(result.selected_option)  # पासवर्ड रीसेट

---

Training on Modal.com

All training, parameter-efficient fine-tuning (LoRA), cross-candidate attention training, temperature calibration, and INT8 ONNX dynamic quantization are contained in a single file: scripts/train_modal.py.

To launch serverless GPU training on Modal:

modal run scripts/train_modal.py

Optional parameters (e.g. smoke testing or custom dataset scale):

modal run scripts/train_modal.py --train-limit 150000

The resulting optimized ONNX models and calibration parameters are automatically saved to your Modal persistent volume (zesyde-models):

• model_fp32.onnx & model_fp32.onnx.data: FP32 full-precision ONNX model
• model_int8.onnx & model_int8.onnx.data: INT8 dynamic quantized model (~1/4 size)
• model.safetensors & pytorch_model.bin: Standard production PyTorch weights
• calibration.json: Task-specific temperature scaling parameters
• config.json & tokenizer vocabulary

---

Architecture at a Glance

• Backbone: google/gemma-4-E2B native omni-modal foundation model (140+ languages, vision, audio, spatial).
• Accuracy Head: SemanticTransformerHead using cross-candidate self-attention so candidate options compete directly against each other before scoring.
• Perception Channels: Native text tokens, visual patch tokens ([IMAGE]), acoustic tokens ([AUDIO]), and normalized coordinate tokens ([SPATIAL]).
• Uncertainty & Escalation Head: Binary logit head assessing epistemic ambiguity. If confidence is low or risk is high, queries gracefully fall back to frontier models or humans.
• Serving: Dynamic INT8 ONNX graphs (model_int8.onnx) runnable anywhere via pure Python with zero heavy GPU dependencies.

---

License

MIT License. Free for commercial and research use.