README.md

---
license: apache-2.0
tags:
  - multimodal
  - decision-router
  - agent-routing
  - onnx
  - gemma
  - multilingual
pipeline_tag: feature-extraction
language:
  - en
  - fr
  - de
  - es
  - zh
  - ar
  - hi
  - ja
  - ru
  - pt
---

# Zesyde: Multimodal Decision & Routing Foundation Model

**Zesyde-2B Multimodal Decision Router** (`zoebchhatriwala/Zesyde-2B-Multimodal-Router`) is a fast, lightweight foundation model built for one specific job: **making calibrated decisions and routing tasks in 15–35 milliseconds on pure CPU**.

Instead of waiting seconds for a giant generative LLM to generate text, Zesyde evaluates all your options at once in a single forward pass. It supports **140+ languages, images, audio clips, and 2D/3D spatial coordinates** natively.

---

## Benchmark Highlights (150,000 Records)

* 🎯 **87.65% Real Validation Accuracy**: Evaluated on 22,500 unseen held-out samples across 7 real-world domains (Belebele QA, Cambridge VSR, ESC-50, SMS Fraud, PAWS-X, CodeAlpaca, XNLI).
* ⚡ **Ultra Fast (15–35ms on CPU)**: Powered by `model_int8.onnx` with direct streaming dynamic quantization. Zero GPU required at inference time.
* 📦 **Dual-Precision Artifacts**: Standard `model_fp32.onnx` (FP32 precision) and `model_int8.onnx` (INT8 linear quantization, ~1/4 size) committed alongside `model.safetensors`.
* 🛡️ **Autonomous Escalation**: Built-in safety check. If a question is too ambiguous, high-risk, or low-confidence, Zesyde automatically flags it to escalate to a human or a heavier frontier model (like GPT-4 or Claude).
* ☁️ **Serverless A100 Training**: Full production training and export pipeline run on Modal.com with a single command.

---

## Installation & Quickstart

```bash
pip install zesyde
```

### 2. Basic Text Decision (15–35ms on CPU)

```python
from zesyde.router import ZesydeRouter

# 1. Zero-config load (defaults to zoebchhatriwala/Zesyde-2B-Multimodal-Router on Hugging Face,
# or loads from local ./models if weights exist locally):
router = ZesydeRouter.from_pretrained()

# Or explicitly select your precision:
# router = ZesydeRouter.from_pretrained(precision="int8")   # Fast CPU mode (default: 15-35ms)
# router = ZesydeRouter.from_pretrained(precision="fp32")   # High-fidelity unquantized mode
# router = ZesydeRouter.from_pretrained(precision="torch")  # Pure PyTorch safetensors mode

# Or point to any custom local weights directory:
# router = ZesydeRouter.from_pretrained("./my_local_model")

# 2. Dynamic candidate options (evaluated simultaneously in 1 forward pass)
result = router.decide(
    context="User: 'Can you show me the quarterly sales report for Q3?'",
    question="Which agent tool should handle this request?",
    options=["search_web", "query_database", "create_calendar_invite", "send_email"]
)

print(f"Choice: {result.selected_option}")       # query_database
print(f"Confidence: {result.confidence:.1%}")     # 94.2%
print(f"Latency: {result.latency_ms:.1f}ms")       # ~18ms
print(f"Escalate: {result.should_escalate}")     # False
```

---

### 3. Multimodal Decision (Vision & Audio)

Zesyde accepts image and audio signals directly alongside text:

```python
# Vision decision: Road hazard assessment
result = router.decide(
    context="Front-facing vehicle camera feed.",
    image="dashcam_frame_042.jpg",
    question="What action should the autonomous navigation system take?",
    options=["Maintain speed", "Slow down and yield", "Emergency brake", "Overtake"]
)
print(result.selected_option)  # Slow down and yield

# Audio decision: Acoustic security event detection
result = router.decide(
    context="Smart home microphone stream.",
    audio="living_room_mic.wav",
    question="Identify the detected acoustic sound event:",
    options=["Speech / Conversation", "Glass break / Alarm", "Dog barking", "Background HVAC"]
)
print(result.selected_option)  # Glass break / Alarm
```

---

### 4. Spatial Awareness (2D & 3D Grounding)

Pass bounding boxes or metric 3D coordinates `[x, y, z]` to let Zesyde route spatial and robotic tasks:

```python
# Spatial tabletop manipulation
result = router.decide(
    context="Robotic arm workspace. Gripper origin at [0.0, 0.0, 0.0].",
    question="Which target object should the robot pick up next?",
    options=[
        "Red Mug at [0.15, 0.20, 0.40]",
        "Laptop at [0.85, 1.10, 0.90]",
        "Desk Lamp at [1.40, 0.30, 0.25]"
    ],
    question_type="spatial"
)

print(result.selected_option)  # Red Mug at [0.15, 0.20, 0.40]
```

---

### 5. Multilingual Across 1,800+ Languages

Zesyde natively understands global languages without translation layers:

```python
# Spanish
result = router.decide(
    context="El cliente solicitó cancelar su suscripción anual inmediatamente.",
    question="¿Cuál es la intención del usuario?",
    options=["Soporte técnico", "Cancelación de cuenta", "Consulta de facturación"]
)
print(result.selected_option)  # Cancelación de cuenta

# Hindi
result = router.decide(
    context="उपयोगकर्ता ने अपना पासवर्ड रीसेट करने का अनुरोध किया है।",
    question="यह किस प्रकार का अनुरोध है?",
    options=["पासवर्ड रीसेट", "खाता हटाना", "भुगतान समस्या"]
)
print(result.selected_option)  # पासवर्ड रीसेट
```

---

## Training on Modal.com

All training, parameter-efficient fine-tuning (LoRA), cross-candidate attention training, temperature calibration, and INT8 ONNX dynamic quantization are contained in **a single file**: `scripts/train_modal.py`.

To launch serverless GPU training on Modal:

```bash
modal run scripts/train_modal.py
```

Optional parameters (e.g. smoke testing or custom dataset scale):

```bash
modal run scripts/train_modal.py --train-limit 150000
```

The resulting optimized ONNX models and calibration parameters are automatically saved to your Modal persistent volume (`zesyde-models`):
- `model_fp32.onnx` & `model_fp32.onnx.data`: FP32 full-precision ONNX model
- `model_int8.onnx` & `model_int8.onnx.data`: INT8 dynamic quantized model (~1/4 size)
- `model.safetensors` & `pytorch_model.bin`: Standard production PyTorch weights
- `calibration.json`: Task-specific temperature scaling parameters
- `config.json` & tokenizer vocabulary

---

## Architecture at a Glance

* **Backbone**: `google/gemma-4-E2B` native omni-modal foundation model (140+ languages, vision, audio, spatial).
* **Accuracy Head**: `SemanticTransformerHead` using cross-candidate self-attention so candidate options compete directly against each other before scoring.
* **Perception Channels**: Native text tokens, visual patch tokens (`[IMAGE]`), acoustic tokens (`[AUDIO]`), and normalized coordinate tokens (`[SPATIAL]`).
* **Uncertainty & Escalation Head**: Binary logit head assessing epistemic ambiguity. If confidence is low or risk is high, queries gracefully fall back to frontier models or humans.
* **Serving**: Dynamic INT8 ONNX graphs (`model_int8.onnx`) runnable anywhere via pure Python with zero heavy GPU dependencies.

---

## License

MIT License. Free for commercial and research use.