README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: any-to-any
base_model: Qwen/Qwen2.5-1.5B-Instruct
language:
- en
metrics:
- accuracy
tags:
- safetensors
- qwen
- lora
- router
- multi-agent
- orchestration
- gradio
- multimodal
- text
- image
- video
- audio
model-index:
- name: RouterBrain
  results:
  - task:
      type: text-classification
      name: Intent Routing
    dataset:
      name: Synthetic Routing Dataset
      type: synthetic
      args: routing
    metrics:
    - type: accuracy
      value: 60.0
      name: Routing Accuracy
      verified: false
---

# 🧟 Frankenstein-3.0: The Unified AI Entity

A production-grade **multi-modal** AI orchestration system with a **trained routing brain**, featuring 4 specialist models, 5 integrated tools, and triple-layer fault tolerance.

> **Modality:** Text → Text • Image • Video • Audio (orchestrated across specialists)
> **Repo weights:** RouterBrain (text router, QLoRA-fine-tuned Qwen2.5-1.5B)

## Architecture Overview

```
┌─────────────────────────────────────────────────────────────┐
│                     USER REQUEST                            │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  LAYER 1: Pre-Flight Regex (100% accurate, hard patterns)   │
│  • "uploaded file" → RAG    • "then" → multi-step chain     │
│  • "explain how to" → chat                                  │
└─────────────────────────────────────────────────────────────┘
                            ↓ (pass-through)
┌─────────────────────────────────────────────────────────────┐
│  LAYER 2: RouterBrain (0.3s instant routing)                │
│  • Qwen2.5-1.5B fine-tuned with QLoRA, merged to fp16       │
│  • Outputs JSON routing decisions                           │
└─────────────────────────────────────────────────────────────┘
                            ↓ (if brain fails)
┌─────────────────────────────────────────────────────────────┐
│  LAYER 3: Qwen3.5-4B Resident (smart fallback)              │
│  • Understands complex/novel requests                       │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  SPECIALISTS:  🧠 Chat   💻 Code   🎨 Image   🎬 Video       │
└─────────────────────────────────────────────────────────────┘
```

## The RouterBrain (This Repo's Weights)

The root weights are the **RouterBrain**: a distilled routing model that reads user requests and outputs JSON routing decisions in ~0.3 seconds.

### Load the RouterBrain

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Questionmarkboy/frankenstein-3.0")
model = AutoModelForCausalLM.from_pretrained(
    "Questionmarkboy/frankenstein-3.0",
    torch_dtype="float16",
    device_map="auto"
)

prompt = "Route to the correct specialist (chat/code/image/video/search/calc/execute/rag). USER: Draw a robot then animate it dancing. Respond ONLY with JSON."

text = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    tokenize=False,
    add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=120, do_sample=False, repetition_penalty=1.05)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

### Expected Output

```json
{"specialist": "multi", "confidence": 0.95, "reasoning": "Chained multi-step task", "steps": [{"specialist": "image", "prompt": "Draw a robot"}, {"specialist": "video", "prompt": "animate it dancing"}]}
```

## The Full System (app.py)

### Specialists
| Specialist | Model | Use Case |
|------------|-------|----------|
| **Chat** | Qwen3.5-4B | Conversation, reasoning, explanations |
| **Code** | Qwen2.5-Coder-14B | Programming, debugging, code generation |
| **Image** | SDXL | Photorealistic image generation |
| **Video** | MiniMax-H3 NF4 | Video + audio generation |

### Tools
- 🔍 **Web Search** — DuckDuckGo real-time search
- 🧮 **Calculator** — SymPy safe mathematical evaluation
- ⚡ **Code Execution** — Sandboxed Python execution (5s timeout)
- 📚 **RAG** — Document upload + keyword search
- 🧠 **Memory** — SQLite persistent conversation history

## Installation

### Prerequisites

| Requirement | Minimum | Notes |
|-------------|---------|-------|
| Python | 3.10+ | 3.11 recommended |
| pip | latest | upgraded in Step 3 |
| Git + Git-LFS | any | LFS needed for the ~3GB weights |
| RAM | 16 GB | 32 GB recommended |
| GPU (optional) | 8 GB VRAM | 16-24 GB VRAM for image/video specialists |
| CUDA (optional) | 12.1+ | only required for GPU features |

### Step 1 — Clone the repository

```bash
git lfs install
git clone https://huggingface.co/Questionmarkboy/frankenstein-3.0
cd frankenstein-3.0
```

### Step 2 — Create a virtual environment (recommended)

```bash
python -m venv venv
source venv/bin/activate        # Linux / macOS
venv\Scripts\activate          # Windows
```

### Step 3 — Install dependencies

```bash
pip install --upgrade pip
pip install -r requirements.txt
```

### Step 4 — Launch the app

```bash
python app.py
```

Then open **http://localhost:7860** in your browser.

### CPU-only machines

Installation is identical. The app auto-detects the absence of CUDA and runs the RouterBrain, chat and code specialists on CPU. Image and video specialists respond with a friendly "GPU required" message instead of crashing.

### Troubleshooting

- **bitsandbytes warnings on CPU** — expected and safely ignored by the app.
- **Slow clone** — the RouterBrain weights are ~3 GB; make sure Git-LFS is installed (Step 1).
- **Port 7860 already in use** — change server_port at the bottom of app.py.
- **Out of memory on GPU** — the app loads specialists on demand and unloads them after use; close other GPU processes or reduce image resolution in app.py.

## Hardware Requirements

| Mode | Minimum | Recommended |
|------|---------|-------------|
| RouterBrain only | 2GB VRAM | CPU works fine |
| Chat + Code (CPU) | 16GB RAM | 32GB RAM |
| Full GPU stack | 16GB VRAM | 24GB+ VRAM |

The app auto-detects CUDA. On CPU-only machines, chat/code work and image/video return a friendly "GPU required" message.

## Training Details

### RouterBrain Training
- **Base Model**: Qwen2.5-1.5B-Instruct
- **Fine-tuning**: QLoRA (r=32, α=64, dropout=0.05)
- **Dataset**: ~10k synthetic routing examples (combinatorial diversity, noise injection, 30+ edge-case patterns)
- **Training**: 2-3 epochs on T4 GPU
- **Merge**: Clean fp16 merge (no quantization rounding)

### Performance Metrics
- **Pre-flight layer**: 100% accuracy on hard patterns
- **RouterBrain**: 60% standalone accuracy on diverse test set
- **Effective routing**: 90%+ via triple-layer fault tolerance

## Multi-Step Task Chaining

```
User: "Draw a robot then animate it dancing"
  ↓
Pre-flight: detects "then" → routes to multi specialist
  ↓
Execution: image specialist → video specialist (sequential)
  ↓
Output: generated image + animated video with audio
```

## Architecture Decisions

### Why Triple-Layer Routing?
1. **Speed**: Pre-flight regex is instant (0.001s)
2. **Efficiency**: RouterBrain is fast (0.3s) vs prompt-based (4-6s)
3. **Robustness**: 4B fallback catches edge cases
4. **Production-ready**: Graceful degradation under all conditions

### Why Distill A Routing Brain?
- **20x faster** than prompt-based routing
- **50% less VRAM** than using the 4B resident for routing
- **Perfect JSON** output (trained to only output JSON)
- **Cheaper** to deploy at scale

## Companion Models

- **[Frankenstein-2.0](https://huggingface.co/Questionmarkboy/frankenstein-2.0)** — Merged Mistral-7B checkpoint

## Technical Stack

- **Framework**: HuggingFace Transformers + Diffusers
- **Quantization**: BitsAndBytes (NF4, 4-bit)
- **UI**: Gradio 4.44
- **Memory**: SQLite
- **Training**: PEFT + LoRA

## Known Limitations

- Image/video generation requires 16GB+ VRAM
- RAG only supports .txt files (no PDF/DOCX yet)
- Web search is basic (DuckDuckGo HTML scraping)
- Code execution has 5s timeout for safety

## Citation

```bibtex
@misc{frankenstein3,
  author = {Questionmarkboy},
  title = {Frankenstein-3.0: Multi-Modal AI Orchestration System},
  year = {2024},
  publisher = {HuggingFace},
  journal = {HuggingFace Repository},
  howpublished = {\url{https://huggingface.co/Questionmarkboy/frankenstein-3.0}}
}
```

## License

Apache 2.0

## Acknowledgments

- Base models: Qwen, Stability AI, MiniMax
- Frameworks: HuggingFace, Gradio, BitsAndBytes
- Training infrastructure: Kaggle T4 GPUs

---

**Built with ❤️ for the AI community**