| --- |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: any-to-any |
| base_model: Qwen/Qwen2.5-1.5B-Instruct |
| language: |
| - en |
| metrics: |
| - accuracy |
| tags: |
| - safetensors |
| - qwen |
| - lora |
| - router |
| - multi-agent |
| - orchestration |
| - gradio |
| - multimodal |
| - text |
| - image |
| - video |
| - audio |
| model-index: |
| - name: RouterBrain |
| results: |
| - task: |
| type: text-classification |
| name: Intent Routing |
| dataset: |
| name: Synthetic Routing Dataset |
| type: synthetic |
| args: routing |
| metrics: |
| - type: accuracy |
| value: 60.0 |
| name: Routing Accuracy |
| verified: false |
| --- |
| |
| # 🧟 Frankenstein-3.0: The Unified AI Entity |
|
|
| A production-grade **multi-modal** AI orchestration system with a **trained routing brain**, featuring 4 specialist models, 5 integrated tools, and triple-layer fault tolerance. |
|
|
| > **Modality:** Text → Text • Image • Video • Audio (orchestrated across specialists) |
| > **Repo weights:** RouterBrain (text router, QLoRA-fine-tuned Qwen2.5-1.5B) |
|
|
| ## Architecture Overview |
|
|
| ``` |
| ┌─────────────────────────────────────────────────────────────┐ |
| │ USER REQUEST │ |
| └─────────────────────────────────────────────────────────────┘ |
| ↓ |
| ┌─────────────────────────────────────────────────────────────┐ |
| │ LAYER 1: Pre-Flight Regex (100% accurate, hard patterns) │ |
| │ • "uploaded file" → RAG • "then" → multi-step chain │ |
| │ • "explain how to" → chat │ |
| └─────────────────────────────────────────────────────────────┘ |
| ↓ (pass-through) |
| ┌─────────────────────────────────────────────────────────────┐ |
| │ LAYER 2: RouterBrain (0.3s instant routing) │ |
| │ • Qwen2.5-1.5B fine-tuned with QLoRA, merged to fp16 │ |
| │ • Outputs JSON routing decisions │ |
| └─────────────────────────────────────────────────────────────┘ |
| ↓ (if brain fails) |
| ┌─────────────────────────────────────────────────────────────┐ |
| │ LAYER 3: Qwen3.5-4B Resident (smart fallback) │ |
| │ • Understands complex/novel requests │ |
| └─────────────────────────────────────────────────────────────┘ |
| ↓ |
| ┌─────────────────────────────────────────────────────────────┐ |
| │ SPECIALISTS: 🧠 Chat 💻 Code 🎨 Image 🎬 Video │ |
| └─────────────────────────────────────────────────────────────┘ |
| ``` |
|
|
| ## The RouterBrain (This Repo's Weights) |
|
|
| The root weights are the **RouterBrain**: a distilled routing model that reads user requests and outputs JSON routing decisions in ~0.3 seconds. |
|
|
| ### Load the RouterBrain |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("Questionmarkboy/frankenstein-3.0") |
| model = AutoModelForCausalLM.from_pretrained( |
| "Questionmarkboy/frankenstein-3.0", |
| torch_dtype="float16", |
| device_map="auto" |
| ) |
| |
| prompt = "Route to the correct specialist (chat/code/image/video/search/calc/execute/rag). USER: Draw a robot then animate it dancing. Respond ONLY with JSON." |
| |
| text = tokenizer.apply_chat_template( |
| [{"role": "user", "content": prompt}], |
| tokenize=False, |
| add_generation_prompt=True |
| ) |
| inputs = tokenizer(text, return_tensors="pt").to(model.device) |
| output = model.generate(**inputs, max_new_tokens=120, do_sample=False, repetition_penalty=1.05) |
| print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ### Expected Output |
|
|
| ```json |
| {"specialist": "multi", "confidence": 0.95, "reasoning": "Chained multi-step task", "steps": [{"specialist": "image", "prompt": "Draw a robot"}, {"specialist": "video", "prompt": "animate it dancing"}]} |
| ``` |
|
|
| ## The Full System (app.py) |
|
|
| ### Specialists |
| | Specialist | Model | Use Case | |
| |------------|-------|----------| |
| | **Chat** | Qwen3.5-4B | Conversation, reasoning, explanations | |
| | **Code** | Qwen2.5-Coder-14B | Programming, debugging, code generation | |
| | **Image** | SDXL | Photorealistic image generation | |
| | **Video** | MiniMax-H3 NF4 | Video + audio generation | |
|
|
| ### Tools |
| - 🔍 **Web Search** — DuckDuckGo real-time search |
| - 🧮 **Calculator** — SymPy safe mathematical evaluation |
| - ⚡ **Code Execution** — Sandboxed Python execution (5s timeout) |
| - 📚 **RAG** — Document upload + keyword search |
| - 🧠 **Memory** — SQLite persistent conversation history |
|
|
| ## Installation |
|
|
| ### Prerequisites |
|
|
| | Requirement | Minimum | Notes | |
| |-------------|---------|-------| |
| | Python | 3.10+ | 3.11 recommended | |
| | pip | latest | upgraded in Step 3 | |
| | Git + Git-LFS | any | LFS needed for the ~3GB weights | |
| | RAM | 16 GB | 32 GB recommended | |
| | GPU (optional) | 8 GB VRAM | 16-24 GB VRAM for image/video specialists | |
| | CUDA (optional) | 12.1+ | only required for GPU features | |
|
|
| ### Step 1 — Clone the repository |
|
|
| ```bash |
| git lfs install |
| git clone https://huggingface.co/Questionmarkboy/frankenstein-3.0 |
| cd frankenstein-3.0 |
| ``` |
|
|
| ### Step 2 — Create a virtual environment (recommended) |
|
|
| ```bash |
| python -m venv venv |
| source venv/bin/activate # Linux / macOS |
| venv\Scripts\activate # Windows |
| ``` |
|
|
| ### Step 3 — Install dependencies |
|
|
| ```bash |
| pip install --upgrade pip |
| pip install -r requirements.txt |
| ``` |
|
|
| ### Step 4 — Launch the app |
|
|
| ```bash |
| python app.py |
| ``` |
|
|
| Then open **http://localhost:7860** in your browser. |
|
|
| ### CPU-only machines |
|
|
| Installation is identical. The app auto-detects the absence of CUDA and runs the RouterBrain, chat and code specialists on CPU. Image and video specialists respond with a friendly "GPU required" message instead of crashing. |
|
|
| ### Troubleshooting |
|
|
| - **bitsandbytes warnings on CPU** — expected and safely ignored by the app. |
| - **Slow clone** — the RouterBrain weights are ~3 GB; make sure Git-LFS is installed (Step 1). |
| - **Port 7860 already in use** — change server_port at the bottom of app.py. |
| - **Out of memory on GPU** — the app loads specialists on demand and unloads them after use; close other GPU processes or reduce image resolution in app.py. |
| |
| ## Hardware Requirements |
| |
| | Mode | Minimum | Recommended | |
| |------|---------|-------------| |
| | RouterBrain only | 2GB VRAM | CPU works fine | |
| | Chat + Code (CPU) | 16GB RAM | 32GB RAM | |
| | Full GPU stack | 16GB VRAM | 24GB+ VRAM | |
| |
| The app auto-detects CUDA. On CPU-only machines, chat/code work and image/video return a friendly "GPU required" message. |
| |
| ## Training Details |
| |
| ### RouterBrain Training |
| - **Base Model**: Qwen2.5-1.5B-Instruct |
| - **Fine-tuning**: QLoRA (r=32, α=64, dropout=0.05) |
| - **Dataset**: ~10k synthetic routing examples (combinatorial diversity, noise injection, 30+ edge-case patterns) |
| - **Training**: 2-3 epochs on T4 GPU |
| - **Merge**: Clean fp16 merge (no quantization rounding) |
| |
| ### Performance Metrics |
| - **Pre-flight layer**: 100% accuracy on hard patterns |
| - **RouterBrain**: 60% standalone accuracy on diverse test set |
| - **Effective routing**: 90%+ via triple-layer fault tolerance |
| |
| ## Multi-Step Task Chaining |
| |
| ``` |
| User: "Draw a robot then animate it dancing" |
| ↓ |
| Pre-flight: detects "then" → routes to multi specialist |
| ↓ |
| Execution: image specialist → video specialist (sequential) |
| ↓ |
| Output: generated image + animated video with audio |
| ``` |
| |
| ## Architecture Decisions |
| |
| ### Why Triple-Layer Routing? |
| 1. **Speed**: Pre-flight regex is instant (0.001s) |
| 2. **Efficiency**: RouterBrain is fast (0.3s) vs prompt-based (4-6s) |
| 3. **Robustness**: 4B fallback catches edge cases |
| 4. **Production-ready**: Graceful degradation under all conditions |
| |
| ### Why Distill A Routing Brain? |
| - **20x faster** than prompt-based routing |
| - **50% less VRAM** than using the 4B resident for routing |
| - **Perfect JSON** output (trained to only output JSON) |
| - **Cheaper** to deploy at scale |
| |
| ## Companion Models |
| |
| - **[Frankenstein-2.0](https://huggingface.co/Questionmarkboy/frankenstein-2.0)** — Merged Mistral-7B checkpoint |
| |
| ## Technical Stack |
| |
| - **Framework**: HuggingFace Transformers + Diffusers |
| - **Quantization**: BitsAndBytes (NF4, 4-bit) |
| - **UI**: Gradio 4.44 |
| - **Memory**: SQLite |
| - **Training**: PEFT + LoRA |
| |
| ## Known Limitations |
| |
| - Image/video generation requires 16GB+ VRAM |
| - RAG only supports .txt files (no PDF/DOCX yet) |
| - Web search is basic (DuckDuckGo HTML scraping) |
| - Code execution has 5s timeout for safety |
| |
| ## Citation |
| |
| ```bibtex |
| @misc{frankenstein3, |
| author = {Questionmarkboy}, |
| title = {Frankenstein-3.0: Multi-Modal AI Orchestration System}, |
| year = {2024}, |
| publisher = {HuggingFace}, |
| journal = {HuggingFace Repository}, |
| howpublished = {\url{https://huggingface.co/Questionmarkboy/frankenstein-3.0}} |
| } |
| ``` |
| |
| ## License |
| |
| Apache 2.0 |
| |
| ## Acknowledgments |
| |
| - Base models: Qwen, Stability AI, MiniMax |
| - Frameworks: HuggingFace, Gradio, BitsAndBytes |
| - Training infrastructure: Kaggle T4 GPUs |
| |
| --- |
| |
| **Built with ❤️ for the AI community** |
| |
