README.md

---
base_model: Qwen/Qwen2.5-0.5B-Instruct
license: apache-2.0
language: [en]
datasets: [podstack/Shaktimaan-routing]
library_name: transformers
pipeline_tag: text-generation
tags: [router, llm-router, orchestration, multi-agent, model-selection, fugu]
model-index:
- name: Shaktimaan v1.0
  results:
  - task: {type: text-classification, name: LLM routing}
    dataset: {type: podstack/Shaktimaan-routing, name: Shaktimaan routing (held-out test, 554 tasks)}
    metrics:
    - {type: accuracy, value: 0.8339, name: end-to-end task accuracy}
    - {type: relative_cost, value: 0.4240, name: cost vs always-strongest agent}
---
# Shaktimaan v1.0 (0.5B)

**Shaktimaan** is a 0.5B-parameter LLM orchestrator in the spirit of Sakana Fugu. It reads a task and decides which
agent(s) from a pool of six LLMs should handle it - one agent (`single`), or several that answer independently with
the last one aggregating (`ensemble`) - trading off cost, capability (image input, context length) and accuracy. It
does not answer tasks itself. Fine-tuned from [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
on [podstack/Shaktimaan-routing](https://huggingface.co/datasets/podstack/Shaktimaan-routing).

## At a glance
- **83.4%** end-to-end accuracy on 554 held-out tasks (math, knowledge,
  science, code, long context, charts) at **0.42x** the cost of always using the
  strongest agent.
- Beats every single agent used alone (best: Always `glm-5.3-flash` at 79.2%) and the
  strongest agent alone (78.5% at 1.00x).
- Learns capability routing from data: charts -> the only vision agent, very long inputs -> the long-context agents.
- **Does not beat hand-written capability rules overall**: Capability rules -> `glm-5.3-flash` reaches
  85.7% at 0.13x. See *Results* and *Findings*.
- 100.0% of outputs on the test set are valid route JSON.

## Output format
```json
{"domain": "math", "difficulty": "easy|medium|hard", "strategy": "single|ensemble", "agents": ["glm-5.3-flash"]}
```

## Agent pool
| agent | model | context | image input (in this pool) | relative cost / token |
|---|---|---|---|---|
| `gemma-3-1b` | `google/gemma-3-1b-it` | 32k | no | 0.02 |
| `kimi-vl` | `Kimi-VL-A3B-Thinking` | 8k | yes | 0.05 |
| `nemotron-lightning` | `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` | 128k | no | 0.1 |
| `glm-5.3-flash` | `zai-org/GLM-5.3-Flash` | 32k | no | 0.2 |
| `qwen3.8-27b` | `Qwen/Qwen3.8-27B` | 32k | no | 0.5 |
| `nemotron-ultra` | `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4` | 256k | no | 1.0 |

Agents are reached through OpenAI-compatible chat-completions endpoints. "Relative cost" is a per-token weight used
to compute routing cost (roughly relative serving cost), not a vendor price.

## Quick start
```python
import json
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer
import importlib.util

repo = "podstack/Shaktimaan"
spec = importlib.util.spec_from_file_location("common", hf_hub_download(repo, "common.py"))
common = importlib.util.module_from_spec(spec); spec.loader.exec_module(common)

tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
task = "Write a Python function that returns the n-th Fibonacci number."
msgs = [{"role": "system", "content": common.ROUTER_SYSTEM},
        {"role": "user", "content": common.router_view(task, has_image=False)}]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(**enc, max_new_tokens=96, do_sample=False)
print(json.loads(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True)))
```

Always build the input with `common.router_view(task, has_image)` - the router was trained on that exact format (a
`[input ~Nk tokens; 1 image attached]` header plus the first 5,000 and last 1,500 characters of the task) with
`common.ROUTER_SYSTEM` as the system prompt.

### End-to-end orchestration
`orchestrate.py` routes with Shaktimaan, drops agents that cannot take the input, then calls the chosen agent(s) and
aggregates ensembles:

```bash
export SHAKTIMAAN_ENDPOINTS='{"kimi-vl": "http://host-a:8000/v1", "nemotron-lightning": "http://host-b:8000/v1", ...}'
python orchestrate.py "If a train travels 120 km in 1.5 hours, what is its average speed?"
python orchestrate.py --code "Write a function that checks whether a string is a palindrome."
python orchestrate.py --image chart.png "What was the revenue in Q2 2021?"
```

## Training data
4989 train / 554 test tasks across eight sources. Every capable agent attempted every
task once and was graded automatically; the label is the cheapest *reliable* agent that solved the task (or an
ensemble when none did). Full details, solve rates per agent and the label policy are in the
[dataset card](https://huggingface.co/datasets/podstack/Shaktimaan-routing).

| source | category | train | test | grading | license |
|---|---|---|---|---|---|
| [GSM8K](https://huggingface.co/datasets/openai/gsm8k) | math word problems | 895 | 105 | numeric match | MIT |
| [MATH](https://huggingface.co/datasets/EleutherAI/hendrycks_math) | competition math | 894 | 106 | normalised LaTeX / numeric match | MIT |
| [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) | knowledge MCQ (10 options) | 1080 | 120 | option letter | MIT |
| [ARC-Challenge](https://huggingface.co/datasets/allenai/ai2_arc) | science MCQ | 725 | 75 | option letter | CC-BY-SA-4.0 |
| [MBPP](https://huggingface.co/datasets/google-research-datasets/mbpp) | Python coding | 633 | 67 | unit tests executed in a network-less sandbox | CC-BY-4.0 |
| [HumanEval](https://huggingface.co/datasets/openai/openai_humaneval) | Python coding | 146 | 18 | unit tests executed in a network-less sandbox | MIT |
| [LongBench-v2](https://huggingface.co/datasets/THUDM/LongBench-v2) | long-context MCQ (up to ~260k tokens) | 347 | 32 | option letter | Apache-2.0 |
| [ChartQA](https://huggingface.co/datasets/HuggingFaceM4/ChartQA) | chart images | 269 | 31 | relaxed match (numbers within 5%) | GPL-3.0 |

## Training
| setting | value |
|---|---|
| base model | Qwen2.5-0.5B-Instruct (494M parameters) |
| method | full fine-tune, supervised; loss on the route JSON only |
| epochs | 3 |
| learning rate | 2e-5, cosine schedule, 20 warm-up steps |
| effective batch size | 32 |
| max sequence length | 3,072 tokens |
| precision | bf16 |
| final eval loss | 0.0484 |

## Results
Held-out test set: 554 tasks the router never saw. A policy's accuracy is whether the agent it picks
actually solved the task. Shaktimaan's 31 ensemble decisions were **executed live** (all agents
plus aggregator called and graded); single-agent decisions use the recorded outcomes. Cost is the sum of
`relative cost x output tokens`, normalised so that always using `nemotron-ultra` = 1.00.

| policy | accuracy | relative cost (always-Ultra = 1.00) |
|---|---|---|
| **Shaktimaan v1.0** | 83.4% | 0.424 |
| Capability rules -> `glm-5.3-flash` | 85.7% | 0.133 |
| Capability rules -> `qwen3.8-27b` | 85.4% | 0.403 |
| Capability rules -> `nemotron-ultra` | 82.7% | 1.001 |
| Always `gemma-3-1b` | 25.6% | 0.002 |
| Always `kimi-vl` | 61.4% | 0.074 |
| Always `nemotron-lightning` | 75.6% | 0.109 |
| Always `glm-5.3-flash` | 79.2% | 0.094 |
| Always `qwen3.8-27b` | 78.9% | 0.364 |
| Always `nemotron-ultra` | 78.5% | 1.000 |
| Oracle (hindsight-best label) | 89.7% | 0.490 |

**Capability rules** are a hand-written baseline: image -> `kimi-vl`; input too long for the default agent -> the
cheapest agent with a large enough context window; otherwise the default agent. **Oracle** is the hindsight-best
training label (an upper bound; its ensembles are scored as the aggregator's solo answer).

### Accuracy by source
| source | n | Shaktimaan | rules -> glm | rules -> qwen | always Ultra | oracle |
|---|---|---|---|---|---|---|
| ARC-Challenge | 75 | 98.7% | 98.7% | 98.7% | 98.7% | 100.0% |
| ChartQA | 31 | 74.2% | 74.2% | 74.2% | 0.0% | 74.2% |
| GSM8K | 105 | 89.5% | 96.2% | 98.1% | 97.1% | 98.1% |
| HumanEval | 18 | 88.9% | 94.4% | 94.4% | 94.4% | 100.0% |
| LongBench-v2 | 32 | 59.4% | 46.9% | 50.0% | 59.4% | 59.4% |
| MATH | 106 | 75.5% | 74.5% | 75.5% | 65.1% | 82.1% |
| MBPP | 67 | 94.0% | 98.5% | 92.5% | 97.0% | 100.0% |
| MMLU-Pro | 120 | 77.5% | 83.3% | 81.7% | 74.2% | 87.5% |

### Routes chosen on the test set
| route chosen by Shaktimaan | test tasks |
|---|---|
| `nemotron-lightning` | 218 |
| `glm-5.3-flash` | 192 |
| `kimi-vl` | 80 |
| `glm-5.3-flash,qwen3.8-27b,nemotron-ultra` | 32 |
| `nemotron-ultra` | 31 |
| `qwen3.8-27b` | 1 |

## Findings
- **Routing beats any single model.** Shaktimaan is 4.1
  points above the best single agent and 4.9
  points above always using the strongest (and most expensive) one, at
  0.42x its cost.
- **Capability routing is learned from data.** On long-context tasks Shaktimaan matches the oracle
  (59.4%) and beats the capability rules (46.9%),
  sending inputs to the agent most likely to solve them rather than merely one that fits. On charts it routes to the
  only vision agent, as the rules do.
- **Simple capability rules are stronger overall** (85.7% at
  0.13x). On ordinary text and code tasks the mid-size agents are close to
  interchangeable, and which one solves a particular task is largely sampling noise that a router cannot predict from
  the prompt. Shaktimaan sometimes picks the cheaper `nemotron-lightning` or `kimi-vl` where the rule's fixed
  `glm-5.3-flash` default is slightly more reliable, and its ensembles (chosen for hard tasks) solved
  11 of the 31 hard tasks they were sent, at a high cost.
- **Practical recommendation:** for this pool, apply capability rules first and use Shaktimaan where its learned
  judgement helps most - choosing among long-context agents and flagging hard tasks.

## Limitations
- Routes only among the six agents above, with their context windows and image support; a different pool needs new
  data and retraining.
- Labels come from a single sample per agent and inherit grading noise (e.g. answer extraction from free text).
- Not trained on agentic / tool-use, multi-turn or open-ended writing tasks; English only.
- `est_tokens` uses a conservative characters/3 estimate, so borderline inputs may be routed to a larger-context
  agent than strictly needed.

## Files
| file | purpose |
|---|---|
| `model.safetensors`, `config.json`, tokenizer files | the router |
| `common.py` | agent pool, prompts, `router_view`, capability checks, graders |
| `orchestrate.py` | end-to-end runtime (route -> call agents -> aggregate) |
| `gen.py` | builds tasks and collects graded agent outcomes |
| `build.py` | labels outcomes and builds the train / test split |
| `train.py` | fine-tuning script |
| `evaluate.py` | held-out evaluation incl. live ensembles and baselines |
| `eval_results.json` | raw evaluation output |

## License
Model weights: Apache-2.0 (as the base model). Training data: see the
[dataset card](https://huggingface.co/datasets/podstack/Shaktimaan-routing) (CC-BY-SA-4.0; ChartQA rows GPL-3.0, not redistributed).