Shaktimaan v1.0 (0.5B)

Shaktimaan is a 0.5B-parameter LLM orchestrator in the spirit of Sakana Fugu. It reads a task and decides which agent(s) from a pool of six LLMs should handle it - one agent (single), or several that answer independently with the last one aggregating (ensemble) - trading off cost, capability (image input, context length) and accuracy. It does not answer tasks itself. Fine-tuned from Qwen2.5-0.5B-Instruct on podstack/Shaktimaan-routing.

At a glance

• 83.4% end-to-end accuracy on 554 held-out tasks (math, knowledge, science, code, long context, charts) at 0.42x the cost of always using the strongest agent.
• Beats every single agent used alone (best: Always glm-5.3-flash at 79.2%) and the strongest agent alone (78.5% at 1.00x).
• Learns capability routing from data: charts -> the only vision agent, very long inputs -> the long-context agents.
• Does not beat hand-written capability rules overall: Capability rules -> glm-5.3-flash reaches 85.7% at 0.13x. See Results and Findings.
• 100.0% of outputs on the test set are valid route JSON.

Output format

{"domain": "math", "difficulty": "easy|medium|hard", "strategy": "single|ensemble", "agents": ["glm-5.3-flash"]}

Agent pool

agent	model	context	image input (in this pool)	relative cost / token
gemma-3-1b	google/gemma-3-1b-it	32k	no	0.02
kimi-vl	Kimi-VL-A3B-Thinking	8k	yes	0.05
nemotron-lightning	nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4	128k	no	0.1
glm-5.3-flash	zai-org/GLM-5.3-Flash	32k	no	0.2
qwen3.8-27b	Qwen/Qwen3.8-27B	32k	no	0.5
nemotron-ultra	nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4	256k	no	1.0

Agents are reached through OpenAI-compatible chat-completions endpoints. "Relative cost" is a per-token weight used to compute routing cost (roughly relative serving cost), not a vendor price.

Quick start

import json
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer
import importlib.util

repo = "podstack/Shaktimaan"
spec = importlib.util.spec_from_file_location("common", hf_hub_download(repo, "common.py"))
common = importlib.util.module_from_spec(spec); spec.loader.exec_module(common)

tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
task = "Write a Python function that returns the n-th Fibonacci number."
msgs = [{"role": "system", "content": common.ROUTER_SYSTEM},
        {"role": "user", "content": common.router_view(task, has_image=False)}]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(**enc, max_new_tokens=96, do_sample=False)
print(json.loads(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True)))

Always build the input with common.router_view(task, has_image) - the router was trained on that exact format (a [input ~Nk tokens; 1 image attached] header plus the first 5,000 and last 1,500 characters of the task) with common.ROUTER_SYSTEM as the system prompt.

End-to-end orchestration

orchestrate.py routes with Shaktimaan, drops agents that cannot take the input, then calls the chosen agent(s) and aggregates ensembles:

export SHAKTIMAAN_ENDPOINTS='{"kimi-vl": "http://host-a:8000/v1", "nemotron-lightning": "http://host-b:8000/v1", ...}'
python orchestrate.py "If a train travels 120 km in 1.5 hours, what is its average speed?"
python orchestrate.py --code "Write a function that checks whether a string is a palindrome."
python orchestrate.py --image chart.png "What was the revenue in Q2 2021?"

Training data

4989 train / 554 test tasks across eight sources. Every capable agent attempted every task once and was graded automatically; the label is the cheapest reliable agent that solved the task (or an ensemble when none did). Full details, solve rates per agent and the label policy are in the dataset card.

source	category	train	test	grading	license
GSM8K	math word problems	895	105	numeric match	MIT
MATH	competition math	894	106	normalised LaTeX / numeric match	MIT
MMLU-Pro	knowledge MCQ (10 options)	1080	120	option letter	MIT
ARC-Challenge	science MCQ	725	75	option letter	CC-BY-SA-4.0
MBPP	Python coding	633	67	unit tests executed in a network-less sandbox	CC-BY-4.0
HumanEval	Python coding	146	18	unit tests executed in a network-less sandbox	MIT
LongBench-v2	long-context MCQ (up to ~260k tokens)	347	32	option letter	Apache-2.0
ChartQA	chart images	269	31	relaxed match (numbers within 5%)	GPL-3.0

Training

setting	value
base model	Qwen2.5-0.5B-Instruct (494M parameters)
method	full fine-tune, supervised; loss on the route JSON only
epochs	3
learning rate	2e-5, cosine schedule, 20 warm-up steps
effective batch size	32
max sequence length	3,072 tokens
precision	bf16
final eval loss	0.0484

Results

Held-out test set: 554 tasks the router never saw. A policy's accuracy is whether the agent it picks actually solved the task. Shaktimaan's 31 ensemble decisions were executed live (all agents plus aggregator called and graded); single-agent decisions use the recorded outcomes. Cost is the sum of relative cost x output tokens, normalised so that always using nemotron-ultra = 1.00.

policy	accuracy	relative cost (always-Ultra = 1.00)
Shaktimaan v1.0	83.4%	0.424
Capability rules -> glm-5.3-flash	85.7%	0.133
Capability rules -> qwen3.8-27b	85.4%	0.403
Capability rules -> nemotron-ultra	82.7%	1.001
Always gemma-3-1b	25.6%	0.002
Always kimi-vl	61.4%	0.074
Always nemotron-lightning	75.6%	0.109
Always glm-5.3-flash	79.2%	0.094
Always qwen3.8-27b	78.9%	0.364
Always nemotron-ultra	78.5%	1.000
Oracle (hindsight-best label)	89.7%	0.490

Capability rules are a hand-written baseline: image -> kimi-vl; input too long for the default agent -> the cheapest agent with a large enough context window; otherwise the default agent. Oracle is the hindsight-best training label (an upper bound; its ensembles are scored as the aggregator's solo answer).

Accuracy by source

source	n	Shaktimaan	rules -> glm	rules -> qwen	always Ultra	oracle
ARC-Challenge	75	98.7%	98.7%	98.7%	98.7%	100.0%
ChartQA	31	74.2%	74.2%	74.2%	0.0%	74.2%
GSM8K	105	89.5%	96.2%	98.1%	97.1%	98.1%
HumanEval	18	88.9%	94.4%	94.4%	94.4%	100.0%
LongBench-v2	32	59.4%	46.9%	50.0%	59.4%	59.4%
MATH	106	75.5%	74.5%	75.5%	65.1%	82.1%
MBPP	67	94.0%	98.5%	92.5%	97.0%	100.0%
MMLU-Pro	120	77.5%	83.3%	81.7%	74.2%	87.5%

Routes chosen on the test set

route chosen by Shaktimaan	test tasks
nemotron-lightning	218
glm-5.3-flash	192
kimi-vl	80
glm-5.3-flash,qwen3.8-27b,nemotron-ultra	32
nemotron-ultra	31
qwen3.8-27b	1

Findings

• Routing beats any single model. Shaktimaan is 4.1 points above the best single agent and 4.9 points above always using the strongest (and most expensive) one, at 0.42x its cost.
• Capability routing is learned from data. On long-context tasks Shaktimaan matches the oracle (59.4%) and beats the capability rules (46.9%), sending inputs to the agent most likely to solve them rather than merely one that fits. On charts it routes to the only vision agent, as the rules do.
• Simple capability rules are stronger overall (85.7% at 0.13x). On ordinary text and code tasks the mid-size agents are close to interchangeable, and which one solves a particular task is largely sampling noise that a router cannot predict from the prompt. Shaktimaan sometimes picks the cheaper nemotron-lightning or kimi-vl where the rule's fixed glm-5.3-flash default is slightly more reliable, and its ensembles (chosen for hard tasks) solved 11 of the 31 hard tasks they were sent, at a high cost.
• Practical recommendation: for this pool, apply capability rules first and use Shaktimaan where its learned judgement helps most - choosing among long-context agents and flagging hard tasks.

Limitations

• Routes only among the six agents above, with their context windows and image support; a different pool needs new data and retraining.
• Labels come from a single sample per agent and inherit grading noise (e.g. answer extraction from free text).
• Not trained on agentic / tool-use, multi-turn or open-ended writing tasks; English only.
• est_tokens uses a conservative characters/3 estimate, so borderline inputs may be routed to a larger-context agent than strictly needed.

Files

file	purpose
model.safetensors, config.json, tokenizer files	the router
common.py	agent pool, prompts, router_view, capability checks, graders
orchestrate.py	end-to-end runtime (route -> call agents -> aggregate)
gen.py	builds tasks and collects graded agent outcomes
build.py	labels outcomes and builds the train / test split
train.py	fine-tuning script
evaluate.py	held-out evaluation incl. live ensembles and baselines
eval_results.json	raw evaluation output

License

Model weights: Apache-2.0 (as the base model). Training data: see the dataset card (CC-BY-SA-4.0; ChartQA rows GPL-3.0, not redistributed).