DifficultyRouter: A Lightweight 3‑Tier Prompt Difficulty Router

DifficultyRouter is a successor to ComplexityRouter, specifically designed for cost optimization in LLM routing. It is finetuned from microsoft/deberta-v3-xsmall (≈22M params, ~8x smaller than the base model used previously) on a 20,000‑prompt dataset (2,000 L0 + 6,000 L1 → Tier 0; 6,000 L2 → Tier 1; 6,000 L3 → Tier 2).

It classifies prompts into 3 difficulty tiers with a single classification head.

Model Details

Model Description

• Model type: Text Classification (3‑tier, single head)
• Language: English
• License: Apache‑2.0
• Finetuned from model: microsoft/deberta-v3-xsmall
• Training data: RowRed/prompts-24000-en (L0 downsampled to 2,000, L1/L2/L3 kept fully)

Model Sources

• Dataset repository: https://huggingface.co/datasets/RowRed/prompts-24000-en
• Old model: RowRed/ComplexityRouter

Uses

Direct Use

Route prompts to appropriate LLM tiers based on predicted difficulty:

Tier	Meaning	Original Levels	Suggested LLM Tier
0 (Easy)	Simple lookups, basic Q&A, light reasoning	L0 + L1	Fast/cheap model
1 (Moderate)	Complex reasoning, deep domain knowledge	L2	Standard model
2 (Complex)	Very complex reasoning, niche expertise, edge cases	L3	Frontier model

Out‑of‑Scope Use

• Multi‑turn conversation routing (single prompts only).
• Non‑English prompts (training data is English‑only).
• Prompts requiring image or multimodal understanding.
• 4‑level classification (use the old ComplexityRouter for 4 classes).

Bias, Risks, and Limitations

• Training data includes synthetic augmentation; distribution may not match all real‑world prompt patterns.
• Tier 0 (merged L0+L1) has inherent ambiguity—some "trivial" and "simple" prompts are hard to distinguish from "moderate".
• DeBERTa‑v3‑xsmall has a smaller representation capacity than the base model, so it may miss very subtle difficulty cues in niche technical domains.

⚠️ Not Production‑Ready

This model is a research prototype.

• Accuracy is ~63.4%, meaning ~4 out of 10 prompts will be misrouted.
• Adjacent accuracy of 87.1% means 1 in 8 prompts will be sent to a tier that is still off by one level, leading to noticeable latency/cost misses.
• The model has not been stress‑tested on real‑world, messy, multi‑domain prompts. It was trained on synthetic augmentations.

Training Details

Training Data

• Source: RowRed/prompts-24000-en (approximately 24,000 raw prompts)
• Used: 20,000 prompts (L0 downsampled to 2,000; L1, L2, L3 kept at 6,000 each)
• Preprocessing: Original difficulty levels (0–3) are mapped to 3 tiers: 0+1 -> 0, 2 -> 1, 3 -> 2.
• L0 downsampling: To combat overfitting, exactly 2,000 L0 samples were randomly sampled (config flag l0_sample_size=2000, locked).
• Split: 70% train / 18% validation / 12% held-out test (stratified).

Training Procedure

• Hardware: NVIDIA T4 (16 GB VRAM, Google Colab)
• Framework: PyTorch + Hugging Face Transformers
• Optimizer: AdamW (lr=3e-5, weight_decay=0.1)
• Scheduler: Linear warmup (6% steps) → linear decay
• Loss: Weighted Cross‑Entropy with label smoothing=0.1
• Batch size: 16 (effective 32 with gradient accumulation)
• Max sequence length: 256 tokens
• Epochs: 12 max (Early stopping patience = 4 on F1-macro; stopped at epoch 7, best checkpoint at epoch 3)
• Class balancing: WeightedRandomSampler only; class weights in loss removed to avoid double-counting
• Head: 256-dim, dropout=0.1
• Precision: FP16 mixed precision

Evaluation Results

Reported on the held‑out test set (~12% of 20k prompts) using the best model checkpoint (epoch 3, selected via validation F1-macro):

Metric	Value
Exact Match Accuracy	63.36%
Adjacent (±1) Accuracy	87.14%
F1 Macro	0.6375
F1 Weighted	0.6314

Training loss continued to decrease but validation metrics peaked at epoch 3; early stopping correctly caught the onset of overfitting.

How to Get Started with the Model

The model uses a single-head architecture and saves via safetensors (load with strict=True). Use the same class as in training:

from transformers import AutoTokenizer, AutoModel
import torch
import torch.nn as nn

class DifficultyRouter(nn.Module):
    def __init__(self, model_name="microsoft/deberta-v3-xsmall", num_labels=3):
        super().__init__()
        self.backbone = AutoModel.from_pretrained(model_name)
        hidden_size = self.backbone.config.hidden_size
        self.classifier = nn.Sequential(
            nn.Dropout(0.1), nn.Linear(hidden_size, 256), nn.GELU(),
            nn.Dropout(0.1), nn.Linear(256, num_labels)
        )

    def forward(self, input_ids, attention_mask):
        outputs = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
        cls_out = outputs.last_hidden_state[:, 0, :].to(torch.float32)
        logits = self.classifier(cls_out)
        return logits

# Load (safe, no pickle)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained("RowRed/DifficultyRouter")
model = DifficultyRouter()
model.load_state_dict(torch.load("model.safetensors", map_location=device), strict=True)
model.to(device).eval()

# Predict
prompts = ["What is 2+2?", "Explain quantum entanglement in detail."]
encoded = tokenizer(prompts, padding=True, truncation=True, max_length=256, return_tensors="pt").to(device)
with torch.no_grad():
    logits = model(encoded["input_ids"], encoded["attention_mask"])
    probs = torch.softmax(logits, dim=-1)
    tiers = torch.argmax(probs, dim=-1)

for prompt, tier in zip(prompts, tiers):
    print(f"Tier {tier.item()}: {prompt}")

Citation

If you use this model, please cite:

@software{DifficultyRouter,
  author = {RowRed},
  title = {DifficultyRouter},
  year = {2026},
  url = {https://huggingface.co/RowRed/DifficultyRouter}
}

Additionally, acknowledge the base model:

@misc{he2021debertav3,
      title={DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing}, 
      author={Pengcheng He and Jianfeng Gao and Weizhu Chen},
      year={2021},
      eprint={2111.09543},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@inproceedings{
he2021deberta,
title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=XPZIaotutsD}
}

License

This model is released under Apache‑2.0. The backbone (microsoft/deberta-v3-xsmall) is MIT‑licensed.