File size: 7,264 Bytes

---
license: apache-2.0
language:
- en
tags:
- prompt-routing
- difficulty-classifier
- deberta-v3
- llm-router
datasets:
- RowRed/prompts-24000-en
base_model:
- microsoft/deberta-v3-xsmall
---

# DifficultyRouter: A Lightweight 3‑Tier Prompt Difficulty Router

DifficultyRouter is a successor to [ComplexityRouter](https://huggingface.co/RowRed/ComplexityRouter), specifically designed for **cost optimization** in LLM routing. It is finetuned from **microsoft/deberta-v3-xsmall** (≈22M params, ~8x smaller than the base model used previously) on a 20,000‑prompt dataset (2,000 L0 + 6,000 L1 → Tier 0; 6,000 L2 → Tier 1; 6,000 L3 → Tier 2).

It classifies prompts into **3 difficulty tiers** with a single classification head.

## Model Details

### Model Description

- **Model type:** Text Classification (3‑tier, single head)
- **Language:** English
- **License:** Apache‑2.0
- **Finetuned from model:** microsoft/deberta-v3-xsmall
- **Training data:** `RowRed/prompts-24000-en` (L0 downsampled to 2,000, L1/L2/L3 kept fully)

### Model Sources

- **Dataset repository:** `https://huggingface.co/datasets/RowRed/prompts-24000-en`
- **Old model:** `RowRed/ComplexityRouter`

## Uses

### Direct Use

Route prompts to appropriate LLM tiers based on predicted difficulty:

| Tier | Meaning | Original Levels | Suggested LLM Tier |
|------|---------|-----------------|--------------------|
| 0 (Easy) | Simple lookups, basic Q&A, light reasoning | L0 + L1 | Fast/cheap model |
| 1 (Moderate) | Complex reasoning, deep domain knowledge | L2 | Standard model |
| 2 (Complex) | Very complex reasoning, niche expertise, edge cases | L3 | Frontier model |

### Out‑of‑Scope Use

- Multi‑turn conversation routing (single prompts only).
- Non‑English prompts (training data is English‑only).
- Prompts requiring image or multimodal understanding.
- 4‑level classification (use the old ComplexityRouter for 4 classes).

## Bias, Risks, and Limitations

- Training data includes synthetic augmentation; distribution may not match all real‑world prompt patterns.
- Tier 0 (merged L0+L1) has inherent ambiguity—some "trivial" and "simple" prompts are hard to distinguish from "moderate".
- DeBERTa‑v3‑xsmall has a smaller representation capacity than the base model, so it may miss very subtle difficulty cues in niche technical domains.

### ⚠️ Not Production‑Ready
This model is a research prototype.

- Accuracy is ~63.4%, meaning ~4 out of 10 prompts will be misrouted.
- Adjacent accuracy of 87.1% means 1 in 8 prompts will be sent to a tier that is still off by one level, leading to noticeable latency/cost misses.
- The model has not been stress‑tested on real‑world, messy, multi‑domain prompts. It was trained on synthetic augmentations.

## Training Details

### Training Data

- **Source:** `RowRed/prompts-24000-en` (approximately 24,000 raw prompts)
- **Used:** 20,000 prompts (L0 downsampled to 2,000; L1, L2, L3 kept at 6,000 each)
- **Preprocessing:** Original difficulty levels (0–3) are mapped to 3 tiers: `0+1 -> 0`, `2 -> 1`, `3 -> 2`.
- **L0 downsampling:** To combat overfitting, exactly `2,000` L0 samples were randomly sampled (config flag `l0_sample_size=2000`, locked).
- **Split:** 70% train / 18% validation / 12% held-out test (stratified).

### Training Procedure

- **Hardware:** NVIDIA T4 (16 GB VRAM, Google Colab)
- **Framework:** PyTorch + Hugging Face Transformers
- **Optimizer:** AdamW (lr=3e-5, weight_decay=0.1)
- **Scheduler:** Linear warmup (6% steps) → linear decay
- **Loss:** Weighted Cross‑Entropy with label smoothing=0.1
- **Batch size:** 16 (effective 32 with gradient accumulation)
- **Max sequence length:** 256 tokens
- **Epochs:** 12 max (Early stopping patience = 4 on F1-macro; stopped at epoch 7, best checkpoint at epoch 3)
- **Class balancing:** WeightedRandomSampler only; class weights in loss removed to avoid double-counting
- **Head:** 256-dim, dropout=0.1
- **Precision:** FP16 mixed precision

## Evaluation Results

Reported on the **held‑out test set** (~12% of 20k prompts) using the **best model checkpoint** (epoch 3, selected via validation F1-macro):

| Metric              | Value  |
|---------------------|--------|
| Exact Match Accuracy | 63.36% |
| Adjacent (±1) Accuracy | 87.14% |
| F1 Macro            | 0.6375 |
| F1 Weighted         | 0.6314 |

Training loss continued to decrease but validation metrics peaked at epoch 3; early stopping correctly caught the onset of overfitting.

## How to Get Started with the Model

The model uses a single-head architecture and saves via **safetensors** (load with `strict=True`). Use the same class as in training:

```python
from transformers import AutoTokenizer, AutoModel
import torch
import torch.nn as nn

class DifficultyRouter(nn.Module):
    def __init__(self, model_name="microsoft/deberta-v3-xsmall", num_labels=3):
        super().__init__()
        self.backbone = AutoModel.from_pretrained(model_name)
        hidden_size = self.backbone.config.hidden_size
        self.classifier = nn.Sequential(
            nn.Dropout(0.1), nn.Linear(hidden_size, 256), nn.GELU(),
            nn.Dropout(0.1), nn.Linear(256, num_labels)
        )

    def forward(self, input_ids, attention_mask):
        outputs = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
        cls_out = outputs.last_hidden_state[:, 0, :].to(torch.float32)
        logits = self.classifier(cls_out)
        return logits

# Load (safe, no pickle)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained("RowRed/DifficultyRouter")
model = DifficultyRouter()
model.load_state_dict(torch.load("model.safetensors", map_location=device), strict=True)
model.to(device).eval()

# Predict
prompts = ["What is 2+2?", "Explain quantum entanglement in detail."]
encoded = tokenizer(prompts, padding=True, truncation=True, max_length=256, return_tensors="pt").to(device)
with torch.no_grad():
    logits = model(encoded["input_ids"], encoded["attention_mask"])
    probs = torch.softmax(logits, dim=-1)
    tiers = torch.argmax(probs, dim=-1)

for prompt, tier in zip(prompts, tiers):
    print(f"Tier {tier.item()}: {prompt}")
```

## Citation
If you use this model, please cite:

```bibtex
@software{DifficultyRouter,
  author = {RowRed},
  title = {DifficultyRouter},
  year = {2026},
  url = {https://huggingface.co/RowRed/DifficultyRouter}
}
```

Additionally, acknowledge the base model:

```bibtex
@misc{he2021debertav3,
      title={DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing}, 
      author={Pengcheng He and Jianfeng Gao and Weizhu Chen},
      year={2021},
      eprint={2111.09543},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```
```bibtex
@inproceedings{
he2021deberta,
title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=XPZIaotutsD}
}
```

## License
This model is released under Apache‑2.0.
The backbone (microsoft/deberta-v3-xsmall) is MIT‑licensed.