Create README.md

README.md

@@ -0,0 +1,188 @@
+---
+license: apache-2.0
+language:
+- en
+tags:
+- prompt-routing
+- difficulty-classifier
+- deberta-v3
+- llm-router
+datasets:
+- RowRed/prompts-24000-en
+base_model:
+- microsoft/deberta-v3-xsmall
+---
+
+# DifficultyRouter: A Lightweight 3‑Tier Prompt Difficulty Router
+
+DifficultyRouter is a successor to [ComplexityRouter](https://huggingface.co/RowRed/ComplexityRouter), specifically designed for **cost optimization** in LLM routing. It is finetuned from **microsoft/deberta-v3-xsmall** (≈22M params, ~8x smaller than the base model used previously) on a 20,000‑prompt dataset (2,000 L0 + 6,000 L1 → Tier 0; 6,000 L2 → Tier 1; 6,000 L3 → Tier 2).
+
+It classifies prompts into **3 difficulty tiers** with a single classification head.
+
+## Model Details
+
+### Model Description
+
+- **Model type:** Text Classification (3‑tier, single head)
+- **Language:** English
+- **Backbone:** microsoft/deberta-v3-xsmall
+- **License:** Apache‑2.0
+- **Finetuned from model:** microsoft/deberta-v3-xsmall
+- **Training data:** `RowRed/prompts-24000-en` (L0 downsampled to 2,000, L1/L2/L3 kept fully)
+
+### Model Sources
+
+- **Dataset repository:** `https://huggingface.co/datasets/RowRed/prompts-24000-en`
+- **Old model:** `RowRed/ComplexityRouter`
+
+## Uses
+
+### Direct Use
+
+Route prompts to appropriate LLM tiers based on predicted difficulty:
+
+| Tier | Meaning | Original Levels | Suggested LLM Tier |
+|------|---------|-----------------|--------------------|
+| 0 (Easy) | Simple lookups, basic Q&A, light reasoning | L0 + L1 | Fast/cheap model |
+| 1 (Moderate) | Complex reasoning, deep domain knowledge | L2 | Standard model |
+| 2 (Complex) | Very complex reasoning, niche expertise, edge cases | L3 | Frontier model |
+
+### Out‑of‑Scope Use
+
+- Multi‑turn conversation routing (single prompts only).
+- Non‑English prompts (training data is English‑only).
+- Prompts requiring image or multimodal understanding.
+- 4‑level classification (use the old ComplexityRouter for 4 classes).
+
+## Bias, Risks, and Limitations
+
+- Training data includes synthetic augmentation; distribution may not match all real‑world prompt patterns.
+- Tier 0 (merged L0+L1) has inherent ambiguity—some "trivial" and "simple" prompts are hard to distinguish from "moderate".
+- DeBERTa‑v3‑xsmall has a smaller representation capacity than the base model, so it may miss very subtle difficulty cues in niche technical domains.
+
+### ⚠️ Not Production‑Ready
+This model is a research prototype.
+
+- Accuracy is ~63.4%, meaning ~4 out of 10 prompts will be misrouted.
+- Adjacent accuracy of 87.1% means 1 in 8 prompts will be sent to a tier that is still off by one level, leading to noticeable latency/cost misses.
+- The model has not been stress‑tested on real‑world, messy, multi‑domain prompts. It was trained on synthetic augmentations.
+
+## Training Details
+
+### Training Data
+
+- **Source:** `RowRed/prompts-24000-en` (approximately 24,000 raw prompts)
+- **Used:** 20,000 prompts (L0 downsampled to 2,000; L1, L2, L3 kept at 6,000 each)
+- **Preprocessing:** Original difficulty levels (0–3) are mapped to 3 tiers: `0+1 -> 0`, `2 -> 1`, `3 -> 2`.
+- **L0 downsampling:** To combat overfitting, exactly `2,000` L0 samples were randomly sampled (config flag `l0_sample_size=2000`, locked).
+- **Split:** 70% train / 18% validation / 12% held-out test (stratified).
+
+### Training Procedure
+
+- **Hardware:** NVIDIA T4 (16 GB VRAM, Google Colab)
+- **Framework:** PyTorch + Hugging Face Transformers
+- **Optimizer:** AdamW (lr=3e-5, weight_decay=0.1)
+- **Scheduler:** Linear warmup (6% steps) → linear decay
+- **Loss:** Weighted Cross‑Entropy with label smoothing=0.1
+- **Batch size:** 16 (effective 32 with gradient accumulation)
+- **Max sequence length:** 256 tokens
+- **Epochs:** 12 max (Early stopping patience = 4 on F1-macro; stopped at epoch 7, best checkpoint at epoch 3)
+- **Class balancing:** WeightedRandomSampler only; class weights in loss removed to avoid double-counting
+- **Head:** 256-dim, dropout=0.1 (simpler head for less overfit)
+- **Precision:** FP16 mixed precision
+
+## Evaluation Results
+
+Reported on the **held‑out test set** (~12% of 20k prompts) using the **best model checkpoint** (epoch 3, selected via validation F1-macro):
+
+| Metric              | Value  |
+|---------------------|--------|
+| Exact Match Accuracy | 63.36% |
+| Adjacent (±1) Accuracy | 87.14% |
+| F1 Macro            | 0.6375 |
+| F1 Weighted         | 0.6314 |
+
+Training loss continued to decrease but validation metrics peaked at epoch 3; early stopping correctly caught the onset of overfitting.
+
+## How to Get Started with the Model
+
+The model uses a single-head architecture and saves via **safetensors** (load with `strict=True`). Use the same class as in training:
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+import torch.nn as nn
+
+class DifficultyRouter(nn.Module):
+    def __init__(self, model_name="microsoft/deberta-v3-xsmall", num_labels=3):
+        super().__init__()
+        self.backbone = AutoModel.from_pretrained(model_name)
+        hidden_size = self.backbone.config.hidden_size
+        self.classifier = nn.Sequential(
+            nn.Dropout(0.1), nn.Linear(hidden_size, 256), nn.GELU(),
+            nn.Dropout(0.1), nn.Linear(256, num_labels)
+        )
+
+    def forward(self, input_ids, attention_mask):
+        outputs = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
+        cls_out = outputs.last_hidden_state[:, 0, :].to(torch.float32)
+        logits = self.classifier(cls_out)
+        return logits
+
+# Load (safe, no pickle)
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+tokenizer = AutoTokenizer.from_pretrained("RowRed/DifficultyRouter")
+model = DifficultyRouter()
+model.load_state_dict(torch.load("model.safetensors", map_location=device), strict=True)
+model.to(device).eval()
+
+# Predict
+prompts = ["What is 2+2?", "Explain quantum entanglement in detail."]
+encoded = tokenizer(prompts, padding=True, truncation=True, max_length=256, return_tensors="pt").to(device)
+with torch.no_grad():
+    logits = model(encoded["input_ids"], encoded["attention_mask"])
+    probs = torch.softmax(logits, dim=-1)
+    tiers = torch.argmax(probs, dim=-1)
+
+for prompt, tier in zip(prompts, tiers):
+    print(f"Tier {tier.item()}: {prompt}")
+```
+
+## Citation
+If you use this model, please cite:
+
+```bibtex
+@software{DifficultyRouter,
+  author = {RowRed},
+  title = {DifficultyRouter},
+  year = {2026},
+  url = {https://huggingface.co/RowRed/DifficultyRouter}
+}
+```
+
+Additionally, acknowledge the base model:
+
+```bibtex
+@misc{he2021debertav3,
+      title={DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing}, 
+      author={Pengcheng He and Jianfeng Gao and Weizhu Chen},
+      year={2021},
+      eprint={2111.09543},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+```bibtex
+@inproceedings{
+he2021deberta,
+title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
+author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
+booktitle={International Conference on Learning Representations},
+year={2021},
+url={https://openreview.net/forum?id=XPZIaotutsD}
+}
+```
+
+## License
+This model is released under Apache‑2.0.
+The backbone (microsoft/deberta-v3-xsmall) is MIT‑licensed.