File size: 5,763 Bytes
71d49e7 463d23e 71d49e7 6d524ae 463d23e 71d49e7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 | ---
license: mit
pipeline_tag: text-classification
base_model: sentence-transformers/all-MiniLM-L6-v2
tags:
- prompt-routing
- llm-routing
- complexity-classification
- text-classification
- onnx
- cpu
datasets:
- TakamaruAI/skala-dataset
language:
- en
---
# Skala
### A prompt-difficulty classifier for simple, medium, and complex model routing.
Skala classifies chat, agent, and tooling prompts into three difficulty tiers to help routers select an appropriate model for each request. Its 23.2 MB model delivers median CPU inference latency as low as 2.65 ms. Skala combines a linear classifier over lexical and structural features with semantic embeddings from a pruned, int8-quantised variant of [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2), trained on 6,948 English prompts.
## Architecture
- **Architecture:** Hybrid prompt classifier combining semantic embeddings, lexical features, and structural features
- **Encoder:** Pruned, int8-quantised variant of [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2), with 5 Transformer layers and 20.9M parameters
- **Encoder configuration:** 12 attention heads, 384-dimensional embeddings, 1,536-dimensional feed-forward layers
- **Feature blocks:** 384 semantic features, 15,000 lexical features, and 21 structural features
- **Classification head:** Multinomial logistic regression, 3 classes
- **Calibration:** Per-class isotonic regression, fitted out-of-fold
- **Decision rule:** Highest calibrated class score
- **Length guard:** Structural features clamped to their training-set 95th percentiles before scoring
- **Maximum sequence length:** 64 tokens
## Datasets
Skala was trained on 6,948 English prompts, 1,698 sourced from the [Nasim435/Multi-label-Prompt-Dataset](https://huggingface.co/datasets/Nasim435/Multi-label-Prompt-Dataset) and 5,250 generated and labelled by two open-weight models, `google/gemma-4-26B-A4B` and `Qwen/Qwen3.8-27B`.
| dataset | rows | licence |
|---|---:|---|
| [TakamaruAI/Skala-Dataset](https://huggingface.co/datasets/TakamaruAI/Skala-Dataset) (published training corpus) | 6,940 | MIT |
| [Nasim435/Multi-label-Prompt-Dataset](https://huggingface.co/datasets/Nasim435/Multi-label-Prompt-Dataset) | 1,698 | MIT |
| [data/generated_prompts.csv](data/generated_prompts.csv) | 5,242 | MIT |
| [data/real_350.csv](data/real_350.csv) | 350 | MIT |
| [tests/behavior_suite2.csv](tests/behavior_suite2.csv) | 52 | MIT |
| [tests/behavior_suite2_clean.csv](tests/behavior_suite2_clean.csv) | 38 | MIT |
| [tests/holdout_150.csv](tests/holdout_150.csv) | 150 | MIT |
## Results
Evaluation covers behavioural test suites and held-out prompts. The 350-prompt set was evaluated using out-of-fold predictions, where each prompt was scored by a model that had not been trained on it. The 150-prompt holdout was kept separate from training and used for final evaluation.
| Benchmark | n | Score |
|---|---:|---:|
| [tests/behavior_suite2.csv](tests/behavior_suite2.csv) | 52 | **44/52 (84.6%)** |
| [tests/behavior_suite2_clean.csv](tests/behavior_suite2_clean.csv) | 38 | **31/38 (81.6%)** |
| [data/real_350.csv](data/real_350.csv) | 350 | **0.7429** |
| [tests/holdout_150.csv](tests/holdout_150.csv) | 150 | **116/150 (77.3%)** |
## Performance
Skala delivers low-latency, single-prompt inference across desktop and mobile-class CPUs.
| Hardware | p50 (median) | p95 | Calls |
|---|---:|---:|---:|
| Intel Core i7-10700K | 2.75 ms | 3.69 ms | 200 |
| Intel Core i5-4460 | 3.79 ms | 4.44 ms | 200 |
| Snapdragon 8 Gen 3 (SM8650) | 2.65 ms | 3.96 ms | 200 |
All measurements use single-prompt inference (batch size 1), with one warmup pass.
### Runtime footprint
- **Model size:** 23.2 MB
- **Peak memory usage:** ~289 MB RSS
## Limitations
- **English only.** Non-English prompts may be misclassified as `simple` without warning.
- **Length bias.** Longer prompts can lean toward `medium`, even when the task itself is simple.
- **Evaluation caveats.** 14 of 52 behavioural-suite prompts overlap with training data. The clean subset scores 31/38 (81.6%). The separate 150-prompt holdout scores 116/150 (77.3%). Its labels are public and should remain excluded from future training.
- **Single-prompt inference.** Batch classification currently processes prompts one at a time.
- **Retraining variability.** Results may shift slightly between training runs. Published scores refer to the released model artifact.
## Requirements
##### Core dependencies:
- [scikit-learn](https://github.com/scikit-learn/scikit-learn) (pinned to 1.9.1: the fitted head will not unpickle under other versions)
- [numpy](https://github.com/numpy/numpy)
- [scipy](https://github.com/scipy/scipy)
- [joblib](https://github.com/joblib/joblib)
- [onnxruntime](https://github.com/microsoft/onnxruntime)
- [transformers](https://github.com/huggingface/transformers) (tokenizer only, no torch)
##### Python:
- 3.12+
## Installation
Install with:
```bash
python3 -m venv .venv
. .venv/bin/activate
pip install .
```
## Usage
```python
import sys
sys.path.insert(0, ".")
from skala import SkalaClassifier
clf = SkalaClassifier("artifacts")
clf.load()
clf.warmup()
# Single prompt
label, scores = clf.classify("I have this Python codebase, can you port it to C++")
print(label, scores)
# Several prompts at once
pairs = clf.classify_batch([
"hi",
"fix my wifi",
"prove the Riemann hypothesis",
])
```
## Examples
Classify a prompt from the command line or reproduce the suite numbers from this folder alone
```bash
python examples/classify_prompt.py "I have this Python codebase, can you port it to C++"
python tests/test_suite.py
```
## License
Skala is licensed under the [MIT License](LICENSE). |