File size: 7,298 Bytes
503332f
 
 
dbe4247
 
 
503332f
 
 
 
dbe4247
 
 
503332f
 
 
 
dbe4247
503332f
dbe4247
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
503332f
 
 
dbe4247
 
 
 
 
 
 
 
503332f
dbe4247
 
503332f
dbe4247
 
503332f
dbe4247
503332f
dbe4247
 
 
 
 
503332f
dbe4247
 
503332f
dbe4247
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
503332f
 
dbe4247
 
 
 
 
 
 
 
 
 
 
 
 
 
503332f
dbe4247
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
---

base_model: Qwen/Qwen3-8B
library_name: peft
pipeline_tag: text-generation
language:
- en
tags:
- tinker
- peft
- lora
- prompt-complexity
- classification
- llm-routing
- base_model:adapter:Qwen/Qwen3-8B
tinker_path: tinker://c2148695-43ab-5758-aaff-bd78ddda980d:train:0/sampler_weights/final
---


# Complexity Scorer

**Complexity Scorer** is a fine-tuned Qwen3-8B model that estimates how difficult a user request is for a general-purpose large language model to complete correctly.

It returns a single complexity score between **0.00 and 1.00** in JSON format.

The goal is simple: help developers make smarter LLM routing decisions without depending on any specific AI provider.

For example, an application might send simple requests to a cheaper model while reserving more capable reasoning models for difficult tasks.

Complexity Scorer doesn't select the model. **It provides a complexity score, and the client decides what to do with it.**

## Complexity levels

| Score | Level | Description |
|---|---|---|
| 0.00 | Trivial | Extraction, formatting, or deterministic transformations |
| 0.25 | Easy | Straightforward tasks requiring limited interpretation |
| 0.50 | Moderate | Multiple steps, domain knowledge, or several constraints |
| 0.75 | Hard | Complex reasoning, difficult tradeoffs, or specialized expertise |
| 1.00 | Very Hard | Deep reasoning, expert analysis, or complex system interactions |

Complexity is based on the reasoning required to produce a correct response, not simply the length of the prompt.

### Example

Input:

```text

Review this distributed locking implementation and identify race conditions that could cause duplicate payments.

```

Example output:

```json

{"complexity":0.75}

```

The score is a classification label, not a calibrated probability of success or failure.

## Model details

| Property | Value |
|---|---|
| Base model | Qwen/Qwen3-8B |
| Fine-tuning platform | Tinker by Thinking Machines Lab |
| Training method | Supervised fine-tuning (LoRA) |
| Training examples | 2,500 |
| Complexity classes | 5 |
| LoRA rank | 32 |
| Learning rate | 1e-4 |
| Batch size | 16 |
| Epochs | 1 |
| Maximum sequence length | 4,096 tokens |
| Evaluation mode | Non-thinking |
| Evaluation temperature | 0 |

Only the LoRA adapter is provided in this repository. The original Qwen3-8B model is required for inference.

## Training dataset

The model was fine-tuned using **2,500 synthetically generated examples**, balanced across five complexity levels, with 500 examples per class.

The dataset covers a variety of domains, including software development, APIs, databases, cybersecurity, business, writing, data analysis, and general reasoning.

The examples were designed to include both short but difficult requests and long but simple requests to discourage the model from using prompt length as a shortcut.

Training targets contain only the expected JSON complexity score. Explanations used during dataset generation were not part of the model's target output.

## Evaluation results

All models were evaluated against the same **50-example synthetic evaluation dataset**, containing 10 examples per complexity level.

| Model | Training examples | Exact accuracy | MAE |
|---|---:|---:|---:|
| Base Qwen3-8B | 0 | 50% | 0.125 |
| Fine-tuned | 50 | **86%** | **0.035** |
| Fine-tuned | 500 | 74% | 0.065 |
| **This model** | **2,500** | **76%** | **0.060** |

Interestingly, the model trained on only 50 examples achieved the highest accuracy on this evaluation set.

Increasing the dataset to 2,500 examples did not improve the measured accuracy beyond that smaller fine-tune.

Possible explanations include inconsistencies in synthetic labels, differences in training configurations, and the limited size of the evaluation dataset.

### Final model performance

For this 2,500-example model:

- **Exact accuracy:** 76% (38/50)
- **Mean absolute error:** 0.060
- **Valid JSON responses:** 100% (50/50)
- **Incorrect predictions:** 12

All 12 incorrect predictions differed from the expected score by just one adjacent complexity level (0.25).

### Limitations of these results

These results are experimental and should not be interpreted as production-level accuracy.

The evaluation dataset is small, synthetic, and was reused during development to compare training configurations. As a result, it is not a fully independent final benchmark.

The model has not been validated against large-scale real-world API traffic or measured for actual cost savings across different LLM providers.

## Usage

Install the required packages:

```bash

pip install torch transformers peft accelerate

```

Load the base model and LoRA adapter:

```python

from transformers import AutoTokenizer, AutoModelForCausalLM

from peft import PeftModel



BASE_MODEL = "Qwen/Qwen3-8B"

ADAPTER = "Gramli/complexity-scorer"



tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)



base_model = AutoModelForCausalLM.from_pretrained(

    BASE_MODEL,

    device_map="auto",

    torch_dtype="auto"

)



model = PeftModel.from_pretrained(base_model, ADAPTER)

model.eval()

```

Run inference:

```python

system_prompt = """

You are a prompt complexity classifier.



Estimate how difficult the user's request is for a capable

general-purpose LLM to answer correctly.



Complexity scores:

0.00 - Trivial

0.25 - Easy

0.50 - Moderate

0.75 - Hard

1.00 - Very Hard



Judge reasoning difficulty, not prompt length.



Return JSON only:

{"complexity":0.50}

"""



messages = [

    {"role": "system", "content": system_prompt},

    {

        "role": "user",

        "content": "Explain distributed transactions in microservices."

    }

]



text = tokenizer.apply_chat_template(

    messages,

    tokenize=False,

    add_generation_prompt=True,

    enable_thinking=False

)



inputs = tokenizer(

    [text],

    return_tensors="pt"

).to(model.device)



outputs = model.generate(

    **inputs,

    max_new_tokens=50,

    do_sample=False

)



generated = outputs[0][inputs["input_ids"].shape[-1]:]



print(tokenizer.decode(generated, skip_special_tokens=True))

```

**Important:** For reproducible evaluation results, use the exact system prompt and inference settings from the project's evaluation implementation. The shorter system prompt above demonstrates usage but is not guaranteed to reproduce the published benchmark.

This example runs the model locally after downloading the necessary weights. A suitable machine with sufficient memory is required.

## Potential applications

- **LLM routing:** Help applications decide when to use cheaper or more capable models.
- **Cost optimization:** Provide an additional signal for deciding how to allocate LLM requests.
- **AI gateways:** Classify incoming requests before forwarding them to a model provider.
- **Usage analytics:** Analyze the complexity distribution of application prompts.

The scorer is provider-agnostic. Applications can interpret its scores according to their own available models, pricing, and requirements.

## Limitations

- Complexity is subjective, particularly near class boundaries.
- The output