Gramli commited on
Commit
dbe4247
·
verified ·
1 Parent(s): 503332f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +197 -13
README.md CHANGED
@@ -1,37 +1,221 @@
1
  ---
2
  base_model: Qwen/Qwen3-8B
3
  library_name: peft
 
 
 
4
  tags:
5
  - tinker
6
  - peft
7
  - lora
 
 
 
8
  - base_model:adapter:Qwen/Qwen3-8B
9
  tinker_path: tinker://c2148695-43ab-5758-aaff-bd78ddda980d:train:0/sampler_weights/final
10
  ---
11
 
12
- # Tinker LoRA Adapter
13
 
14
- This repository contains a LoRA adapter exported from Tinker.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
15
 
16
  ## Usage
17
 
 
 
 
 
 
 
 
 
18
  ```python
19
- from transformers import AutoModelForCausalLM
 
20
 
21
- adapter_id = "Gramli/complexity-scorer"
22
- base_model = "Qwen/Qwen3-8B"
23
 
24
- model = AutoModelForCausalLM.from_pretrained(adapter_id, device_map="auto")
25
- ```
26
 
27
- ## Source
 
 
 
 
28
 
 
 
29
  ```
30
- tinker://c2148695-43ab-5758-aaff-bd78ddda980d:train:0/sampler_weights/final
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
  ```
32
 
33
- ## Details
 
 
 
 
 
 
 
 
 
 
 
 
 
34
 
35
- - Base model: Qwen/Qwen3-8B
36
- - LoRA rank: 32
37
- - Trained modules: attn=True, mlp=True, unembed=True
 
1
  ---
2
  base_model: Qwen/Qwen3-8B
3
  library_name: peft
4
+ pipeline_tag: text-generation
5
+ language:
6
+ - en
7
  tags:
8
  - tinker
9
  - peft
10
  - lora
11
+ - prompt-complexity
12
+ - classification
13
+ - llm-routing
14
  - base_model:adapter:Qwen/Qwen3-8B
15
  tinker_path: tinker://c2148695-43ab-5758-aaff-bd78ddda980d:train:0/sampler_weights/final
16
  ---
17
 
18
+ # Complexity Scorer
19
 
20
+ **Complexity Scorer** is a fine-tuned Qwen3-8B model that estimates how difficult a user request is for a general-purpose large language model to complete correctly.
21
+
22
+ It returns a single complexity score between **0.00 and 1.00** in JSON format.
23
+
24
+ The goal is simple: help developers make smarter LLM routing decisions without depending on any specific AI provider.
25
+
26
+ For example, an application might send simple requests to a cheaper model while reserving more capable reasoning models for difficult tasks.
27
+
28
+ Complexity Scorer doesn't select the model. **It provides a complexity score, and the client decides what to do with it.**
29
+
30
+ ## Complexity levels
31
+
32
+ | Score | Level | Description |
33
+ |---|---|---|
34
+ | 0.00 | Trivial | Extraction, formatting, or deterministic transformations |
35
+ | 0.25 | Easy | Straightforward tasks requiring limited interpretation |
36
+ | 0.50 | Moderate | Multiple steps, domain knowledge, or several constraints |
37
+ | 0.75 | Hard | Complex reasoning, difficult tradeoffs, or specialized expertise |
38
+ | 1.00 | Very Hard | Deep reasoning, expert analysis, or complex system interactions |
39
+
40
+ Complexity is based on the reasoning required to produce a correct response, not simply the length of the prompt.
41
+
42
+ ### Example
43
+
44
+ Input:
45
+
46
+ ```text
47
+ Review this distributed locking implementation and identify race conditions that could cause duplicate payments.
48
+ ```
49
+
50
+ Example output:
51
+
52
+ ```json
53
+ {"complexity":0.75}
54
+ ```
55
+
56
+ The score is a classification label, not a calibrated probability of success or failure.
57
+
58
+ ## Model details
59
+
60
+ | Property | Value |
61
+ |---|---|
62
+ | Base model | Qwen/Qwen3-8B |
63
+ | Fine-tuning platform | Tinker by Thinking Machines Lab |
64
+ | Training method | Supervised fine-tuning (LoRA) |
65
+ | Training examples | 2,500 |
66
+ | Complexity classes | 5 |
67
+ | LoRA rank | 32 |
68
+ | Learning rate | 1e-4 |
69
+ | Batch size | 16 |
70
+ | Epochs | 1 |
71
+ | Maximum sequence length | 4,096 tokens |
72
+ | Evaluation mode | Non-thinking |
73
+ | Evaluation temperature | 0 |
74
+
75
+ Only the LoRA adapter is provided in this repository. The original Qwen3-8B model is required for inference.
76
+
77
+ ## Training dataset
78
+
79
+ The model was fine-tuned using **2,500 synthetically generated examples**, balanced across five complexity levels, with 500 examples per class.
80
+
81
+ The dataset covers a variety of domains, including software development, APIs, databases, cybersecurity, business, writing, data analysis, and general reasoning.
82
+
83
+ The examples were designed to include both short but difficult requests and long but simple requests to discourage the model from using prompt length as a shortcut.
84
+
85
+ Training targets contain only the expected JSON complexity score. Explanations used during dataset generation were not part of the model's target output.
86
+
87
+ ## Evaluation results
88
+
89
+ All models were evaluated against the same **50-example synthetic evaluation dataset**, containing 10 examples per complexity level.
90
+
91
+ | Model | Training examples | Exact accuracy | MAE |
92
+ |---|---:|---:|---:|
93
+ | Base Qwen3-8B | 0 | 50% | 0.125 |
94
+ | Fine-tuned | 50 | **86%** | **0.035** |
95
+ | Fine-tuned | 500 | 74% | 0.065 |
96
+ | **This model** | **2,500** | **76%** | **0.060** |
97
+
98
+ Interestingly, the model trained on only 50 examples achieved the highest accuracy on this evaluation set.
99
+
100
+ Increasing the dataset to 2,500 examples did not improve the measured accuracy beyond that smaller fine-tune.
101
+
102
+ Possible explanations include inconsistencies in synthetic labels, differences in training configurations, and the limited size of the evaluation dataset.
103
+
104
+ ### Final model performance
105
+
106
+ For this 2,500-example model:
107
+
108
+ - **Exact accuracy:** 76% (38/50)
109
+ - **Mean absolute error:** 0.060
110
+ - **Valid JSON responses:** 100% (50/50)
111
+ - **Incorrect predictions:** 12
112
+
113
+ All 12 incorrect predictions differed from the expected score by just one adjacent complexity level (0.25).
114
+
115
+ ### Limitations of these results
116
+
117
+ These results are experimental and should not be interpreted as production-level accuracy.
118
+
119
+ The evaluation dataset is small, synthetic, and was reused during development to compare training configurations. As a result, it is not a fully independent final benchmark.
120
+
121
+ The model has not been validated against large-scale real-world API traffic or measured for actual cost savings across different LLM providers.
122
 
123
  ## Usage
124
 
125
+ Install the required packages:
126
+
127
+ ```bash
128
+ pip install torch transformers peft accelerate
129
+ ```
130
+
131
+ Load the base model and LoRA adapter:
132
+
133
  ```python
134
+ from transformers import AutoTokenizer, AutoModelForCausalLM
135
+ from peft import PeftModel
136
 
137
+ BASE_MODEL = "Qwen/Qwen3-8B"
138
+ ADAPTER = "Gramli/complexity-scorer"
139
 
140
+ tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
 
141
 
142
+ base_model = AutoModelForCausalLM.from_pretrained(
143
+ BASE_MODEL,
144
+ device_map="auto",
145
+ torch_dtype="auto"
146
+ )
147
 
148
+ model = PeftModel.from_pretrained(base_model, ADAPTER)
149
+ model.eval()
150
  ```
151
+
152
+ Run inference:
153
+
154
+ ```python
155
+ system_prompt = """
156
+ You are a prompt complexity classifier.
157
+
158
+ Estimate how difficult the user's request is for a capable
159
+ general-purpose LLM to answer correctly.
160
+
161
+ Complexity scores:
162
+ 0.00 - Trivial
163
+ 0.25 - Easy
164
+ 0.50 - Moderate
165
+ 0.75 - Hard
166
+ 1.00 - Very Hard
167
+
168
+ Judge reasoning difficulty, not prompt length.
169
+
170
+ Return JSON only:
171
+ {"complexity":0.50}
172
+ """
173
+
174
+ messages = [
175
+ {"role": "system", "content": system_prompt},
176
+ {
177
+ "role": "user",
178
+ "content": "Explain distributed transactions in microservices."
179
+ }
180
+ ]
181
+
182
+ text = tokenizer.apply_chat_template(
183
+ messages,
184
+ tokenize=False,
185
+ add_generation_prompt=True,
186
+ enable_thinking=False
187
+ )
188
+
189
+ inputs = tokenizer(
190
+ [text],
191
+ return_tensors="pt"
192
+ ).to(model.device)
193
+
194
+ outputs = model.generate(
195
+ **inputs,
196
+ max_new_tokens=50,
197
+ do_sample=False
198
+ )
199
+
200
+ generated = outputs[0][inputs["input_ids"].shape[-1]:]
201
+
202
+ print(tokenizer.decode(generated, skip_special_tokens=True))
203
  ```
204
 
205
+ **Important:** For reproducible evaluation results, use the exact system prompt and inference settings from the project's evaluation implementation. The shorter system prompt above demonstrates usage but is not guaranteed to reproduce the published benchmark.
206
+
207
+ This example runs the model locally after downloading the necessary weights. A suitable machine with sufficient memory is required.
208
+
209
+ ## Potential applications
210
+
211
+ - **LLM routing:** Help applications decide when to use cheaper or more capable models.
212
+ - **Cost optimization:** Provide an additional signal for deciding how to allocate LLM requests.
213
+ - **AI gateways:** Classify incoming requests before forwarding them to a model provider.
214
+ - **Usage analytics:** Analyze the complexity distribution of application prompts.
215
+
216
+ The scorer is provider-agnostic. Applications can interpret its scores according to their own available models, pricing, and requirements.
217
+
218
+ ## Limitations
219
 
220
+ - Complexity is subjective, particularly near class boundaries.
221
+ - The output