ArkidMitra commited on
Commit
80262fe
·
verified ·
1 Parent(s): 4e40d7e

Card: long menus (more than 26 options) from code release 1.1.1; weights and map unchanged

Browse files
Files changed (1) hide show
  1. README.md +31 -0
README.md CHANGED
@@ -196,6 +196,37 @@ not on this fold.
196
  - Expect the held-out hard items to score below the public ones, and expect the judge tier, which
197
  we have never seen, to be the least predictable part.
198
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
199
  ## Limitations
200
 
201
  - **One pass of a 4B model.** Hopper does not reason step by step. Any problem that needs a chain
 
196
  - Expect the held-out hard items to score below the public ones, and expect the judge tier, which
197
  we have never seen, to be the least predictable part.
198
 
199
+ ## Long menus (more than 26 options), from code release 1.1.1
200
+
201
+ The adapter answers by reading one letter per option, so a single forward pass can weigh at most 26
202
+ options. From release 1.1.1 of the serving code (github.com/hopit-ai/hopper, tag v1.1.1), a choice
203
+ question with more options is answered in two disclosed stages, using this same adapter and map:
204
+
205
+ 1. **First stage.** By default a *tournament*: the menu is dealt with a fixed seed into
206
+ `ceil(n / 26)` chunks, each chunk is read by the ordinary single pass, and the 10 options that
207
+ score best within their chunks go on. Optionally (`--shortlist embedding`), an embedding model
208
+ (`Qwen/Qwen3-Embedding-0.6B`, Apache-2.0, pinned) picks them instead.
209
+ 2. **Final pass.** The ordinary single pass, with the same prompt and calibration map, decides among
210
+ the survivors.
211
+
212
+ Every option in the request gets a probability: the final distribution mixed with 5 % of a uniform
213
+ distribution over the whole menu, so an eliminated option gets 0.05 / n and never zero. That
214
+ residual stands for the chance that the first stage dropped the right answer; it does not change
215
+ the answer. A 77-option menu costs 4 forward passes and a 150-option menu 7. Questions with 26
216
+ options or fewer are answered exactly as before, with one pass.
217
+
218
+ Measured on an A10G through the server, 300 public test items each, default settings:
219
+
220
+ | | options | top-1 accuracy | right answer survived the first stage | latency p50 |
221
+ | --- | ---: | ---: | ---: | ---: |
222
+ | BANKING77 | 77 | 0.673 | 0.887 | 321 ms |
223
+ | CLINC150 (out-of-scope dropped) | 150 | 0.863 | 0.977 | 569 ms |
224
+
225
+ The adapter was trained on menus of at most six options; on these long menus it is level with the
226
+ frozen base model within sampling error. On BANKING77 the served top choice is overconfident (mean
227
+ top probability 0.78 against 0.67 accuracy). The 5 % residual is a default, not yet refitted to
228
+ held-out long menus. Research and demo use only, as for the rest of this card.
229
+
230
  ## Limitations
231
 
232
  - **One pass of a 4B model.** Hopper does not reason step by step. Any problem that needs a chain