Henry Ndubuaku commited on
Commit
c472016
·
verified ·
1 Parent(s): 3b6e883

Fill DeepSeek V4 Flash benchmark row

Browse files
Files changed (1) hide show
  1. README.md +31 -2
README.md CHANGED
@@ -45,7 +45,7 @@ Tool calling is exact-match accuracy on the full test splits, extraction is fiel
45
 
46
  | Model | Params | Mobile Actions | DroidCall | BFCL v4 | DSTC8 F1 | SNIPS gold F1 | SNIPS 7-way F1 |
47
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
48
- | DeepSeek V4 Flash (cloud) | - | 87.5 | 58.4 | - | - | - | - |
49
  | **Needle3-20L-121M** | 121M | 86.0 | 47.0 | 50.2 | 40.7 | 30.2 | 24.7 |
50
  | LFM2.5 1.2B | 1.2B | 82.4 | 35.5 | 62.0 | 48.0 | 43.0 | 38.0 |
51
  | **Needle3-16L-98M** | 98M | 80.7 | 40.0 | 41.3 | 28.5 | 23.5 | 19.2 |
@@ -119,6 +119,21 @@ def send_money(
119
 
120
  `Field` supports `description`, `enum`, `const`, `ge`/`le`/`gt`/`lt`, `multiple_of`, `min_length`/`max_length`, `pattern`, `format`, `min_items`/`max_items`, and `unique_items`.
121
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
122
  **Extraction**: to pull structured data out of text, declare the shape and call `extract()`. Pass a Pydantic model and you get a typed object back.
123
 
124
  ```python
@@ -179,7 +194,20 @@ Every turn returns one JSON object:
179
  "peak_ram_mb": 88.5
180
  }
181
 
182
- A call the engine does not trust comes back in `suppressed_calls` with `function_calls` empty; show it to the user to confirm, or treat the turn as a refusal.
 
 
 
 
 
 
 
 
 
 
 
 
 
183
  ```
184
 
185
  ## Behaviour
@@ -194,6 +222,7 @@ Needle solves every problem as a function call. The context declares what may be
194
  - Decoded arguments pass through a deterministic repair step grounded in the request text before they are returned: a full name placed in one of a first/last pair is split, a truncated phone number is completed from the request, a quoted title or body is restored verbatim, a place query recovers the words the model dropped between its first and last matched word plus a following street address, `next <weekday>`, `this <weekday>`, bare weekdays and `March 4th` style dates are resolved against the `date:` fact, request casing is restored for free-text strings, a boolean or on/off enum is flipped to match an unambiguous request verb (unlock, turn on, resume), an optional origin copied from the destination is dropped, optional strings absent from the request are dropped, celsius values become fahrenheit for fahrenheit parameters, file extensions become MIME types, number-typed integers render as floats, empty optional strings and duplicate calls are dropped. Enumerated parameters and tool-result turns are never touched.
195
  - A call is withheld when its confidence is below 0.1 or when the request negates the called tool's verb, quotes a command reported by someone else, fills a required enumerated parameter with an option the request never mentions, or copies the destination into a required origin: `function_calls` is empty and the call sits in `suppressed_calls`, so a client that does nothing gets a refusal and a client that wants to confirm with the user can show the withheld call. `--forced` bypasses the gate.
196
  - The grammar refuses runaway values: a bounded number cannot grow past its maximum, a numeric literal stops at twenty characters, an array stops after seven identical consecutive items or sixty-four items, and a repeating token cycle inside a call is banned.
 
197
  - A turn after a call is a tool result only when it parses as JSON; any other text is a new user turn. `--fail-input-overflow` refuses a turn that would not fit the context window instead of trimming it.
198
 
199
  ## System facts
 
45
 
46
  | Model | Params | Mobile Actions | DroidCall | BFCL v4 | DSTC8 F1 | SNIPS gold F1 | SNIPS 7-way F1 |
47
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
48
+ | DeepSeek V4 Flash (cloud) | - | 88.4 | 60.5 | 77.2 | 80.0 | 69.4 | 66.7 |
49
  | **Needle3-20L-121M** | 121M | 86.0 | 47.0 | 50.2 | 40.7 | 30.2 | 24.7 |
50
  | LFM2.5 1.2B | 1.2B | 82.4 | 35.5 | 62.0 | 48.0 | 43.0 | 38.0 |
51
  | **Needle3-16L-98M** | 98M | 80.7 | 40.0 | 41.3 | 28.5 | 23.5 | 19.2 |
 
119
 
120
  `Field` supports `description`, `enum`, `const`, `ge`/`le`/`gt`/`lt`, `multiple_of`, `min_length`/`max_length`, `pattern`, `format`, `min_items`/`max_items`, and `unique_items`.
121
 
122
+ **Route by pattern**: when a description cannot enumerate every phrasing, give a tool `triggers`, regular expressions matched against each request. A match restricts the decode to the matched tools and requires a call, so the request reaches the tool you named instead of being refused or misrouted, and the call ships even below the confidence floor. Raw schemas take the same `"triggers"` list beside `description`.
123
+
124
+ ```python
125
+ from typing import Literal
126
+
127
+ @needle.tool(triggers=[r"\b(turn|switch|power|flip)\b.*\b(on|off)\b", r"\btoggle\b"])
128
+ def control_device(device: str, action: Literal["on", "off", "toggle"]):
129
+ "Switch or toggle any named smart-home device."
130
+ return {"device": device, "action": action}
131
+
132
+ agent = needle.Needle(tools=[control_device, get_weather])
133
+ agent.complete("toggle the garage door")
134
+ # function_calls [{"name": "control_device", "arguments": {"device": "garage door", "action": "toggle"}}]
135
+ ```
136
+
137
  **Extraction**: to pull structured data out of text, declare the shape and call `extract()`. Pass a Pydantic model and you get a typed object back.
138
 
139
  ```python
 
194
  "peak_ram_mb": 88.5
195
  }
196
 
197
+ **Confidence gating and routing**: every response carries a `confidence` score from a calibrated head, and the engine already applies a floor of 0.1: below it the call is withheld into `suppressed_calls` and `function_calls` is empty. Above it the score is yours to route on: act at once when it is high, show the call and ask when it is middling, and treat an empty result as a refusal. A tool with `triggers` always produces a call for a matching request, so the score is what tells you whether to run it or confirm it.
198
+
199
+ ```python
200
+ r = agent.complete(user_text)
201
+ calls = r["function_calls"]
202
+ held = r["suppressed_calls"]
203
+
204
+ if calls and r["confidence"] >= 0.7:
205
+ execute(calls) # sure: act
206
+ elif calls or held:
207
+ confirm(calls or held, r["reasoning"]) # unsure: show the call, ask
208
+ else:
209
+ say("I can't do that here") # nothing to do: refuse
210
+ ```
211
  ```
212
 
213
  ## Behaviour
 
222
  - Decoded arguments pass through a deterministic repair step grounded in the request text before they are returned: a full name placed in one of a first/last pair is split, a truncated phone number is completed from the request, a quoted title or body is restored verbatim, a place query recovers the words the model dropped between its first and last matched word plus a following street address, `next <weekday>`, `this <weekday>`, bare weekdays and `March 4th` style dates are resolved against the `date:` fact, request casing is restored for free-text strings, a boolean or on/off enum is flipped to match an unambiguous request verb (unlock, turn on, resume), an optional origin copied from the destination is dropped, optional strings absent from the request are dropped, celsius values become fahrenheit for fahrenheit parameters, file extensions become MIME types, number-typed integers render as floats, empty optional strings and duplicate calls are dropped. Enumerated parameters and tool-result turns are never touched.
223
  - A call is withheld when its confidence is below 0.1 or when the request negates the called tool's verb, quotes a command reported by someone else, fills a required enumerated parameter with an option the request never mentions, or copies the destination into a required origin: `function_calls` is empty and the call sits in `suppressed_calls`, so a client that does nothing gets a refusal and a client that wants to confirm with the user can show the withheld call. `--forced` bypasses the gate.
224
  - The grammar refuses runaway values: a bounded number cannot grow past its maximum, a numeric literal stops at twenty characters, an array stops after seven identical consecutive items or sixty-four items, and a repeating token cycle inside a call is banned.
225
+ - A tool may declare `triggers`, a list of regular expressions (case-insensitive) beside its `description`. When a user turn matches a tool's trigger, the decode is restricted to the matched tools and a call is required, so a request the model would refuse or route elsewhere still reaches the tool the author named. A triggered call ships even below the confidence floor, since the author asked for it; the negation and quotation gates still apply. Declare triggers for a catch-all tool whose phrasing the description cannot enumerate, e.g. `"triggers": ["\\b(turn|switch|power)\\b.*\\b(on|off)\\b", "\\btoggle\\b"]`.
226
  - A turn after a call is a tool result only when it parses as JSON; any other text is a new user turn. `--fail-input-overflow` refuses a turn that would not fit the context window instead of trimming it.
227
 
228
  ## System facts