Henry Ndubuaku commited on
Fill DeepSeek V4 Flash benchmark row
Browse files
README.md
CHANGED
|
@@ -45,7 +45,7 @@ Tool calling is exact-match accuracy on the full test splits, extraction is fiel
|
|
| 45 |
|
| 46 |
| Model | Params | Mobile Actions | DroidCall | BFCL v4 | DSTC8 F1 | SNIPS gold F1 | SNIPS 7-way F1 |
|
| 47 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 48 |
-
| DeepSeek V4 Flash (cloud) | - |
|
| 49 |
| **Needle3-20L-121M** | 121M | 86.0 | 47.0 | 50.2 | 40.7 | 30.2 | 24.7 |
|
| 50 |
| LFM2.5 1.2B | 1.2B | 82.4 | 35.5 | 62.0 | 48.0 | 43.0 | 38.0 |
|
| 51 |
| **Needle3-16L-98M** | 98M | 80.7 | 40.0 | 41.3 | 28.5 | 23.5 | 19.2 |
|
|
@@ -119,6 +119,21 @@ def send_money(
|
|
| 119 |
|
| 120 |
`Field` supports `description`, `enum`, `const`, `ge`/`le`/`gt`/`lt`, `multiple_of`, `min_length`/`max_length`, `pattern`, `format`, `min_items`/`max_items`, and `unique_items`.
|
| 121 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 122 |
**Extraction**: to pull structured data out of text, declare the shape and call `extract()`. Pass a Pydantic model and you get a typed object back.
|
| 123 |
|
| 124 |
```python
|
|
@@ -179,7 +194,20 @@ Every turn returns one JSON object:
|
|
| 179 |
"peak_ram_mb": 88.5
|
| 180 |
}
|
| 181 |
|
| 182 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 183 |
```
|
| 184 |
|
| 185 |
## Behaviour
|
|
@@ -194,6 +222,7 @@ Needle solves every problem as a function call. The context declares what may be
|
|
| 194 |
- Decoded arguments pass through a deterministic repair step grounded in the request text before they are returned: a full name placed in one of a first/last pair is split, a truncated phone number is completed from the request, a quoted title or body is restored verbatim, a place query recovers the words the model dropped between its first and last matched word plus a following street address, `next <weekday>`, `this <weekday>`, bare weekdays and `March 4th` style dates are resolved against the `date:` fact, request casing is restored for free-text strings, a boolean or on/off enum is flipped to match an unambiguous request verb (unlock, turn on, resume), an optional origin copied from the destination is dropped, optional strings absent from the request are dropped, celsius values become fahrenheit for fahrenheit parameters, file extensions become MIME types, number-typed integers render as floats, empty optional strings and duplicate calls are dropped. Enumerated parameters and tool-result turns are never touched.
|
| 195 |
- A call is withheld when its confidence is below 0.1 or when the request negates the called tool's verb, quotes a command reported by someone else, fills a required enumerated parameter with an option the request never mentions, or copies the destination into a required origin: `function_calls` is empty and the call sits in `suppressed_calls`, so a client that does nothing gets a refusal and a client that wants to confirm with the user can show the withheld call. `--forced` bypasses the gate.
|
| 196 |
- The grammar refuses runaway values: a bounded number cannot grow past its maximum, a numeric literal stops at twenty characters, an array stops after seven identical consecutive items or sixty-four items, and a repeating token cycle inside a call is banned.
|
|
|
|
| 197 |
- A turn after a call is a tool result only when it parses as JSON; any other text is a new user turn. `--fail-input-overflow` refuses a turn that would not fit the context window instead of trimming it.
|
| 198 |
|
| 199 |
## System facts
|
|
|
|
| 45 |
|
| 46 |
| Model | Params | Mobile Actions | DroidCall | BFCL v4 | DSTC8 F1 | SNIPS gold F1 | SNIPS 7-way F1 |
|
| 47 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 48 |
+
| DeepSeek V4 Flash (cloud) | - | 88.4 | 60.5 | 77.2 | 80.0 | 69.4 | 66.7 |
|
| 49 |
| **Needle3-20L-121M** | 121M | 86.0 | 47.0 | 50.2 | 40.7 | 30.2 | 24.7 |
|
| 50 |
| LFM2.5 1.2B | 1.2B | 82.4 | 35.5 | 62.0 | 48.0 | 43.0 | 38.0 |
|
| 51 |
| **Needle3-16L-98M** | 98M | 80.7 | 40.0 | 41.3 | 28.5 | 23.5 | 19.2 |
|
|
|
|
| 119 |
|
| 120 |
`Field` supports `description`, `enum`, `const`, `ge`/`le`/`gt`/`lt`, `multiple_of`, `min_length`/`max_length`, `pattern`, `format`, `min_items`/`max_items`, and `unique_items`.
|
| 121 |
|
| 122 |
+
**Route by pattern**: when a description cannot enumerate every phrasing, give a tool `triggers`, regular expressions matched against each request. A match restricts the decode to the matched tools and requires a call, so the request reaches the tool you named instead of being refused or misrouted, and the call ships even below the confidence floor. Raw schemas take the same `"triggers"` list beside `description`.
|
| 123 |
+
|
| 124 |
+
```python
|
| 125 |
+
from typing import Literal
|
| 126 |
+
|
| 127 |
+
@needle.tool(triggers=[r"\b(turn|switch|power|flip)\b.*\b(on|off)\b", r"\btoggle\b"])
|
| 128 |
+
def control_device(device: str, action: Literal["on", "off", "toggle"]):
|
| 129 |
+
"Switch or toggle any named smart-home device."
|
| 130 |
+
return {"device": device, "action": action}
|
| 131 |
+
|
| 132 |
+
agent = needle.Needle(tools=[control_device, get_weather])
|
| 133 |
+
agent.complete("toggle the garage door")
|
| 134 |
+
# function_calls [{"name": "control_device", "arguments": {"device": "garage door", "action": "toggle"}}]
|
| 135 |
+
```
|
| 136 |
+
|
| 137 |
**Extraction**: to pull structured data out of text, declare the shape and call `extract()`. Pass a Pydantic model and you get a typed object back.
|
| 138 |
|
| 139 |
```python
|
|
|
|
| 194 |
"peak_ram_mb": 88.5
|
| 195 |
}
|
| 196 |
|
| 197 |
+
**Confidence gating and routing**: every response carries a `confidence` score from a calibrated head, and the engine already applies a floor of 0.1: below it the call is withheld into `suppressed_calls` and `function_calls` is empty. Above it the score is yours to route on: act at once when it is high, show the call and ask when it is middling, and treat an empty result as a refusal. A tool with `triggers` always produces a call for a matching request, so the score is what tells you whether to run it or confirm it.
|
| 198 |
+
|
| 199 |
+
```python
|
| 200 |
+
r = agent.complete(user_text)
|
| 201 |
+
calls = r["function_calls"]
|
| 202 |
+
held = r["suppressed_calls"]
|
| 203 |
+
|
| 204 |
+
if calls and r["confidence"] >= 0.7:
|
| 205 |
+
execute(calls) # sure: act
|
| 206 |
+
elif calls or held:
|
| 207 |
+
confirm(calls or held, r["reasoning"]) # unsure: show the call, ask
|
| 208 |
+
else:
|
| 209 |
+
say("I can't do that here") # nothing to do: refuse
|
| 210 |
+
```
|
| 211 |
```
|
| 212 |
|
| 213 |
## Behaviour
|
|
|
|
| 222 |
- Decoded arguments pass through a deterministic repair step grounded in the request text before they are returned: a full name placed in one of a first/last pair is split, a truncated phone number is completed from the request, a quoted title or body is restored verbatim, a place query recovers the words the model dropped between its first and last matched word plus a following street address, `next <weekday>`, `this <weekday>`, bare weekdays and `March 4th` style dates are resolved against the `date:` fact, request casing is restored for free-text strings, a boolean or on/off enum is flipped to match an unambiguous request verb (unlock, turn on, resume), an optional origin copied from the destination is dropped, optional strings absent from the request are dropped, celsius values become fahrenheit for fahrenheit parameters, file extensions become MIME types, number-typed integers render as floats, empty optional strings and duplicate calls are dropped. Enumerated parameters and tool-result turns are never touched.
|
| 223 |
- A call is withheld when its confidence is below 0.1 or when the request negates the called tool's verb, quotes a command reported by someone else, fills a required enumerated parameter with an option the request never mentions, or copies the destination into a required origin: `function_calls` is empty and the call sits in `suppressed_calls`, so a client that does nothing gets a refusal and a client that wants to confirm with the user can show the withheld call. `--forced` bypasses the gate.
|
| 224 |
- The grammar refuses runaway values: a bounded number cannot grow past its maximum, a numeric literal stops at twenty characters, an array stops after seven identical consecutive items or sixty-four items, and a repeating token cycle inside a call is banned.
|
| 225 |
+
- A tool may declare `triggers`, a list of regular expressions (case-insensitive) beside its `description`. When a user turn matches a tool's trigger, the decode is restricted to the matched tools and a call is required, so a request the model would refuse or route elsewhere still reaches the tool the author named. A triggered call ships even below the confidence floor, since the author asked for it; the negation and quotation gates still apply. Declare triggers for a catch-all tool whose phrasing the description cannot enumerate, e.g. `"triggers": ["\\b(turn|switch|power)\\b.*\\b(on|off)\\b", "\\btoggle\\b"]`.
|
| 226 |
- A turn after a call is a tool result only when it parses as JSON; any other text is a new user turn. `--fail-input-overflow` refuses a turn that would not fit the context window instead of trimming it.
|
| 227 |
|
| 228 |
## System facts
|