Trio-Spark v1.4: unversioned model ID, current benchmarks

README.md

@@ -19,13 +19,13 @@ license_link: https://machinefi.com/terms-of-service
 
 # Trio-Spark
 
-**Current release: Trio-Spark v1.3 · October 15, 2026**
+**Current release: Trio-Spark v1.4 · October 28, 2026**
 
 Trio-Spark is MachineFi's hosted **Situated World Model** for bounded decisions. Give it the current state and 2–8 allowed actions. One pass returns a selected action and a probability for every option, with no free-form answer to parse.
 
 **This repository is a model card for the commercial API. Model weights and training code are not distributed here.**
 
-[Try the playground](https://platform.machinefi.com/spark) · [Get an API key](https://platform.machinefi.com/login?callback=%2Fspark) · [API docs](https://platform.machinefi.com/spark/docs) · [Demos and clients](https://github.com/machinefi/trio-spark) · [v1.3 release post](https://machinefi.com/blog/trio-spark-v13-decisions)
+[Try the playground](https://platform.machinefi.com/spark) · [Get an API key](https://platform.machinefi.com/login?callback=%2Fspark) · [API docs](https://machinefi.com/docs) · [Pricing](https://machinefi.com/pricing) · [Demos and clients](https://github.com/machinefi/trio-spark) · [v1.4 release post](https://machinefi.com/blog/trio-spark-v14-decisions)
 
 ## What it does
 
@@ -37,7 +37,7 @@ environment signals → current state → Trio-Spark → selected action → exe
 
 Your application supplies the state, legal actions, executor and safety controls. Spark handles the bounded judgment step: routing, tool selection, GUI actions, game moves, robot skills and operational decisions.
 
-Input can be structured text or JSON, one image or camera frame, or a 2–4-frame sampled video window. v1.3 updates text decisions; image and sampled-video behavior is unchanged. The API accepts 2–8 choices and at most 1,024 expanded input tokens, including task, state, choices and any visual input.
+Input can be structured text or JSON, one image or camera frame, or a 2–4-frame sampled video window. In v1.4 one model serves text, images and video. The API accepts 2–8 choices and at most 1,024 expanded input tokens, including task, state, choices and any visual input.
 
 ## API quickstart
 
@@ -49,7 +49,7 @@ curl https://platform.machinefi.com/api/spark/v1/decisions \
   -H "Content-Type: application/json" \
   -H "Idempotency-Key: $(uuidgen)" \
   -d '{
-    "model": "trio-spark-v1.3",
+    "model": "trio-spark",
     "task": "The user asked to export the current report as CSV. Choose the next safe UI action from the available controls.",
     "state": {
       "app": "Reports",
@@ -66,57 +66,46 @@ curl https://platform.machinefi.com/api/spark/v1/decisions \
 
 Use `choice_id` to identify the selection. The response includes the complete probability distribution, confidence, a `decide`, `review` or `abstain` disposition, usage and latency. Reuse the same idempotency key and identical request body when retrying an uncertain request.
 
-The recommended public model ID is **`trio-spark-v1.3`**. Older aliases remain compatible and route to the current served model; they do not pin historical weights. See the [API docs](https://platform.machinefi.com/spark/docs) for visual requests, System One compatibility and the full response schema.
+Use the model ID **`trio-spark`**. It always runs the current release, and each response reports the exact release in `model_version`. Earlier identifiers (`trio-spark-preview`, `trio-spark-v1.0` to `trio-spark-v1.3`) remain accepted and route to the current model; they do not pin historical weights. The [API docs](https://machinefi.com/docs) cover visual requests, errors, limits and the full response schema.
 
-## v1.3 evaluation snapshot
+## Benchmarks
 
-On a ten-task panel of **2,977 decisions**, v1.3 scored **90.9%** (2,706 correct), versus 83.1% for v1.0 and 83.0% for v1.2. Its expected calibration error on that panel, using 10 bins, was **0.019**.
+Trio-Spark v1.4 against three other fast decision models, on the same items for every model. Scores are accuracy percentages; the best score in each row is bold. Trio-Spark is best or tied for best on seven of thirteen.
 
-The following developer-run evaluation uses the same items for each model. Scores are accuracy percentages on the listed samples, not full-dataset or independently certified leaderboard scores. The ten-task total covers the first ten rows; the remaining three rows are additional evaluations.
-
-| Task (items) | Spark v1.3 | Spark v1.2 | Spark v1.0 | Jev 1.13 | Clef | Clef-Flash | Laya |
-| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
-| AG News (300) | 90.3 | 83.0 | 83.3 | 82.7 | 87.0 | 88.7 | 91.7 |
-| ARC-Easy (300) | 98.3 | 95.7 | 95.0 | 99.0 | 99.7 | 99.7 | 48.0 |
-| BoolQ (300) | 91.0 | 87.0 | 87.3 | 92.0 | 92.7 | 91.7 | 80.7 |
-| HellaSwag (300) | 93.7 | 80.7 | 80.3 | 95.0 | 98.0 | 98.7 | 28.3 |
-| PIQA (300) | 88.0 | 78.0 | 82.0 | 95.7 | 91.0 | 90.3 | 53.7 |
-| PubMedQA (300) | 81.0 | 70.3 | 67.3 | 77.0 | 77.7 | 79.7 | 57.0 |
-| RTE (277) | 94.2 | 88.8 | 89.2 | 91.7 | 91.7 | 91.0 | 83.0 |
-| SST-2 (300) | 95.3 | 93.7 | 94.0 | 95.3 | 94.3 | 96.3 | 91.0 |
-| TweetEval offensive (300) | 86.3 | 81.7 | 82.7 | 82.3 | 82.7 | 84.3 | 78.3 |
-| WinoGrande (300) | 91.0 | 72.0 | 70.7 | 89.3 | 93.7 | 98.0 | 50.3 |
-| SNLI (150) | 85.3 | 84.0 | 63.3 | 75.3 | 82.7 | 82.7 | 46.0 |
-| HelpSteer2 (0–4 score) (300) | 42.3 | 41.3 | 40.0 | 41.5 | 41.3 | 42.7 | 35.3 |
-| Banking77 (77-way)† (300) | 88.1 | 80.5 | — | 80.0 | 94.6 | 94.7 | 38.7 |
-| **Ten-task panel (2,977)** | **90.9** | **83.0** | **83.1** | **90.0** | **90.8** | **91.8** | **66.1** |
+| Benchmark (items) | Trio-Spark | Jev 1.14 | Clef-Flash | Clef |
+| --- | ---: | ---: | ---: | ---: |
+| AG News (300) | **90.7** | 82.7 | 88.7 | 87.0 |
+| SST-2 (300) | **96.3** | 95.3 | **96.3** | 94.3 |
+| TweetEval offensive (300) | **87.0** | 84.3 | 84.3 | 82.7 |
+| Banking77 (77-way)† (300) | 88.2 | 80.1 | **94.7** | 94.6 |
+| RTE (277) | **92.8** | 91.3 | 91.0 | 91.7 |
+| SNLI (150) | **88.7** | 76.0 | 82.7 | 82.7 |
+| PubMedQA (300) | **82.7** | 76.7 | 79.7 | 77.7 |
+| BoolQ (300) | 91.7 | 90.3 | 91.7 | **92.7** |
+| HelpSteer2 (0–4 score) (300) | **44.3** | 38.9 | 42.7 | 41.3 |
+| ARC-Easy (300) | 97.0 | **99.7** | **99.7** | **99.7** |
+| HellaSwag (300) | 94.7 | 94.7 | **98.7** | 98.0 |
+| PIQA (300) | 93.0 | **95.7** | 90.3 | 91.0 |
+| WinoGrande (300) | 89.7 | 89.3 | **98.0** | 93.7 |
 
 Method and limitations:
 
-- Spark was evaluated through production API code and its model server, one request per item. Requests rejected by the input limit or content filter count as wrong.
-- Peer results came from official APIs or the official SDK, recorded October 7–8, 2026. HelpSteer2 and Banking77 peer scores average five option orderings.
-- † Banking77 has 77 choices and was evaluated offline; it is not a supported 77-choice request to the public API. v1.0 was not measured on this row.
-- These numbers measure text decisions. They do not establish image, video, GUI-control or real-world task performance. Calibration on this panel does not guarantee calibrated confidence in a different application.
-
-See the [release post and protocol](https://machinefi.com/blog/trio-spark-v13-decisions) and [downloadable results](https://machinefi.com/spark/benchmarks/v13-launch-table-2026-10-15.json) for the complete published context.
-
-## Version history
-
-**v1.3 · October 15, 2026 (current).** Updated text decisions; image and sampled-video behavior unchanged. Ten-task panel 90.9% (see above). Public model ID `trio-spark-v1.3`.
+- Developer-run evaluation on the listed samples, not full-dataset or independently certified leaderboard scores. Trio-Spark was evaluated through the production API code path, one request per item; requests rejected by the input limit or content filter count as wrong.
+- Jev 1.14 through its official API on October 28, 2026; Clef and Clef-Flash recorded October 7–8, 2026. HelpSteer2 and Banking77 peer scores average five orderings of the options.
+- † Banking77 has 77 choices and was read out offline from the same weights; the public API accepts at most 8 choices per request.
+- These numbers measure text decisions. They do not establish image, video, GUI-control or real-world task performance.
 
-**v1.0 · September 30, 2026.** First release, served as `trio-spark-preview` (the alias now routes to the current model). Launch evaluation on the production API: SST-2 92.89%, AG News 88.46%, PubMedQA 85.33%, TweetEval 80.58% ([launch post](https://machinefi.com/blog/trio-spark-decisions-single-pass)). Replayable demo runs from the [public demo repository](https://github.com/machinefi/trio-spark):
+The [release post](https://machinefi.com/blog/trio-spark-v14-decisions) has the full protocol, and the [downloadable results](https://machinefi.com/spark/benchmarks/v14-launch-table-2026-10-28.json) include every count.
 
-| Environment | Result |
-| --- | --- |
-| Autonomous drone | 65 s, 25 calls, 75.7 m flown, 0 collisions, course completed |
-| Browser agent | 4.002 s, 3 calls, requested item selected, autonomous `DONE` |
-| Robot arm | 13.68 s, 13 calls, physical success, 0 forbidden contacts |
+## Releases
 
-These demos send structured environment signals to Spark; they do not claim direct control from raw pixels. The former `MachineFi/Trio-Spark-v1.0` card is merged into this page.
+- **v1.4 · October 28, 2026 (current).** One model for text, images and video; served as `trio-spark`.
+- **v1.3 · October 15, 2026.** Text decision update.
+- **v1.0 · September 30, 2026.** First release, with replayable [drone, browser and robot-arm demos](https://github.com/machinefi/trio-spark).
 
 ## Limitations and intended use
 
-v1.3 scores below v1.0 on MachineFi's internal GUI-action suite despite improving on the published text panel. Evaluate it on your own screens and actions before increasing automation. Latency depends on the request and workload; the evaluation table is not a latency guarantee.
+Evaluate Spark on your own screens, frames and actions before increasing automation. Latency depends on the request and workload; nothing on this card is a latency guarantee.
 
 Spark selects among supplied choices. It does not execute actions, operate a browser by itself, predict future frames, or provide general conversational answers. Include choices such as **wait** or **ask for review** when appropriate.
 
@@ -124,8 +113,8 @@ Applications remain responsible for permissions, legal-action filtering, hard sa
 
 ## Access and licensing
 
-The hosted API is a commercial service subject to [MachineFi's terms](https://machinefi.com/terms-of-service). Text and visual input cost **$0.042 per million billed input tokens**; output is free. New accounts receive 100 free decisions. Refer to the console for current access and billing details.
+The hosted API is a commercial service subject to [MachineFi's terms](https://machinefi.com/terms-of-service). Text and visual input cost **$0.042 per million billed input tokens**; output is free. New accounts receive 100 free decisions. See [pricing](https://machinefi.com/pricing) for current details.
 
 The [public Trio-Spark repository](https://github.com/machinefi/trio-spark) contains Apache-2.0 clients and demo integrations. That code license does not grant access to model weights or training artifacts. This Hugging Face repository does not provide downloadable weights or a local-inference package.
 
-Open research weights: [MachineFi/Trio-Spark-DI](https://huggingface.co/MachineFi/Trio-Spark-DI) is a separate, text-only research checkpoint for the Decision Index, released under CC BY-NC 4.0. It is not the model the API serves.
+Research weights: [MachineFi/Trio-Spark-DI-1.4](https://huggingface.co/MachineFi/Trio-Spark-DI-1.4) is a text-only Q8_0 build of the v1.4 model the API serves, released under CC BY-NC 4.0 for research and evaluation.