husky8 commited on
Commit
ded2ef4
·
verified ·
1 Parent(s): d3047b2

Embed the benchmark film and rank five verified browser tasks

Browse files
Files changed (1) hide show
  1. README.md +13 -16
README.md CHANGED
@@ -8,27 +8,24 @@ tags:
8
  - browser-automation
9
  ---
10
 
11
- # Inkling 2 Mini Fast Browser Use
12
 
13
- <!-- ordered-browser-release -->
14
- **8.95× faster on WebArena’s completed-order lookup.**
15
 
16
- WebArena task 190 asks for the total of the most recent completed order. The table shows the faster of two paired runs on an M5 Max MacBook Pro. Both models made the same clicks, returned $65.32 and earned full credit.
17
 
18
- | WebArena 190 · Completed-order lookup | Original Inkling 2 Mini | Edited Inkling 2 Mini |
19
- |---|---:|---:|
20
- | Total task time | 45.89 s | 5.13 s |
21
- | Model inference time | 43.03 s | 2.84 s |
22
- | Generated tokens | 1,555 | 28 |
23
- | Thinking tokens | 1,509 | 0 |
 
24
 
25
- **Weight-space programming.** We edited four rows in Inkling 2 Mini’s output matrix, `model.llm.unembed.weight`, without additional training. We swapped the rows for answer text (`200004`) and thinking (`200008`), then adjusted the rows for the click command `C` (`34`) and end of generation (`200006`). A byte comparison found 9,192 changed BF16 values across those four rows. [Weight audit](https://github.com/jtc268/inkling-2-fast-computer-use/blob/main/evidence/webarena-order-checkpoints/complete-cross/weight-audit.json)
26
 
27
- **Method.** Both checkpoints ran in MLX and read visible text and numbered controls in Brave. We reset the storefront, warmed each model and used the same prompt, greedy decoding, reasoning effort (`none`) and cursor timing. Total time includes inference and browser actions. We developed the edit on this task and reversed model order for the second run.
28
 
29
- [Watch the demo](https://jtc268.github.io/inkling-2-fast-computer-use/) · [Download the tested weights](https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Use/tree/686faa4d1748f0120008ec007dc97677678e1643) · [Reproduce the run](https://github.com/jtc268/inkling-2-fast-computer-use/blob/faa71279f5bb3a64f6e37546c3664288c83f7f64/benchmark/REPRODUCE-BROWSER.md) · [Paired runs and grades](https://github.com/jtc268/inkling-2-fast-computer-use/blob/main/benchmark/WEBARENA-ORDER-CHECKPOINTS.md)
30
- <!-- /ordered-browser-release -->
31
-
32
- The default download contains the earlier two-row form-and-cart edit. [Accuracy results](https://github.com/jtc268/inkling-2-fast-computer-use#browsergym-accuracy)
33
 
34
  Base model by Thinking Machines Lab. [Inkling 2 Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) · [Apache 2.0 and modification notice](https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Use/blob/main/NOTICE)
 
8
  - browser-automation
9
  ---
10
 
11
+ <video controls playsinline preload="metadata" width="100%" poster="https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Use/resolve/d3047b233dd402005cca0bfecb000798a1df5fc0/benchmark/browser-speedups-poster.jpg" src="https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Use/resolve/d3047b233dd402005cca0bfecb000798a1df5fc0/benchmark/browser-speedups.mp4"></video>
12
 
13
+ ## Five fastest browser tasks
 
14
 
15
+ Fastest verified run per task on an M5 Max MacBook Pro. Both models earned full credit. Edited times link to the weights used for each run.
16
 
17
+ | Task | Original Inkling 2 Mini | Edited Inkling 2 Mini | Speedup |
18
+ |---|---:|---:|---:|
19
+ | Latest completed order total · WebArena 190 | 45.89 s | [5.13 s](https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Use/tree/686faa4d1748f0120008ec007dc97677678e1643) | **8.95×** |
20
+ | Latest canceled order total · WebArena 188 | 45.75 s | [6.44 s](https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Use/tree/cd27dd36403a165685abd5bf202c326ec9590b20) | **7.10×** |
21
+ | Latest pending order total · WebArena 189 | 33.13 s | [5.25 s](https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Use/tree/686faa4d1748f0120008ec007dc97677678e1643) | **6.31×** |
22
+ | Complete an autocomplete field · MiniWoB++ 103 | 19.86 s | [3.84 s](https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Use/tree/d4562716ff6d8d0281bf09814a1df0db570f6be3) | **5.18×** |
23
+ | Reply to an email · MiniWoB++ 104 | 26.01 s | [5.98 s](https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Use/tree/d4562716ff6d8d0281bf09814a1df0db570f6be3) | **4.35×** |
24
 
25
+ On the completed-order lookup, inference fell from 43.03 s to 2.84 s. The original model generated 1,509 thinking tokens. The edit generated zero. Both made the same clicks and returned $65.32.
26
 
27
+ **Weight-space programming.** We edited Inkling 2 Mini’s output matrix (`model.llm.unembed.weight`) without training. The order checkpoints swap the answer-text and thinking rows (`200004` and `200008`) and adjust the click-command and end-of-generation rows (`34` and `200006`). The completed-order edit changed 9,192 BF16 values. The autocomplete and email runs used the two-row swap available on `main`. [Weight audit](https://github.com/jtc268/inkling-2-fast-computer-use/blob/main/evidence/webarena-order-checkpoints/complete-cross/weight-audit.json)
28
 
29
+ **Method.** Both models ran in MLX and read visible text and numbered controls in Brave. Each pair used the same task, initial page, prompt, greedy decoding, reasoning effort (`none`) and cursor timing. Task time includes inference and browser actions after warmup. We developed the order edits on order-lookup tasks. [WebArena runs](https://github.com/jtc268/inkling-2-fast-computer-use/blob/main/benchmark/WEBARENA-ORDER-CHECKPOINTS.md) · [MiniWoB++ runs](https://github.com/jtc268/inkling-2-fast-computer-use/blob/main/benchmark/HEADLINERS.md) · [Reproduction instructions](https://github.com/jtc268/inkling-2-fast-computer-use/blob/main/benchmark/REPRODUCE-BROWSER.md)
 
 
 
30
 
31
  Base model by Thinking Machines Lab. [Inkling 2 Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) · [Apache 2.0 and modification notice](https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Use/blob/main/NOTICE)