husky8 commited on
Commit
d3047b2
路
verified 路
1 Parent(s): f864ee6

Add paired inference timings, exact output rows and WebArena method

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -13,7 +13,7 @@ tags:
13
  <!-- ordered-browser-release -->
14
  **8.95脳 faster on WebArena鈥檚 completed-order lookup.**
15
 
16
- WebArena task 190 asks for the total of the most recent completed order. In the fastest paired run on an M5 Max MacBook Pro, both models made the same clicks, returned $65.32 and earned full credit.
17
 
18
  | WebArena 190 路 Completed-order lookup | Original Inkling 2 Mini | Edited Inkling 2 Mini |
19
  |---|---:|---:|
@@ -24,9 +24,9 @@ WebArena task 190 asks for the total of the most recent completed order. In the
24
 
25
  **Weight-space programming.** We edited four rows in Inkling 2 Mini鈥檚 output matrix, `model.llm.unembed.weight`, without additional training. We swapped the rows for answer text (`200004`) and thinking (`200008`), then adjusted the rows for the click command `C` (`34`) and end of generation (`200006`). A byte comparison found 9,192 changed BF16 values across those four rows. [Weight audit](https://github.com/jtc268/inkling-2-fast-computer-use/blob/main/evidence/webarena-order-checkpoints/complete-cross/weight-audit.json)
26
 
27
- **Method.** Both checkpoints ran in MLX and read visible text and numbered controls in Brave. We reset the storefront, warmed each model and used the same prompt, greedy decoding, reasoning effort (`none`) and cursor timing. Total time includes inference and browser actions. We developed the edit on this task and tested both run orders.
28
 
29
- [Watch the demo](https://jtc268.github.io/inkling-2-fast-computer-use/) 路 [Download the tested weights](https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Use/tree/686faa4d1748f0120008ec007dc97677678e1643) 路 [Reproduce the run](https://github.com/jtc268/inkling-2-fast-computer-use/blob/1177b0a576bba406f5c8fcd3224f5ca7ebf7d4a7/benchmark/REPRODUCE-BROWSER.md) 路 [Paired runs and grades](https://github.com/jtc268/inkling-2-fast-computer-use/blob/main/benchmark/WEBARENA-ORDER-CHECKPOINTS.md)
30
  <!-- /ordered-browser-release -->
31
 
32
  The default download contains the earlier two-row form-and-cart edit. [Accuracy results](https://github.com/jtc268/inkling-2-fast-computer-use#browsergym-accuracy)
 
13
  <!-- ordered-browser-release -->
14
  **8.95脳 faster on WebArena鈥檚 completed-order lookup.**
15
 
16
+ WebArena task 190 asks for the total of the most recent completed order. The table shows the faster of two paired runs on an M5 Max MacBook Pro. Both models made the same clicks, returned $65.32 and earned full credit.
17
 
18
  | WebArena 190 路 Completed-order lookup | Original Inkling 2 Mini | Edited Inkling 2 Mini |
19
  |---|---:|---:|
 
24
 
25
  **Weight-space programming.** We edited four rows in Inkling 2 Mini鈥檚 output matrix, `model.llm.unembed.weight`, without additional training. We swapped the rows for answer text (`200004`) and thinking (`200008`), then adjusted the rows for the click command `C` (`34`) and end of generation (`200006`). A byte comparison found 9,192 changed BF16 values across those four rows. [Weight audit](https://github.com/jtc268/inkling-2-fast-computer-use/blob/main/evidence/webarena-order-checkpoints/complete-cross/weight-audit.json)
26
 
27
+ **Method.** Both checkpoints ran in MLX and read visible text and numbered controls in Brave. We reset the storefront, warmed each model and used the same prompt, greedy decoding, reasoning effort (`none`) and cursor timing. Total time includes inference and browser actions. We developed the edit on this task and reversed model order for the second run.
28
 
29
+ [Watch the demo](https://jtc268.github.io/inkling-2-fast-computer-use/) 路 [Download the tested weights](https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Use/tree/686faa4d1748f0120008ec007dc97677678e1643) 路 [Reproduce the run](https://github.com/jtc268/inkling-2-fast-computer-use/blob/faa71279f5bb3a64f6e37546c3664288c83f7f64/benchmark/REPRODUCE-BROWSER.md) 路 [Paired runs and grades](https://github.com/jtc268/inkling-2-fast-computer-use/blob/main/benchmark/WEBARENA-ORDER-CHECKPOINTS.md)
30
  <!-- /ordered-browser-release -->
31
 
32
  The default download contains the earlier two-row form-and-cart edit. [Accuracy results](https://github.com/jtc268/inkling-2-fast-computer-use#browsergym-accuracy)