husky8 commited on
Commit
f864ee6
路
verified 路
1 Parent(s): fe490d9

Add paired inference timings, exact output rows and WebArena method

Browse files
Files changed (1) hide show
  1. README.md +12 -3
README.md CHANGED
@@ -13,11 +13,20 @@ tags:
13
  <!-- ordered-browser-release -->
14
  **8.95脳 faster on WebArena鈥檚 completed-order lookup.**
15
 
16
- In the fastest run on an M5 Max MacBook Pro, stock found the correct total in 45.89 seconds. The edited model took 5.13 seconds.
17
 
18
- We changed four rows of model weights directly, a method called weight-space programming. Inkling 2 Mini then made the same clicks without writing out its reasoning.
 
 
 
 
 
19
 
20
- [Watch the demo](https://jtc268.github.io/inkling-2-fast-computer-use/) 路 [Download the tested weights](https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Use/tree/686faa4d1748f0120008ec007dc97677678e1643) 路 [Reproduce the run](https://github.com/jtc268/inkling-2-fast-computer-use/blob/56b4ed941ce6174c26c000667135e99be69246df/benchmark/REPRODUCE-BROWSER.md) 路 [Results and weight audit](https://github.com/jtc268/inkling-2-fast-computer-use/blob/main/benchmark/WEBARENA-ORDER-CHECKPOINTS.md)
 
 
 
 
21
  <!-- /ordered-browser-release -->
22
 
23
  The default download contains the earlier two-row form-and-cart edit. [Accuracy results](https://github.com/jtc268/inkling-2-fast-computer-use#browsergym-accuracy)
 
13
  <!-- ordered-browser-release -->
14
  **8.95脳 faster on WebArena鈥檚 completed-order lookup.**
15
 
16
+ WebArena task 190 asks for the total of the most recent completed order. In the fastest paired run on an M5 Max MacBook Pro, both models made the same clicks, returned $65.32 and earned full credit.
17
 
18
+ | WebArena 190 路 Completed-order lookup | Original Inkling 2 Mini | Edited Inkling 2 Mini |
19
+ |---|---:|---:|
20
+ | Total task time | 45.89 s | 5.13 s |
21
+ | Model inference time | 43.03 s | 2.84 s |
22
+ | Generated tokens | 1,555 | 28 |
23
+ | Thinking tokens | 1,509 | 0 |
24
 
25
+ **Weight-space programming.** We edited four rows in Inkling 2 Mini鈥檚 output matrix, `model.llm.unembed.weight`, without additional training. We swapped the rows for answer text (`200004`) and thinking (`200008`), then adjusted the rows for the click command `C` (`34`) and end of generation (`200006`). A byte comparison found 9,192 changed BF16 values across those four rows. [Weight audit](https://github.com/jtc268/inkling-2-fast-computer-use/blob/main/evidence/webarena-order-checkpoints/complete-cross/weight-audit.json)
26
+
27
+ **Method.** Both checkpoints ran in MLX and read visible text and numbered controls in Brave. We reset the storefront, warmed each model and used the same prompt, greedy decoding, reasoning effort (`none`) and cursor timing. Total time includes inference and browser actions. We developed the edit on this task and tested both run orders.
28
+
29
+ [Watch the demo](https://jtc268.github.io/inkling-2-fast-computer-use/) 路 [Download the tested weights](https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Use/tree/686faa4d1748f0120008ec007dc97677678e1643) 路 [Reproduce the run](https://github.com/jtc268/inkling-2-fast-computer-use/blob/1177b0a576bba406f5c8fcd3224f5ca7ebf7d4a7/benchmark/REPRODUCE-BROWSER.md) 路 [Paired runs and grades](https://github.com/jtc268/inkling-2-fast-computer-use/blob/main/benchmark/WEBARENA-ORDER-CHECKPOINTS.md)
30
  <!-- /ordered-browser-release -->
31
 
32
  The default download contains the earlier two-row form-and-cart edit. [Accuracy results](https://github.com/jtc268/inkling-2-fast-computer-use#browsergym-accuracy)