Image-Text-to-Text
Transformers
Safetensors
inkling2_loop
weight-edit
browser-automation
conversational
8-bit precision
modelopt
Instructions to use husky8/Inkling-2-Mini-Fast-Browser-Use with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use husky8/Inkling-2-Mini-Fast-Browser-Use with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="husky8/Inkling-2-Mini-Fast-Browser-Use") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("husky8/Inkling-2-Mini-Fast-Browser-Use") model = AutoModelForMultimodalLM.from_pretrained("husky8/Inkling-2-Mini-Fast-Browser-Use", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use husky8/Inkling-2-Mini-Fast-Browser-Use with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "husky8/Inkling-2-Mini-Fast-Browser-Use" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "husky8/Inkling-2-Mini-Fast-Browser-Use", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/husky8/Inkling-2-Mini-Fast-Browser-Use
- SGLang
How to use husky8/Inkling-2-Mini-Fast-Browser-Use with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "husky8/Inkling-2-Mini-Fast-Browser-Use" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "husky8/Inkling-2-Mini-Fast-Browser-Use", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "husky8/Inkling-2-Mini-Fast-Browser-Use" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "husky8/Inkling-2-Mini-Fast-Browser-Use", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use husky8/Inkling-2-Mini-Fast-Browser-Use with Docker Model Runner:
docker model run hf.co/husky8/Inkling-2-Mini-Fast-Browser-Use
Add paired inference timings, exact output rows and WebArena method
Browse files
README.md
CHANGED
|
@@ -13,11 +13,20 @@ tags:
|
|
| 13 |
<!-- ordered-browser-release -->
|
| 14 |
**8.95脳 faster on WebArena鈥檚 completed-order lookup.**
|
| 15 |
|
| 16 |
-
In the fastest run on an M5 Max MacBook Pro,
|
| 17 |
|
| 18 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
<!-- /ordered-browser-release -->
|
| 22 |
|
| 23 |
The default download contains the earlier two-row form-and-cart edit. [Accuracy results](https://github.com/jtc268/inkling-2-fast-computer-use#browsergym-accuracy)
|
|
|
|
| 13 |
<!-- ordered-browser-release -->
|
| 14 |
**8.95脳 faster on WebArena鈥檚 completed-order lookup.**
|
| 15 |
|
| 16 |
+
WebArena task 190 asks for the total of the most recent completed order. In the fastest paired run on an M5 Max MacBook Pro, both models made the same clicks, returned $65.32 and earned full credit.
|
| 17 |
|
| 18 |
+
| WebArena 190 路 Completed-order lookup | Original Inkling 2 Mini | Edited Inkling 2 Mini |
|
| 19 |
+
|---|---:|---:|
|
| 20 |
+
| Total task time | 45.89 s | 5.13 s |
|
| 21 |
+
| Model inference time | 43.03 s | 2.84 s |
|
| 22 |
+
| Generated tokens | 1,555 | 28 |
|
| 23 |
+
| Thinking tokens | 1,509 | 0 |
|
| 24 |
|
| 25 |
+
**Weight-space programming.** We edited four rows in Inkling 2 Mini鈥檚 output matrix, `model.llm.unembed.weight`, without additional training. We swapped the rows for answer text (`200004`) and thinking (`200008`), then adjusted the rows for the click command `C` (`34`) and end of generation (`200006`). A byte comparison found 9,192 changed BF16 values across those four rows. [Weight audit](https://github.com/jtc268/inkling-2-fast-computer-use/blob/main/evidence/webarena-order-checkpoints/complete-cross/weight-audit.json)
|
| 26 |
+
|
| 27 |
+
**Method.** Both checkpoints ran in MLX and read visible text and numbered controls in Brave. We reset the storefront, warmed each model and used the same prompt, greedy decoding, reasoning effort (`none`) and cursor timing. Total time includes inference and browser actions. We developed the edit on this task and tested both run orders.
|
| 28 |
+
|
| 29 |
+
[Watch the demo](https://jtc268.github.io/inkling-2-fast-computer-use/) 路 [Download the tested weights](https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Use/tree/686faa4d1748f0120008ec007dc97677678e1643) 路 [Reproduce the run](https://github.com/jtc268/inkling-2-fast-computer-use/blob/1177b0a576bba406f5c8fcd3224f5ca7ebf7d4a7/benchmark/REPRODUCE-BROWSER.md) 路 [Paired runs and grades](https://github.com/jtc268/inkling-2-fast-computer-use/blob/main/benchmark/WEBARENA-ORDER-CHECKPOINTS.md)
|
| 30 |
<!-- /ordered-browser-release -->
|
| 31 |
|
| 32 |
The default download contains the earlier two-row form-and-cart edit. [Accuracy results](https://github.com/jtc268/inkling-2-fast-computer-use#browsergym-accuracy)
|