Instructions to use infinitylogesh/GUI-Decisions-2B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use infinitylogesh/GUI-Decisions-2B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="infinitylogesh/GUI-Decisions-2B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("infinitylogesh/GUI-Decisions-2B")
model = AutoModelForMultimodalLM.from_pretrained("infinitylogesh/GUI-Decisions-2B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use infinitylogesh/GUI-Decisions-2B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "infinitylogesh/GUI-Decisions-2B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "infinitylogesh/GUI-Decisions-2B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/infinitylogesh/GUI-Decisions-2B

SGLang

How to use infinitylogesh/GUI-Decisions-2B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "infinitylogesh/GUI-Decisions-2B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "infinitylogesh/GUI-Decisions-2B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "infinitylogesh/GUI-Decisions-2B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "infinitylogesh/GUI-Decisions-2B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use infinitylogesh/GUI-Decisions-2B with Docker Model Runner:
```
docker model run hf.co/infinitylogesh/GUI-Decisions-2B
```

infinitylogesh commited on 21 days ago

Commit

299888d

verified ·

1 Parent(s): 190c74a

card: same-session latency (lazy and full reads)

Browse files

Files changed (1) hide show

README.md +24 -8

README.md CHANGED Viewed

@@ -16,13 +16,15 @@ slots. The small-model result of the blog post
 The LoRA alone: [GUI-Decisions-2B-LoRA](https://huggingface.co/infinitylogesh/GUI-Decisions-2B-LoRA); the 31B version:
 [GUI-Decisions-31B](https://huggingface.co/infinitylogesh/GUI-Decisions-31B).
-## Results (blog post; vLLM, one RTX PRO 6000)
-|  | computer-use step (held-out AGUVIS, 1,600) | ScreenSpot clicks (1,272) | time per step |
-| --- | --- | --- | --- |
-| **GUI-Owl-2B + the LoRA** | **0.626** | **0.770** | **133 ms** |
-| GUI-Owl-2B (its own tool-call format), native resolution | 0.626 | 0.645 | 444 ms |
-| GUI-Owl-2B, same 1 MP images | 0.610 | 0.626 | 225 ms |
 Vanilla GUI-Owl's click numbers use its agent prompt, not its dedicated grounding prompt.
@@ -30,12 +32,26 @@ This merged model, read in-process with transformers (bf16): ScreenSpot clicks 0
 LoRA's 0.770 on the same cases (paired difference −0.006 [−0.013, +0.001], not significant; the unmerged
 LoRA read the same way scores 0.764), and computer-use steps 0.627 on the held-out AGUVIS set.
 ## Use
 ### With vLLM (recommended)
-Served on one RTX PRO 6000 a computer-use step takes **83 ms** (median over 60 cold screenshots, through
-the code below; the LoRA on the base model: 87 ms).
 **1. Install** vLLM 0.30 and the reader (`slotread`, from the code repository):

 The LoRA alone: [GUI-Decisions-2B-LoRA](https://huggingface.co/infinitylogesh/GUI-Decisions-2B-LoRA); the 31B version:
 [GUI-Decisions-31B](https://huggingface.co/infinitylogesh/GUI-Decisions-31B).
+## Results
+Accuracy (blog post):
+|  | computer-use step (held-out AGUVIS, 1,600) | ScreenSpot clicks (1,272) |
+| --- | --- | --- |
+| **GUI-Owl-2B + the LoRA** | **0.626** | **0.770** |
+| GUI-Owl-2B (its own tool-call format), native resolution | 0.626 | 0.645 |
+| GUI-Owl-2B, same 1 MP images | 0.610 | 0.626 |
 Vanilla GUI-Owl's click numbers use its agent prompt, not its dedicated grounding prompt.
 LoRA's 0.770 on the same cases (paired difference −0.006 [−0.013, +0.001], not significant; the unmerged
 LoRA read the same way scores 0.764), and computer-use steps 0.627 on the held-out AGUVIS set.
+Latency of one computer-use step: vLLM 0.30, one RTX PRO 6000, cold screenshots, one request at a time,
+all conditions in one session (`scripts/gui_owl.py bench 80`):
+| how the step is decided | median (p90) |
+| --- | --- |
+| **slots, lazy**: `action`, `x`, `y`, then only the slots the action needs (what `decide` does) | **65 ms** (71) |
+| slots, the whole template (8 slots) | 93 ms (104) |
+| GUI-Owl-2B writing its tool call (~50 tokens), same 1 MP images | 194 ms (217) |
+| GUI-Owl-2B writing its tool call (~50 tokens), native resolution | 323 ms (391) |
+The blog's earlier session measured 133, 225 and 444 ms for the last three rows on another box; the
+ratios agree (whole-template slots at ~30% of native-resolution GUI-Owl). End to end through `decide`,
+including the client's JPEG encoding, a step takes ~85 ms.
 ## Use
 ### With vLLM (recommended)
+Served on one RTX PRO 6000 a computer-use step takes ~83 ms end to end through the code below (median over
+60 cold screenshots, client JPEG encoding included; the LoRA on the base model: ~87 ms).
 **1. Install** vLLM 0.30 and the reader (`slotread`, from the code repository):