---
title: DeskForge Live
emoji: 🖱️
colorFrom: yellow
colorTo: green
sdk: gradio
sdk_version: 6.29.1
app_file: app.py
short_description: Give DeskForge a web task, or watch it race its base model
python_version: "3.12"
startup_duration_timeout: 30m
license: apache-2.0
models:
  - docling-project/DeskForge-Qwen3.5-4B
  - docling-project/DeskForge-Gemma4-E4B
  - Qwen/Qwen3.5-9B
  - Qwen/Qwen3.5-4B
  - google/gemma-4-E4B-it
datasets:
  - docling-project/DeskForge-1M
tags:
  - computer-use
  - gui-agent
  - gui-grounding
  - web-agent
---

# DeskForge Live

Live demo of **DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents**
([paper](https://arxiv.org/abs/2610.02320) · [project page](https://saidgurbuz.github.io/deskforge/) ·
[code](https://github.com/Saidgurbuz/deskforge) · [DeskForge-1M dataset](https://huggingface.co/datasets/docling-project/DeskForge-1M)).
It has three views:

**Web agent.** Type any task (or pick a suggestion) and watch a planner–grounder pair work in a real Chromium
running in this Space: [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) reads each screenshot and names one
action, describing its target in words, and
[DeskForge-Qwen3.5-4B](https://huggingface.co/docling-project/DeskForge-Qwen3.5-4B) finds the exact point to click.
Each step is one ZeroGPU call. (The paper's long-horizon study planned with Qwen3.6-27B; on our demo tasks the 9B
planner did as well and is about four times faster per step.)

**Live duel.** A real Xfce desktop (Xvfb) runs inside this Space. At each step of a challenge, a DeskForge model and
the base model it was fine-tuned from both see the same screenshot and predict a click. Pick the pair from
a dropdown: [DeskForge-Qwen3.5-4B](https://huggingface.co/docling-project/DeskForge-Qwen3.5-4B) vs.
[Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B), or
[DeskForge-Gemma4-E4B](https://huggingface.co/docling-project/DeskForge-Gemma4-E4B) vs.
[Gemma4-E4B](https://huggingface.co/google/gemma-4-E4B-it). A referee scores each prediction against the app's real
accessibility tree (AT-SPI), and DeskForge's click is executed live. **Theme roulette** restyles the
desktop between steps (GTK theme, fonts, DPI, panel position, wallpaper) to show the robustness
that DeskForge-1M's appearance presets buy.

**Try a screenshot.** Upload a screenshot and an instruction to see where DeskForge would click. Its `predict`
API/MCP endpoint is kept.

Every GPU step uses the visitor's own daily [ZeroGPU](https://huggingface.co/docs/hub/spaces-zerogpu) quota;
signing in to Hugging Face gives more.

## How it works

| File | Role |
| --- | --- |
| `deskforge.py` | Loads both families' DeskForge and base models (one ZeroGPU call grounds with a pair); parses the pyautogui reply with `ast` (never `exec`) and replays it with xdotool |
| `desktop_local.py` | Starts Xvfb + xfwm4/xfdesktop/xfce4-panel, resets scenes, applies themes |
| `referee.py` + `a11y_dump.py` | Finds the true on-screen box of a target through AT-SPI (system Python + python3-gi) |
| `duel.py` | Challenges, scoring, and the step loop; on a DeskForge miss the referee clicks the real target, so both models always face the same screen |
| `web_plan.py` | The web planner's prompt, its JSON action format and checks, and the demo's safety rules |
| `web_models.py` | Loads the planner (Qwen3.5-9B) next to the duel models; one ZeroGPU call plans a step and grounds its targets |
| `web_browser.py` | One Chromium (Playwright), a private context per run, and the request guard |
| `web_agent.py` + `web_tab.py` | Runs, limits and the step loop; the Web agent view (the page drives each step and animates it) |
| `gpu_bridge.py` | Runs the views' GPU work as real Gradio events, so ZeroGPU charges each visitor's own quota |

## Safety

- **Do not add any secrets to this Space.** Desktop apps run as the same user as the app, so a
  determined visitor could open files such as `/proc/*/environ` through a file dialog. The Space
  needs no secrets, so there is nothing to leak.
- Desktop programs get a scrubbed environment and a throwaway `HOME`. No terminal, browser or
  app finder is installed, and the bottom dock (terminal/browser launchers) is removed.
- One shared desktop: runs are queued (`concurrency_limit=1`) and the desktop is reset before each run.
- The "all visitors" tally lives in memory and resets when the Space restarts.
- Web agent: Chromium runs as an unprivileged user, with a fresh private context per run that is closed afterwards.
  Requests to private or loopback addresses are blocked, as are sign-in and payment pages and form posts that
  navigate; typing email addresses or long numbers is refused; the planner is told never to sign in, buy, book,
  post or send anything. Bing's SafeSearch is always Strict, and pages on domains that Cloudflare's family
  resolver flags as adult or malware are not opened. Sites that tell AI agents not to use them (Amazon) are never
  opened, and the agent is told to leave any
  site that says so. Runs stop after 30 steps, 8 minutes, or six steps without any change on screen, and at
  most four run at once.

## Acknowledgements

Special thanks to [@multimodalart](https://huggingface.co/multimodalart) at Hugging Face, whose
[DeskForge GUI Grounding](https://huggingface.co/spaces/hugging-apps/deskforge-gui-grounding) Space this demo
started from and builds on.