Spaces:
Running on Zero
Running on Zero
Plan with Qwen3.5-9B instead of Qwen3.6-27B-FP8: as good on the demo tasks, about 4x faster per step
3767612 verified | title: DeskForge Live | |
| emoji: 🖱️ | |
| colorFrom: yellow | |
| colorTo: green | |
| sdk: gradio | |
| sdk_version: 6.29.1 | |
| app_file: app.py | |
| short_description: Give DeskForge a web task, or watch it race its base model | |
| python_version: "3.12" | |
| startup_duration_timeout: 30m | |
| license: apache-2.0 | |
| models: | |
| - docling-project/DeskForge-Qwen3.5-4B | |
| - docling-project/DeskForge-Gemma4-E4B | |
| - Qwen/Qwen3.5-9B | |
| - Qwen/Qwen3.5-4B | |
| - google/gemma-4-E4B-it | |
| datasets: | |
| - docling-project/DeskForge-1M | |
| tags: | |
| - computer-use | |
| - gui-agent | |
| - gui-grounding | |
| - web-agent | |
| # DeskForge Live | |
| Live demo of **DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents** | |
| ([paper](https://arxiv.org/abs/2610.02320) · [project page](https://saidgurbuz.github.io/deskforge/) · | |
| [code](https://github.com/Saidgurbuz/deskforge) · [DeskForge-1M dataset](https://huggingface.co/datasets/docling-project/DeskForge-1M)). | |
| It has three views: | |
| **Web agent.** Type any task (or pick a suggestion) and watch a planner–grounder pair work in a real Chromium | |
| running in this Space: [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) reads each screenshot and names one | |
| action, describing its target in words, and | |
| [DeskForge-Qwen3.5-4B](https://huggingface.co/docling-project/DeskForge-Qwen3.5-4B) finds the exact point to click. | |
| Each step is one ZeroGPU call. (The paper's long-horizon study planned with Qwen3.6-27B; on our demo tasks the 9B | |
| planner did as well and is about four times faster per step.) | |
| **Live duel.** A real Xfce desktop (Xvfb) runs inside this Space. At each step of a challenge, a DeskForge model and | |
| the base model it was fine-tuned from both see the same screenshot and predict a click. Pick the pair from | |
| a dropdown: [DeskForge-Qwen3.5-4B](https://huggingface.co/docling-project/DeskForge-Qwen3.5-4B) vs. | |
| [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B), or | |
| [DeskForge-Gemma4-E4B](https://huggingface.co/docling-project/DeskForge-Gemma4-E4B) vs. | |
| [Gemma4-E4B](https://huggingface.co/google/gemma-4-E4B-it). A referee scores each prediction against the app's real | |
| accessibility tree (AT-SPI), and DeskForge's click is executed live. **Theme roulette** restyles the | |
| desktop between steps (GTK theme, fonts, DPI, panel position, wallpaper) to show the robustness | |
| that DeskForge-1M's appearance presets buy. | |
| **Try a screenshot.** Upload a screenshot and an instruction to see where DeskForge would click. Its `predict` | |
| API/MCP endpoint is kept. | |
| Every GPU step uses the visitor's own daily [ZeroGPU](https://huggingface.co/docs/hub/spaces-zerogpu) quota; | |
| signing in to Hugging Face gives more. | |
| ## How it works | |
| | File | Role | | |
| | --- | --- | | |
| | `deskforge.py` | Loads both families' DeskForge and base models (one ZeroGPU call grounds with a pair); parses the pyautogui reply with `ast` (never `exec`) and replays it with xdotool | | |
| | `desktop_local.py` | Starts Xvfb + xfwm4/xfdesktop/xfce4-panel, resets scenes, applies themes | | |
| | `referee.py` + `a11y_dump.py` | Finds the true on-screen box of a target through AT-SPI (system Python + python3-gi) | | |
| | `duel.py` | Challenges, scoring, and the step loop; on a DeskForge miss the referee clicks the real target, so both models always face the same screen | | |
| | `web_plan.py` | The web planner's prompt, its JSON action format and checks, and the demo's safety rules | | |
| | `web_models.py` | Loads the planner (Qwen3.5-9B) next to the duel models; one ZeroGPU call plans a step and grounds its targets | | |
| | `web_browser.py` | One Chromium (Playwright), a private context per run, and the request guard | | |
| | `web_agent.py` + `web_tab.py` | Runs, limits and the step loop; the Web agent view (the page drives each step and animates it) | | |
| | `gpu_bridge.py` | Runs the views' GPU work as real Gradio events, so ZeroGPU charges each visitor's own quota | | |
| ## Safety | |
| - **Do not add any secrets to this Space.** Desktop apps run as the same user as the app, so a | |
| determined visitor could open files such as `/proc/*/environ` through a file dialog. The Space | |
| needs no secrets, so there is nothing to leak. | |
| - Desktop programs get a scrubbed environment and a throwaway `HOME`. No terminal, browser or | |
| app finder is installed, and the bottom dock (terminal/browser launchers) is removed. | |
| - One shared desktop: runs are queued (`concurrency_limit=1`) and the desktop is reset before each run. | |
| - The "all visitors" tally lives in memory and resets when the Space restarts. | |
| - Web agent: Chromium runs as an unprivileged user, with a fresh private context per run that is closed afterwards. | |
| Requests to private or loopback addresses are blocked, as are sign-in and payment pages and form posts that | |
| navigate; typing email addresses or long numbers is refused; the planner is told never to sign in, buy, book, | |
| post or send anything. Bing's SafeSearch is always Strict, and pages on domains that Cloudflare's family | |
| resolver flags as adult or malware are not opened. Sites that tell AI agents not to use them (Amazon) are never | |
| opened, and the agent is told to leave any | |
| site that says so. Runs stop after 30 steps, 8 minutes, or six steps without any change on screen, and at | |
| most four run at once. | |
| ## Acknowledgements | |
| Special thanks to [@multimodalart](https://huggingface.co/multimodalart) at Hugging Face, whose | |
| [DeskForge GUI Grounding](https://huggingface.co/spaces/hugging-apps/deskforge-gui-grounding) Space this demo | |
| started from and builds on. | |