--- title: SEER emoji: 👁 colorFrom: green colorTo: yellow sdk: gradio sdk_version: 5.6.0 app_file: app.py pinned: false license: cc-by-nc-nd-4.0 --- # SEER: Enhancing Multimodal Action Grounding with Semantic UI Parsing ## Overview **SEER** (Semantic Extraction for Enhanced Reasoning) enhances GPT-4V's capability to interact with graphical user interfaces (GUIs) by extracting semantic meanings from UI elements and identifying interactable regions. This enables region-grounded actions, making multimodal environments more intuitive and interactive. SEER pushes the boundaries of multimodal AI by combining vision-based UI parsing and advanced machine learning techniques to create structured, actionable insights from GUI screenshots. --- ## Features - **Semantic Parsing**: Extracts UI elements' semantic meanings, categorizing buttons, icons, and text regions. - **Interactive Region Detection**: Accurately identifies clickable or interactable regions. - **Local Semantics Integration**: Enriches region data with descriptive functionality to improve context comprehension. - **Region-Grounded Actions**: Enables action generation contextualized to GUI regions. - **Gradio Demo**: Provides an interactive interface for testing SEER's capabilities. --- ## Methodology Highlights SEER is designed to decompose complex tasks into structured steps, leveraging multiple components to alleviate computational burdens on GPT-4V and enhance decision-making accuracy. ### 1. **Screen Parsing** SEER integrates outputs from three key components to produce a structured, DOM-like representation of the UI, overlayed with bounding boxes for interactable elements: - **Finetuned Interactable Icon Detection Model** - **Finetuned Icon Description Model** - **OCR Module** This parsing process simplifies GPT-4V's tasks by focusing on semantic and functional information extraction. ### 2. **Interactable Region Detection** Identifying interactable regions is a foundational step: - A custom dataset of **67k UI screenshots** was curated, with bounding boxes derived from DOM trees of public web pages. - Bounding boxes from interactable region detection and OCR modules are merged while minimizing overlaps (overlap threshold > 90%). - Each region is assigned a unique ID to facilitate precise action mapping. ### 3. **Local Semantics of Functionality** To improve understanding of UI elements: - A dataset of **7k icon-description pairs** was curated using GPT-4o and used to finetune a BLIP-v2 model. - This finetuned model generates accurate descriptions of icon functionality. - The descriptions and detected texts are integrated into prompts alongside the UI screenshot. By incorporating local semantics, SEER addresses limitations in GPT-4V's ability to simultaneously identify semantic information and predict actions. This integration significantly enhances its performance in multimodal tasks. --- ## Installation 1. **Create and activate the Python environment:** ```bash conda create -n "seer" python=3.12 conda activate seer pip install -r requirements.txt ``` 2. **Download pre-trained model checkpoints:** Access models from [HuggingFace](https://huggingface.co/strikerhell/SEER-model) and place them in the appropriate directories: - `weights/icon_detect/` - `weights/icon_caption_florence/` - `weights/icon_caption_blip2/` 3. **Convert safetensor files to PyTorch format:** ```bash python weights/convert_safetensor_to_pt.py ``` --- ## Usage ### Run Gradio Demo Test SEER's capabilities with the Gradio-powered demo: ```bash python gradio_demo.py ``` ### Example Notebook Explore SEER with sample use cases provided in `demo.ipynb`. --- ## Project Details ### Pipeline Components - **UI Parsing Model**: Decomposes screenshots into actionable semantic data. - **Interactive Region Detection**: Recognizes clickable and functional UI areas. - **Action Grounding Framework**: Maps detected actions to the corresponding GUI elements. - **Local Semantics Integration**: Enriches UI parsing with descriptive labels for improved task performance. --- ## Contributing Feel free to fork the repository, make changes, and submit pull requests. Contributions are welcome! ## License This project is licensed under the CC 4.0 License. See the [LICENSE](LICENSE) file for details. ## Author **Pirate-Emperor** [![Twitter](https://skillicons.dev/icons?i=twitter)](https://twitter.com/PirateKingRahul) [![Discord](https://skillicons.dev/icons?i=discord)](https://discord.com/users/1200728704981143634) [![LinkedIn](https://skillicons.dev/icons?i=linkedin)](https://www.linkedin.com/in/piratekingrahul) [![Reddit](https://img.shields.io/badge/Reddit-FF5700?style=for-the-badge&logo=reddit&logoColor=white)](https://www.reddit.com/u/PirateKingRahul) [![Medium](https://img.shields.io/badge/Medium-42404E?style=for-the-badge&logo=medium&logoColor=white)](https://medium.com/@piratekingrahul) - GitHub: [Pirate-Emperor](https://github.com/Pirate-Emperor) - Reddit: [PirateKingRahul](https://www.reddit.com/u/PirateKingRahul/) - Twitter: [PirateKingRahul](https://twitter.com/PirateKingRahul) - Discord: [PirateKingRahul](https://discord.com/users/1200728704981143634) - LinkedIn: [PirateKingRahul](https://www.linkedin.com/in/piratekingrahul) - Skype: [Join Skype](https://join.skype.com/invite/yfjOJG3wv9Ki) - Medium: [PirateKingRahul](https://medium.com/@piratekingrahul) Thank you for visiting the SEER project! ---