adamlu1 commited on
Commit
d10c468
·
verified ·
1 Parent(s): 9c41f16

Credit MultimediaTechLab/YOLO (MIT) as the base implementation for icon_detect_v3

Browse files
Files changed (1) hide show
  1. README.md +5 -3
README.md CHANGED
@@ -11,7 +11,7 @@ tags:
11
  OmniParser is a general screen parsing tool, which interprets/converts UI screenshot to structured format, to improve existing LLM based UI agent.
12
  Training Datasets include: 1) an interactable icon detection dataset, which was curated from popular web pages and automatically annotated to highlight clickable and actionable regions, and 2) an icon description dataset, designed to associate each UI element with its corresponding function.
13
 
14
- This model hub includes a finetuned version of YOLOv8 and a finetuned Florence-2 base model on the above dataset respectively. It additionally includes `icon_detect_v3`, an MIT licensed interactable region detector finetuned from YOLOv9-E (see [New: MIT licensed detector weights](#new-mit-licensed-detector-weights-icon_detect_v3)). For more details of the models used and finetuning, please refer to the [paper](https://arxiv.org/abs/2408.00203).
15
 
16
  # What's new in V2?
17
  - Larger and cleaner set of icon caption + grounding dataset
@@ -23,6 +23,8 @@ This model hub includes a finetuned version of YOLOv8 and a finetuned Florence-2
23
  # New: MIT licensed detector weights (`icon_detect_v3`)
24
  The original `icon_detect` weights are finetuned from YOLOv8 and are therefore distributed under the AGPL-3.0 license, which is a blocker for many downstream users. We now additionally release `icon_detect_v3/model.pt`, an interactable region detector finetuned from YOLOv9-E on the same detection data and released under the **MIT license** (see `icon_detect_v3/LICENSE`).
25
 
 
 
26
  Pairing `icon_detect_v3` with `icon_caption` gives a fully MIT licensed OmniParser pipeline. `icon_detect` is still available and unchanged for users who are fine with AGPL-3.0.
27
 
28
  The weights are shipped as a self-contained TorchScript module, so inference only requires `torch` (no `ultralytics` and no AGPL licensed code at runtime):
@@ -106,8 +108,8 @@ Licenses differ per model folder, please refer to the LICENSE file in the folder
106
 
107
  | Folder | Model | License |
108
  | --- | --- | --- |
109
- | `icon_detect` | YOLOv8 based interactable region detector | AGPL-3.0 |
110
- | `icon_detect_v3` | YOLOv9-E based interactable region detector | MIT |
111
  | `icon_caption` | Florence-2 based icon captioner | MIT |
112
 
113
  Using `icon_detect_v3` together with `icon_caption` yields an OmniParser pipeline that is entirely MIT licensed.
 
11
  OmniParser is a general screen parsing tool, which interprets/converts UI screenshot to structured format, to improve existing LLM based UI agent.
12
  Training Datasets include: 1) an interactable icon detection dataset, which was curated from popular web pages and automatically annotated to highlight clickable and actionable regions, and 2) an icon description dataset, designed to associate each UI element with its corresponding function.
13
 
14
+ This model hub includes a finetuned version of YOLOv8 and a finetuned Florence-2 base model on the above dataset respectively. It additionally includes `icon_detect_v3`, an MIT licensed interactable region detector finetuned from YOLOv9-E using the MIT licensed [MultimediaTechLab/YOLO](https://github.com/MultimediaTechLab/YOLO) implementation (see [New: MIT licensed detector weights](#new-mit-licensed-detector-weights-icon_detect_v3)). For more details of the models used and finetuning, please refer to the [paper](https://arxiv.org/abs/2408.00203).
15
 
16
  # What's new in V2?
17
  - Larger and cleaner set of icon caption + grounding dataset
 
23
  # New: MIT licensed detector weights (`icon_detect_v3`)
24
  The original `icon_detect` weights are finetuned from YOLOv8 and are therefore distributed under the AGPL-3.0 license, which is a blocker for many downstream users. We now additionally release `icon_detect_v3/model.pt`, an interactable region detector finetuned from YOLOv9-E on the same detection data and released under the **MIT license** (see `icon_detect_v3/LICENSE`).
25
 
26
+ The model is finetuned from YOLOv9-E using the [MultimediaTechLab/YOLO](https://github.com/MultimediaTechLab/YOLO) implementation, which is MIT licensed. This is what makes the permissive relicensing possible: unlike the Ultralytics YOLOv8 codebase behind `icon_detect`, neither the training code nor the resulting weights carry AGPL-3.0 obligations.
27
+
28
  Pairing `icon_detect_v3` with `icon_caption` gives a fully MIT licensed OmniParser pipeline. `icon_detect` is still available and unchanged for users who are fine with AGPL-3.0.
29
 
30
  The weights are shipped as a self-contained TorchScript module, so inference only requires `torch` (no `ultralytics` and no AGPL licensed code at runtime):
 
108
 
109
  | Folder | Model | License |
110
  | --- | --- | --- |
111
+ | `icon_detect` | YOLOv8 based interactable region detector ([Ultralytics](https://github.com/ultralytics/ultralytics)) | AGPL-3.0 |
112
+ | `icon_detect_v3` | YOLOv9-E based interactable region detector ([MultimediaTechLab/YOLO](https://github.com/MultimediaTechLab/YOLO), MIT) | MIT |
113
  | `icon_caption` | Florence-2 based icon captioner | MIT |
114
 
115
  Using `icon_detect_v3` together with `icon_caption` yields an OmniParser pipeline that is entirely MIT licensed.