| --- |
| language: en |
| library_name: pytorch |
| license: apache-2.0 |
| tags: |
| - browser-automation |
| - web-agents |
| - decision-model |
| - mind2web |
| - laya |
| base_model: convaiinnovations/laya |
| --- |
| |
| # Laya Fine-Tuned for Browser Automation (Mind2Web) |
|
|
| This is a fine-tuned version of the [Laya](https://huggingface.co/convaiinnovations/laya) decision model, specialized for browser automation tasks. It predicts the next browser action (operation and target element) from a web page's DOM structure and a user goal. |
|
|
| ## Model Details |
|
|
| - **Developed by:** Shaun Andrade ([@ShaunSpark](https://huggingface.co/ShaunSpark)) |
| - **Base model:** [convaiinnovations/laya](https://huggingface.co/convaiinnovations/laya) (421M parameters) |
| - **Training data:** [Mind2Web](https://huggingface.co/datasets/osunlp/Mind2Web) (671 usable training examples) |
| - **Validation accuracy:** 74.3% on a held-out set of 68 examples |
|
|
| ## Training Procedure |
|
|
| - **Hardware:** Google Colab, single NVIDIA T4 GPU (16 GB VRAM) |
| - **Optimizer:** 8-bit AdamW (bitsandbytes), learning rate 1e-5 |
| - **Effective batch size:** 8 (micro-batch 1, gradient accumulation 8) |
| - **Precision:** FP16 mixed precision (torch.cuda.amp) |
| - **Memory optimizations:** Gradient checkpointing, expandable_segments |
| - **Epochs:** 2 (best checkpoint; further training overfits) |
| |
| ## Intended Use |
| |
| Decision engine for a browser automation agent. Given a goal and a list of interactive DOM elements, the model answers two questions in one forward pass: |
| |
| - **Operation:** CLICK, TYPE, or SELECT |
| - **Target:** the correct element index from the candidate list |
| |
| ## Limitations |
| |
| - Trained on a single dataset (Mind2Web); may not generalize to all websites. |
| - Overfits beyond 2 epochs on 671 examples. |
| - Requires the base Laya tokenizer and architecture. |
| |
| ## How to Use |
| |
| ```python |
| import torch |
| import laya |
| |
| agent = laya.load("convaiinnovations/laya") |
| model = agent.model |
| model.load_state_dict(torch.load("pytorch_model.bin", map_location="cpu")) |
| model.eval() |
| |
| ``` |
| |
| ## Citation |
| |
| - Laya: https://huggingface.co/convaiinnovations/laya |
| - Mind2Web: https://arxiv.org/abs/2306.06070 |
| |
