Instructions to use LaplAI/gua with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use LaplAI/gua with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("LaplAI/gua") config = load_config("LaplAI/gua") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use LaplAI/gua with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "LaplAI/gua"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "LaplAI/gua" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use LaplAI/gua with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "LaplAI/gua"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default LaplAI/gua
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use LaplAI/gua with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "LaplAI/gua"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "LaplAI/gua" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
gua v2: continue v1 on 12,000 balanced questions from macos-cua v3 (adds Logic Pro, OBS, Plasticity)
Browse files- README.md +90 -60
- adapter/adapters.safetensors +1 -1
- merge_check.json +1 -1
- model-00001-of-00002.safetensors +1 -1
- model-00002-of-00002.safetensors +1 -1
- temperatures.json +35 -35
README.md
CHANGED
|
@@ -22,7 +22,8 @@ on confident answers and defer on the rest.
|
|
| 22 |
|
| 23 |
It is [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) fine-tuned (supervised
|
| 24 |
fine-tuning with LoRA, rank 8 on every linear layer of the language model) on GUI
|
| 25 |
-
trajectories extracted from screen-recorded
|
|
|
|
| 26 |
LoRA merged into the base weights (MLX, bf16) and, in `adapter/`, the LoRA itself.
|
| 27 |
|
| 28 |
> Research model. The training data was derived from public YouTube tutorials and is not
|
|
@@ -79,41 +80,43 @@ Without it, only text candidates are offered, which lowers recall on icon-only c
|
|
| 79 |
|
| 80 |
## Training
|
| 81 |
|
|
|
|
|
|
|
| 82 |
| | |
|
| 83 |
|---|---|
|
| 84 |
| Method | Supervised fine-tuning with LoRA: rank 8, alpha 16, on every linear layer of the language model (`q/k/v/o_proj`, linear-attention `in_proj_*`/`out_proj`, MLP `gate/up/down_proj`); vision tower frozen |
|
| 85 |
| Objective | Log loss of the right option label (and end-of-turn token) only, never of the prompt |
|
| 86 |
-
|
|
| 87 |
-
|
|
| 88 |
-
|
|
| 89 |
-
|
| 90 |
-
The recipe was chosen on a held-out dev split, never on the eval split: on the same 2,000
|
| 91 |
-
samples, LoRA on every linear layer matched rank 64 and edged out attention-only LoRA, and
|
| 92 |
-
a full-parameter run of the same model on earlier data did not beat LoRA.
|
| 93 |
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|---|---|---|---|---|---|---|
|
| 98 |
-
| Dev score | 0.322 | 0.429 | 0.421 | 0.441 | 0.445 | 0.447 |
|
| 99 |
|
| 100 |
## Training data
|
| 101 |
|
| 102 |
Trajectories were extracted automatically from screen-recorded software tutorials on
|
| 103 |
YouTube: videos were searched and screened (recording quality, platform), segmented and
|
| 104 |
annotated into step-by-step actions by vision-language models, each click was located on
|
| 105 |
-
the frame, and before/after screenshots were cut. The **training split** of *macos-cua
|
|
|
|
| 106 |
|
| 107 |
-
| Topic | Videos | Steps |
|
| 108 |
-
|---|---|---|
|
| 109 |
-
|
|
| 110 |
-
|
|
| 111 |
-
|
|
| 112 |
-
|
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 117 |
would be wrong on a Mac (Windows shortcuts, taskbar, native dialogs) were removed.
|
| 118 |
|
| 119 |
**Label quality.** Labels are model-generated, not human-verified. A stronger model audited
|
|
@@ -121,67 +124,94 @@ a random sample per topic:
|
|
| 121 |
|
| 122 |
| Topic | Audited steps | Element located correctly | Action correct |
|
| 123 |
|---|---|---|---|
|
| 124 |
-
|
|
|
|
|
| 125 |
| macOS video editing | 395 | 92.4% | 78.2% |
|
| 126 |
-
|
|
|
|
|
|
|
|
| 127 |
| Godot | 270 | 87.0% | 54.8% |
|
| 128 |
|
| 129 |
The dataset itself is not released.
|
| 130 |
|
| 131 |
## Evaluation
|
| 132 |
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
|
| 137 |
-
| Question |
|
| 138 |
|---|---|---|---|
|
| 139 |
-
| `action` accuracy | 0.
|
| 140 |
-
| `grounding` accuracy | 0.
|
| 141 |
-
| `next_target` accuracy | 0.
|
| 142 |
-
| `action` and `next_target` both right | 0.
|
| 143 |
|
| 144 |
-
On **applications never seen in training**
|
| 145 |
-
|
| 146 |
-
0.221 → 0.313. A candidate covers the target in 83.4% of the element steps, which bounds
|
| 147 |
-
the element questions.
|
| 148 |
|
| 149 |
-
|
| 150 |
-
|
| 151 |
-
|
| 152 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 153 |
|
| 154 |
-
**Selective accuracy
|
| 155 |
-
|
| 156 |
|
| 157 |
-
**Calibration.** `temperatures.json` holds one temperature per question, fitted on the
|
| 158 |
-
split by log loss and applied as `p^(1/T)` renormalised (`inference/cua_decide.py
|
| 159 |
-
--temperatures`). Expected calibration error on eval, which the fit never saw:
|
| 160 |
|
| 161 |
| Question | Temperature | ECE before | ECE after |
|
| 162 |
|---|---|---|---|
|
| 163 |
-
| `action` | 2.
|
| 164 |
-
| `grounding` | 1.
|
| 165 |
-
| `next_target` |
|
| 166 |
|
| 167 |
-
|
| 168 |
-
|
| 169 |
-
while slightly raising ECE.
|
| 170 |
|
| 171 |
## Limitations
|
| 172 |
|
| 173 |
-
- **macOS screenshots only**, English UI and tasks
|
| 174 |
-
Blender.
|
| 175 |
- **Candidate-bound.** The model picks among candidates; if no candidate covers the target
|
| 176 |
-
(
|
| 177 |
-
|
|
|
|
|
|
|
|
|
|
| 178 |
- **Not an agent by itself.** It decides the action kind and target; the text to type, the
|
| 179 |
-
keys of a shortcut and drag end points are not predicted.
|
| 180 |
- **Merged weights round to bf16.** Against base plus adapter, option probabilities differ
|
| 181 |
-
by at most 0.
|
| 182 |
- Confidence is only meaningful after temperature scaling, and only for the question types
|
| 183 |
and candidate sources it was fitted with.
|
| 184 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 185 |
|
| 186 |
## Licence and provenance
|
| 187 |
|
|
|
|
| 22 |
|
| 23 |
It is [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) fine-tuned (supervised
|
| 24 |
fine-tuning with LoRA, rank 8 on every linear layer of the language model) on GUI
|
| 25 |
+
trajectories extracted from screen-recorded tutorials of seven kinds of software: macOS
|
| 26 |
+
system apps, video editing, Blender, Godot, Logic Pro, OBS Studio and Plasticity. This repository holds the
|
| 27 |
LoRA merged into the base weights (MLX, bf16) and, in `adapter/`, the LoRA itself.
|
| 28 |
|
| 29 |
> Research model. The training data was derived from public YouTube tutorials and is not
|
|
|
|
| 80 |
|
| 81 |
## Training
|
| 82 |
|
| 83 |
+
gua v2 continues the v1 adapter on a larger, more varied dataset.
|
| 84 |
+
|
| 85 |
| | |
|
| 86 |
|---|---|
|
| 87 |
| Method | Supervised fine-tuning with LoRA: rank 8, alpha 16, on every linear layer of the language model (`q/k/v/o_proj`, linear-attention `in_proj_*`/`out_proj`, MLP `gate/up/down_proj`); vision tower frozen |
|
| 88 |
| Objective | Log loss of the right option label (and end-of-turn token) only, never of the prompt |
|
| 89 |
+
| Stage 1 (v1) | 8,000 questions from *macos-cua v2* (macOS system, video editing, Blender, Godot) |
|
| 90 |
+
| Stage 2 (v2) | 12,000 further questions from *macos-cua v3*, shared evenly across its seven topics (each topic at most about 1,940; smaller topics give all they have) |
|
| 91 |
+
| Optimiser | Adam, learning rate 5e-5, batch 1, gradient clipping 1.0 |
|
| 92 |
+
| Hardware | One Apple Silicon Mac (MLX), gradient checkpointing, about 110 h in total, peak 182 GB |
|
|
|
|
|
|
|
|
|
|
| 93 |
|
| 94 |
+
The recipe was chosen on held-out dev splits, never on an eval split. The released weights
|
| 95 |
+
are the end of stage 2, which scored higher on the full v3 dev split (mean accuracy 0.453)
|
| 96 |
+
than the checkpoint with the best interim dev check (0.442).
|
|
|
|
|
|
|
| 97 |
|
| 98 |
## Training data
|
| 99 |
|
| 100 |
Trajectories were extracted automatically from screen-recorded software tutorials on
|
| 101 |
YouTube: videos were searched and screened (recording quality, platform), segmented and
|
| 102 |
annotated into step-by-step actions by vision-language models, each click was located on
|
| 103 |
+
the frame, and before/after screenshots were cut. The **training split** of *macos-cua v3*,
|
| 104 |
+
and the questions stage 2 drew from it:
|
| 105 |
|
| 106 |
+
| Topic | Videos | Steps | Questions used in stage 2 |
|
| 107 |
+
|---|---|---|---|
|
| 108 |
+
| Plasticity (CAD) | 162 | 16,656 | 1,937 |
|
| 109 |
+
| Godot 4 (game engine) | 79 | 6,271 | 1,937 |
|
| 110 |
+
| Blender (3D) | 40 | 3,833 | 1,937 |
|
| 111 |
+
| OBS Studio (recording) | 37 | 2,890 | 1,937 |
|
| 112 |
+
| Logic Pro (audio) | 43 | 2,020 | 1,937 |
|
| 113 |
+
| macOS video editing (Final Cut Pro, Premiere Pro, CapCut, iMovie) | 7 | 640 | 1,468 |
|
| 114 |
+
| macOS system and productivity apps | 6 | 420 | 844 |
|
| 115 |
+
|
| 116 |
+
Splits are by video (dev 44 videos, eval 99 videos, 38 applications appear only in eval);
|
| 117 |
+
every video of the earlier v2 release kept its split, so v2's eval is part of v3's eval and
|
| 118 |
+
was never trained on. Godot typing steps were left out of training: an audit found their
|
| 119 |
+
labels mostly wrong. Many tutorials were recorded on Windows or full screen; steps that
|
| 120 |
would be wrong on a Mac (Windows shortcuts, taskbar, native dialogs) were removed.
|
| 121 |
|
| 122 |
**Label quality.** Labels are model-generated, not human-verified. A stronger model audited
|
|
|
|
| 124 |
|
| 125 |
| Topic | Audited steps | Element located correctly | Action correct |
|
| 126 |
|---|---|---|---|
|
| 127 |
+
| Logic Pro | 150 | 94.0% | 84.0% |
|
| 128 |
+
| OBS Studio | 150 | 92.0% | 81.3% |
|
| 129 |
| macOS video editing | 395 | 92.4% | 78.2% |
|
| 130 |
+
| macOS system and productivity | 200 | 91.5% | 75.5% |
|
| 131 |
+
| Blender | 180 | 89.4% | 71.7% |
|
| 132 |
+
| Plasticity | 570 | 79.6% | 71.4% |
|
| 133 |
| Godot | 270 | 87.0% | 54.8% |
|
| 134 |
|
| 135 |
The dataset itself is not released.
|
| 136 |
|
| 137 |
## Evaluation
|
| 138 |
|
| 139 |
+
**macos-cua v3 eval split** (9,972 steps from 99 videos never seen in training, 6,509 with
|
| 140 |
+
an element target), gua v2 against gua v1 with the same prompt and candidates, 95% CI by
|
| 141 |
+
bootstrap paired by trajectory:
|
| 142 |
|
| 143 |
+
| Question | gua v1 | gua v2 | Difference |
|
| 144 |
|---|---|---|---|
|
| 145 |
+
| `action` accuracy | 0.507 | 0.537 | +0.030 (+0.021, +0.040) |
|
| 146 |
+
| `grounding` accuracy | 0.516 | 0.524 | +0.008 (+0.002, +0.015) |
|
| 147 |
+
| `next_target` accuracy | 0.303 | 0.323 | +0.020 (+0.012, +0.029) |
|
| 148 |
+
| `action` and `next_target` both right | 0.182 | 0.216 | |
|
| 149 |
|
| 150 |
+
On the 1,767 steps from **applications never seen in training**: action 0.602, grounding
|
| 151 |
+
0.628, next target 0.357. By topic (gua v2):
|
|
|
|
|
|
|
| 152 |
|
| 153 |
+
| Topic | `action` | `grounding` | `next_target` |
|
| 154 |
+
|---|---|---|---|
|
| 155 |
+
| OBS Studio | 0.806 | 0.803 | 0.453 |
|
| 156 |
+
| macOS system and productivity | 0.720 | 0.643 | 0.465 |
|
| 157 |
+
| Logic Pro | 0.687 | 0.566 | 0.341 |
|
| 158 |
+
| macOS video editing | 0.592 | 0.644 | 0.388 |
|
| 159 |
+
| Blender | 0.539 | 0.503 | 0.253 |
|
| 160 |
+
| Godot | 0.519 | 0.657 | 0.275 |
|
| 161 |
+
| Plasticity | 0.462 | 0.381 | 0.308 |
|
| 162 |
+
|
| 163 |
+
**macos-cua v2 eval split** (3,442 steps, the eval gua v1 was published with), against the
|
| 164 |
+
untrained base:
|
| 165 |
+
|
| 166 |
+
| Question | Base | gua v1 | gua v2 |
|
| 167 |
+
|---|---|---|---|
|
| 168 |
+
| `action` accuracy | 0.481 | 0.525 | 0.542 |
|
| 169 |
+
| `grounding` accuracy | 0.443 | 0.603 | 0.605 |
|
| 170 |
+
| `next_target` accuracy | 0.155 | 0.261 | 0.289 |
|
| 171 |
+
|
| 172 |
+
A candidate covers the target in 70.2% of v3's element steps (83.4% in v2's), which bounds
|
| 173 |
+
the element questions; Plasticity's small, dense CAD icons are often missed. Excluding the
|
| 174 |
+
Godot typing steps, whose labels are mostly wrong, v3 action accuracy is 0.563. Because
|
| 175 |
+
those steps were left out of training, the model rarely answers `type`.
|
| 176 |
|
| 177 |
+
**Selective accuracy** (v3 eval). Acting only on the most confident half of the answers:
|
| 178 |
+
grounding 0.711, action 0.665, next target 0.444.
|
| 179 |
|
| 180 |
+
**Calibration.** `temperatures.json` holds one temperature per question, fitted on the v3
|
| 181 |
+
dev split by log loss and applied as `p^(1/T)` renormalised (`inference/cua_decide.py
|
| 182 |
+
--temperatures`). Expected calibration error on the v3 eval split, which the fit never saw:
|
| 183 |
|
| 184 |
| Question | Temperature | ECE before | ECE after |
|
| 185 |
|---|---|---|---|
|
| 186 |
+
| `action` | 2.15 | 0.294 | 0.060 |
|
| 187 |
+
| `grounding` | 1.25 | 0.259 | 0.173 |
|
| 188 |
+
| `next_target` | 1.20 | 0.253 | 0.141 |
|
| 189 |
|
| 190 |
+
Without scaling the model is overconfident on every question; after it, element answers
|
| 191 |
+
remain somewhat overconfident.
|
|
|
|
| 192 |
|
| 193 |
## Limitations
|
| 194 |
|
| 195 |
+
- **macOS screenshots only**, English UI and tasks.
|
|
|
|
| 196 |
- **Candidate-bound.** The model picks among candidates; if no candidate covers the target
|
| 197 |
+
(about 30% of element steps in v3's eval, most often in CAD), the element questions have
|
| 198 |
+
no right answer. On a live Mac, the accessibility tree is a better candidate source.
|
| 199 |
+
- **Noisy labels**, as audited above; action-type labels are the weakest, and Plasticity's
|
| 200 |
+
element labels the least reliable.
|
| 201 |
+
- **Weak on dense professional UIs**: Plasticity grounding is 0.38.
|
| 202 |
- **Not an agent by itself.** It decides the action kind and target; the text to type, the
|
| 203 |
+
keys of a shortcut and drag end points are not predicted. It rarely predicts `type`.
|
| 204 |
- **Merged weights round to bf16.** Against base plus adapter, option probabilities differ
|
| 205 |
+
by at most 0.035 in total variation on our check; use `adapter/` for exact reproduction.
|
| 206 |
- Confidence is only meaningful after temperature scaling, and only for the question types
|
| 207 |
and candidate sources it was fitted with.
|
| 208 |
+
|
| 209 |
+
## Versions
|
| 210 |
+
|
| 211 |
+
| Version | Training | Notes |
|
| 212 |
+
|---|---|---|
|
| 213 |
+
| v2 (this) | v1 continued on 12,000 balanced questions from macos-cua v3 | adds Logic Pro, OBS Studio, Plasticity; better on every question |
|
| 214 |
+
| v1 | 8,000 questions from macos-cua v2 | first release; tagged `v1` in this repository |
|
| 215 |
|
| 216 |
## Licence and provenance
|
| 217 |
|
adapter/adapters.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 64991946
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e125d445722f9f931e5473988f3cc44a942d230af1aff1669d68aa271ef79966
|
| 3 |
size 64991946
|
merge_check.json
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
{
|
| 2 |
"fused_layers": 248,
|
| 3 |
"check_steps": 5,
|
| 4 |
-
"worst_total_variation_vs_adapter": 0.
|
| 5 |
}
|
|
|
|
| 1 |
{
|
| 2 |
"fused_layers": 248,
|
| 3 |
"check_steps": 5,
|
| 4 |
+
"worst_total_variation_vs_adapter": 0.03447
|
| 5 |
}
|
model-00001-of-00002.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 5366980383
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b7c089d993c6939edf367da5cf6a1d250f0414cd9e9dcd38981541d0d7506b92
|
| 3 |
size 5366980383
|
model-00002-of-00002.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 3711641426
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b85549825e9a4785f4728ee780f114e1775f9545817208653f79294151236b84
|
| 3 |
size 3711641426
|
temperatures.json
CHANGED
|
@@ -1,68 +1,68 @@
|
|
| 1 |
{
|
| 2 |
-
"dev": "experiments/decision/dev-
|
| 3 |
-
"eval": "experiments/decision/eval-
|
| 4 |
"questions": {
|
| 5 |
"grounding": {
|
| 6 |
-
"temperature": 1.
|
| 7 |
-
"dev_items":
|
| 8 |
"dev_ece": {
|
| 9 |
-
"before": 0.
|
| 10 |
-
"after": 0.
|
| 11 |
},
|
| 12 |
"dev_log_loss": {
|
| 13 |
-
"before": 1.
|
| 14 |
-
"after": 1.
|
| 15 |
},
|
| 16 |
-
"eval_items":
|
| 17 |
"eval_ece": {
|
| 18 |
-
"before": 0.
|
| 19 |
-
"after": 0.
|
| 20 |
},
|
| 21 |
"eval_log_loss": {
|
| 22 |
-
"before": 1.
|
| 23 |
-
"after": 1.
|
| 24 |
}
|
| 25 |
},
|
| 26 |
"action": {
|
| 27 |
-
"temperature": 2.
|
| 28 |
-
"dev_items":
|
| 29 |
"dev_ece": {
|
| 30 |
-
"before": 0.
|
| 31 |
-
"after": 0.
|
| 32 |
},
|
| 33 |
"dev_log_loss": {
|
| 34 |
-
"before":
|
| 35 |
-
"after": 1.
|
| 36 |
},
|
| 37 |
-
"eval_items":
|
| 38 |
"eval_ece": {
|
| 39 |
-
"before": 0.
|
| 40 |
-
"after": 0.
|
| 41 |
},
|
| 42 |
"eval_log_loss": {
|
| 43 |
-
"before":
|
| 44 |
-
"after": 1.
|
| 45 |
}
|
| 46 |
},
|
| 47 |
"next_target": {
|
| 48 |
-
"temperature":
|
| 49 |
-
"dev_items":
|
| 50 |
"dev_ece": {
|
| 51 |
-
"before": 0.
|
| 52 |
-
"after": 0.
|
| 53 |
},
|
| 54 |
"dev_log_loss": {
|
| 55 |
-
"before": 2.
|
| 56 |
-
"after": 2.
|
| 57 |
},
|
| 58 |
-
"eval_items":
|
| 59 |
"eval_ece": {
|
| 60 |
-
"before": 0.
|
| 61 |
-
"after": 0.
|
| 62 |
},
|
| 63 |
"eval_log_loss": {
|
| 64 |
-
"before":
|
| 65 |
-
"after":
|
| 66 |
}
|
| 67 |
}
|
| 68 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"dev": "experiments/decision/dev-v3-lora-v3.jsonl",
|
| 3 |
+
"eval": "experiments/decision/eval-v3-lora-v3.jsonl",
|
| 4 |
"questions": {
|
| 5 |
"grounding": {
|
| 6 |
+
"temperature": 1.25,
|
| 7 |
+
"dev_items": 2555,
|
| 8 |
"dev_ece": {
|
| 9 |
+
"before": 0.2877,
|
| 10 |
+
"after": 0.2073
|
| 11 |
},
|
| 12 |
"dev_log_loss": {
|
| 13 |
+
"before": 1.0868,
|
| 14 |
+
"after": 1.0408
|
| 15 |
},
|
| 16 |
+
"eval_items": 6508,
|
| 17 |
"eval_ece": {
|
| 18 |
+
"before": 0.259,
|
| 19 |
+
"after": 0.1725
|
| 20 |
},
|
| 21 |
"eval_log_loss": {
|
| 22 |
+
"before": 1.1036,
|
| 23 |
+
"after": 1.0632
|
| 24 |
}
|
| 25 |
},
|
| 26 |
"action": {
|
| 27 |
+
"temperature": 2.15,
|
| 28 |
+
"dev_items": 4006,
|
| 29 |
"dev_ece": {
|
| 30 |
+
"before": 0.3158,
|
| 31 |
+
"after": 0.0655
|
| 32 |
},
|
| 33 |
"dev_log_loss": {
|
| 34 |
+
"before": 1.9669,
|
| 35 |
+
"after": 1.4436
|
| 36 |
},
|
| 37 |
+
"eval_items": 9972,
|
| 38 |
"eval_ece": {
|
| 39 |
+
"before": 0.2941,
|
| 40 |
+
"after": 0.0603
|
| 41 |
},
|
| 42 |
"eval_log_loss": {
|
| 43 |
+
"before": 1.9069,
|
| 44 |
+
"after": 1.4122
|
| 45 |
}
|
| 46 |
},
|
| 47 |
"next_target": {
|
| 48 |
+
"temperature": 1.2,
|
| 49 |
+
"dev_items": 2555,
|
| 50 |
"dev_ece": {
|
| 51 |
+
"before": 0.2854,
|
| 52 |
+
"after": 0.1744
|
| 53 |
},
|
| 54 |
"dev_log_loss": {
|
| 55 |
+
"before": 2.5119,
|
| 56 |
+
"after": 2.4338
|
| 57 |
},
|
| 58 |
+
"eval_items": 6508,
|
| 59 |
"eval_ece": {
|
| 60 |
+
"before": 0.2527,
|
| 61 |
+
"after": 0.1405
|
| 62 |
},
|
| 63 |
"eval_log_loss": {
|
| 64 |
+
"before": 2.5645,
|
| 65 |
+
"after": 2.483
|
| 66 |
}
|
| 67 |
}
|
| 68 |
}
|