Per-channel int8 model, figure, demo video, updated card
Browse files- .gitattributes +2 -0
- README.md +19 -1
- fig_router.png +3 -0
- milo_demo_clean.mp4 +3 -0
- onnx/pointer_int8_pc.onnx +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
fig_router.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
milo_demo_clean.mp4 filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -5,9 +5,25 @@ tags: [gui-agent, android, accessibility, on-device, androidcontrol]
|
|
| 5 |
---
|
| 6 |
# Milo Pointer (149M)
|
| 7 |
|
| 8 |
-
Paper: *A
|
| 9 |
(Chaitanya Talapalli, 2026). Code, per-step results and the registration log: https://github.com/chayworks/milo-pointer
|
| 10 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
A 149M-parameter model that picks the next action on an Android screen from its accessibility tree, in about 0.3 s on a laptop.
|
| 12 |
It is meant to sit **in front of** a frontier agent: it answers the steps it is sure about and returns `defer: true` for the rest.
|
| 13 |
|
|
@@ -72,6 +88,8 @@ navigate_home, open_app (name copied from the instruction), wait. `last_step` de
|
|
| 72 |
| Pixel 6a CPU, ONNX Runtime 1.20, 4 threads, int8 | 3.4 s (p90 11.2 s) | 1.45 GB (agrees with fp32 on 34/40 steps) |
|
| 73 |
| Same, empty layout containers removed from the input | 1.18 s (p90 4.9 s) | 1.45 GB (0.8 points lower accuracy, routing intact) |
|
| 74 |
|
|
|
|
|
|
|
| 75 |
Phone CPU latency is not yet sub-second. `android/` has the export, the benchmark and a live demo.
|
| 76 |
|
| 77 |
## Limits
|
|
|
|
| 5 |
---
|
| 6 |
# Milo Pointer (149M)
|
| 7 |
|
| 8 |
+
Paper: *A 149M Text Encoder as the First Tier for Phone Agents: Pre-Registered Routing to Frontier Models*
|
| 9 |
(Chaitanya Talapalli, 2026). Code, per-step results and the registration log: https://github.com/chayworks/milo-pointer
|
| 10 |
|
| 11 |
+
- **A 149M text-only model scores 86.1% on AndroidControl-Low** under the protocol screenshot agents report, above the 85.2%
|
| 12 |
+
reported for OS-Atlas-7B (step sets differ slightly).
|
| 13 |
+
- **Small model + frontier beats the frontier alone.** The small model answers about 70% of steps on the device and hands the
|
| 14 |
+
rest to Claude Opus 5.5. In a pre-registered confirmatory test routing beat Opus alone by 2.8 points (p = 0.003); pooled over
|
| 15 |
+
all 1,650 test steps the gain is 1.5 points (95% CI [+0.5, +2.4]), at under a third of Opus's API cost.
|
| 16 |
+
- **What did not work is documented too.** About 40 registered experiments, negative ones included, with timestamps
|
| 17 |
+
(`paper/PREREGISTRATION.md`).
|
| 18 |
+
|
| 19 |
+
All Claude API calls behind the AndroidControl results cost **$89.73** in total (Batch prices); each training run takes about
|
| 20 |
+
**6.4 hours on one NVIDIA T4**.
|
| 21 |
+
|
| 22 |
+

|
| 23 |
+
|
| 24 |
+
Demo: the model drives a Pixel 6a through a five-step Clock task ("set a 5 minute timer"), every step answered locally
|
| 25 |
+
([video](https://huggingface.co/coolnow/milo-pointer/resolve/main/milo_demo_clean.mp4)).
|
| 26 |
+
|
| 27 |
A 149M-parameter model that picks the next action on an Android screen from its accessibility tree, in about 0.3 s on a laptop.
|
| 28 |
It is meant to sit **in front of** a frontier agent: it answers the steps it is sure about and returns `defer: true` for the rest.
|
| 29 |
|
|
|
|
| 88 |
| Pixel 6a CPU, ONNX Runtime 1.20, 4 threads, int8 | 3.4 s (p90 11.2 s) | 1.45 GB (agrees with fp32 on 34/40 steps) |
|
| 89 |
| Same, empty layout containers removed from the input | 1.18 s (p90 4.9 s) | 1.45 GB (0.8 points lower accuracy, routing intact) |
|
| 90 |
|
| 91 |
+
int8 accuracy on 1,000 held-out AndroidControl steps (fp32: 869): per-tensor `pointer_int8.onnx` 851, per-channel `onnx/pointer_int8_pc.onnx` (151 MB, recommended; built by `android/quant_variants.py`) 862. Per-channel was chosen on a separate calibration sample. Phone latencies above are for the per-tensor model; on the phone the per-channel model runs within 5% of it.
|
| 92 |
+
|
| 93 |
Phone CPU latency is not yet sub-second. `android/` has the export, the benchmark and a live demo.
|
| 94 |
|
| 95 |
## Limits
|
fig_router.png
ADDED
|
Git LFS Details
|
milo_demo_clean.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6c102f0bcaa33d1ad858d498e976ef0c14bedcaba81040fd4c8622983416fb9c
|
| 3 |
+
size 184724
|
onnx/pointer_int8_pc.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:28c3be79edb70d06fd70fdac37fc2c2c151249cb142c45eee06a700c12ee9822
|
| 3 |
+
size 150673275
|