laya-browser-v1

A continuation fine-tune of cklxx/laya-browser v17s that adds Turkish web coverage and real click-trace data. Same architecture (mmBERT-base 322M encoder + decision heads), same format (laya_fmt v3), same checkpoint layout — a drop-in replacement for v17s/ in the laya-browser repo.

Like all laya System-1 models, it does not generate text. It answers typed questions (choice, noul, score) about a page state and returns the answer plus calibrated probabilities. The browser agent loop, the HTTP server and the text side (TYPE_TEXT values) come from the harness, not from this model.

• Checkpoint size: 647 MB (fp16), 5 files — model.safetensors, encoder/, tokenizer/, rl_agent_config.json
• Latency (RX 7800 XT, fp16, stock forward): ~170–480 ms per decision at 37–59 options, mean ≈ 300 ms
• Training: ~5 h, 15,987 updates, 4 epochs, peak VRAM 6.1 GB

Benchmarks at a glance

Every number below was measured locally on one RX 7800 XT 16 GB, with the same harness for every model (laya-jev-benchmark for the decision suites, finetune/eval.py for the held-out browser split). Rows marked * are quoted from the vendor, not re-measured. parent = cklxx/laya-browser v17s, base = convaiinnovations/laya. Detail and per-source breakdowns follow in Results below.

benchmark	base	parent (v17s)	this model	Jev 1.13.0*	Claude Haiku 4.5*
browser agent tasks — Suite B (18 domains held out from all training sources)	—	18/18	18/18	—	—
browser agent tasks — Suite A (16 real-site tasks)	—	14/16	14/16	—	—
held-out browser decisions — operation accuracy (n = 1,288)	0.54 †	0.663	0.703	—	—
held-out browser decisions — element top-1 (n = 1,107)	0.10 †	0.548	0.539	—	—
typed-decisions (Jev's four published workflows, 2,000 decisions) — accuracy	0.361	0.343	0.344	0.727	—
typed-decisions — ECE (lower is better)	0.175	0.257	0.211	0.144	—
probe suite — failed assertions (fewer is better; 11 assertions)	6	11	6	—	—
PhishNChips (2,000 emails) — AUROC	0.682	0.577	0.759	0.689	0.837
PhishNChips — accuracy	0.505	0.502	0.511	0.626	0.813
PhishNChips — phishing recall	0.012	0.004	0.024	0.432	0.764
Turkish Wikipedia task (absent from every arm's training data)	—	FAIL	PASS	—	—
latency per typed-decisions decision (p50)	106 ms	56 ms	55 ms	710 ms	687 ms
latency per browser decision (37–59 options, 12-option chunks)	—	~300 ms mean	~300 ms mean	—	—

† measured by the upstream cklxx/laya-browser repo on the same held-out split (quoted base-model row).

How much better, plainly.

• Versus its parent (v17s): +4.0 points of held-out operation accuracy (0.663 → 0.703), driven by NNetNav TYPE_TEXT (0.49 → 0.62, +13) and live DONE (0.41 → 0.72, +31); element top-1 is flat (0.548 → 0.539, inside noise) and SCROLL_DOWN slips on a 59-case sample. Calibration error drops 0.257 → 0.211 (−18 % relative) and probe-suite failures halve (11 → 6, back to the base model's level). The Turkish Wikipedia task goes from FAIL — the parent clicks "Oturum aç" and declares DONE on the login page — to PASS in 2 steps.
• Versus the base model: general decision accuracy stays at the same level (0.344 vs 0.361, −1.7 points) and calibration is worse than the base's (0.211 vs 0.175) but better than the parent's (0.257). The decisive difference is capability: the base model cannot do this job at all — on the same held-out split its operation accuracy is 0.54 and element top-1 0.10 (quoted from the upstream repo) against 0.703 / 0.539 here — and phishing ranking improves by +7.7 AUROC over the base (0.759 vs 0.682).
• Versus Jev 1.13.0: on general workflow decisions Jev wins by a wide margin (0.727 vs 0.344) — it is a far larger, purpose-built model, and this checkpoint does not claim to match it. What this 322M model wins is speed and ranking: 55 ms vs 710 ms p50 (~13x faster) and a better phishing AUROC (0.759 vs 0.689), while losing badly on phishing recall (0.024 vs 0.432) — never use it as a phishing threshold without fitting a bias term.

What went into it

Initialized from v17s weights, trained with laya's RLCD recipe (finetune/train_desktop.py, LAYA_FMT=v3, LAYA_HEAD=768, MAX_TARGETS=40, FINAL_P=0.2, micro-batch 4 × accum 8).

51,557 tokenized items (1,611 steps/epoch), from 28,499 cases:

source	cases	notes
NNetNav live crawl (nnetnav-live/train.jsonl, converted)	19,587
Mind2Web train (converted)	7,296
Turkish + general live crawl (crawl_tr)	843	goals generated per page; labels from a hosted teacher model
second-step cases from real traces	573	teacher-generated variants of the cases below
real click/DONE traces (CDP, 200 cases)	200	executed in a real Chrome, outcome-checked
pages	80	incl. 14 Turkish sites (migros, bim, vatanbilgisayar, mediamarkt, dr.com.tr, kitapyurdu, idefix, turkcell, vodafone, turktelekom, garanti, ziraat, yapıkredi, ptt.gov.tr)

Held-out eval set: 7,456 cases (runs/browser_v1/eval_cases.jsonl), never used for training; the calibration temperature (1.7046) was fitted on a separate split of it (n=6787).

Results

All numbers below were measured on one machine (RX 7800 XT 16 GB, Chrome headless via CDP, systemone_server.py with 12-option chunks) unless marked published. Nothing here is quoted from another run.

1. Browser agent suites (the author's suites, 1 run per task, same teacher for both arms)

suite	v17s (parent)	this model
Suite B — 18 tasks on 18 domains held out from all training sources	18/18	18/18
Suite A — 16 real-site tasks	14/16	14/16
Suite A, median wall time	5.5 s	4.0 s
Suite B, median wall time	4.1 s	3.3 s

Failures are identical in both arms (books-page2, flights). One run of wiki-search in this model's arm first failed with a Chrome IPC timeout (_IPCResponseTimeout, a harness error, not a model error); the retry passed in 6.3 s.

Caveat, stated plainly: 7 Suite-A domains (en.wikipedia, github, news.ycombinator, arxiv, quotes.toscrape, the-internet.herokuapp, google) also appear in this model's training crawl, so Suite A is not a clean holdout for it. Suite B is verified clean (held-out check: OK). On the clean suite the two models tie.

For reference, the parent's own published numbers on the same suites (3 runs per task, the author's machine) are Suite A 85% (41/48) and Suite B 100% (54/54) — our single-run arm reproduces the same pass/fail pattern (books-page2 and flights fail in both arms).

2. Turkish task (not in training data for either arm)

Goal (Turkish): "Vikipedi'de 'yapay zeka' araması yap ve yapay zekâ maddesini aç." — start page tr.wikipedia.org/wiki/Anasayfa, 12 steps max.

	result	steps	wall
v17s	FAIL — clicked "Oturum aç" (Log in), declared DONE on the login page	1	3.6 s
this model	PASS — typed into "Vikipedi üzerinde ara", Enter, landed on /wiki/Yapay_zekâ	2	8.0 s

3. Held-out decision accuracy (1,288-case stratified sample of the 7,456-case set)

Same sample, same machine, finetune/eval.py, LAYA_HEAD=768. target top-1 is measured only on cases with a known gold element (n=1,107).

bucket (source · operation)	n	op acc v17s → this	target top-1 v17s → this
all	1,288	0.663 → 0.703	0.548 → 0.539
mind2web (test split) · CLICK	300	0.81 → 0.80	0.47 → 0.46
mind2web · SELECT	38	0.45 → 0.53	0.53 → 0.58
mind2web · TYPE_TEXT	62	0.90 → 0.85	0.87 → 0.84
nnetnav (test split) · CLICK	322	0.64 → 0.67	0.28 → 0.25
nnetnav · DONE	80	0.29 → 0.33	—
nnetnav · TYPE_TEXT	136	0.49 → 0.62	0.87 → 0.85
nnetnav · SCROLL_DOWN	59	0.12 → 0.07	0.00 → 0.00
live (Turkish/general crawl) · CLICK	218	0.88 → 0.94	0.72 → 0.75
live · DONE	39	0.41 → 0.72	—
live · TYPE_TEXT	30	0.90 → 0.93	0.87 → 0.83

Reading it honestly: operation accuracy improves by +4.0 points overall, driven by NNetNav TYPE_TEXT (+13) and live DONE (+31); target top-1 is flat (0.548 → 0.539, within noise) and SCROLL_DOWN regressed slightly on a small sample (n=59). The Mind2Web and NNetNav buckets are clean holdouts (test splits, never trained on). The live bucket is in-domain-ish: 85 of its 106 labelled cases sit on domains that also appear in the training crawl (same sites, different goals and states; one exact URL overlap), so treat that gain as expected in-domain behaviour, not as a generalization result.

4. Decision-quality benchmarks on Jev's published tasks (local run of the laya-jev-benchmark harness)

Jev publishes accuracy/ECE numbers for its hosted model; the open harness in datasets/laya-jev-benchmark re-runs the same tasks locally. This checkpoint is a browser/UI decision head, so these two tables are mainly a "did fine-tuning damage anything?" check — reported in full, including where it loses.

typed-decisions test split (400 cases / 2,000 decisions; these are Jev's four published workflows: security incidents, agent trace observability, invoice processing, customer service):

model	accuracy	soft acc	ECE	Brier	p50 ms/case
this model	0.344	0.370	0.211	0.388	55 ms
v17s (parent)	0.343	0.366	0.257	0.472	56 ms
convaiinnovations/laya base	0.361	0.366	0.175	0.329	106 ms
Jev 1.13.0 (published)	0.727	0.580	0.144	0.148	710 ms
task-specific fine-tune (published, 2×T4 notebook)	0.766	0.511	0.212	0.063	—
teacher self-agreement (published ceiling)	0.735	—	—	—	—

Per workflow (accuracy / ECE): this model 0.322/0.171 · 0.410/0.106 · 0.304/0.333 · 0.340/0.240 — base 0.388/0.204 · 0.382/0.105 · 0.362/0.226 · 0.312/0.233 (agent_trace · customer_service · invoice · security).

Reading: on general workflow decisions a 322M browser head is nowhere near a 4B-class specialised model (0.34 vs 0.727), and this model does not claim to be one. Against its own parent it is level on accuracy (0.344 vs 0.343) with a clearly better ECE (0.211 vs 0.257, Brier 0.388 vs 0.472). The base model's raw ECE (0.175) is still the best of the three — the three shipped temperatures (3 values) were fitted for it, ours only refits the first.

probe suite (11 behavioural assertions: contradiction, stability, grounding, overconfidence — fewer failures is better):

model	total failures
this model	6
v17s (parent)	11
convaiinnovations/laya base	6

The parent's failures are concentrated in contradiction pairs (P(A)+P(B) collapsing to ≈0.1) and wrong answers given at confidence 1.00. That is consistent with a shipped configuration bug, not a knowledge gap: v17s/rl_agent_config.json carries temperature = [5.0256, 1.0, 1.0], and laya clamps the first value (valid range [0.5, 5]) with RuntimeWarning: this checkpoint ships invalid temperatures ... Treat confidence from the affected entries as uncalibrated. This checkpoint carries [1.7046, 1.0, 1.0] (valid, refit), which brings the suite back to the base model's level.

PhishNChips core split (2,000 emails, 1,000 phishing / 1,000 legitimate; Jev published 0.626 acc / ECE 0.154 / AUROC 0.689 / recall 0.432):

model	acc	ECE	AUROC	recall	prec	p50 ms
this model	0.511	0.415	0.759	0.024	0.923	18.3
v17s (parent)	0.502	0.496	0.577	0.004	1.000	18.1
convaiinnovations/laya base	0.505	0.441	0.682	0.012	0.857	25.3
Jev (published)	0.626	0.154	0.689	0.432	—	239
Claude Haiku 4.5 (published)	0.813	0.097	0.837	0.764	—	687

This is the one benchmark where this checkpoint beats both its parent and the base model on ranking (AUROC 0.759 vs 0.577 / 0.682) and accuracy (0.511). It is still a chance-level classifier — recall 2.4%, i.e. it says "not phishing" to almost everything, exactly the bias documented in the harness README (the ranking is fine, the threshold is wrong; Platt scaling on 1,000 emails lifts base laya to 0.611 but cannot be added at inference by the checkpoint). Do not use this model as a phishing filter without fitting a bias term.

Reproducibility note: our base-model row (0.505 / 0.441 / 0.682) reproduces the harness author's published base-laya row (0.505 / 0.441 / 0.678) to two decimals on a different GPU, which is a useful sanity check that the harness itself is being run correctly.

5. Latency

hardware	setting	result
RX 7800 XT, fp16	browser decision, 37–59 options, 12-option chunks	171–480 ms (mean ≈ 300 ms)
RX 7800 XT, fp16	typed-decisions case (2,000 decisions)	p50 55 ms, p95 83 ms
Raspberry Pi 5 16 GB, CPU only	166 tokens / 5 options	899 ms
Raspberry Pi 5 16 GB, CPU only	1,717 tokens	9.7 s
Raspberry Pi 5 16 GB, CPU only	2,534 tokens / 65 options	18.1 s
Raspberry Pi 5 16 GB, CPU only	8 decisions in one batch	779 ms/decision

The Pi has no GPU offload (VideoCore is not usable by torch/onnxruntime); ~6–8 ms per input token, input-bound. Use it for nightly/batch work with states ≤300 tokens.

How to use

import laya
agent = laya.load("ikaanakkok/laya-browser-v1")   # or a local checkpoint dir
state, questions = ...   # see code/sample_request.json in the laya-browser repo
answers = agent.predict(state, questions)["answers"]

HTTP (the same server the suites above were run with):

python code/apps/systemone_server.py 8791 /path/to/this/ckpt 12   # 12 = chunk width for wide choice questions
curl -s localhost:8791/health

With the jev-ultrafast agent (MIT, Browser Use): apply code/jev-ultrafast.patch, start Chrome with --remote-debugging-port=9222, then set BU_CDP_URL=http://127.0.0.1:9222 and TYPESAFE_BASE_URL=http://127.0.0.1:8791 and run the agent loop. A recorded browser step can be checked with the repo's verify.py.

Limitations

• No text generation, no vision. It reads the accessibility text of the page; elements with no accessible name cannot be targeted. Screenshots are not consumed.
• The target head degrades as the candidate list grows — measured on the parent architecture: 2 candidates 61.7%, 4 → 36.7%, 8 → 33.3%, 16 → 26.7%, 32 → 11.7%, full list (~151) ≈ 2%. Real desktop screens average ~159 candidates. Use chunked selection (systemone_server.py does this) and treat wide-list target picks as low-confidence.
• An opt-in embedding shortlist ships with laya (laya.shortlist), but on the desktop head's held-out set mean-pooled encoder embeddings rank candidates poorly (recall@20 12.5%, versus 34% for a simple lexical baseline and 52% top-1 for a perfect shortlist at 8 candidates), so chunked selection remains the recommended wide-list path.
• Trained for browser/desktop UI decisions, English + Turkish. Do not use it for authentication, payments, or irreversible actions without a human check.
• Part of the Turkish crawl labels (1,416 of 51,557 items, ≈2.7%: 843 goal/label cases + 573 second-step variants) were produced by a hosted teacher model; the 200 click traces and everything else are public data or generated from real executions.

Provenance & credits

• Base: convaiinnovations/laya (Apache-2.0) → cklxx/laya-browser v17s (Apache-2.0) → this checkpoint. This repository ships weights + this model card only (no code/): harness references below point at the upstream cklxx/laya-browser repo. The parent is Apache-2.0; this derivative is published under MIT with the upstream notices kept.
• Suites A/B and the decision benchmarks: cklxx/laya-browser (code/apps/browser_suite.py, browser_suite_b.py, datasets/laya-jev-benchmark), Apache-2.0.
• Reference numbers for the decision benchmarks are TypeSafe's published Jev results (quoted as such).
• Agent harness: browser-use/jev-ultrafast, MIT.
• Training data licenses: Mind2Web and NNetNav are used for training only, under their own terms.

Türkçe özet

cklxx/laya-browser v17s'in devamı niteliğinde bir ince ayar: Türkçe site kapsaması ve gerçek tıklama izleriyle eğitildi. Aynı mimari, aynı format — v17s'in yerine doğrudan kullanılabilir. Metin üretmez; sayfa durumu hakkında tipli soruları yanıtlar ve kalibre edilmiş olasılık döndürür. İngilizce suite'lerde v17s ile başa baş (Suite B 18/18, Suite A 14/16) ve biraz daha hızlı; Türkçe görevde v17s başarısız olurken bu model başarılı (tr.wikipedia "yapay zeka" görevi, 2 adım, 8,0 s).