laya typed-decisions · q4 split ONNX for the browser (WebGPU)

A 4-bit weight-only quantization of the typed-decisions checkpoint of convaiinnovations/laya (ModernBERT-large, English, fine-tuned for typed choice / score / noul decisions), in the split layout that laya-ts loads with ONNX Runtime Web. 467 MB vs 1.69 GB fp32.

Recipe (reproducible, see laya-skill):

  1. laya-ts/scripts/export_onnx.py --repo convaiinnovations/laya --subfolder typed-decisions (laya @ 6d942c9, torch vs ONNX within 1e-4)
  2. MatMulNBitsQuantizer (ONNX Runtime 1.30, 4 bit, block 32, symmetric) on every weight MatMul of the encoder (112) and the head (8). The token embedding stays fp32.

Measured (Apple M4 Pro, Chrome, laya-ts + onnxruntime-web 1.30)

variant size WebGPU p50 WASM p50 agreement with fp32 (120 decisions) max prob drift
fp32 1.69 GB 265 ms 2443 ms — —
q4 (this repo) 467 MB 361 ms ~2.8 s 110 / 120 0.22
q8 (not published) 705 MB ≈ WASM: 8-bit MatMulNBits has no WebGPU kernel 2.8 s 119 / 120 0.013

The 120 decisions are 40 labelled support tickets × 3 questions. Accuracy on them (dept / urgency / churn): fp32 0.875 / 0.425 / 0.700, q4 0.875 / 0.450 / 0.625. q4 flips about 8 % of decisions, so re-calibrate thresholds on your own held-out data. English only; use the multilingual export for other languages.

Use

import { Agent } from "laya-ts";
const agent = await Agent.load("https://huggingface.co/Steven10429/laya-typed-decisions-webgpu-q4/resolve/main/");

Live demo: https://stevenli-phoenix-work.itch.io/laya-webgpu

License & attribution

Apache-2.0, same as the original weights by ConvAI Innovations / Nandakishor M (NandhaKishorM/laya). This repository only changes format and precision.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Steven10429/laya-typed-decisions-webgpu-q4

Quantized
(94)
this model