| --- |
| license: apache-2.0 |
| base_model: convaiinnovations/laya |
| language: |
| - en |
| library_name: pytorch |
| pipeline_tag: text-classification |
| tags: |
| - typed-decisions |
| - classification |
| - routing |
| - modernbert |
| - coreml |
| - on-device |
| --- |
| |
| # Snap 1 |
|
|
| **Fast, local, typed decisions.** Give Snap a state (text or JSON) and a few typed questions — |
| yes/no, pick-one, or a level — and it answers all of them in a single forward pass, with |
| probabilities. It never generates text, so there is nothing to parse and nothing to hallucinate. |
| It runs on your machine: no API key, no network, nothing leaves the device. |
|
|
| `Otron/snap-1` · 421M parameters · English · up to 4,096 tokens per pass |
|
|
| ## Quick start |
|
|
| ```python |
| # pip install "laya==0.3.4" torch transformers safetensors huggingface_hub |
| # then download snap.py from this repo |
| from snap import Snap |
| |
| snap = Snap("Otron/snap-1") |
| snap.decide( |
| state={"message": "my card got charged twice for the same order"}, |
| questions={ |
| "topic": {"type": "choice", "instructions": "What is the message about?", |
| "criteria": {"billing": "Payments, charges, refunds", |
| "shipping": "Delivery and tracking", |
| "other": "Anything else"}}, |
| "urgent": {"type": "noul", "instructions": "The customer needs a reply today"}, |
| "mood": {"type": "score", "instructions": "How upset is the customer?", |
| "criteria": ["calm", "annoyed", "angry", "furious"]}, |
| }, |
| ) |
| # {"topic": {"choice": "billing", "confidence": 0.997, "probabilities": {...}}, |
| # "urgent": {"noul": 0.42, "confidence": 0.58}, |
| # "mood": {"score": 2, "confidence": 0.41, "probabilities": {"0": 0.05, "1": 0.38, "2": 0.41, "3": 0.15}}} |
| ``` |
|
|
| Question types: |
|
|
| | type | answers | `criteria` | |
| |---|---|---| |
| | `noul` | probability the statement in `instructions` is true | — | |
| | `choice` | one key | dict of key → description (or a list of keys) | |
| | `score` | one ordered level (0-based) | list of level descriptions, lowest first | |
|
|
| Descriptions matter: Snap reads every option's text, so options it has never seen work as well as |
| familiar ones. Option lists can be long (hundreds of options in one question). |
|
|
| ### Core ML (macOS 14+) |
|
|
| `coreml/Snap.mlpackage` is the same model converted for Core ML (fp16 weights, 819 MB). It |
| takes one sequence (`input_ids`, `tok_type`) and returns a score per token; `coreml/snap.json` |
| holds the token ids, limits and confidence temperatures needed to build the input and read the |
| answers, and `coreml/tokenizer.json` is the tokenizer. The input format is exactly the one |
| `snap.py` builds (`Snap._block` / `decide`). Run it on `.cpuAndGPU`: the Neural Engine |
| recompiles for every new input length. |
|
|
| Apps that download the model on demand can use `coreml/manifest.json`: every Core ML file with |
| its size and SHA-256, at URLs pinned to the commit they were published in. |
|
|
| ## Speed |
|
|
| Measured on an Apple M5 (32 GB), one decision at a time, model already loaded: |
|
|
| | | PyTorch (MPS, fp16) | Core ML (GPU, fp16) | |
| |---|---|---| |
| | short decision — one sentence, 3 questions (109 tokens) | 25 ms | **16 ms** | |
| | long decision — a spoken request routed over ~100 options plus context (1.6k tokens median, up to 3.7k) | 214 ms median · 545 ms p90 | 191 ms median · 557 ms p90 | |
| | load | 18.9 s | 5.8 s first time (compiles), then faster | |
|
|
| Cost grows with the total length of the state and the options, not with the number of questions: |
| every question about the same state shares one pass when they fit together. |
|
|
| ## Results |
|
|
| Accuracy against each dataset's own labels, on 500 examples per dataset from the **official** |
| evaluation split (never trained on). Questions are phrased as plain typed decisions with the |
| labels' names and descriptions as options; no task-specific heads or prompts. |
|
|
| | dataset (split) | majority class | Laya (base) | **Snap 1** | |
| |---|---|---|---| |
| | Banking77 (test) — 77 intents | 0.028 | 0.464 | **0.812** | |
| | CLINC150 plus (test) — 151 intents incl. out-of-scope | 0.196 | 0.514 | **0.838** | |
| | ContractNLI (test) — whole NDAs | 0.488 | 0.488 | **0.712** | |
| | MNLI (validation matched) | 0.356 | **0.872** | 0.870 | |
| | ANLI (test r1–r3) | 0.366 | 0.506 | **0.520** | |
| | BoolQ (validation) — accuracy · AUROC | 0.626 | 0.850 · 0.913 | **0.860 · 0.933** | |
| | Civil Comments (test) — AUROC | 0.5 | **0.944** | 0.923 | |
|
|
| Laya is scored on the same examples in its own input format, with its 512-token limit and with |
| Snap's 4,096; the better of the two is shown (with its own 192-token option budget it cannot fit |
| Banking77's or CLINC150's option lists at all). Snap's gains are where the fine-tuning aimed — |
| many-option routing and long documents; it is level on reasoning-style NLI and slightly behind the |
| base model at ranking toxicity. |
|
|
| Civil Comments is reported as AUROC: its label (≥50% of raters said toxic) is narrower than a plain |
| "is this rude or disrespectful" question, so raw accuracy mostly measures the cut-off. |
|
|
| **Unseen tools.** On held-out synthetic decisions for 300 invented tools (spoken requests routed to |
| each tool's own actions, none seen in training), Snap picks the reference action 69.8% of the time, |
| with expected calibration error 0.025. |
|
|
| **Calibration.** Confidences are temperature-calibrated per question type and option count; across |
| all held-out decisions the expected calibration error is 0.026. |
|
|
| ## JevBench (public half) |
|
|
| [JevBench](https://github.com/fstandhartinger/jevbench) is an independent benchmark for typed-decision |
| models. We ran Snap 1 on its 231 public items with the benchmark's own harness and scorer (commit |
| `609a664`). The Laya and Jev columns are the per-item results JevBench publishes for the same items |
| (release v1.3.0). This is a self-run of the public half, not an official board entry: the board also |
| scores sealed items, calibration, speed and cost. Snap 1 was not trained on any JevBench item. |
|
|
| | tier (items) | Laya (base) | **Snap 1** | Jev 1.13 | |
| |---|---|---|---| |
| | easy (48) | 0.958 | **1.000** | 1.000 | |
| | standard (72) | 0.694 | **0.806** | 0.986 | |
| | hard (111) | 0.351 | **0.342** | 0.730 | |
| | **all (231)** | 0.584 | **0.623** | 0.866 | |
| | median latency per decision | — | **24 ms** on-device (M5) | 665 ms over the network | |
|
|
| The hard tier (long insurance policies, contracts and rule sets with multi-step reasoning, written |
| by frontier models) is where a 421M encoder falls behind; none of those items exceed Snap's 4,096- |
| token window. Jev's latency is as JevBench measured it, over the network; Snap's runs locally. |
|
|
| ## How it was made |
|
|
| - **Base:** [Laya](https://huggingface.co/convaiinnovations/laya) by ConvAI Innovations, a typed- |
| decision model on [ModernBERT-large](https://huggingface.co/answerdotai/ModernBERT-large). |
| - **Training:** full-parameter fine-tuning (all weights) on the data below. |
| - **Data:** training splits of Banking77, CLINC150, MNLI, ANLI, BoolQ, Civil Comments, ContractNLI |
| and QuALITY, plus ~30k synthetic decisions for 300 invented tools generated with GPT-6 Luna, and a |
| small amount of private data that is not released. Examples were labelled with TypeSafe's Jev, |
| combined with each dataset's own labels where they exist. |
| - **Layout:** Snap packs all questions about one state into a single sequence (each question's |
| options are `[MASK]` markers that the model scores), instead of one sequence per question. This |
| is what makes many questions cost one pass. `extra.safetensors` holds the extra token-type |
| embedding this layout uses; `snap.py` implements it. |
|
|
| ## Limitations |
|
|
| - **Snap learned from model-generated labels and inherits their mistakes.** It is weakest on |
| hard multi-step reasoning (ANLI; JevBench's hard tier) and less sure of itself on tools it has |
| never seen. |
| - **English only**, and 4,096 tokens per pass (state + questions); longer states are truncated. |
| - **Text in the state can steer the answer** (prompt injection). Treat decisions on untrusted input |
| as advice, and keep a human or a hard rule in the loop for anything consequential. |
| - **Not for safety-critical, medical, legal or financial decisions.** |
|
|
| ## License and attribution |
|
|
| Apache 2.0 (see `LICENSE` and `NOTICE`). Built on Laya (Apache 2.0) and ModernBERT (Apache 2.0). |
| Trained on public datasets under their own licenses: ANLI is CC BY-NC 4.0 and BoolQ CC BY-SA 3.0; |
| the datasets themselves are not redistributed here. |
|
|
| Not affiliated with or endorsed by TypeSafe or ConvAI Innovations. |
|
|
