while-ai
/

ecommerce-1b

@@ -18,112 +18,73 @@ tags:
   - structured-output
 ---
-# ZeroProof Intent Model 0 — `v0`
-> **Status: v0 — first iteration, private preview.** A compact adapter over `gemma-3-1b`
-> that reads a payment/agent conversation and returns **one structured intent object**.
-> It is the model half of ZeroProof's behavioral-attestation stack: it decides *what the
-> user actually intended* so a downstream layer can verify an agent's action matches it.
-![Model 0 evaluation](eval_report.png)
-Built by **[ZeroProof](https://www.zeroproofai.com/)** — a lab building the trust layer for
-AI agents. The data, the training, and the rubric are ours; that's the part that compounds.
----
-## What it does
-**In:** a short user↔assistant conversation. **Out:** a single structured intent object —
-the detected intent, whether it's actionable, the salient details for that intent, and a
-confidence signal. Ground truth comes from the **user's** turns; assistant turns are context.
-It routes to one of **7 intents:** `spend` (buy from a merchant) · `send` (pay a
-person/wallet) · `exchange` (swap/convert an asset) · `recur` (subscription/renewal) ·
-`bill` (pay a specific bill) · `reverse` (refund/return/dispute) · `none` (no actionable
-intent — the safety class; a false positive here is a *false attestation*, the worst
-failure mode). This is **routing-then-extraction**, not one-word classification.
-*(The exact output schema and integration spec are part of the ZeroProof runtime and are
-not published here — reach out for access.)*
----
-## How well did the fine-tuning work?
-Base and Model 0 scored in **one engine, on the identical 4,000 held-out conversations**
-(greedy decoding, de-duplicated so nothing leaks from training). Only the adapter differs.
-| Metric | Base `gemma-3-1b` | **Model 0** | Δ |
 |---|---|---|---|
-| Valid structured output | 87.5% | **99.8%** | +12.3 |
-| Intent detected (act vs. `none`) | 35.0% | **81.9%** | +46.9 |
-| **Intent routing — macro** | **14.4%** | **70.0%** | **+55.6** |
-| Detail extraction | 19.7% | **61.3%** | +41.6 |
-*Macro = equal weight per intent (the honest headline; the eval is 57% `none`, so a
-none-weighted average would flatter us — we don't report it as the headline).*
-## Why this beats a base 1B — and why it isn't luck
-The base model **doesn't fail randomly. It collapses to one class.** Across all 4,000 eval
-conversations it predicted `spend` **80.7%** of the time — a refund request, a status
-question, a "yo what's up," all called `spend`. It can produce well-formed output but it
-**cannot route**: it latches onto one intent and marks nearly everything actionable.
-Fine-tuning teaches the two things the base lacks — **routing across all seven intents**
-and **the `none` boundary**:
-| Intent | n | Base | **Model 0** |
-|---|---|---|---|
-| none | 2286 | 0 | **90** |
-| reverse | 300 | 0 | **74** |
-| send | 300 | 10 | **72** |
-| exchange | 214 | 0 | **68** |
-| bill | 300 | 6 | **65** |
-| spend | 300 | 84 | 63 |
-| recur | 300 | 0 | **59** |
-`spend` is the *only* column where the base looks good — because it guesses `spend` for
-everything. Model 0 trades that false strength for real coverage of all seven.
-## Why a fine-tuned 1B instead of calling the frontier model?
-This model runs in the **hot path of every agent transaction**. Against a frontier
-parent, the case is **economic and operational**, not "we're more accurate":
-- **Cost / latency / energy** — a self-hosted 1B is far cheaper and faster per call.
-  Attestation only works if you can afford to run it on *every* action.
-- **Self-hostable & pinned** — a fixed version you control, no external dependency in the
-  trust path.
-- **Accuracy vs. the parent** — that's the **next milestone (Model 1)**, measured on this
-  exact eval. We do **not** claim to beat the parent on accuracy in v0; we claim to close
-  enough of the gap to be worth running everywhere.
----
-## What we noticed · what we're watching · where our rubric is weak
-*(We publish these because the thinking is the product.)*
-- **Noticed:** the base's failure is *structured* (collapse-to-`spend`) — which is exactly
-  why teaching it to route moves the needle so much.
-- **Rubric weaknesses we own:** detail scoring is strict exact-match and under-credits
-  near-right extractions; the confidence field isn't calibrated yet — treat it as a rough
-  signal, not a probability.
-- **What we're watching next (attestation metrics, not accuracy bars):** how often the
-  model catches a **human switching intent mid-conversation**, and how often it correctly
-  flags **"this needs verification"** before an action fires. Those matter more for a trust
-  layer than raw accuracy.
-## Limitations
-v0, single base family. Weakest intents are `recur` (59) and `spend` (63) — the
-status-vs-action and purchase-vs-defer boundaries — which the next iteration targets. No
-frontier/peer or long-context comparison is claimed yet (Model 1).
-## Links
-- **ZeroProof** — [zeroproofai.com](https://www.zeroproofai.com/) ·
-  [huggingface.co/zero-proof-ai](https://huggingface.co/zero-proof-ai)
-- Technical report (v0): *Honest Evaluation for 1B-Scale Payment-Intent Detection* — coming soon.

   - structured-output
 ---
+# ZeroProof Intent Model 0 (v0)
+A QLoRA adapter over `gemma-3-1b` that reads a payment or ecommerce conversation and returns one structured intent object. It is the model side of ZeroProof's behavioral attestation stack: it decides what the user actually intended so a downstream layer can verify an agent's action against it. This is v0, our first release, evaluated against the frozen base on a de-duplicated, per-intent balanced held-out set.
+![evaluation](eval_report.png)
+## Model description
+- Base: `gemma-3-1b` (frozen, pinned snapshot)
+- Method: QLoRA supervised fine tuning, rank-16 adapter, response-only supervision, 2 epochs
+- Task: route a conversation to one of 7 intents (`spend`, `send`, `exchange`, `recur`, `bill`, `reverse`, `none`) and extract the fields for that intent
+- Language: English
+Ground truth is taken from the user turns only. `none` is first class: in an attestation setting a false positive is a false attestation, so the model is trained and measured on getting the no-action boundary right.
+The output schema and integration spec are part of the ZeroProof runtime and are not published here.
+## Model sources
+- Org: [zero-proof-ai](https://huggingface.co/zero-proof-ai) · Collection: ZeroProof Intent Models
+- Website: [zeroproofai.com](https://www.zeroproofai.com/)
+- Paper: in progress
+## Versions
+| Version | Base | Status | Notes |
 |---|---|---|---|
+| **v0 (this)** | gemma-3-1b | released | first controlled base-vs-SFT result |
+| v1 | TBD | planned | balanced trap-laden eval, frontier and peer comparison, long-context |
+## Performance
+Base and the adapter are scored in one engine on the identical 4,000 held-out conversations, greedy decoding. Only the adapter differs. Core-type is reported macro (equal weight per intent) because the eval is 57% `none`; the micro number is shown for completeness only.
+| Metric | Base gemma-3-1b | **Model 0** |
+|---|---|---|
+| Valid structured output | 87.5 | **99.8** |
+| Intent detected (act vs none) | 35.0 | **81.9** |
+| **Core-type routing, macro** | 14.4 | **70.0** |
+| Core-type routing, micro | 7.6 | 79.8 |
+| Detail extraction | 19.7 | **61.3** |
+Per-intent core-type accuracy (n = held-out count):
+| | none | reverse | send | exchange | bill | spend | recur |
+|---|---|---|---|---|---|---|---|
+| n | 2286 | 300 | 300 | 214 | 300 | 300 | 300 |
+| base | 0 | 0 | 10 | 0 | 6 | 84 | 0 |
+| **Model 0** | **90** | **74** | **72** | **68** | **65** | 63 | **59** |
+The base does not fail at random. It collapses to a single class, predicting `spend` on 80.7% of all inputs regardless of the true intent, so its only strong column is `spend`. The adapter trades that false strength for real coverage of all seven intents. Columns for a peer 1B (Llama) and a frontier parent are the v1 comparison, run on this same eval.
+## Training
+QLoRA SFT on Modal, effective batch 16, 2 epochs, learning rate 2e-4, response-only masking on the frozen base. Trained on ZeroProof's proprietary conversation data: 6,000 train / 4,000 eval, human-realistic conversations across many people and situations, near-duplicates removed so eval gains are not memorized. How we build and label that data is proprietary.
+## Intended use and limitations
+Intended for intent detection inside the ZeroProof attestation pipeline. v0, single base family, English. Weakest intents are `recur` (59) and `spend` (63), the status-vs-action and purchase-vs-defer boundaries, which the next data cycle targets. The confidence field is not yet calibrated; treat it as a rough signal. No frontier or peer comparison is claimed in v0.
+## Citation
+```bibtex
+@misc{zeroproof2026intentmodel0,
+  title        = {ZeroProof Intent Model 0},
+  author       = {ZeroProof},
+  year         = {2026},
+  howpublished = {\url{https://huggingface.co/zero-proof-ai/intent-model-0}}
+}
+```