Zero Proof AI commited on
Commit
d9d92d8
路
verified 路
1 Parent(s): 882299f

Lean v0 card + simple headline figure

Browse files
Files changed (1) hide show
  1. README.md +16 -42
README.md CHANGED
@@ -20,61 +20,33 @@ tags:
20
 
21
  # ZeroProof Intent Model 0 (v0)
22
 
23
- A QLoRA adapter over `gemma-3-1b` that reads a payment or ecommerce conversation and returns one structured intent object. It is the model side of ZeroProof's behavioral attestation stack: it decides what the user actually intended so a downstream layer can verify an agent's action against it. This is v0, our first release, evaluated against the frozen base on a de-duplicated, per-intent balanced held-out set.
24
 
25
  ![evaluation](eval_report.png)
26
 
27
- ## Model description
28
 
29
- - Base: `gemma-3-1b` (frozen, pinned snapshot)
30
- - Method: QLoRA supervised fine tuning, rank-16 adapter, response-only supervision, 2 epochs
31
- - Task: route a conversation to one of 7 intents (`spend`, `send`, `exchange`, `recur`, `bill`, `reverse`, `none`) and extract the fields for that intent
32
- - Language: English
33
 
34
- Ground truth is taken from the user turns only. `none` is first class: in an attestation setting a false positive is a false attestation, so the model is trained and measured on getting the no-action boundary right.
35
 
36
- ## Model sources
37
 
38
- - Org: [zero-proof-ai](https://huggingface.co/zero-proof-ai) 路 Collection: ZeroProof Intent Models
39
- - Website: [zeroproofai.com](https://www.zeroproofai.com/)
40
- - Paper: in progress
41
-
42
- ## Versions
43
-
44
- | Version | Base | Status | Notes |
45
- |---|---|---|---|
46
- | **v0 (this)** | gemma-3-1b | released | first controlled base-vs-SFT result |
47
- | v1 | TBD | planned | balanced trap-laden eval, frontier and peer comparison, long-context |
48
-
49
- ## Performance
50
-
51
- Base and the adapter are scored in one engine on the identical 4,000 held-out conversations, greedy decoding. Only the adapter differs. Core-type is reported macro (equal weight per intent) because the eval is 57% `none`; the micro number is shown for completeness only.
52
-
53
- | Metric | Base gemma-3-1b | **Model 0** |
54
  |---|---|---|
55
- | Valid structured output | 87.5 | **99.8** |
56
- | Intent detected (act vs none) | 35.0 | **81.9** |
57
- | **Core-type routing, macro** | 14.4 | **70.0** |
58
- | Core-type routing, micro | 7.6 | 79.8 |
59
- | Detail extraction | 19.7 | **61.3** |
60
-
61
- Per-intent core-type accuracy (n = held-out count):
62
 
63
- | | none | reverse | send | exchange | bill | spend | recur |
64
- |---|---|---|---|---|---|---|---|
65
- | n | 2286 | 300 | 300 | 214 | 300 | 300 | 300 |
66
- | base | 0 | 0 | 10 | 0 | 6 | 84 | 0 |
67
- | **Model 0** | **90** | **74** | **72** | **68** | **65** | 63 | **59** |
68
 
69
- The base does not fail at random. It collapses to a single class, predicting `spend` on 80.7% of all inputs regardless of the true intent, so its only strong column is `spend`. The adapter trades that false strength for real coverage of all seven intents. Columns for a peer 1B (Llama) and a frontier parent are the v1 comparison, run on this same eval.
70
 
71
- ## Training
72
 
73
- QLoRA SFT on Modal, effective batch 16, 2 epochs, learning rate 2e-4, response-only masking on the frozen base. Trained on ZeroProof's own conversation dataset: 6,000 train / 4,000 eval, human-realistic conversations across a wide range of people and situations, with near-duplicates removed so eval gains are not memorized.
74
 
75
- ## Intended use and limitations
76
-
77
- Intended for intent detection inside the ZeroProof attestation pipeline. v0, single base family, English. Weakest intents are `recur` (59) and `spend` (63), the status-vs-action and purchase-vs-defer boundaries, which the next data cycle targets. The confidence field is not yet calibrated; treat it as a rough signal. No frontier or peer comparison is claimed in v0.
78
 
79
  ## Citation
80
 
@@ -86,3 +58,5 @@ Intended for intent detection inside the ZeroProof attestation pipeline. v0, sin
86
  howpublished = {\url{https://huggingface.co/zero-proof-ai/intent-model-0}}
87
  }
88
  ```
 
 
 
20
 
21
  # ZeroProof Intent Model 0 (v0)
22
 
23
+ A QLoRA adapter over `gemma-3-1b` that reads a payment or ecommerce conversation and returns one structured intent object. It is the model side of ZeroProof's behavioral attestation stack: it decides what the user intended so a downstream layer can verify an agent's action against it. Small on purpose, so it can run on every transaction. This is v0, our first release.
24
 
25
  ![evaluation](eval_report.png)
26
 
27
+ ## What it does
28
 
29
+ In: a short conversation. Out: one intent object, the intent plus the fields that matter for it. It routes to one of 7 intents: `spend`, `send`, `exchange`, `recur`, `bill`, `reverse`, `none`. `none` is first class. In an attestation setting a false positive is a false attestation, so getting the no-action boundary right matters most.
 
 
 
30
 
31
+ ## Results
32
 
33
+ Scored against the frozen base on 4,000 held-out conversations, greedy decoding, both models in one engine. Core-type is reported macro (equal weight per intent).
34
 
35
+ | Metric | Base gemma-3-1b | Model 0 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
  |---|---|---|
37
+ | Valid output | 87.5 | 99.8 |
38
+ | Intent detected | 35.0 | 81.9 |
39
+ | Core-type (macro) | 14.4 | 70.0 |
 
 
 
 
40
 
41
+ The base defaults to a single class. Model 0 routes across all seven.
 
 
 
 
42
 
43
+ ## Method
44
 
45
+ QLoRA SFT on a frozen, pinned `gemma-3-1b`, rank-16 adapter, 2 epochs, trained on Modal. Trained on ZeroProof's own conversation dataset: 6,000 train / 4,000 eval, balanced across the action intents, with near-duplicates removed so eval gains are not memorized.
46
 
47
+ ## Status
48
 
49
+ v0, first release, English. Next: a broader, harder eval, comparisons against peer 1B and frontier models, and more intents and domains.
 
 
50
 
51
  ## Citation
52
 
 
58
  howpublished = {\url{https://huggingface.co/zero-proof-ai/intent-model-0}}
59
  }
60
  ```
61
+
62
+ Built by [ZeroProof](https://www.zeroproofai.com/) 路 [zero-proof-ai](https://huggingface.co/zero-proof-ai)