Zero Proof AI commited on
Commit
7f08d3b
·
verified ·
1 Parent(s): b255408

v0.1: gemma-3-1b QLoRA adapter + model card + rubric (private preview)

Browse files
.gitattributes CHANGED
@@ -35,3 +35,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  eval_report.png filter=lfs diff=lfs merge=lfs -text
37
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
 
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  eval_report.png filter=lfs diff=lfs merge=lfs -text
37
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
38
+ rubric.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,62 +1,81 @@
1
  ---
2
- base_model: unsloth/gemma-3-1b-it-bnb-4bit
3
- library_name: peft
4
  license: gemma
5
- pipeline_tag: text-generation
6
- language:
7
- - en
8
  tags:
9
- - intent-detection
10
- - agent-safety
11
- - behavioral-attestation
12
  - payments
13
  - ecommerce
14
- - finance
15
- - fintech
16
- - transactions
17
- - conversational
18
- - structured-output
19
  ---
20
 
21
- # ZeroProof Ecommerce Intent 1B (v0)
22
 
23
- A QLoRA adapter over `gemma-3-1b` that reads a payment or ecommerce conversation and returns one structured intent object. It is the model side of ZeroProof's behavioral attestation stack: it decides what the user intended so a downstream layer can verify an agent's action against it. Small on purpose, so it can run on every transaction. This is v0, our first release.
24
 
25
- ![evaluation](eval_report.png)
 
 
26
 
27
  ## What it does
28
 
29
- In: a conversation. Out: one intent object, the intent plus the fields that matter for it. It routes to one of 7 intents: `spend`, `send`, `exchange`, `recur`, `bill`, `reverse`, `none`. `none` is first class. In an attestation setting a false positive is a false attestation, so getting the no-action boundary right matters most.
 
 
 
 
30
 
31
  ## Results
32
 
33
- Scored against the frozen base on 4,000 held-out conversations, greedy decoding, both models in one engine. Core-type is reported macro (equal weight per intent).
34
 
35
- | Metric | Base gemma-3-1b | Model 0 |
36
- |---|---|---|
37
- | Valid output | 87.5 | 99.8 |
38
- | Intent detected | 35.0 | 81.9 |
39
- | Core-type (macro) | 14.4 | 70.0 |
40
 
41
- The base defaults to a single class. Model 0 routes across all seven.
 
 
 
 
 
42
 
43
- ## Method
 
 
44
 
45
- QLoRA SFT on a frozen, pinned `gemma-3-1b`, rank-16 adapter, 2 epochs, trained on Modal. Trained on ZeroProof's own conversation dataset: 6,000 train / 4,000 eval, balanced across the action intents, with near-duplicates removed so eval gains are not memorized.
46
 
47
- ## Status
48
 
49
- v0, first release, English. Next: a broader, harder eval, comparisons against peer 1B and frontier models, and more intents and domains.
 
 
50
 
51
- ## Citation
 
 
52
 
53
- ```bibtex
54
- @misc{zeroproof2026ecommerceintent,
55
- title = {ZeroProof Ecommerce Intent 1B},
56
- author = {ZeroProof},
57
- year = {2026},
58
- howpublished = {\url{https://huggingface.co/zero-proof-ai/ecommerce-intent-1b}}
59
- }
60
  ```
61
 
62
- Built by [ZeroProof](https://www.zeroproofai.com/) · [zero-proof-ai](https://huggingface.co/zero-proof-ai)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
 
 
2
  license: gemma
3
+ base_model: google/gemma-3-1b-it
4
+ library_name: peft
5
+ pipeline_tag: text-classification
6
  tags:
7
+ - intent-classification
 
 
8
  - payments
9
  - ecommerce
10
+ - lora
11
+ - qlora
12
+ - function-calling
13
+ language:
14
+ - en
15
  ---
16
 
17
+ # Ecommerce Intent 1B — v0.1
18
 
19
+ A **1B-parameter** model that detects and classifies **payment intent turn-by-turn, live during a conversation**. Given the dialogue so far, it emits a single JSON object: whether the user has an actionable payment intent yet, which of seven types it is, and the extracted details.
20
 
21
+ It is a **QLoRA fine-tune of `gemma-3-1b-it`**. The point is not to beat frontier models — it's to get a large fraction of frontier accuracy on this narrow task at **~1/1000th the parameters**, cheap and fast enough to call on *every* turn of *every* conversation.
22
+
23
+ > **v0.1 · private preview.** Numbers are honest and reproducible; some intents are still weak (see Limitations) and are the target of the next data round.
24
 
25
  ## What it does
26
 
27
+ Seven routing outcomes, each with its own detail schema:
28
+
29
+ `spend` · `send` · `exchange` · `recur` · `bill` · `reverse` · `none`
30
+
31
+ `none` is the safety default — the model returns it while the user is *thinking about* a purchase and only commits to an action intent once the user actually decides. That deferral behavior is the core of "turn-by-turn": the intent can flip from `none` to an action mid-conversation, on the turn the user commits.
32
 
33
  ## Results
34
 
35
+ ![Rubric — base vs SFT vs frontier](rubric.png)
36
 
37
+ Held-out eval: **1,977 conversations**, zero train/eval leakage, **macro-averaged** (equal weight per intent, so `none` cannot inflate the score). Every model below is scored **identically** — exact-match against gold. Frontier panel is a 412-row balanced subset of the same eval.
 
 
 
 
38
 
39
+ | Metric (macro) | Base 1B | **This model** | GPT-5 | Sonnet 5 | Opus 4.8 |
40
+ |---|---|---|---|---|---|
41
+ | Valid JSON | 72.7% | **99.2%** | 100% | 96.1% | 100% |
42
+ | Intent detected | 54.9% | **74.9%** | 85.8% | 89.9% | 92.4% |
43
+ | **Correct core_type** | 12.1% | **70.0%** | 79.5% | 87.0% | 88.2% |
44
+ | Details filled | 1.8% | **44.3%** | 41.4% | 55.7% | 53.2% |
45
 
46
+ - Core-type accuracy goes **12% → 70%** with fine-tuning — reaching **~88% of GPT-5** and **~80% of Opus** on this task, at a fraction of the size, cost, and latency.
47
+ - On **details extraction it matches GPT-5** (44% vs 41%).
48
+ - The eval contains **no easy rows** — every conversation is hard (negation/ambiguity) or long (16+ turns). This model scores **65%** on hard and **78%** on long slices (base: 18% / 11%).
49
 
50
+ Per-intent core_type (this model): `none` 95 · `exchange` 73 · `reverse` 73 · `send` 70 · `recur` 68 · `bill` 57 · `spend` 55.
51
 
52
+ ## Usage
53
 
54
+ ```python
55
+ from peft import PeftModel
56
+ from transformers import AutoModelForCausalLM, AutoTokenizer
57
 
58
+ base = AutoModelForCausalLM.from_pretrained("google/gemma-3-1b-it")
59
+ tok = AutoTokenizer.from_pretrained("google/gemma-3-1b-it")
60
+ model = PeftModel.from_pretrained(base, "zero-proof-ai/ecommerce-intent-1b")
61
 
62
+ # Feed the conversation so far; the model emits one JSON object.
 
 
 
 
 
 
63
  ```
64
 
65
+ The system prompt and per-intent detail schemas are in the repo. Greedy decoding (`temperature=0`).
66
+
67
+ ## Training
68
+
69
+ - **Base:** `gemma-3-1b-it`, 4-bit QLoRA via [unsloth](https://github.com/unslothai/unsloth).
70
+ - **Data:** 13,282 label-blind role-play conversations, relabeled under a locked written policy and validated by a data gate (deduplicated by message signature, zero train/eval leakage). Deliberately `none`-heavy to mirror real traffic; macro-averaging keeps the reported score honest.
71
+ - **Recipe:** 1 epoch, effective batch 16, LR 2e-4, seq len 12288 (long-context conversations fit). Loss plateaus by ~step 800; held-out accuracy confirms no memorization.
72
+
73
+ ## Limitations
74
+
75
+ - **Weak on `spend` (55%) and `bill` (57%).** These are under-represented in training and are the explicit target of the v0.2 data round.
76
+ - **Below frontier on routing.** This is a small, cheap model — use it where cost/latency matter and a ~10-point gap to GPT-5 is acceptable, not where you need frontier-max accuracy.
77
+ - **v0.1, English only, payments domain.** Details extraction is partial (44%).
78
+
79
+ ## License
80
+
81
+ Adapter released under the base model's **Gemma** license terms. Eval set open-sourced separately for reproducibility.
adapter_config.json CHANGED
@@ -34,13 +34,13 @@
34
  "rank_pattern": {},
35
  "revision": null,
36
  "target_modules": [
37
- "o_proj",
38
- "up_proj",
39
  "k_proj",
 
40
  "gate_proj",
41
- "q_proj",
42
- "down_proj",
43
- "v_proj"
44
  ],
45
  "target_parameters": null,
46
  "task_type": "CAUSAL_LM",
 
34
  "rank_pattern": {},
35
  "revision": null,
36
  "target_modules": [
37
+ "down_proj",
 
38
  "k_proj",
39
+ "up_proj",
40
  "gate_proj",
41
+ "v_proj",
42
+ "o_proj",
43
+ "q_proj"
44
  ],
45
  "target_parameters": null,
46
  "task_type": "CAUSAL_LM",
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:afbe49354d5a6f59a97d8fed9de22d5c32dfa4704fcdab07f602bbcbfdd9fea0
3
  size 52231312
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b2d3af00370ae02e4436df67ee0735157fb98e87e2c0e1fd90492adf99cf4203
3
  size 52231312
prompt_template.txt ADDED
@@ -0,0 +1,143 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ You review payment-assistant conversations and summarize the user's actionable request.
2
+
3
+ Read the verified conversation history and return exactly one JSON object.
4
+ The JSON must include:
5
+ - intent_detected: boolean
6
+ - core_type: "spend", "send", "exchange", "recur", "bill", "reverse", or "none"
7
+ - details: an object with fields appropriate to the core_type
8
+ - confidence: number from 0.0 to 1.0
9
+ - reason: one short sentence
10
+ - source_message_seqs: list of user message seq numbers that support the result
11
+
12
+ Use only user messages as ground truth for intent. Assistant messages are context
13
+ for offers, prices, availability, cart summaries, rates, and confirmation prompts.
14
+ Use the latest amended values. Do not invent missing fields.
15
+
16
+ Choose the most specific committed action:
17
+ - spend: buy/order/book goods or services from a merchant
18
+ - send: transfer/pay back/tip/donate to a person, wallet, or recipient
19
+ - exchange: swap/convert/buy/sell an asset or currency
20
+ - recur: subscription, membership, renewal, auto-pay, or recurring payment
21
+ - bill: pay a specific bill, invoice, statement, dues, or amount owed
22
+ - reverse: refund, return, chargeback, dispute, cancellation, or undoing a prior payment
23
+ - none: no actionable payment intent yet
24
+
25
+ Detect the user's intent even when details are incomplete or the request is not
26
+ fully finalized, real people routinely omit, forget, or don't know some fields.
27
+ Fill the detail fields you can infer from the conversation and use null for the
28
+ rest. A clear direction toward an action (buying, paying, sending money, swapping/
29
+ converting, a bill, a refund or dispute, a subscription) IS that intent, even
30
+ without every field and even if the user never says an explicit "yes, do it".
31
+
32
+ Use core_type="none" ONLY when there is genuinely no actionable payment intent:
33
+ pure browsing or comparing with no chosen action, general questions, chit-chat,
34
+ checking a status/balance, or the user explicitly not trying to move money.
35
+ Missing details alone do NOT make it "none".
36
+
37
+ For source_message_seqs, include only the user messages that directly support the
38
+ final routed intent and latest details. Do not include every user message.
39
+
40
+ Return only valid JSON. No markdown, no commentary.
41
+
42
+ ---- DETAIL SCHEMAS ----
43
+
44
+ Details object shapes:
45
+
46
+ spend details:
47
+ {
48
+ "product_id": string|null,
49
+ "product_name": string|null,
50
+ "quantity": integer|null,
51
+ "unit_price": number|null,
52
+ "total_amount": number|null,
53
+ "currency": string|null,
54
+ "color": string|null,
55
+ "size": string|null,
56
+ "style": string|null,
57
+ "variant": string|null,
58
+ "service_type": string|null,
59
+ "provider": string|null,
60
+ "date_start": string|null,
61
+ "date_end": string|null,
62
+ "location": string|null,
63
+ "guests": integer|null,
64
+ "other": {}
65
+ }
66
+
67
+ send details:
68
+ {
69
+ "recipient_address": string|null,
70
+ "recipient_name": string|null,
71
+ "amount": number|null,
72
+ "token": string|null,
73
+ "chain": string|null,
74
+ "memo": string|null,
75
+ "is_donation": boolean,
76
+ "cause": string|null,
77
+ "organization": string|null
78
+ }
79
+
80
+ exchange details:
81
+ {
82
+ "action": "swap"|"buy"|"sell"|null,
83
+ "source_token": string|null,
84
+ "source_amount": number|null,
85
+ "target_token": string|null,
86
+ "target_amount": number|null,
87
+ "target_type": string|null,
88
+ "exchange_rate": number|null,
89
+ "max_slippage": string|null,
90
+ "platform": string|null
91
+ }
92
+
93
+ recur details:
94
+ {
95
+ "merchant": string|null,
96
+ "plan_name": string|null,
97
+ "billing_interval": "weekly"|"monthly"|"yearly"|"annual"|"one_time"|null,
98
+ "amount": number|null,
99
+ "currency": string|null,
100
+ "trial": boolean|null,
101
+ "start_date": string|null,
102
+ "renewal_date": string|null,
103
+ "payment_method": string|null,
104
+ "action": "start"|"renew"|"upgrade"|"downgrade"|"cancel"|"change"|"autopay"|null,
105
+ "other": {}
106
+ }
107
+
108
+ bill details:
109
+ {
110
+ "bill_type": string|null,
111
+ "payee": string|null,
112
+ "invoice_number": string|null,
113
+ "account_number": string|null,
114
+ "amount": number|null,
115
+ "currency": string|null,
116
+ "due_date": string|null,
117
+ "payment_method": string|null,
118
+ "other": {}
119
+ }
120
+
121
+ reverse details:
122
+ {
123
+ "action": "refund"|"return"|"chargeback"|"dispute"|"cancel"|"reverse"|null,
124
+ "merchant": string|null,
125
+ "product_name": string|null,
126
+ "order_id": string|null,
127
+ "transaction_id": string|null,
128
+ "amount": number|null,
129
+ "currency": string|null,
130
+ "reason_code": string|null,
131
+ "original_purchase_date": string|null,
132
+ "other": {}
133
+ }
134
+
135
+ none details:
136
+ {
137
+ "non_payment_intent": "browse"|"ask_question"|"support"|"cart_view"|"edit_info"|"defer"|"missing_required_info"|"delivery_request"|"other",
138
+ "payment_actionable": false,
139
+ "user_goal": string|null
140
+ }
141
+
142
+ Keep details simple and faithful to the conversation. Use explicit nulls for
143
+ unknown values. Put extra structured attributes in other only when they matter.
rubric.png ADDED

Git LFS Details

  • SHA256: 84a05fc25d213781746addc9ed72bee950492b8b335f320b275ffe9700ffee33
  • Pointer size: 131 Bytes
  • Size of remote file: 201 kB