Zero Proof AI commited on
Commit
a29dc55
·
verified ·
1 Parent(s): afd2305

Refresh README.md

Browse files
Files changed (1) hide show
  1. README.md +6 -7
README.md CHANGED
@@ -10,7 +10,6 @@ tags:
10
  - payments
11
  - ecommerce
12
  - lora
13
- - qlora
14
  - function-calling
15
  language:
16
  - en
@@ -20,7 +19,7 @@ language:
20
 
21
  **Frontier-approaching e-commerce intent detection in a 1B model.** Given a live customer conversation, `zeroproof-ecommerce-1b` returns a single JSON object with the customer's payment intent and its structured details. It runs inline on live traffic, at a cost and latency where calling a frontier model is not an option.
22
 
23
- On a hard, held-out benchmark it reaches **77.5 macro-averaged intent-type accuracy, 96% of GPT-5's**, at **$0.18 per 1M output tokens** and about a second per response.
24
 
25
  ## Model Details
26
 
@@ -30,7 +29,7 @@ On a hard, held-out benchmark it reaches **77.5 macro-averaged intent-type accur
30
  - **Model type:** E-commerce payment-intent classifier; structured JSON output over seven intent types
31
  - **Language:** English
32
  - **License:** Gemma (inherited from the base model)
33
- - **Finetuned from:** `google/gemma-3-1b-it`, via 4-bit QLoRA
34
 
35
  ### Model Sources
36
 
@@ -49,7 +48,7 @@ English-language e-commerce and payments only. It is a narrow intent classifier,
49
 
50
  ## Bias, Risks, and Limitations
51
 
52
- Trained and evaluated on synthetic role-play conversations from a multi-agent marketplace simulation, labeled in a separate model pass under a locked labeling policy; labels are model-assigned and carry residual noise, and live-traffic distribution may differ. Accuracy is currently strongest on refunds, exchanges, and transfers; checkout and bill-pay are the focus of the current data round.
53
 
54
  ### Recommendations
55
 
@@ -90,7 +89,7 @@ The differentiator is the data. ZeroProof builds e-commerce intent training data
90
 
91
  #### Training Hyperparameters
92
 
93
- - **Method:** 4-bit QLoRA (via [unsloth](https://github.com/unslothai/unsloth))
94
  - **Epochs:** 1
95
  - **Effective batch size:** 16
96
  - **Learning rate:** 2e-4
@@ -106,9 +105,9 @@ Held-out eval of 1,977 conversations, zero train/eval leakage, macro-averaged (e
106
 
107
  ![ZeroProof e-commerce intent evaluation: accuracy, per-action accuracy, and cost to serve](1b-v1.png)
108
 
109
- - Fine-tuning takes intent-type accuracy from **12 to 77.5** (macro), landing at **96% of GPT-5's 80.2**.
110
  - On structured detail extraction it **exceeds the full frontier panel**: 64 against Sonnet 5's 56, Opus 4.8's 53, and GPT-5's 41.
111
- - On the no-action boundary it errs toward inaction: **9.2%** of true no-action conversations are marked actionable, with remaining errors concentrated in the safe direction (missed actions, not invented ones).
112
 
113
  ## Technical Specifications
114
 
 
10
  - payments
11
  - ecommerce
12
  - lora
 
13
  - function-calling
14
  language:
15
  - en
 
19
 
20
  **Frontier-approaching e-commerce intent detection in a 1B model.** Given a live customer conversation, `zeroproof-ecommerce-1b` returns a single JSON object with the customer's payment intent and its structured details. It runs inline on live traffic, at a cost and latency where calling a frontier model is not an option.
21
 
22
+ On a hard, held-out benchmark it reaches **75.8 macro-averaged intent-type accuracy, 89% of GPT-5's**, at **$0.18 per 1M output tokens** and about a second per response.
23
 
24
  ## Model Details
25
 
 
29
  - **Model type:** E-commerce payment-intent classifier; structured JSON output over seven intent types
30
  - **Language:** English
31
  - **License:** Gemma (inherited from the base model)
32
+ - **Finetuned from:** `google/gemma-3-1b-it`, via LoRA
33
 
34
  ### Model Sources
35
 
 
48
 
49
  ## Bias, Risks, and Limitations
50
 
51
+ Trained and evaluated on synthetic role-play conversations from a multi-agent marketplace simulation, labeled in a separate model pass under a locked labeling policy; labels are model-assigned and carry residual noise, and live-traffic distribution may differ. Accuracy is currently strongest on refunds, subscriptions, and checkout; bill-pay is the focus of the current data round.
52
 
53
  ### Recommendations
54
 
 
89
 
90
  #### Training Hyperparameters
91
 
92
+ - **Method:** LoRA on the bf16 base (via [unsloth](https://github.com/unslothai/unsloth))
93
  - **Epochs:** 1
94
  - **Effective batch size:** 16
95
  - **Learning rate:** 2e-4
 
105
 
106
  ![ZeroProof e-commerce intent evaluation: accuracy, per-action accuracy, and cost to serve](1b-v1.png)
107
 
108
+ - Fine-tuning takes intent-type accuracy from **12.1 to 75.8** (macro); on the frontier panel's own rows it scores **71.4, 89% of GPT-5's 80.2**.
109
  - On structured detail extraction it **exceeds the full frontier panel**: 64 against Sonnet 5's 56, Opus 4.8's 53, and GPT-5's 41.
110
+ - On the no-action boundary it errs toward inaction: **7.5%** of true no-action conversations are marked actionable, with remaining errors concentrated in the safe direction (missed actions, not invented ones).
111
 
112
  ## Technical Specifications
113