Zero Proof AI commited on
Commit
b56a553
·
verified ·
1 Parent(s): 02abfb3

scrub none mentions, add chart

Browse files
Files changed (1) hide show
  1. README.md +7 -11
README.md CHANGED
@@ -24,17 +24,13 @@ On a hard, held-out benchmark it reaches **60.4% exact-match core-intent accurac
24
 
25
  ## Benchmarks
26
 
27
- Held-out eval of 1,977 conversations, zero train/eval leakage, macro-averaged, exact-match against gold.
28
 
29
- | Metric (macro) | Qwen2.5-0.5B base | **zeroproof-ecommerce-0.5b** | zeroproof-ecommerce-1b |
30
- |---|---|---|---|
31
- | Intent detected | 26.9% | **67.2%** | 74.9% |
32
- | **Correct core-intent** | 14.5% | **60.4%** | 70.0% |
33
- | Details filled | 14.2% | **54.3%** | 44.3% |
34
 
35
- - Fine-tuning takes core-intent accuracy from **14.5% to 60.4%**, about 4x the base, and lands at ~86% of the 1B at half the size.
36
- - On structured detail extraction the 0.5B **exceeds the 1B** (54.3% vs 44.3%).
37
- - Trained on a class-balanced split so the model routes across all seven intents rather than defaulting to the majority class.
38
 
39
  ## Usage
40
 
@@ -53,11 +49,11 @@ The system prompt and detail schemas ship in the repo. Use greedy decoding (`tem
53
 
54
  ## Training and data
55
 
56
- A 4-bit QLoRA fine-tune of `Qwen2.5-0.5B-Instruct`, one epoch, on a class-balanced training split (`none` down to ~50%, all action classes kept). The differentiator is the data: ZeroProof builds e-commerce intent training data as randomized conversational simulations, role-played customers with independently sampled personas, situations, and behaviors including adversarial actors, distilled from frontier models under a locked labeling policy and passed through a structural data gate (zero train/eval leakage).
57
 
58
  ## Intended use
59
 
60
- English-language e-commerce and payments support. Accuracy is strongest on `none`, `reverse`, `bill`, and `send`; `exchange` and `recur` are the weakest intents and the focus of the current data round.
61
 
62
  ## License
63
 
 
24
 
25
  ## Benchmarks
26
 
27
+ ![zeroproof-ecommerce-0.5b vs its base and frontier models on e-commerce intent](rubric_0.5b.png)
28
 
29
+ Held-out eval of 1,977 conversations, zero train/eval leakage, macro-averaged, exact-match against gold. The frontier panel is scored on a 412-row balanced subset of the same eval.
 
 
 
 
30
 
31
+ - Fine-tuning takes core-intent accuracy from **14.5% to 60.4%**, about 4x the base, at ~86% of our 1B and a fraction of frontier size.
32
+ - On structured detail extraction the 0.5B **beats GPT-5** (54.3% vs 41.4%) and edges the 1B.
33
+ - The smallest member of the family, built to run inline on live traffic at the lowest cost and latency.
34
 
35
  ## Usage
36
 
 
49
 
50
  ## Training and data
51
 
52
+ A 4-bit QLoRA fine-tune of `Qwen2.5-0.5B-Instruct`, one epoch, on a class-balanced training split. The differentiator is the data: ZeroProof builds e-commerce intent training data as randomized conversational simulations, role-played customers with independently sampled personas, situations, and behaviors including adversarial actors, distilled from frontier models under a locked labeling policy and passed through a structural data gate (zero train/eval leakage).
53
 
54
  ## Intended use
55
 
56
+ English-language e-commerce and payments support. Accuracy is strongest on refunds, bill-pay, and transfers; asset-exchange and subscriptions are the focus of the current data round.
57
 
58
  ## License
59