Zero Proof AI commited on
Commit
2c30adf
·
verified ·
1 Parent(s): d293434

Update results

Browse files
Files changed (1) hide show
  1. README.md +4 -5
README.md CHANGED
@@ -20,7 +20,7 @@ language:
20
 
21
  **Frontier-approaching e-commerce intent detection in a 1B model.** Given a live customer conversation, `zeroproof-ecommerce-1b` returns a single JSON object with the customer's payment intent and its structured details. It runs inline on live traffic, at a cost and latency where calling a frontier model is not an option.
22
 
23
- On a hard, held-out benchmark it reaches **70 macro-averaged intent-type accuracy, 88% of GPT-5's**, at **$0.18 per 1M output tokens** and about a second per response.
24
 
25
  ## Model Details
26
 
@@ -106,10 +106,9 @@ Held-out eval of 1,977 conversations, zero train/eval leakage, macro-averaged (e
106
 
107
  ![ZeroProof e-commerce intent evaluation: accuracy, per-action accuracy, and cost to serve](1b-v1.png)
108
 
109
- - Fine-tuning takes intent-type accuracy from **12 to 70** (macro), landing at **88% of GPT-5's 79.5**.
110
- - On structured detail extraction it **exceeds the full frontier panel**: 63 against Sonnet 5's 56, Opus 4.8's 53, and GPT-5's 41.
111
- - On the no-action boundary it errs toward inaction: **4.2%** of true no-action conversations are marked actionable, with remaining errors concentrated in the safe direction (missed actions, not invented ones).
112
- - No easy rows: every conversation is adversarial (negation, ambiguity) or long. The model holds **65** on the hard slice and **70** on the long slice, where the base scores 18 and 10.
113
 
114
  ## Technical Specifications
115
 
 
20
 
21
  **Frontier-approaching e-commerce intent detection in a 1B model.** Given a live customer conversation, `zeroproof-ecommerce-1b` returns a single JSON object with the customer's payment intent and its structured details. It runs inline on live traffic, at a cost and latency where calling a frontier model is not an option.
22
 
23
+ On a hard, held-out benchmark it reaches **77.5 macro-averaged intent-type accuracy, 96% of GPT-5's**, at **$0.18 per 1M output tokens** and about a second per response.
24
 
25
  ## Model Details
26
 
 
106
 
107
  ![ZeroProof e-commerce intent evaluation: accuracy, per-action accuracy, and cost to serve](1b-v1.png)
108
 
109
+ - Fine-tuning takes intent-type accuracy from **12 to 77.5** (macro), landing at **96% of GPT-5's 80.2**.
110
+ - On structured detail extraction it **exceeds the full frontier panel**: 64 against Sonnet 5's 56, Opus 4.8's 53, and GPT-5's 41.
111
+ - On the no-action boundary it errs toward inaction: **9.2%** of true no-action conversations are marked actionable, with remaining errors concentrated in the safe direction (missed actions, not invented ones).
 
112
 
113
  ## Technical Specifications
114