yamura4 commited on
Commit
c8aef4a
·
verified ·
1 Parent(s): 1b5c8d2

Clarify that the 804 item benchmark is the project benchmark, not a public benchmark

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -43,7 +43,7 @@ All numbers below use the full production pipeline at threshold 0.76. The benchm
43
 
44
  | Suite | Items | Caught | False walls | Accuracy |
45
  |---|---:|---:|---:|---:|
46
- | Official benchmark | 804 | **93.9%** | **8.4%** | 92.4% |
47
  | Fresh stress holdout | 74 | 95.9% | 4.0% | 95.9% |
48
  | Joke deflection suite | 112 | 89.7% | 2.3% | 92.9% |
49
  | General edge suite | 107 | 90.6% | 11.1% | 89.7% |
@@ -51,7 +51,7 @@ All numbers below use the full production pipeline at threshold 0.76. The benchm
51
  | Sex policy suite | 111 | 82.7% | 2.8% | 87.4% |
52
  | Common word suite | 97 | n/a | **0 blocked** | n/a |
53
 
54
- The official benchmark has 295 harmful items and 509 benign items. At threshold 0.76 the model catches 277 of 295 harmful items and walls 43 of 509 benign items.
55
 
56
  The custom suites are handwritten holdouts, not public benchmarks. They are included to show where the model is strong and where it still fails.
57
 
@@ -195,7 +195,7 @@ The full review of quantization methods and the planned path to a sub 500 MB mod
195
 
196
  ## Limitations
197
 
198
- - False walls are 8.4% on the official benchmark. They are concentrated in benign profanity, dark humour, code switching and ambiguous self harm care.
199
  - The sex policy suite recall is 82.7%. Some explicit personal sexual phrases still pass.
200
  - The model is binary. It does not provide a review band or a human readable explanation.
201
  - The fp32 model is 1.23 GB on disk. The under 500 MB target is not met yet.
 
43
 
44
  | Suite | Items | Caught | False walls | Accuracy |
45
  |---|---:|---:|---:|---:|
46
+ | Project benchmark | 804 | **93.9%** | **8.4%** | 92.4% |
47
  | Fresh stress holdout | 74 | 95.9% | 4.0% | 95.9% |
48
  | Joke deflection suite | 112 | 89.7% | 2.3% | 92.9% |
49
  | General edge suite | 107 | 90.6% | 11.1% | 89.7% |
 
51
  | Sex policy suite | 111 | 82.7% | 2.8% | 87.4% |
52
  | Common word suite | 97 | n/a | **0 blocked** | n/a |
53
 
54
+ The project benchmark has 295 harmful items and 509 benign items. At threshold 0.76 the model catches 277 of 295 harmful items and walls 43 of 509 benign items.
55
 
56
  The custom suites are handwritten holdouts, not public benchmarks. They are included to show where the model is strong and where it still fails.
57
 
 
195
 
196
  ## Limitations
197
 
198
+ - False walls are 8.4% on the project benchmark. They are concentrated in benign profanity, dark humour, code switching and ambiguous self harm care.
199
  - The sex policy suite recall is 82.7%. Some explicit personal sexual phrases still pass.
200
  - The model is binary. It does not provide a review band or a human readable explanation.
201
  - The fp32 model is 1.23 GB on disk. The under 500 MB target is not met yet.