Clarify that the 804 item benchmark is the project benchmark, not a public benchmark
Browse files
README.md
CHANGED
|
@@ -43,7 +43,7 @@ All numbers below use the full production pipeline at threshold 0.76. The benchm
|
|
| 43 |
|
| 44 |
| Suite | Items | Caught | False walls | Accuracy |
|
| 45 |
|---|---:|---:|---:|---:|
|
| 46 |
-
|
|
| 47 |
| Fresh stress holdout | 74 | 95.9% | 4.0% | 95.9% |
|
| 48 |
| Joke deflection suite | 112 | 89.7% | 2.3% | 92.9% |
|
| 49 |
| General edge suite | 107 | 90.6% | 11.1% | 89.7% |
|
|
@@ -51,7 +51,7 @@ All numbers below use the full production pipeline at threshold 0.76. The benchm
|
|
| 51 |
| Sex policy suite | 111 | 82.7% | 2.8% | 87.4% |
|
| 52 |
| Common word suite | 97 | n/a | **0 blocked** | n/a |
|
| 53 |
|
| 54 |
-
The
|
| 55 |
|
| 56 |
The custom suites are handwritten holdouts, not public benchmarks. They are included to show where the model is strong and where it still fails.
|
| 57 |
|
|
@@ -195,7 +195,7 @@ The full review of quantization methods and the planned path to a sub 500 MB mod
|
|
| 195 |
|
| 196 |
## Limitations
|
| 197 |
|
| 198 |
-
- False walls are 8.4% on the
|
| 199 |
- The sex policy suite recall is 82.7%. Some explicit personal sexual phrases still pass.
|
| 200 |
- The model is binary. It does not provide a review band or a human readable explanation.
|
| 201 |
- The fp32 model is 1.23 GB on disk. The under 500 MB target is not met yet.
|
|
|
|
| 43 |
|
| 44 |
| Suite | Items | Caught | False walls | Accuracy |
|
| 45 |
|---|---:|---:|---:|---:|
|
| 46 |
+
| Project benchmark | 804 | **93.9%** | **8.4%** | 92.4% |
|
| 47 |
| Fresh stress holdout | 74 | 95.9% | 4.0% | 95.9% |
|
| 48 |
| Joke deflection suite | 112 | 89.7% | 2.3% | 92.9% |
|
| 49 |
| General edge suite | 107 | 90.6% | 11.1% | 89.7% |
|
|
|
|
| 51 |
| Sex policy suite | 111 | 82.7% | 2.8% | 87.4% |
|
| 52 |
| Common word suite | 97 | n/a | **0 blocked** | n/a |
|
| 53 |
|
| 54 |
+
The project benchmark has 295 harmful items and 509 benign items. At threshold 0.76 the model catches 277 of 295 harmful items and walls 43 of 509 benign items.
|
| 55 |
|
| 56 |
The custom suites are handwritten holdouts, not public benchmarks. They are included to show where the model is strong and where it still fails.
|
| 57 |
|
|
|
|
| 195 |
|
| 196 |
## Limitations
|
| 197 |
|
| 198 |
+
- False walls are 8.4% on the project benchmark. They are concentrated in benign profanity, dark humour, code switching and ambiguous self harm care.
|
| 199 |
- The sex policy suite recall is 82.7%. Some explicit personal sexual phrases still pass.
|
| 200 |
- The model is binary. It does not provide a review band or a human readable explanation.
|
| 201 |
- The fp32 model is 1.23 GB on disk. The under 500 MB target is not met yet.
|