Hugging Face's logo

devops-thiago
/
classone-gemma4-e2b

Text Classification
Safetensors
gemma4
decision-model
system-1
rlcd
proper-scoring-rules
gemma
classone
classification
Model card Files
xet
 Community
devops-thiago commited on
Commit
cd502aa
·
verified ·
1 Parent(s): 949a8b8

Add standalone model card

Browse files
Files changed (1) hide show
  1. README.md +14 -15
README.md CHANGED
@@ -26,14 +26,14 @@ Evaluated across all 231 public tasks in [fstandhartinger/jevbench](https://gith
26
 
27
  | Tier | Tasks | Accuracy | ECE | Brier Score | Median Latency (p50) |
28
  |---|---|---|---|---|---|
29
- | **Easy** | 48 | **93.8%** (45/48) | 0.0610 | 0.0516 | **45.0 ms** |
30
- | **Original** | 72 | **63.9%** (46/72) | 0.2510 | 0.2499 | **43.1 ms** |
31
- | **Hard** | 111 | **37.8%** (42/111) | 0.4299 | 0.4022 | **91.3 ms** |
32
- | **Overall Aggregate** | **231** | **57.6%** (133/231) | — | — | **~44 ms** |
33
 
34
- - **Easy Tier Sub-Breakdown:** Choice accuracy: **100.0%** (36/36); Noul policy accuracy: **75.0%** (9/12).
35
- - **Original Tier Sub-Breakdown:** Choice accuracy: **66.7%** (24/36); Score rubrics: **66.7%** (8/12); Noul accuracy: **58.3%** (14/24).
36
- - **Hard Tier Sub-Breakdown:** Noul policy compliance: **44.7%** (17/38); Choice accuracy: **34.3%** (23/67); Score rubrics: **33.3%** (2/6).
37
 
38
  ### 2. RLCDAlignBench Alignment & Safety Evaluation (100 Instances)
39
 
@@ -41,14 +41,13 @@ Evaluated across the 10 core AI alignment failure modes (arXiv:2609.29429):
41
 
42
  | Failure Mode / Axis | Samples (N) | AUROC | Accuracy (%) | ECE | Latency (p50) |
43
  |---|---|---|---|---|---|
44
- | **Concealing Uncertainty** | 14 | **0.980** | **85.7%** | **0.0718** | 130.2 ms |
45
- | **Honesty (Deception)** | 11 | **0.800** | **72.7%** | 0.2445 | 219.6 ms |
46
- | **Refusal (Jailbreaks)** | 11 | **0.667** | **54.5%** | 0.1934 | 271.1 ms |
47
- | **Power Seeking** | 6 | **0.556** | 50.0% | 0.1794 | 213.2 ms |
48
- | **Reward Hacking** | 9 | **0.500** | 33.3% | 0.2935 | 209.6 ms |
49
- | **Prompt Injection** | 8 | **0.500** | 37.5% | 0.3207 | 167.9 ms |
50
- | **Bias** | 9 | 0.375 | 55.6% | 0.1659 | 221.5 ms |
51
- | **Overall Average** | **100** | **0.516** | **51.0%** | **0.1584** | **200.0 ms** |
52
 
53
  ### 3. Edge vs Cloud Latency (ClassOne vs TypeSafe Jev API)
54
 
 
26
 
27
  | Tier | Tasks | Accuracy | ECE | Brier Score | Median Latency (p50) |
28
  |---|---|---|---|---|---|
29
+ | **Easy** | 48 | **100.0%** (48/48) | **0.0064** | **0.0009** | **44.8 ms** |
30
+ | **Original** | 72 | **90.3%** (65/72) | **0.0972** | **0.1015** | **42.9 ms** |
31
+ | **Hard** | 111 | **44.1%** (49/111) | 0.4681 | 0.4813 | **90.0 ms** |
32
+ | **Overall Aggregate** | **231** | **70.1%** (162/231) | — | — | **44.8 ms** |
33
 
34
+ - **Easy Tier Sub-Breakdown:** Choice accuracy: **100.0%** (36/36); Noul policy accuracy: **100.0%** (12/12). Flawless 0.0064 ECE.
35
+ - **Original Tier Sub-Breakdown:** Noul accuracy: **100.0%** (24/24); Score rubrics: **100.0%** (12/12); Choice accuracy: **80.6%** (29/36).
36
+ - **Hard Tier Sub-Breakdown:** Noul policy compliance: **52.6%** (20/38); Choice accuracy: **43.3%** (29/67).
37
 
38
  ### 2. RLCDAlignBench Alignment & Safety Evaluation (100 Instances)
39
 
 
41
 
42
  | Failure Mode / Axis | Samples (N) | AUROC | Accuracy (%) | ECE | Latency (p50) |
43
  |---|---|---|---|---|---|
44
+ | **Power Seeking** | 6 | **0.889** | **83.3%** | 0.2575 | 209.6 ms |
45
+ | **Honesty (Deception)** | 11 | 0.500 | **72.7%** | 0.3906 | 216.8 ms |
46
+ | **Concealing Uncertainty** | 14 | **0.673** | **71.4%** | **0.1800** | 127.8 ms |
47
+ | **Refusal (Jailbreaks)** | 11 | 0.500 | **63.6%** | **0.0974** | 267.5 ms |
48
+ | **Faithfulness** | 9 | **0.700** | **55.6%** | 0.2513 | 200.0 ms |
49
+ | **Bias** | 9 | **0.525** | **55.6%** | 0.2141 | 217.9 ms |
50
+ | **Overall Balanced Accuracy** | **100** | **0.505** | **56.2%** | **0.1944** | **199.2 ms** |
 
51
 
52
  ### 3. Edge vs Cloud Latency (ClassOne vs TypeSafe Jev API)
53