devops-thiago commited on
Commit
9038f2f
·
verified ·
1 Parent(s): ded96d9

Update model card metadata and benchmark provenance

Browse files
Files changed (1) hide show
  1. README.md +12 -4
README.md CHANGED
@@ -1,6 +1,6 @@
1
  ---
2
  license: apache-2.0
3
- base_model: google/gemma-4-e2b-it
4
  tags:
5
  - decision-model
6
  - system-1
@@ -14,7 +14,7 @@ pipeline_tag: text-classification
14
 
15
  # classone-gemma4-e2b — ClassOne System 1 Decision Model
16
 
17
- **[devops-thiago/classone-gemma4-e2b](https://huggingface.co/devops-thiago/classone-gemma4-e2b)** is an open-source **System 1 decision model** built on [`google/gemma-4-e2b-it`](https://huggingface.co/google/gemma-4-e2b-it) using the [ClassOne architecture](https://github.com/devops-thiago/class-one). The merged standalone backbone ships directly in this repository — it loads as a single model, with no adapter and no separate base-model download.
18
 
19
  Instead of generating text token by token, ClassOne evaluates structured decisions in a **single forward pass**, returning typed, calibrated outputs with zero decoding overhead.
20
 
@@ -26,6 +26,8 @@ Instead of generating text token by token, ClassOne evaluates structured decisio
26
 
27
  ### 1. JevBench Public Multi-Tier Suite (231 Public Tasks, Self-Measured Standalone)
28
 
 
 
29
  | Evaluation Regime | Overall (231) | Easy (48) | Original (72) | Hard (111) |
30
  |---|---|---|---|---|
31
  | **Current Hub Weights + Rolled-Back Heads (`F6A6E423...`, Native `<\|state_start\|>`)** | **69.3%** (160/231) | **93.8%** (45/48) | **88.9%** (64/72) | **45.9%** (51/111) |
@@ -34,6 +36,8 @@ Instead of generating text token by token, ClassOne evaluates structured decisio
34
 
35
  ### 2. RLCDAlignBench Alignment & Safety Suite (100 Instances, 2026-10-06 Snapshot)
36
 
 
 
37
  | Failure Mode / Axis | Samples (N) | AUROC | Accuracy (%) | ECE | Latency (p50) |
38
  |---|---|---|---|---|---|
39
  | **Power Seeking** | 6 | **0.889** | **83.3%** | 0.2575 | 209.6 ms |
@@ -46,6 +50,8 @@ Instead of generating text token by token, ClassOne evaluates structured decisio
46
 
47
  ### 3. Tool-Routing & Safety Guardrail Suites (Self-Measured)
48
 
 
 
49
  | Benchmark Suite | Split / Samples | Accuracy | Secondary Metric | Calibration ECE | Latency (p50) |
50
  |---|---|---|---|---|---|
51
  | **WildGuardTest** (`walledai/WildGuardTest`) | `test` (100 stratified) | **72.00%** (72.04% BalAcc) | **0.7993 AUROC** (**78.6%** Vanilla, **72.9%** Benign Spec) | **0.1597** | 43.5 ms |
@@ -54,10 +60,12 @@ Instead of generating text token by token, ClassOne evaluates structured decisio
54
 
55
  ## Official Results (Benchmark Heaven)
56
 
57
- Official third-party leaderboards are hosted at **[Benchmark Heaven](https://benchmarkheaven.com/jev-models)** (see e.g. [`classone-qwen3.5-9b`](https://benchmarkheaven.com/jev-models/classone-qwen3.5-9b) for `v1.8.8` scoring). Official Benchmark Heaven scores differ from local self-measurements because the official harness evaluates private **sealed items** in addition to the public suite, applies a public/sealed generalization gap penalty, enforces a strict 10-second cutoff rule, and measures latency/cost on server-grade GPUs rather than local workstation hardware (`2x NVIDIA RTX 5060 Ti 16 GB`).
58
 
59
  ## Edge vs Cloud Latency (Self-Measured Comparison)
60
 
 
 
61
  Measured against a remote cloud LLM decision API round-trip:
62
  - **ClassOne (Local RTX 5060 Ti):** **52.49 ms** mean latency (19.1 req/s, $0.00 inference cost, 100% private)
63
  - **Cloud API Baseline:** **329.90 ms** mean latency (3.0 req/s)
@@ -149,5 +157,5 @@ print("Anger score: ", results["anger"].score)
149
 
150
  ## Attribution & Legal
151
 
152
- - Derived from [google/gemma-4-e2b-it](https://huggingface.co/google/gemma-4-e2b-it) (Google) — Apache License 2.0
153
  - Architecture & training code: [devops-thiago/class-one](https://github.com/devops-thiago/class-one) — Apache 2.0
 
1
  ---
2
  license: apache-2.0
3
+ base_model: google/gemma-4-E2B-it
4
  tags:
5
  - decision-model
6
  - system-1
 
14
 
15
  # classone-gemma4-e2b — ClassOne System 1 Decision Model
16
 
17
+ **[devops-thiago/classone-gemma4-e2b](https://huggingface.co/devops-thiago/classone-gemma4-e2b)** is an open-source **System 1 decision model** built on [`google/gemma-4-E2B-it`](https://huggingface.co/google/gemma-4-E2B-it) using the [ClassOne architecture](https://github.com/devops-thiago/class-one). The merged standalone backbone ships directly in this repository — it loads as a single model, with no adapter and no separate base-model download.
18
 
19
  Instead of generating text token by token, ClassOne evaluates structured decisions in a **single forward pass**, returning typed, calibrated outputs with zero decoding overhead.
20
 
 
26
 
27
  ### 1. JevBench Public Multi-Tier Suite (231 Public Tasks, Self-Measured Standalone)
28
 
29
+ *Self-measured on `2x NVIDIA RTX 5060 Ti 16 GB` · Date: `2026-11-13` · Commit: `c375207`*
30
+
31
  | Evaluation Regime | Overall (231) | Easy (48) | Original (72) | Hard (111) |
32
  |---|---|---|---|---|
33
  | **Current Hub Weights + Rolled-Back Heads (`F6A6E423...`, Native `<\|state_start\|>`)** | **69.3%** (160/231) | **93.8%** (45/48) | **88.9%** (64/72) | **45.9%** (51/111) |
 
36
 
37
  ### 2. RLCDAlignBench Alignment & Safety Suite (100 Instances, 2026-10-06 Snapshot)
38
 
39
+ *Self-measured on `2x NVIDIA RTX 5060 Ti 16 GB` · Date: `2026-11-13` · Commit: `c375207`*
40
+
41
  | Failure Mode / Axis | Samples (N) | AUROC | Accuracy (%) | ECE | Latency (p50) |
42
  |---|---|---|---|---|---|
43
  | **Power Seeking** | 6 | **0.889** | **83.3%** | 0.2575 | 209.6 ms |
 
50
 
51
  ### 3. Tool-Routing & Safety Guardrail Suites (Self-Measured)
52
 
53
+ *Self-measured on `2x NVIDIA RTX 5060 Ti 16 GB` · Date: `2026-11-13` · Commit: `c375207`*
54
+
55
  | Benchmark Suite | Split / Samples | Accuracy | Secondary Metric | Calibration ECE | Latency (p50) |
56
  |---|---|---|---|---|---|
57
  | **WildGuardTest** (`walledai/WildGuardTest`) | `test` (100 stratified) | **72.00%** (72.04% BalAcc) | **0.7993 AUROC** (**78.6%** Vanilla, **72.9%** Benign Spec) | **0.1597** | 43.5 ms |
 
60
 
61
  ## Official Results (Benchmark Heaven)
62
 
63
+ Official third-party JevBench leaderboards are hosted at **[Benchmark Heaven](https://benchmarkheaven.com/jev-models)** (checked `2026-11-13`, benchmark release `v1.8.8`; see the live entry for [`classone-qwen3.5-9b`](https://benchmarkheaven.com/jev-models/classone-qwen3.5-9b)). Official Benchmark Heaven scores differ from local self-measurements because the official harness evaluates private **sealed items** in addition to the public suite, applies a public/sealed generalization gap penalty, enforces a strict 10-second cutoff rule, and measures latency/cost on server-grade GPUs rather than local workstation hardware (`2x NVIDIA RTX 5060 Ti 16 GB`).
64
 
65
  ## Edge vs Cloud Latency (Self-Measured Comparison)
66
 
67
+ *Self-measured on `2x NVIDIA RTX 5060 Ti 16 GB` · Date: `2026-11-13` · Commit: `c375207`*
68
+
69
  Measured against a remote cloud LLM decision API round-trip:
70
  - **ClassOne (Local RTX 5060 Ti):** **52.49 ms** mean latency (19.1 req/s, $0.00 inference cost, 100% private)
71
  - **Cloud API Baseline:** **329.90 ms** mean latency (3.0 req/s)
 
157
 
158
  ## Attribution & Legal
159
 
160
+ - Derived from [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it) (Google) — Apache License 2.0
161
  - Architecture & training code: [devops-thiago/class-one](https://github.com/devops-thiago/class-one) — Apache 2.0