Update model card metadata and benchmark provenance
Browse files
README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
-
base_model: google/gemma-4-
|
| 4 |
tags:
|
| 5 |
- decision-model
|
| 6 |
- system-1
|
|
@@ -14,7 +14,7 @@ pipeline_tag: text-classification
|
|
| 14 |
|
| 15 |
# classone-gemma4-e2b — ClassOne System 1 Decision Model
|
| 16 |
|
| 17 |
-
**[devops-thiago/classone-gemma4-e2b](https://huggingface.co/devops-thiago/classone-gemma4-e2b)** is an open-source **System 1 decision model** built on [`google/gemma-4-
|
| 18 |
|
| 19 |
Instead of generating text token by token, ClassOne evaluates structured decisions in a **single forward pass**, returning typed, calibrated outputs with zero decoding overhead.
|
| 20 |
|
|
@@ -26,6 +26,8 @@ Instead of generating text token by token, ClassOne evaluates structured decisio
|
|
| 26 |
|
| 27 |
### 1. JevBench Public Multi-Tier Suite (231 Public Tasks, Self-Measured Standalone)
|
| 28 |
|
|
|
|
|
|
|
| 29 |
| Evaluation Regime | Overall (231) | Easy (48) | Original (72) | Hard (111) |
|
| 30 |
|---|---|---|---|---|
|
| 31 |
| **Current Hub Weights + Rolled-Back Heads (`F6A6E423...`, Native `<\|state_start\|>`)** | **69.3%** (160/231) | **93.8%** (45/48) | **88.9%** (64/72) | **45.9%** (51/111) |
|
|
@@ -34,6 +36,8 @@ Instead of generating text token by token, ClassOne evaluates structured decisio
|
|
| 34 |
|
| 35 |
### 2. RLCDAlignBench Alignment & Safety Suite (100 Instances, 2026-10-06 Snapshot)
|
| 36 |
|
|
|
|
|
|
|
| 37 |
| Failure Mode / Axis | Samples (N) | AUROC | Accuracy (%) | ECE | Latency (p50) |
|
| 38 |
|---|---|---|---|---|---|
|
| 39 |
| **Power Seeking** | 6 | **0.889** | **83.3%** | 0.2575 | 209.6 ms |
|
|
@@ -46,6 +50,8 @@ Instead of generating text token by token, ClassOne evaluates structured decisio
|
|
| 46 |
|
| 47 |
### 3. Tool-Routing & Safety Guardrail Suites (Self-Measured)
|
| 48 |
|
|
|
|
|
|
|
| 49 |
| Benchmark Suite | Split / Samples | Accuracy | Secondary Metric | Calibration ECE | Latency (p50) |
|
| 50 |
|---|---|---|---|---|---|
|
| 51 |
| **WildGuardTest** (`walledai/WildGuardTest`) | `test` (100 stratified) | **72.00%** (72.04% BalAcc) | **0.7993 AUROC** (**78.6%** Vanilla, **72.9%** Benign Spec) | **0.1597** | 43.5 ms |
|
|
@@ -54,10 +60,12 @@ Instead of generating text token by token, ClassOne evaluates structured decisio
|
|
| 54 |
|
| 55 |
## Official Results (Benchmark Heaven)
|
| 56 |
|
| 57 |
-
Official third-party leaderboards are hosted at **[Benchmark Heaven](https://benchmarkheaven.com/jev-models)** (
|
| 58 |
|
| 59 |
## Edge vs Cloud Latency (Self-Measured Comparison)
|
| 60 |
|
|
|
|
|
|
|
| 61 |
Measured against a remote cloud LLM decision API round-trip:
|
| 62 |
- **ClassOne (Local RTX 5060 Ti):** **52.49 ms** mean latency (19.1 req/s, $0.00 inference cost, 100% private)
|
| 63 |
- **Cloud API Baseline:** **329.90 ms** mean latency (3.0 req/s)
|
|
@@ -149,5 +157,5 @@ print("Anger score: ", results["anger"].score)
|
|
| 149 |
|
| 150 |
## Attribution & Legal
|
| 151 |
|
| 152 |
-
- Derived from [google/gemma-4-
|
| 153 |
- Architecture & training code: [devops-thiago/class-one](https://github.com/devops-thiago/class-one) — Apache 2.0
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
base_model: google/gemma-4-E2B-it
|
| 4 |
tags:
|
| 5 |
- decision-model
|
| 6 |
- system-1
|
|
|
|
| 14 |
|
| 15 |
# classone-gemma4-e2b — ClassOne System 1 Decision Model
|
| 16 |
|
| 17 |
+
**[devops-thiago/classone-gemma4-e2b](https://huggingface.co/devops-thiago/classone-gemma4-e2b)** is an open-source **System 1 decision model** built on [`google/gemma-4-E2B-it`](https://huggingface.co/google/gemma-4-E2B-it) using the [ClassOne architecture](https://github.com/devops-thiago/class-one). The merged standalone backbone ships directly in this repository — it loads as a single model, with no adapter and no separate base-model download.
|
| 18 |
|
| 19 |
Instead of generating text token by token, ClassOne evaluates structured decisions in a **single forward pass**, returning typed, calibrated outputs with zero decoding overhead.
|
| 20 |
|
|
|
|
| 26 |
|
| 27 |
### 1. JevBench Public Multi-Tier Suite (231 Public Tasks, Self-Measured Standalone)
|
| 28 |
|
| 29 |
+
*Self-measured on `2x NVIDIA RTX 5060 Ti 16 GB` · Date: `2026-11-13` · Commit: `c375207`*
|
| 30 |
+
|
| 31 |
| Evaluation Regime | Overall (231) | Easy (48) | Original (72) | Hard (111) |
|
| 32 |
|---|---|---|---|---|
|
| 33 |
| **Current Hub Weights + Rolled-Back Heads (`F6A6E423...`, Native `<\|state_start\|>`)** | **69.3%** (160/231) | **93.8%** (45/48) | **88.9%** (64/72) | **45.9%** (51/111) |
|
|
|
|
| 36 |
|
| 37 |
### 2. RLCDAlignBench Alignment & Safety Suite (100 Instances, 2026-10-06 Snapshot)
|
| 38 |
|
| 39 |
+
*Self-measured on `2x NVIDIA RTX 5060 Ti 16 GB` · Date: `2026-11-13` · Commit: `c375207`*
|
| 40 |
+
|
| 41 |
| Failure Mode / Axis | Samples (N) | AUROC | Accuracy (%) | ECE | Latency (p50) |
|
| 42 |
|---|---|---|---|---|---|
|
| 43 |
| **Power Seeking** | 6 | **0.889** | **83.3%** | 0.2575 | 209.6 ms |
|
|
|
|
| 50 |
|
| 51 |
### 3. Tool-Routing & Safety Guardrail Suites (Self-Measured)
|
| 52 |
|
| 53 |
+
*Self-measured on `2x NVIDIA RTX 5060 Ti 16 GB` · Date: `2026-11-13` · Commit: `c375207`*
|
| 54 |
+
|
| 55 |
| Benchmark Suite | Split / Samples | Accuracy | Secondary Metric | Calibration ECE | Latency (p50) |
|
| 56 |
|---|---|---|---|---|---|
|
| 57 |
| **WildGuardTest** (`walledai/WildGuardTest`) | `test` (100 stratified) | **72.00%** (72.04% BalAcc) | **0.7993 AUROC** (**78.6%** Vanilla, **72.9%** Benign Spec) | **0.1597** | 43.5 ms |
|
|
|
|
| 60 |
|
| 61 |
## Official Results (Benchmark Heaven)
|
| 62 |
|
| 63 |
+
Official third-party JevBench leaderboards are hosted at **[Benchmark Heaven](https://benchmarkheaven.com/jev-models)** (checked `2026-11-13`, benchmark release `v1.8.8`; see the live entry for [`classone-qwen3.5-9b`](https://benchmarkheaven.com/jev-models/classone-qwen3.5-9b)). Official Benchmark Heaven scores differ from local self-measurements because the official harness evaluates private **sealed items** in addition to the public suite, applies a public/sealed generalization gap penalty, enforces a strict 10-second cutoff rule, and measures latency/cost on server-grade GPUs rather than local workstation hardware (`2x NVIDIA RTX 5060 Ti 16 GB`).
|
| 64 |
|
| 65 |
## Edge vs Cloud Latency (Self-Measured Comparison)
|
| 66 |
|
| 67 |
+
*Self-measured on `2x NVIDIA RTX 5060 Ti 16 GB` · Date: `2026-11-13` · Commit: `c375207`*
|
| 68 |
+
|
| 69 |
Measured against a remote cloud LLM decision API round-trip:
|
| 70 |
- **ClassOne (Local RTX 5060 Ti):** **52.49 ms** mean latency (19.1 req/s, $0.00 inference cost, 100% private)
|
| 71 |
- **Cloud API Baseline:** **329.90 ms** mean latency (3.0 req/s)
|
|
|
|
| 157 |
|
| 158 |
## Attribution & Legal
|
| 159 |
|
| 160 |
+
- Derived from [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it) (Google) — Apache License 2.0
|
| 161 |
- Architecture & training code: [devops-thiago/class-one](https://github.com/devops-thiago/class-one) — Apache 2.0
|