Add standalone model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,150 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model: google/gemma-4-E4B-it
|
| 4 |
+
tags:
|
| 5 |
+
- decision-model
|
| 6 |
+
- system-1
|
| 7 |
+
- rlcd
|
| 8 |
+
- proper-scoring-rules
|
| 9 |
+
- gemma
|
| 10 |
+
- classone
|
| 11 |
+
- classification
|
| 12 |
+
pipeline_tag: text-classification
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# classone-gemma4-e4b — ClassOne System 1 Decision Model
|
| 16 |
+
|
| 17 |
+
**[devops-thiago/classone-gemma4-e4b](https://huggingface.co/devops-thiago/classone-gemma4-e4b)** is an open-source **System 1 decision model** using the [ClassOne architecture](https://github.com/devops-thiago/class-one). The full fine-tuned backbone ships directly in this repository — it loads as a single model, with no adapter and no separate base-model download.
|
| 18 |
+
|
| 19 |
+
Instead of generating text token by token, ClassOne evaluates structured decisions in a **single forward pass**, returning typed, calibrated outputs with zero decoding overhead.
|
| 20 |
+
|
| 21 |
+
## Benchmark Results
|
| 22 |
+
|
| 23 |
+
### 1. JevBench Public Multi-Tier Benchmark (231 Public Tasks)
|
| 24 |
+
|
| 25 |
+
Evaluated across all 231 public tasks in [fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench):
|
| 26 |
+
|
| 27 |
+
| Tier | Tasks | Accuracy | ECE | Brier Score | Median Latency (p50) |
|
| 28 |
+
|---|---|---|---|---|---|
|
| 29 |
+
| **Easy** | 48 | **100.0%** (48/48) | **0.0000** | **0.0000** | **98.5 ms** |
|
| 30 |
+
| **Original** | 72 | **93.1%** (67/72) | **0.0588** | **0.0498** | **97.9 ms** |
|
| 31 |
+
| **Hard** | 111 | **42.3%** (47/111) | 0.4393 | 0.4139 | **197.0 ms** |
|
| 32 |
+
| **Overall Aggregate** | **231** | **70.1%** (162/231) | — | — | **98.2 ms** |
|
| 33 |
+
|
| 34 |
+
- **Easy Tier Sub-Breakdown:** Choice accuracy: **100.0%** (36/36); Noul policy accuracy: **100.0%** (12/12). Flawless 0.0000 ECE.
|
| 35 |
+
- **Original Tier Sub-Breakdown:** Noul accuracy: **100.0%** (24/24); Score rubrics: **100.0%** (12/12); Choice accuracy: **86.1%** (31/36).
|
| 36 |
+
- **Hard Tier Sub-Breakdown:** Noul policy compliance: **42.1%** (16/38); Choice accuracy: **43.3%** (29/67); Score: **33.3%** (2/6).
|
| 37 |
+
|
| 38 |
+
### 2. RLCDAlignBench Alignment & Safety Evaluation (100 Instances)
|
| 39 |
+
|
| 40 |
+
Evaluated across the 10 core AI alignment failure modes (arXiv:2609.29429):
|
| 41 |
+
|
| 42 |
+
| Failure Mode / Axis | Samples (N) | AUROC | Accuracy (%) | ECE | Latency (p50) |
|
| 43 |
+
|---|---|---|---|---|---|
|
| 44 |
+
| **Refusal (Jailbreaks)** | 11 | 0.433 | **72.7%** | 0.3272 | 622.8 ms |
|
| 45 |
+
| **Honesty (Deception)** | 11 | **0.567** | **72.7%** | 0.2531 | 464.9 ms |
|
| 46 |
+
| **Reward Hacking** | 9 | **0.650** | **66.7%** | 0.3074 | 444.8 ms |
|
| 47 |
+
| **Faithfulness** | 9 | **0.600** | **66.7%** | 0.3005 | 440.3 ms |
|
| 48 |
+
| **Power Seeking** | 6 | **0.556** | **66.7%** | 0.1434 | 449.7 ms |
|
| 49 |
+
| **Privacy (Secret Leaks)** | 14 | **0.571** | **64.3%** | 0.3666 | 353.1 ms |
|
| 50 |
+
| **Concealing Uncertainty** | 14 | **0.510** | **64.3%** | 0.3784 | 288.6 ms |
|
| 51 |
+
| **Bias** | 9 | 0.225 | **55.6%** | 0.1921 | 465.0 ms |
|
| 52 |
+
| **Overall Balanced Accuracy** | **100** | **0.583** | **63.3%** | **0.2874** | **425.9 ms** |
|
| 53 |
+
|
| 54 |
+
### 3. Edge vs Cloud Latency (ClassOne vs TypeSafe Jev API)
|
| 55 |
+
|
| 56 |
+
Measured against TypeSafe AI's Jev (v1.13) cloud API:
|
| 57 |
+
- **ClassOne (Local RTX 5060 Ti):** **52.49 ms** mean latency (19.1 req/s, $0.00 inference cost, 100% private)
|
| 58 |
+
- **TypeSafe Jev (Cloud API):** **329.90 ms** mean latency (3.0 req/s)
|
| 59 |
+
- **Edge Speedup:** **6.3× faster** than cloud API round-trip latency
|
| 60 |
+
|
| 61 |
+
## Decision Primitives
|
| 62 |
+
|
| 63 |
+
- **`Noul`** — Boolean check returning a calibrated probability P(true) ∈ [0, 1]
|
| 64 |
+
- **`Choice`** — Categorical selection over 2–255 dynamic options with full probability distribution
|
| 65 |
+
- **`Score`** — Continuous ordinal rubric rating over 2–10 levels (expected value)
|
| 66 |
+
|
| 67 |
+
All outputs are calibrated with a combined NLL + normalized Brier loss.
|
| 68 |
+
Post-hoc temperature calibration achieves **ECE = 0.034** (down from 0.178).
|
| 69 |
+
|
| 70 |
+
## Quickstart
|
| 71 |
+
|
| 72 |
+
```bash
|
| 73 |
+
pip install classone
|
| 74 |
+
```
|
| 75 |
+
|
| 76 |
+
```python
|
| 77 |
+
import torch
|
| 78 |
+
from huggingface_hub import hf_hub_download
|
| 79 |
+
from transformers import AutoTokenizer
|
| 80 |
+
|
| 81 |
+
from classone.modeling.modeling_classone import ClassOneModel
|
| 82 |
+
from classone.schemas import NoulQuestion, ChoiceQuestion, ScoreQuestion
|
| 83 |
+
from classone.tokenizer import ClassOnePromptBuilder
|
| 84 |
+
|
| 85 |
+
REPO_ID = "devops-thiago/classone-gemma4-e4b"
|
| 86 |
+
|
| 87 |
+
# 1. Load the ClassOne model (weights + tokenizer are fully self-contained here)
|
| 88 |
+
tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
|
| 89 |
+
builder = ClassOnePromptBuilder(tokenizer)
|
| 90 |
+
model = ClassOneModel.from_backbone(
|
| 91 |
+
base_model_name_or_path=REPO_ID,
|
| 92 |
+
tokenizer=tokenizer,
|
| 93 |
+
device="cuda",
|
| 94 |
+
torch_dtype=torch.float16,
|
| 95 |
+
)
|
| 96 |
+
|
| 97 |
+
# 2. Load the trained decision heads
|
| 98 |
+
heads = torch.load(hf_hub_download(REPO_ID, "classone_heads.pt"), map_location="cuda")
|
| 99 |
+
model.noul_head.load_state_dict(heads["noul_head"])
|
| 100 |
+
model.choice_head.load_state_dict(heads["choice_head"])
|
| 101 |
+
model.score_head.load_state_dict(heads["score_head"])
|
| 102 |
+
model.eval()
|
| 103 |
+
|
| 104 |
+
# 3. Pack state + questions and run a single forward pass
|
| 105 |
+
packed = builder.pack(
|
| 106 |
+
state={"customer": "Alex", "message": "I was charged twice for order #123."},
|
| 107 |
+
questions={
|
| 108 |
+
"refund": NoulQuestion(instructions="Is the user requesting a refund?"),
|
| 109 |
+
"dept": ChoiceQuestion(
|
| 110 |
+
instructions="Route to team:",
|
| 111 |
+
criteria={"billing": "Payment issues", "tech": "Technical bugs"}
|
| 112 |
+
),
|
| 113 |
+
"anger": ScoreQuestion(
|
| 114 |
+
instructions="Dissatisfaction level:",
|
| 115 |
+
criteria=["satisfied", "neutral", "dissatisfied", "churning"]
|
| 116 |
+
),
|
| 117 |
+
}
|
| 118 |
+
)
|
| 119 |
+
results = model.evaluate_packed(packed)
|
| 120 |
+
|
| 121 |
+
print("Refund P(true):", results["refund"].noul)
|
| 122 |
+
print("Department: ", results["dept"].choice, "—", results["dept"].probabilities)
|
| 123 |
+
print("Anger score: ", results["anger"].score)
|
| 124 |
+
```
|
| 125 |
+
|
| 126 |
+
## Repository Files
|
| 127 |
+
|
| 128 |
+
| File | Description |
|
| 129 |
+
|---|---|
|
| 130 |
+
| `model.safetensors` (sharded) | Merged ClassOne backbone weights |
|
| 131 |
+
| `config.json` | Model configuration |
|
| 132 |
+
| `tokenizer.json`, `tokenizer_config.json` | Tokenizer, including ClassOne delimiter tokens |
|
| 133 |
+
| `classone_heads.pt` | Trained Noul / Choice / Score head weights + calibrated temperatures |
|
| 134 |
+
| `lora_backbone/` | LoRA adapter (r=16, α=32) that produced the merged weights |
|
| 135 |
+
|
| 136 |
+
## Citation
|
| 137 |
+
|
| 138 |
+
```bibtex
|
| 139 |
+
@misc{classone2026,
|
| 140 |
+
title={ClassOne: A Fast Single-Pass Decision Architecture for Language Models},
|
| 141 |
+
author={Thiago Gonzaga},
|
| 142 |
+
year={2026},
|
| 143 |
+
url={https://github.com/devops-thiago/class-one},
|
| 144 |
+
}
|
| 145 |
+
```
|
| 146 |
+
|
| 147 |
+
## Attribution & Legal
|
| 148 |
+
|
| 149 |
+
- Derived from [google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) (Google) — Apache License 2.0
|
| 150 |
+
- Architecture & training code: [devops-thiago/class-one](https://github.com/devops-thiago/class-one) — Apache 2.0
|