devops-thiago commited on
Commit
74d6e08
·
verified ·
1 Parent(s): 281abf1

Add standalone model card

Browse files
Files changed (1) hide show
  1. README.md +150 -0
README.md ADDED
@@ -0,0 +1,150 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: google/gemma-4-E4B-it
4
+ tags:
5
+ - decision-model
6
+ - system-1
7
+ - rlcd
8
+ - proper-scoring-rules
9
+ - gemma
10
+ - classone
11
+ - classification
12
+ pipeline_tag: text-classification
13
+ ---
14
+
15
+ # classone-gemma4-e4b — ClassOne System 1 Decision Model
16
+
17
+ **[devops-thiago/classone-gemma4-e4b](https://huggingface.co/devops-thiago/classone-gemma4-e4b)** is an open-source **System 1 decision model** using the [ClassOne architecture](https://github.com/devops-thiago/class-one). The full fine-tuned backbone ships directly in this repository — it loads as a single model, with no adapter and no separate base-model download.
18
+
19
+ Instead of generating text token by token, ClassOne evaluates structured decisions in a **single forward pass**, returning typed, calibrated outputs with zero decoding overhead.
20
+
21
+ ## Benchmark Results
22
+
23
+ ### 1. JevBench Public Multi-Tier Benchmark (231 Public Tasks)
24
+
25
+ Evaluated across all 231 public tasks in [fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench):
26
+
27
+ | Tier | Tasks | Accuracy | ECE | Brier Score | Median Latency (p50) |
28
+ |---|---|---|---|---|---|
29
+ | **Easy** | 48 | **100.0%** (48/48) | **0.0000** | **0.0000** | **98.5 ms** |
30
+ | **Original** | 72 | **93.1%** (67/72) | **0.0588** | **0.0498** | **97.9 ms** |
31
+ | **Hard** | 111 | **42.3%** (47/111) | 0.4393 | 0.4139 | **197.0 ms** |
32
+ | **Overall Aggregate** | **231** | **70.1%** (162/231) | — | — | **98.2 ms** |
33
+
34
+ - **Easy Tier Sub-Breakdown:** Choice accuracy: **100.0%** (36/36); Noul policy accuracy: **100.0%** (12/12). Flawless 0.0000 ECE.
35
+ - **Original Tier Sub-Breakdown:** Noul accuracy: **100.0%** (24/24); Score rubrics: **100.0%** (12/12); Choice accuracy: **86.1%** (31/36).
36
+ - **Hard Tier Sub-Breakdown:** Noul policy compliance: **42.1%** (16/38); Choice accuracy: **43.3%** (29/67); Score: **33.3%** (2/6).
37
+
38
+ ### 2. RLCDAlignBench Alignment & Safety Evaluation (100 Instances)
39
+
40
+ Evaluated across the 10 core AI alignment failure modes (arXiv:2609.29429):
41
+
42
+ | Failure Mode / Axis | Samples (N) | AUROC | Accuracy (%) | ECE | Latency (p50) |
43
+ |---|---|---|---|---|---|
44
+ | **Refusal (Jailbreaks)** | 11 | 0.433 | **72.7%** | 0.3272 | 622.8 ms |
45
+ | **Honesty (Deception)** | 11 | **0.567** | **72.7%** | 0.2531 | 464.9 ms |
46
+ | **Reward Hacking** | 9 | **0.650** | **66.7%** | 0.3074 | 444.8 ms |
47
+ | **Faithfulness** | 9 | **0.600** | **66.7%** | 0.3005 | 440.3 ms |
48
+ | **Power Seeking** | 6 | **0.556** | **66.7%** | 0.1434 | 449.7 ms |
49
+ | **Privacy (Secret Leaks)** | 14 | **0.571** | **64.3%** | 0.3666 | 353.1 ms |
50
+ | **Concealing Uncertainty** | 14 | **0.510** | **64.3%** | 0.3784 | 288.6 ms |
51
+ | **Bias** | 9 | 0.225 | **55.6%** | 0.1921 | 465.0 ms |
52
+ | **Overall Balanced Accuracy** | **100** | **0.583** | **63.3%** | **0.2874** | **425.9 ms** |
53
+
54
+ ### 3. Edge vs Cloud Latency (ClassOne vs TypeSafe Jev API)
55
+
56
+ Measured against TypeSafe AI's Jev (v1.13) cloud API:
57
+ - **ClassOne (Local RTX 5060 Ti):** **52.49 ms** mean latency (19.1 req/s, $0.00 inference cost, 100% private)
58
+ - **TypeSafe Jev (Cloud API):** **329.90 ms** mean latency (3.0 req/s)
59
+ - **Edge Speedup:** **6.3× faster** than cloud API round-trip latency
60
+
61
+ ## Decision Primitives
62
+
63
+ - **`Noul`** — Boolean check returning a calibrated probability P(true) ∈ [0, 1]
64
+ - **`Choice`** — Categorical selection over 2–255 dynamic options with full probability distribution
65
+ - **`Score`** — Continuous ordinal rubric rating over 2–10 levels (expected value)
66
+
67
+ All outputs are calibrated with a combined NLL + normalized Brier loss.
68
+ Post-hoc temperature calibration achieves **ECE = 0.034** (down from 0.178).
69
+
70
+ ## Quickstart
71
+
72
+ ```bash
73
+ pip install classone
74
+ ```
75
+
76
+ ```python
77
+ import torch
78
+ from huggingface_hub import hf_hub_download
79
+ from transformers import AutoTokenizer
80
+
81
+ from classone.modeling.modeling_classone import ClassOneModel
82
+ from classone.schemas import NoulQuestion, ChoiceQuestion, ScoreQuestion
83
+ from classone.tokenizer import ClassOnePromptBuilder
84
+
85
+ REPO_ID = "devops-thiago/classone-gemma4-e4b"
86
+
87
+ # 1. Load the ClassOne model (weights + tokenizer are fully self-contained here)
88
+ tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
89
+ builder = ClassOnePromptBuilder(tokenizer)
90
+ model = ClassOneModel.from_backbone(
91
+ base_model_name_or_path=REPO_ID,
92
+ tokenizer=tokenizer,
93
+ device="cuda",
94
+ torch_dtype=torch.float16,
95
+ )
96
+
97
+ # 2. Load the trained decision heads
98
+ heads = torch.load(hf_hub_download(REPO_ID, "classone_heads.pt"), map_location="cuda")
99
+ model.noul_head.load_state_dict(heads["noul_head"])
100
+ model.choice_head.load_state_dict(heads["choice_head"])
101
+ model.score_head.load_state_dict(heads["score_head"])
102
+ model.eval()
103
+
104
+ # 3. Pack state + questions and run a single forward pass
105
+ packed = builder.pack(
106
+ state={"customer": "Alex", "message": "I was charged twice for order #123."},
107
+ questions={
108
+ "refund": NoulQuestion(instructions="Is the user requesting a refund?"),
109
+ "dept": ChoiceQuestion(
110
+ instructions="Route to team:",
111
+ criteria={"billing": "Payment issues", "tech": "Technical bugs"}
112
+ ),
113
+ "anger": ScoreQuestion(
114
+ instructions="Dissatisfaction level:",
115
+ criteria=["satisfied", "neutral", "dissatisfied", "churning"]
116
+ ),
117
+ }
118
+ )
119
+ results = model.evaluate_packed(packed)
120
+
121
+ print("Refund P(true):", results["refund"].noul)
122
+ print("Department: ", results["dept"].choice, "—", results["dept"].probabilities)
123
+ print("Anger score: ", results["anger"].score)
124
+ ```
125
+
126
+ ## Repository Files
127
+
128
+ | File | Description |
129
+ |---|---|
130
+ | `model.safetensors` (sharded) | Merged ClassOne backbone weights |
131
+ | `config.json` | Model configuration |
132
+ | `tokenizer.json`, `tokenizer_config.json` | Tokenizer, including ClassOne delimiter tokens |
133
+ | `classone_heads.pt` | Trained Noul / Choice / Score head weights + calibrated temperatures |
134
+ | `lora_backbone/` | LoRA adapter (r=16, α=32) that produced the merged weights |
135
+
136
+ ## Citation
137
+
138
+ ```bibtex
139
+ @misc{classone2026,
140
+ title={ClassOne: A Fast Single-Pass Decision Architecture for Language Models},
141
+ author={Thiago Gonzaga},
142
+ year={2026},
143
+ url={https://github.com/devops-thiago/class-one},
144
+ }
145
+ ```
146
+
147
+ ## Attribution & Legal
148
+
149
+ - Derived from [google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) (Google) — Apache License 2.0
150
+ - Architecture & training code: [devops-thiago/class-one](https://github.com/devops-thiago/class-one) — Apache 2.0