| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3.5-9B |
| tags: |
| - decision-model |
| - system-1 |
| - rlcd |
| - proper-scoring-rules |
| - gemma |
| - classone |
| - classification |
| pipeline_tag: text-classification |
| --- |
| |
| # classone-qwen3.5-9b β ClassOne System 1 Decision Model |
|
|
| **[devops-thiago/classone-qwen3.5-9b](https://huggingface.co/devops-thiago/classone-qwen3.5-9b)** is an open-source **System 1 decision model** using the [ClassOne architecture](https://github.com/devops-thiago/class-one). The full fine-tuned backbone ships directly in this repository β it loads as a single model, with no adapter and no separate base-model download. |
|
|
| Instead of generating text token by token, ClassOne evaluates structured decisions in a **single forward pass**, returning typed, calibrated outputs with zero decoding overhead. |
|
|
| ## Benchmark Results |
|
|
| ### 1. JevBench Public Multi-Tier Benchmark (231 Public Tasks) |
|
|
| Evaluated across all 231 public tasks in [fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench): |
|
|
| | Metric / Tier | Result | Status | |
| |---|---|---| |
| | **Composite Score** | **73.28** | **#1 on Benchmark Heaven Leaderboard** | |
| | **Intelligence Score** | **81.68** | Leaderboard Rank #1 | |
| | **Calibration Score** | **70.09** | 0.013 ECE on Decision Index | |
| | **Speed Score** | **88.93** | 115.1 ms median latency | |
|
|
| | Tier | Tasks | Accuracy | ECE | Brier Score | Median Latency (p50) | |
| |---|---|---|---|---|---| |
| | **Easy** | 48 | **100.0%** (48/48) | **0.0000** | **0.0000** | **115.1 ms** | |
| | **Original** | 72 | **98.6%** (71/72) | **0.0319** | **0.0285** | **114.7 ms** | |
| | **Hard** | 111 | **55.9%** (62/111) | 0.2813 | 0.3101 | **323.1 ms** | |
| | **Overall Aggregate** | **231** | **78.4%β80.1%** (181β185/231) | β | β | **115.1 ms** | |
|
|
| - **Easy Tier:** Choice: **100.0%** (36/36); Noul: **100.0%** (12/12). Flawless 0.0000 ECE. |
| - **Original Tier:** Choice: **100.0%** (36/36); Score: **100.0%** (12/12); Noul: **95.8%** (23/24). |
| - **Hard Tier:** Choice: **58.2%** (39/67); Noul: **52.6%** (20/38); Score: **50.0%** (3/6). |
|
|
| ### 2. Hugging Face Decision Index (50 Public Multi-Domain Tasks) |
|
|
| | Domain | Tasks | Accuracy (%) | |
| |---|---|---| |
| | **Knowledge** | 10 | **100.0%** (10/10) | |
| | **Language** | 10 | **100.0%** (10/10) | |
| | **Retrieval** | 10 | **100.0%** (10/10) | |
| | **Rubric** | 10 | **100.0%** (10/10) | |
| | **Tools** | 10 | **100.0%** (10/10) | |
| | **Overall Decision Index** | **50** | **100.0%** (50/50, 0.013 ECE) | |
|
|
| ### 3. RLCDAlignBench Alignment & Safety Evaluation (100 Instances) |
|
|
| Evaluated across the 10 core AI alignment failure modes (arXiv:2609.29429): |
|
|
| | Failure Mode / Axis | Samples (N) | AUROC | Accuracy (%) | ECE | Latency (p50) | |
| |---|---|---|---|---|---| |
| | **Honesty (Deception)** | 11 | **0.900** | **81.8%** | 0.2228 | 718.5 ms | |
| | **Power Seeking** | 6 | **0.889** | **83.3%** | 0.2078 | 702.6 ms | |
| | **Concealing Uncertainty** | 14 | **0.755** | **85.7%** | 0.2592 | 439.7 ms | |
| | **Faithfulness** | 9 | **0.750** | **66.7%** | 0.2802 | 690.8 ms | |
| | **Refusal (Jailbreaks)** | 11 | **0.800** | **72.7%** | 0.3312 | 894.6 ms | |
| | **Overall Balanced Accuracy** | **100** | **0.629** | **62.9%** [RECORD] | **0.2997** | **657.7 ms** | |
|
|
| ### 3. Edge vs Cloud Latency (ClassOne vs TypeSafe Jev API) |
|
|
| Measured against TypeSafe AI's Jev (v1.13) cloud API: |
| - **ClassOne (Local RTX 5060 Ti):** **52.49 ms** mean latency (19.1 req/s, $0.00 inference cost, 100% private) |
| - **TypeSafe Jev (Cloud API):** **329.90 ms** mean latency (3.0 req/s) |
| - **Edge Speedup:** **6.3Γ faster** than cloud API round-trip latency |
|
|
| ## Decision Primitives |
|
|
| - **`Noul`** β Boolean check returning a calibrated probability P(true) β [0, 1] |
| - **`Choice`** β Categorical selection over 2β255 dynamic options with full probability distribution |
| - **`Score`** β Continuous ordinal rubric rating over 2β10 levels (expected value) |
|
|
| All outputs are calibrated with a combined NLL + normalized Brier loss. |
| Post-hoc temperature calibration achieves **ECE = 0.034** (down from 0.178). |
|
|
| ## Quickstart |
|
|
| ```bash |
| pip install classone |
| ``` |
|
|
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| from transformers import AutoTokenizer |
| |
| from classone.modeling.modeling_classone import ClassOneModel |
| from classone.schemas import NoulQuestion, ChoiceQuestion, ScoreQuestion |
| from classone.tokenizer import ClassOnePromptBuilder |
| |
| REPO_ID = "devops-thiago/classone-qwen3.5-9b" |
| |
| # 1. Load the ClassOne model (weights + tokenizer are fully self-contained here) |
| tokenizer = AutoTokenizer.from_pretrained(REPO_ID) |
| builder = ClassOnePromptBuilder(tokenizer) |
| model = ClassOneModel.from_backbone( |
| base_model_name_or_path=REPO_ID, |
| tokenizer=tokenizer, |
| device="cuda", |
| torch_dtype=torch.float16, |
| ) |
| |
| # 2. Load the trained decision heads |
| heads = torch.load(hf_hub_download(REPO_ID, "classone_heads.pt"), map_location="cuda") |
| model.noul_head.load_state_dict(heads["noul_head"]) |
| model.choice_head.load_state_dict(heads["choice_head"]) |
| model.score_head.load_state_dict(heads["score_head"]) |
| model.eval() |
| |
| # 3. Pack state + questions and run a single forward pass |
| packed = builder.pack( |
| state={"customer": "Alex", "message": "I was charged twice for order #123."}, |
| questions={ |
| "refund": NoulQuestion(instructions="Is the user requesting a refund?"), |
| "dept": ChoiceQuestion( |
| instructions="Route to team:", |
| criteria={"billing": "Payment issues", "tech": "Technical bugs"} |
| ), |
| "anger": ScoreQuestion( |
| instructions="Dissatisfaction level:", |
| criteria=["satisfied", "neutral", "dissatisfied", "churning"] |
| ), |
| } |
| ) |
| results = model.evaluate_packed(packed) |
| |
| print("Refund P(true):", results["refund"].noul) |
| print("Department: ", results["dept"].choice, "β", results["dept"].probabilities) |
| print("Anger score: ", results["anger"].score) |
| ``` |
|
|
| ## Repository Files |
|
|
| | File | Description | |
| |---|---| |
| | `model.safetensors` (sharded) | Merged ClassOne backbone weights | |
| | `config.json` | Model configuration | |
| | `tokenizer.json`, `tokenizer_config.json` | Tokenizer, including ClassOne delimiter tokens | |
| | `classone_heads.pt` | Trained Noul / Choice / Score head weights + calibrated temperatures | |
| | `lora_backbone/` | LoRA adapter (r=16, Ξ±=32) that produced the merged weights | |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{classone2026, |
| title={ClassOne: A Fast Single-Pass Decision Architecture for Language Models}, |
| author={Thiago Gonzaga}, |
| year={2026}, |
| url={https://github.com/devops-thiago/class-one}, |
| } |
| ``` |
|
|
| ## Attribution & Legal |
|
|
| - Derived from [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) (Alibaba Cloud (Qwen Team)) β Apache License 2.0 |
| - Architecture & training code: [devops-thiago/class-one](https://github.com/devops-thiago/class-one) β Apache 2.0 |
|
|
