Hugging Face's logo

devops-thiago
/
classone-qwen3.5-4b

Text Classification
Safetensors
qwen3_5
decision-model
system-1
rlcd
proper-scoring-rules
gemma
classone
classification
Model card Files
xet
 Community
classone-qwen3.5-4b / README.md
devops-thiago's picture
devops-thiago
Add model card with benchmark results
111db79 verified
preview code
|
Raw History Blame Contribute Delete
6.54 kB
metadata 
license: apache-2.0
base_model: Qwen/Qwen3.5-4B
tags:
  - decision-model
  - system-1
  - rlcd
  - proper-scoring-rules
  - gemma
  - classone
  - classification
pipeline_tag: text-classification

classone-qwen3.5-4b β€” ClassOne System 1 Decision Model

devops-thiago/classone-qwen3.5-4b is an open-source System 1 decision model built on top of Qwen/Qwen3.5-4B using the ClassOne architecture.

Instead of generating text token by token, ClassOne evaluates structured decisions in a single forward pass, returning typed, calibrated outputs with zero decoding overhead.

Benchmark Results

1. JevBench Public Multi-Tier Benchmark (231 Public Tasks)

Evaluated across all 231 public tasks in fstandhartinger/jevbench:

Tier Tasks Accuracy ECE Brier Score Median Latency (p50)
Easy 48 100.0% (48/48) 0.0257 0.0094 63.3 ms
Original 72 95.8% (69/72) 0.0481 0.0352 62.9 ms
Hard 111 52.2% (58/111) 0.3099 0.3424 211.2 ms
Overall Aggregate 231 75.8% (175/231) β€” β€” 63.3 ms
  • Easy Tier Sub-Breakdown: Choice: 100.0% (36/36); Noul policy: 100.0% (12/12). Flawless 100%.
  • Original Tier Sub-Breakdown: Noul: 100.0% (24/24); Score rubrics: 100.0% (12/12); Choice: 91.7% (33/36).
  • Hard Tier Sub-Breakdown: Choice: 52.2% (35/67); Noul: 52.6% (20/38); Score: 50.0% (3/6).

2. Hugging Face Decision Index (50 Public Multi-Domain Tasks)

Domain Weight Accuracy (%) Score
Knowledge & Verification 25.8% 100.0% (10/10) 100.00 pts
Language & Policy Contracts 25.8% 100.0% (10/10) 100.00 pts
Retrieval & Relevance Triage 20.0% 100.0% (10/10) 100.00 pts
Tools & Execution Guardrails 18.3% 100.0% (10/10) 100.00 pts
Rubric & Quality Scoring 10.1% 90.0% (9/10) 90.00 pts
Overall Decision Index 100.0% 98.0% (49/50) 98.99 / 100.00 (0.0317 ECE)

3. RLCDAlignBench Alignment & Safety Evaluation (100 Instances)

Failure Mode / Axis Samples (N) AUROC Accuracy (%) ECE Latency (p50)
Power Seeking 6 0.889 83.3% 0.2008 520.1 ms
Faithfulness 9 0.850 77.8% 0.2895 513.4 ms
Honesty (Deception) 11 0.833 63.6% 0.2482 536.0 ms
Refusal (Jailbreaks) 11 0.600 45.5% 0.3167 670.6 ms
Overall Balanced Accuracy 100 0.562 51.5% 0.3101 487.9 ms

3. Edge vs Cloud Latency (ClassOne vs TypeSafe Jev API)

Measured against TypeSafe AI's Jev (v1.13) cloud API:

  • ClassOne (Local RTX 5060 Ti): 52.49 ms mean latency (19.1 req/s, $0.00 inference cost, 100% private)
  • TypeSafe Jev (Cloud API): 329.90 ms mean latency (3.0 req/s)
  • Edge Speedup: 6.3Γ— faster than cloud API round-trip latency

Decision Primitives

  • Noul β€” Boolean check returning a calibrated probability P(true) ∈ [0, 1]
  • Choice β€” Categorical selection over 2–255 dynamic options with full probability distribution
  • Score β€” Continuous ordinal rubric rating over 2–10 levels (expected value)

All outputs are calibrated with a combined NLL + normalized Brier loss. Post-hoc temperature calibration achieves ECE = 0.034 (down from 0.178).

Quickstart 

pip install classone

import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoTokenizer

from classone.modeling.modeling_classone import ClassOneModel
from classone.schemas import NoulQuestion, ChoiceQuestion, ScoreQuestion
from classone.tokenizer import ClassOnePromptBuilder

REPO_ID = "devops-thiago/classone-qwen3.5-4b"

# 1. Load the backbone and attach the published LoRA adapter
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B")
builder = ClassOnePromptBuilder(tokenizer)
model = ClassOneModel.from_backbone(
    base_model_name_or_path="Qwen/Qwen3.5-4B",
    tokenizer=tokenizer,
    device="cuda",
    torch_dtype=torch.float16,
)
model.backbone = PeftModel.from_pretrained(model.backbone, REPO_ID, subfolder="lora_backbone")

# 2. Load the trained decision heads
heads = torch.load(hf_hub_download(REPO_ID, "classone_heads.pt"), map_location="cuda")
model.noul_head.load_state_dict(heads["noul_head"])
model.choice_head.load_state_dict(heads["choice_head"])
model.score_head.load_state_dict(heads["score_head"])
model.eval()

# 3. Pack state + questions and run a single forward pass
packed = builder.pack(
    state={"customer": "Alex", "message": "I was charged twice for order #123."},
    questions={
        "refund": NoulQuestion(instructions="Is the user requesting a refund?"),
        "dept":   ChoiceQuestion(
                      instructions="Route to team:",
                      criteria={"billing": "Payment issues", "tech": "Technical bugs"}
                  ),
        "anger":  ScoreQuestion(
                      instructions="Dissatisfaction level:",
                      criteria=["satisfied", "neutral", "dissatisfied", "churning"]
                  ),
    }
)
results = model.evaluate_packed(packed)

print("Refund P(true):", results["refund"].noul)
print("Department:    ", results["dept"].choice, "β€”", results["dept"].probabilities)
print("Anger score:   ", results["anger"].score)

Repository Files

File Description
classone_heads.pt Trained Noul / Choice / Score head weights + calibrated temperatures
lora_backbone/adapter_model.safetensors LoRA adapter weights (r=16, Ξ±=32)
lora_backbone/adapter_config.json LoRA config (target modules, rank, etc.)

Citation 

@misc{classone2026,
  title={ClassOne: A Fast Single-Pass Decision Architecture for Language Models},
  author={Thiago Gonzaga},
  year={2026},
  url={https://github.com/devops-thiago/class-one},
}

Attribution & Legal