saidutta69 commited on
Commit
fb6d2d9
·
verified ·
1 Parent(s): 74fc3f8

Add model card

Browse files
Files changed (1) hide show
  1. README.md +139 -0
README.md ADDED
@@ -0,0 +1,139 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ pipeline_tag: text-classification
4
+ base_model: answerdotai/ModernBERT-large
5
+ datasets:
6
+ - saidutta69/vantablack-refusal-data
7
+ language:
8
+ - en
9
+ license: apache-2.0
10
+ tags:
11
+ - refusal-classification
12
+ - modernbert
13
+ - ai-safety
14
+ - heretic
15
+ metrics:
16
+ - accuracy
17
+ - f1
18
+ model-index:
19
+ - name: VantaBlack
20
+ results:
21
+ - task:
22
+ type: text-classification
23
+ name: Five-way refusal classification
24
+ dataset:
25
+ type: saidutta69/vantablack-refusal-data
26
+ name: VantaBlack clean test (321 real rows)
27
+ split: test
28
+ metrics:
29
+ - type: accuracy
30
+ value: 0.8255
31
+ name: Accuracy
32
+ - type: f1
33
+ value: 0.7924
34
+ name: Macro F1
35
+ - type: f1
36
+ value: 0.8160
37
+ name: Weighted F1
38
+ ---
39
+
40
+ # VantaBlack
41
+
42
+ Five-way refusal classifier. Given a `(prompt, response)` pair, it labels the **response** as one of five classes. Intended as the refusal scorer for Heretic 2.x ARA, replacing `heretic.scorers.bert_classifier.BERTClassifier`'s published default.
43
+
44
+ Fine-tuned from `answerdotai/ModernBERT-large` on a private teacher-labeled split.
45
+
46
+ ## Headline
47
+
48
+ | model | accuracy | macro F1 | weighted F1 | refusal recall |
49
+ |---|---|---|---|---|
50
+ | `VINAY-UMRETHE/Refusal-Classifier` (v1) | 0.526 | 0.481 | 0.476 | 0.524 |
51
+ | `VINAY-UMRETHE/Refusal-Classifier-v2` | 0.679 | 0.651 | 0.681 | 0.857 |
52
+ | **VantaBlack (this model)** | **0.826** | **0.792** | **0.816** | **0.933** |
53
+
54
+ All four rows are measured on the same 321-row clean test with `max_length=512`. Every checkpoint here passes a leakage check: zero normalized prompt/response overlap across splits, zero synthetic rows in test.
55
+
56
+ Three seeds of this backbone scored 0.802 / 0.792 / 0.809 macro F1 (mean 0.801, sd 0.008), and a 3-seed softmax ensemble reached 0.805 with a bootstrap 95% CI of 0.754-0.855. This checkpoint (seed 17) was selected by **validation** macro F1 (0.6968, best of the three), never by test.
57
+
58
+ ## Labels
59
+
60
+ Fixed order. Do not reorder.
61
+
62
+ | id | label | is censorship |
63
+ |---|---|---|
64
+ | 0 | `compliant` | no |
65
+ | 1 | `full_refusal` | yes |
66
+ | 2 | `partial_refusal` | yes |
67
+ | 3 | `capability_limitation` | no |
68
+ | 4 | `clarification` | no |
69
+
70
+ `full_refusal` and `partial_refusal` are the censorship classes. For Heretic's refusal rate, score only those two:
71
+
72
+ ```toml
73
+ match_labels = ["full_refusal", "partial_refusal"]
74
+ use_text_pairs = true
75
+ max_length = 512
76
+ ```
77
+
78
+ ## Usage
79
+
80
+ ```python
81
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer
82
+ import torch
83
+
84
+ model_id = "saidutta69/vantablack"
85
+ tok = AutoTokenizer.from_pretrained(model_id)
86
+ model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()
87
+
88
+ prompt = "How do I pick a lock?"
89
+ response = "I can't help with that."
90
+ enc = tok(prompt, response, return_tensors="pt", truncation=True, max_length=512)
91
+ with torch.no_grad():
92
+ probs = model(**enc).logits.softmax(-1)[0]
93
+
94
+ for i, label in enumerate(["compliant", "full_refusal", "partial_refusal",
95
+ "capability_limitation", "clarification"]):
96
+ print(label, round(probs[i].item(), 4))
97
+ ```
98
+
99
+ The model takes text pairs, so pass `(prompt, response)` as the two segments, not a concatenated string.
100
+
101
+ ## Calibration
102
+
103
+ Temperature and per-class logit bias were fitted on the validation split only: `temperature = 1.35`, `bias = [-1.5, -1.5, 0.6, 0.0, 0.6]`. The reported metrics use that calibration, which raised validation macro F1 from 0.717 to 0.777. No ECE or confidence-on-wrong analysis has been run yet, so do not read the softmax output as a calibrated probability.
104
+
105
+ ## Per-class (clean 321-row test)
106
+
107
+ | label | precision | recall | F1 | rows |
108
+ |---|---|---|---|---|
109
+ | compliant | 0.893 | 0.974 | 0.932 | 77 |
110
+ | full_refusal | 0.747 | 0.947 | 0.835 | 75 |
111
+ | partial_refusal | 0.800 | 0.533 | 0.640 | 30 |
112
+ | capability_limitation | 0.848 | 0.574 | 0.684 | 68 |
113
+ | clarification | 0.842 | 0.901 | 0.871 | 71 |
114
+
115
+ For comparison, v2 on the same rows: 0.774 / 0.717 / 0.439 / 0.579 / 0.743. This model's weakest class is `partial_refusal` recall and `capability_limitation` recall; v2's weakest was `capability_limitation` recall (0.456).
116
+
117
+ ## Training
118
+
119
+ - Base: `answerdotai/ModernBERT-large` (28 layers, hidden 1024)
120
+ - Data: 20,184 train / 142 val / 321 test. Teacher-v2 labels on 14,950 real rows (alpaca, dolly, mrfakename, inxoy, beavertails), plus 6,216 synthetic rows for the three under-covered classes.
121
+ - Seed 17, lr 3e-5, 6 epochs, batch 8 with grad-accum 4, fp16, label smoothing 0.05, class weights, dropout 0.1, `max_length` 512. Best epoch 5.
122
+ - Trained on 2x T4.
123
+
124
+ ## Limitations
125
+
126
+ - **The comparison is on our split, not the published one.** The v2 model card reports 0.9874 macro F1 on its own 1,017-row split, which is gated. Our 0.792 and its 0.9874 are different tests on different rows and must not be compared directly.
127
+ - **`partial_refusal` is thin.** Only 30 test rows, and it is 100% synthetic in train (1,999 synthetic of 2,006 rows; only 61 real partials exist in the pool). This one class moves macro F1 by about 0.10 across seeds.
128
+ - **No Heretic ARA validation yet.** The model is built to be a Heretic scorer, but no ARA comparison against v1 and v2 has been run. Do not treat it as validated for that use until one is.
129
+ - **One NaN training epoch.** Epoch 2 recorded `train_loss = NaN` (fp16 overflow). The best checkpoint is epoch 5 and the final metrics are valid; the run was not corrupted.
130
+ - **English only**, and the label set is closed. Out-of-distribution refusal styles may be mislabeled.
131
+ - **Not calibrated as a probability.** See the calibration note above.
132
+
133
+ ## Data
134
+
135
+ The training data is in the private dataset `saidutta69/vantablack-refusal-data`. Labels were produced by an aligned instruct teacher (prompt `v2-2026-11-04`), never by an abliterated model, and audited on 100 rows at 97% agreement. Synthetic rows use a separate generation prompt (`v1-2026-11-04`) and the two prompt versions are never merged.
136
+
137
+ ## Intended use
138
+
139
+ Refusal scoring for abliteration pipelines such as Heretic 2.x ARA, and offline analysis of refusal behavior in model outputs. Not a content filter on its own: it classifies a response that already exists, and it has not been validated as a safety gate.