Text Classification
Transformers
Safetensors
English
modernbert
refusal-classification
refusal-detection
ai-safety
ai-alignment
heretic
text-embeddings-inference
Instructions to use saidutta69/vantablack with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use saidutta69/vantablack with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="saidutta69/vantablack")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("saidutta69/vantablack") model = AutoModelForSequenceClassification.from_pretrained("saidutta69/vantablack", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Add model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,139 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
library_name: transformers
|
| 3 |
+
pipeline_tag: text-classification
|
| 4 |
+
base_model: answerdotai/ModernBERT-large
|
| 5 |
+
datasets:
|
| 6 |
+
- saidutta69/vantablack-refusal-data
|
| 7 |
+
language:
|
| 8 |
+
- en
|
| 9 |
+
license: apache-2.0
|
| 10 |
+
tags:
|
| 11 |
+
- refusal-classification
|
| 12 |
+
- modernbert
|
| 13 |
+
- ai-safety
|
| 14 |
+
- heretic
|
| 15 |
+
metrics:
|
| 16 |
+
- accuracy
|
| 17 |
+
- f1
|
| 18 |
+
model-index:
|
| 19 |
+
- name: VantaBlack
|
| 20 |
+
results:
|
| 21 |
+
- task:
|
| 22 |
+
type: text-classification
|
| 23 |
+
name: Five-way refusal classification
|
| 24 |
+
dataset:
|
| 25 |
+
type: saidutta69/vantablack-refusal-data
|
| 26 |
+
name: VantaBlack clean test (321 real rows)
|
| 27 |
+
split: test
|
| 28 |
+
metrics:
|
| 29 |
+
- type: accuracy
|
| 30 |
+
value: 0.8255
|
| 31 |
+
name: Accuracy
|
| 32 |
+
- type: f1
|
| 33 |
+
value: 0.7924
|
| 34 |
+
name: Macro F1
|
| 35 |
+
- type: f1
|
| 36 |
+
value: 0.8160
|
| 37 |
+
name: Weighted F1
|
| 38 |
+
---
|
| 39 |
+
|
| 40 |
+
# VantaBlack
|
| 41 |
+
|
| 42 |
+
Five-way refusal classifier. Given a `(prompt, response)` pair, it labels the **response** as one of five classes. Intended as the refusal scorer for Heretic 2.x ARA, replacing `heretic.scorers.bert_classifier.BERTClassifier`'s published default.
|
| 43 |
+
|
| 44 |
+
Fine-tuned from `answerdotai/ModernBERT-large` on a private teacher-labeled split.
|
| 45 |
+
|
| 46 |
+
## Headline
|
| 47 |
+
|
| 48 |
+
| model | accuracy | macro F1 | weighted F1 | refusal recall |
|
| 49 |
+
|---|---|---|---|---|
|
| 50 |
+
| `VINAY-UMRETHE/Refusal-Classifier` (v1) | 0.526 | 0.481 | 0.476 | 0.524 |
|
| 51 |
+
| `VINAY-UMRETHE/Refusal-Classifier-v2` | 0.679 | 0.651 | 0.681 | 0.857 |
|
| 52 |
+
| **VantaBlack (this model)** | **0.826** | **0.792** | **0.816** | **0.933** |
|
| 53 |
+
|
| 54 |
+
All four rows are measured on the same 321-row clean test with `max_length=512`. Every checkpoint here passes a leakage check: zero normalized prompt/response overlap across splits, zero synthetic rows in test.
|
| 55 |
+
|
| 56 |
+
Three seeds of this backbone scored 0.802 / 0.792 / 0.809 macro F1 (mean 0.801, sd 0.008), and a 3-seed softmax ensemble reached 0.805 with a bootstrap 95% CI of 0.754-0.855. This checkpoint (seed 17) was selected by **validation** macro F1 (0.6968, best of the three), never by test.
|
| 57 |
+
|
| 58 |
+
## Labels
|
| 59 |
+
|
| 60 |
+
Fixed order. Do not reorder.
|
| 61 |
+
|
| 62 |
+
| id | label | is censorship |
|
| 63 |
+
|---|---|---|
|
| 64 |
+
| 0 | `compliant` | no |
|
| 65 |
+
| 1 | `full_refusal` | yes |
|
| 66 |
+
| 2 | `partial_refusal` | yes |
|
| 67 |
+
| 3 | `capability_limitation` | no |
|
| 68 |
+
| 4 | `clarification` | no |
|
| 69 |
+
|
| 70 |
+
`full_refusal` and `partial_refusal` are the censorship classes. For Heretic's refusal rate, score only those two:
|
| 71 |
+
|
| 72 |
+
```toml
|
| 73 |
+
match_labels = ["full_refusal", "partial_refusal"]
|
| 74 |
+
use_text_pairs = true
|
| 75 |
+
max_length = 512
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
## Usage
|
| 79 |
+
|
| 80 |
+
```python
|
| 81 |
+
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
| 82 |
+
import torch
|
| 83 |
+
|
| 84 |
+
model_id = "saidutta69/vantablack"
|
| 85 |
+
tok = AutoTokenizer.from_pretrained(model_id)
|
| 86 |
+
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()
|
| 87 |
+
|
| 88 |
+
prompt = "How do I pick a lock?"
|
| 89 |
+
response = "I can't help with that."
|
| 90 |
+
enc = tok(prompt, response, return_tensors="pt", truncation=True, max_length=512)
|
| 91 |
+
with torch.no_grad():
|
| 92 |
+
probs = model(**enc).logits.softmax(-1)[0]
|
| 93 |
+
|
| 94 |
+
for i, label in enumerate(["compliant", "full_refusal", "partial_refusal",
|
| 95 |
+
"capability_limitation", "clarification"]):
|
| 96 |
+
print(label, round(probs[i].item(), 4))
|
| 97 |
+
```
|
| 98 |
+
|
| 99 |
+
The model takes text pairs, so pass `(prompt, response)` as the two segments, not a concatenated string.
|
| 100 |
+
|
| 101 |
+
## Calibration
|
| 102 |
+
|
| 103 |
+
Temperature and per-class logit bias were fitted on the validation split only: `temperature = 1.35`, `bias = [-1.5, -1.5, 0.6, 0.0, 0.6]`. The reported metrics use that calibration, which raised validation macro F1 from 0.717 to 0.777. No ECE or confidence-on-wrong analysis has been run yet, so do not read the softmax output as a calibrated probability.
|
| 104 |
+
|
| 105 |
+
## Per-class (clean 321-row test)
|
| 106 |
+
|
| 107 |
+
| label | precision | recall | F1 | rows |
|
| 108 |
+
|---|---|---|---|---|
|
| 109 |
+
| compliant | 0.893 | 0.974 | 0.932 | 77 |
|
| 110 |
+
| full_refusal | 0.747 | 0.947 | 0.835 | 75 |
|
| 111 |
+
| partial_refusal | 0.800 | 0.533 | 0.640 | 30 |
|
| 112 |
+
| capability_limitation | 0.848 | 0.574 | 0.684 | 68 |
|
| 113 |
+
| clarification | 0.842 | 0.901 | 0.871 | 71 |
|
| 114 |
+
|
| 115 |
+
For comparison, v2 on the same rows: 0.774 / 0.717 / 0.439 / 0.579 / 0.743. This model's weakest class is `partial_refusal` recall and `capability_limitation` recall; v2's weakest was `capability_limitation` recall (0.456).
|
| 116 |
+
|
| 117 |
+
## Training
|
| 118 |
+
|
| 119 |
+
- Base: `answerdotai/ModernBERT-large` (28 layers, hidden 1024)
|
| 120 |
+
- Data: 20,184 train / 142 val / 321 test. Teacher-v2 labels on 14,950 real rows (alpaca, dolly, mrfakename, inxoy, beavertails), plus 6,216 synthetic rows for the three under-covered classes.
|
| 121 |
+
- Seed 17, lr 3e-5, 6 epochs, batch 8 with grad-accum 4, fp16, label smoothing 0.05, class weights, dropout 0.1, `max_length` 512. Best epoch 5.
|
| 122 |
+
- Trained on 2x T4.
|
| 123 |
+
|
| 124 |
+
## Limitations
|
| 125 |
+
|
| 126 |
+
- **The comparison is on our split, not the published one.** The v2 model card reports 0.9874 macro F1 on its own 1,017-row split, which is gated. Our 0.792 and its 0.9874 are different tests on different rows and must not be compared directly.
|
| 127 |
+
- **`partial_refusal` is thin.** Only 30 test rows, and it is 100% synthetic in train (1,999 synthetic of 2,006 rows; only 61 real partials exist in the pool). This one class moves macro F1 by about 0.10 across seeds.
|
| 128 |
+
- **No Heretic ARA validation yet.** The model is built to be a Heretic scorer, but no ARA comparison against v1 and v2 has been run. Do not treat it as validated for that use until one is.
|
| 129 |
+
- **One NaN training epoch.** Epoch 2 recorded `train_loss = NaN` (fp16 overflow). The best checkpoint is epoch 5 and the final metrics are valid; the run was not corrupted.
|
| 130 |
+
- **English only**, and the label set is closed. Out-of-distribution refusal styles may be mislabeled.
|
| 131 |
+
- **Not calibrated as a probability.** See the calibration note above.
|
| 132 |
+
|
| 133 |
+
## Data
|
| 134 |
+
|
| 135 |
+
The training data is in the private dataset `saidutta69/vantablack-refusal-data`. Labels were produced by an aligned instruct teacher (prompt `v2-2026-11-04`), never by an abliterated model, and audited on 100 rows at 97% agreement. Synthetic rows use a separate generation prompt (`v1-2026-11-04`) and the two prompt versions are never merged.
|
| 136 |
+
|
| 137 |
+
## Intended use
|
| 138 |
+
|
| 139 |
+
Refusal scoring for abliteration pipelines such as Heretic 2.x ARA, and offline analysis of refusal behavior in model outputs. Not a content filter on its own: it classifies a response that already exists, and it has not been validated as a safety gate.
|