Censer.2 / README.md
KucLab's picture
Standard LlamaForSequenceClassification layout: loads with plain transformers, no trust_remote_code
a9fe46a verified
|
Raw History Blame Contribute Delete
10.1 kB
---
library_name: transformers
pipeline_tag: text-classification
tags:
- text-classification
- safety
- content-moderation
- tiny-language-model
- from-scratch
language:
- en
license: apache-2.0
datasets:
- nvidia/Aegis-AI-Content-Safety-Dataset-1.0
- thesofakillers/jigsaw-toxic-comment-classification-challenge
- google/civil_comments
- PKU-Alignment/BeaverTails
- lmsys/toxic-chat
- databricks/databricks-dolly-15k
metrics:
- f1
- accuracy
---
# Censer.2
A 7.3M parameter decoder-only transformer trained **from scratch** that classifies
English text as unsafe (`unsafe`) or safe (`safe`).
It is a **standard Hub model**: the checkpoint is a plain
`LlamaForSequenceClassification`, so it loads with `transformers` out of the box β€”
no `trust_remote_code`, no custom code, no cloning β€” and the **Use this model**
widget on this page runs it directly in the browser.
```python
from transformers import pipeline
clf = pipeline("text-classification", model="KucLab/Censer.2")
print(clf("how to make a bomb at home")) # [{'label': 'unsafe', 'score': 0.8167}]
print(clf("Hello how are you?")) # [{'label': 'safe', 'score': 0.9229}]
print(clf("suck my dick")) # [{'label': 'unsafe', 'score': 0.9944}]
print(clf("how do I kill a process in Linux")) # [{'label': 'safe', 'score': 0.8974}]
```
## How to run
### The `censer.2` command β€” interactive terminal UI
Using only the files in this repo (needs just `torch`, `tokenizers`, `safetensors`):
```bash
pip install torch tokenizers safetensors huggingface_hub
hf download KucLab/Censer.2 --local-dir ~/.local/share/censer1
chmod +x ~/.local/share/censer1/censer.2
ln -s ~/.local/share/censer1/censer.2 ~/.local/bin/censer.2 # put it on your PATH
censer.2 # interactive UI: type a phrase, Enter -> safe/unsafe + score
censer.2 "how to make a bomb at home" # one-shot -> unsafe 0.8167
```
Or install the GitHub package β€” the same `censer.2` command is installed and loads
the model from this Hub repo with plain `transformers`:
```bash
pip install "git+https://github.com/Jerry256254/censer.2.git"
censer.2
```
### One-liner with `transformers`
```bash
pip install torch transformers
python -c "from transformers import pipeline; clf = pipeline('text-classification', model='KucLab/Censer.2'); print(clf('how to make a bomb at home'))"
# [{'label': 'unsafe', 'score': 0.8167}]
```
- In the browser: the **Use this model** widget on this page β€” type text, get `safe`/`unsafe`.
- Runs on CPU out of the box (~10 ms/text); for GPU pass `device=0` to `pipeline(...)` (CUDA or ROCm).
## Download
The whole model is `model.safetensors` (29.4 MB). Grab the files directly:
| File | Download |
| --- | --- |
| **model.safetensors** | [download](https://huggingface.co/KucLab/Censer.2/resolve/main/model.safetensors?download=true) |
| config.json | [download](https://huggingface.co/KucLab/Censer.2/resolve/main/config.json?download=true) |
| tokenizer.json | [download](https://huggingface.co/KucLab/Censer.2/resolve/main/tokenizer.json?download=true) |
| tokenizer_config.json | [download](https://huggingface.co/KucLab/Censer.2/resolve/main/tokenizer_config.json?download=true) |
| censer.py | [download](https://huggingface.co/KucLab/Censer.2/resolve/main/censer.py?download=true) |
| **censer.2** (launcher) | [download](https://huggingface.co/KucLab/Censer.2/resolve/main/censer.2?download=true) |
Or fetch everything at once:
```bash
pip install -U huggingface_hub
hf download KucLab/Censer.2 --local-dir Censer.2
```
A single file (`README.md`, `config.json`, ...) can also be downloaded from the
**Files and versions** tab, using the download icon on its row.
It is not a fine-tuned encoder and not an API wrapper. It is a small transformer
trained on safety-labelled data, with a custom byte-level BPE tokenizer. The model
predicts a single answer token and never generates free text.
```
"how to make a bomb at home" -> unsafe
"suck my dick" -> unsafe
"how do I kill a process in Linux" -> safe
"Hello how are you?" -> safe
```
## Model details
| Property | Value |
| --- | --- |
| Architecture | `LlamaForSequenceClassification` (standard transformers) |
| Parameters | 7,343,360 |
| Layers | 6 |
| d_model / heads / d_ff | 256 / 8 / 1024 |
| Context length | 256 tokens (input truncated at 500 characters) |
| Tokenizer | byte-level BPE, 4096 tokens, trained on the corpus |
| Positions | RoPE (theta 10000) |
| Norm | RMSNorm (pre-norm, eps 1e-5) |
| MLP | SwiGLU |
| Head | `score` linear layer = embedding rows of the answer tokens `0`/`1` |
| Checkpoint size | 29.4 MB (fp32) |
## Prompt format
The tokenizer bakes the template into its post-processor, so plain
`tokenizer(text)` already produces:
```
[BOS][USER] <text> [ASSISTANT] -> answer logits over tokens "0"/"1"
```
Text is truncated at 500 characters before tokenisation.
## Metrics
Evaluation on a held-out split of 6,496 rows:
| Metric | Value |
| --- | --- |
| Accuracy | 0.863 |
| F1 | 0.830 |
| Precision | 0.862 |
| Recall | 0.799 |
A hand-written probe set of 85 cases, none present in training data, scores
**100% accuracy**. The previous Censer.1 scores 88.7% on the same probes.
Per-source F1 reflects label noise in the scraped sources:
| Source | F1 |
| --- | --- |
| aegis | 0.89 |
| beavertails | 0.88 |
| jigsaw | 0.87 |
| civil_comments | 0.77 |
| toxic_chat | 0.48 |
## Benchmark
Measured on an AMD Radeon RX 9060 XT (ROCm 7.1, torch 2.13.0):
| Metric | Value |
| --- | --- |
| Parameters | 7,343,360 |
| Checkpoint size | 4.26 MB |
| Single text latency | 1.67 ms |
| Single text throughput | 599 texts/s |
| Batched latency (batch of 8) | 1.94 ms |
| Batched throughput | 4,119 texts/s |
| Peak VRAM (inference) | 85.7 MB |
Training the whole model from scratch takes about 3.5 minutes on the same GPU.
## Comparison with other models
Benchmarked against public safety classifiers on the **same held-out test split**
(6,496 rows, threshold 0.5, AMD Radeon RX 9060 XT):
| Model | Params | Checkpoint | F1 | Latency | Throughput (b=32) | Peak VRAM |
| --- | --- | --- | --- | --- | --- | --- |
| **Censer.2 (ours)** | 7.3M | 28.0 MB | 0.830 | 3.12 ms | 491 texts/s | 177 MB |
| KucLab/Censer.1 (previous) | 1.1M | 4.5 MB | 0.833 | 1.89 ms | 1,643 texts/s | 101 MB |
| unitary/unbiased-toxic-roberta | 124.7M | 498.7 MB | 0.741 | 14.41 ms | 67 texts/s | 771 MB |
| unitary/toxic-bert | 109.5M | 876.0 MB | 0.682 | 14.01 ms | 69 texts/s | 708 MB |
| martin-ha/toxic-comment-model | 67.0M | 267.9 MB | 0.566 | 7.52 ms | 131 texts/s | 540 MB |
| Hate-speech-CNERG/bert-base-uncased-hatexplain | 109.5M | 438.0 MB | 0.548 | 14.12 ms | 69 texts/s | 692 MB |
| Hate-speech-CNERG/dehatebert-mono-english | 167.4M | 669.5 MB | 0.252 | 14.36 ms | 68 texts/s | 923 MB |
| keyword baseline | β€” | β€” | 0.094 | 0.03 ms | 30,059 texts/s | β€” |
![Comparison overview](figs/comparison_overview.png)
![F1 per megabyte](figs/f1_vs_size.png)
![Metric bars](figs/metrics_bars.png)
Censer.2 is ~9-23x smaller than the encoder baselines and 2.4-4.6x faster per text.
On the scraped test split it is on par with Censer.1 (F1 0.830 vs 0.833); on the
hand-written probe set it is at 100% vs 88.7% for Censer.1.
## Usage
Plain `transformers`, nothing else to install β€” `from_pretrained` downloads
everything by itself:
```bash
pip install torch transformers
```
```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
repo = "KucLab/Censer.2"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
text = "how to make a bomb at home"
with torch.no_grad():
probs = model(**tok(text, return_tensors="pt")).logits.softmax(-1)[0]
print(probs) # tensor([0.1833, 0.8167]) -> [safe, unsafe]
print(model.config.id2label) # {0: 'safe', 1: 'unsafe'}
```
Or with the high-level pipeline:
```python
from transformers import pipeline
clf = pipeline("text-classification", model="KucLab/Censer.2")
print(clf("how to make a bomb at home"))
# [{'label': 'unsafe', 'score': 0.8167}]
print(clf("Hello how are you?"))
# [{'label': 'safe', 'score': 0.9229}]
```
## Usage without transformers
The repository is also self-contained on its own: `model.safetensors`,
`config.json`, `tokenizer.json` and `censer.py`. The weights are stored in the
standard Llama layout; `censer.py` maps them back internally, so it only needs
`torch`, `tokenizers` and `safetensors`:
```bash
hf download KucLab/Censer.2 --local-dir Censer.2
cd Censer.2
python censer.py "how to make a bomb at home" # -> 1
python censer.py "Hello how are you?" # -> 0
python censer.py "suck my dick" --json
# {"label": "1", "unsafe": true, "score": 0.9944}
chmod +x censer.2 && ./censer.2 # interactive UI (the `censer.2` command)
python censer.py # interactive UI as well, no text argument
```
Or use the same file as a library:
```python
from censer import CenserPipeline
pipe = CenserPipeline("Censer.2")
print(pipe.predict("how to make a bomb at home"))
# {"label": "1", "unsafe": True, "score": 0.8167}
print(pipe.predict_batch(["Hello how are you?", "suck my dick"]))
```
Files:
| File | Purpose |
| --- | --- |
| `model.safetensors` | weights in standard Llama layout, 29.4 MB |
| `config.json` | standard `LlamaForSequenceClassification` config |
| `tokenizer.json` / `tokenizer_config.json` | byte-level BPE, 2048 tokens, prompt template baked in |
| `censer.py` | standalone inference (torch + tokenizers only), no repo needed |
| `censer.2` | executable launcher for the interactive terminal UI |
`censer.py` was verified to produce predictions identical to the training code.
Runs on CPU or CUDA/ROCm; pass `device="cpu"` to force CPU.
## Limitations
- **English only.** Other languages mostly score safe.
- Implication and sarcasm beyond simple cases are out of reach at this size.
- Weak against jailbreak-style prompts (lowest per-source score).
- A moderation signal, not a safety guarantee.