KucLab commited on
Commit
45cfdd3
·
verified ·
1 Parent(s): 3698f79

Standard LlamaForSequenceClassification layout: loads with plain transformers, no trust_remote_code

Browse files
Files changed (7) hide show
  1. README.md +292 -0
  2. censer.5 +30 -0
  3. censer.py +295 -0
  4. config.json +47 -0
  5. model.safetensors +3 -0
  6. tokenizer.json +0 -0
  7. tokenizer_config.json +6 -0
README.md ADDED
@@ -0,0 +1,292 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ pipeline_tag: text-classification
4
+ tags:
5
+ - text-classification
6
+ - safety
7
+ - content-moderation
8
+ - tiny-language-model
9
+ - from-scratch
10
+ - distilled
11
+ language:
12
+ - en
13
+ license: apache-2.0
14
+ datasets:
15
+ - nvidia/Aegis-AI-Content-Safety-Dataset-1.0
16
+ - thesofakillers/jigsaw-toxic-comment-classification-challenge
17
+ - google/civil_comments
18
+ - PKU-Alignment/BeaverTails
19
+ - lmsys/toxic-chat
20
+ - databricks/databricks-dolly-15k
21
+ metrics:
22
+ - f1
23
+ - accuracy
24
+ ---
25
+
26
+ # Censer.5
27
+
28
+ A **902K parameter** decoder-only transformer trained **from scratch** that
29
+ classifies English text as unsafe (`unsafe`) or safe (`safe`) — the smallest
30
+ *and* the most accurate model in the comparison below.
31
+
32
+ It is a **standard Hub model**: the checkpoint is a plain
33
+ `LlamaForSequenceClassification`, so it loads with `transformers` out of the box —
34
+ no `trust_remote_code`, no custom code, no cloning — and the **Use this model**
35
+ widget on this page runs it directly in the browser.
36
+
37
+ ```python
38
+ from transformers import pipeline
39
+
40
+ clf = pipeline("text-classification", model="KucLab/Censer.5")
41
+ print(clf("how to make a bomb at home")) # [{'label': 'unsafe', 'score': 0.9942}]
42
+ print(clf("Hello how are you?")) # [{'label': 'safe', 'score': 0.8780}]
43
+ print(clf("suck my dick")) # [{'label': 'unsafe', 'score': 0.9955}]
44
+ print(clf("kill you")) # [{'label': 'unsafe', 'score': 0.9263}]
45
+ print(clf("how to filter wate")) # [{'label': 'safe', 'score': 0.9479}] <- typo-robust
46
+ print(clf("how do I kill a process in Linux")) # [{'label': 'safe', 'score': 0.9985}]
47
+ ```
48
+
49
+ ## How to run
50
+
51
+ ### The `censer.5` command — interactive terminal UI
52
+
53
+ Using only the files in this repo (needs just `torch`, `tokenizers`, `safetensors`):
54
+
55
+ ```bash
56
+ pip install torch tokenizers safetensors huggingface_hub
57
+ hf download KucLab/Censer.5 --local-dir ~/.local/share/censer5
58
+ chmod +x ~/.local/share/censer5/censer.5
59
+ ln -s ~/.local/share/censer5/censer.5 ~/.local/bin/censer.5 # put it on your PATH
60
+ censer.5 # interactive UI: type a phrase, Enter -> safe/unsafe + score
61
+ censer.5 "how to make a bomb at home" # one-shot -> unsafe 0.9942
62
+ ```
63
+
64
+ Or install the GitHub package — the same `censer.5` command is installed and loads
65
+ the model from this Hub repo with plain `transformers`:
66
+
67
+ ```bash
68
+ pip install "git+https://github.com/Jerry256254/censer.5.git"
69
+ censer.5
70
+ ```
71
+
72
+ ### One-liner with `transformers`
73
+
74
+ ```bash
75
+ pip install torch transformers
76
+ python -c "from transformers import pipeline; clf = pipeline('text-classification', model='KucLab/Censer.5'); print(clf('how to make a bomb at home'))"
77
+ # [{'label': 'unsafe', 'score': 0.994}]
78
+ ```
79
+
80
+ - In the browser: the **Use this model** widget on this page — type text, get `safe`/`unsafe`.
81
+ - Runs on CPU out of the box; for GPU pass `device=0` to `pipeline(...)` (CUDA or ROCm).
82
+
83
+ ## Download
84
+
85
+ The whole model is `model.safetensors` (3.6 MB). Grab the files directly:
86
+
87
+ | File | Download |
88
+ | --- | --- |
89
+ | **model.safetensors** | [download](https://huggingface.co/KucLab/Censer.5/resolve/main/model.safetensors?download=true) |
90
+ | config.json | [download](https://huggingface.co/KucLab/Censer.5/resolve/main/config.json?download=true) |
91
+ | tokenizer.json | [download](https://huggingface.co/KucLab/Censer.5/resolve/main/tokenizer.json?download=true) |
92
+ | tokenizer_config.json | [download](https://huggingface.co/KucLab/Censer.5/resolve/main/tokenizer_config.json?download=true) |
93
+ | censer.py | [download](https://huggingface.co/KucLab/Censer.5/resolve/main/censer.py?download=true) |
94
+ | **censer.5** (launcher) | [download](https://huggingface.co/KucLab/Censer.5/resolve/main/censer.5?download=true) |
95
+
96
+ Or fetch everything at once:
97
+
98
+ ```bash
99
+ pip install -U huggingface_hub
100
+ hf download KucLab/Censer.5 --local-dir Censer.5
101
+ ```
102
+
103
+ It is not a fine-tuned encoder and not an API wrapper. It is a small transformer
104
+ trained on safety-labelled data with a custom byte-level BPE tokenizer. The model
105
+ predicts a single answer token and never generates free text. The tiny 3-layer
106
+ student is **distilled from a 5-teacher ensemble** (Censer.1–.4 + a dedicated
107
+ 8-layer teacher), on 2x-augmented training data with a 192-token context.
108
+
109
+ ```
110
+ "how to make a bomb at home" -> unsafe 0.994
111
+ "kill you" / "selling gun" -> unsafe 0.93-0.94
112
+ "suck my dick" -> unsafe 0.996
113
+ "how do I kill a process in Linux" -> safe 0.9985
114
+ "how to filter wate" (typo) -> safe 0.948
115
+ "Hello how are you?" -> safe 0.878
116
+ ```
117
+
118
+ ## Model details
119
+
120
+ | Property | Value |
121
+ | --- | --- |
122
+ | Architecture | `LlamaForSequenceClassification` (standard transformers) |
123
+ | Parameters | 902,016 |
124
+ | Layers | 3 |
125
+ | d_model / heads / d_ff | 128 / 4 / 384 |
126
+ | Context length | 192 tokens (input truncated at 500 characters) |
127
+ | Tokenizer | byte-level BPE, 2048 tokens, trained on the corpus |
128
+ | Positions | RoPE (theta 10000) |
129
+ | Norm | RMSNorm (pre-norm, eps 1e-5) |
130
+ | MLP | SwiGLU |
131
+ | Head | `score` linear layer = embedding rows of the answer tokens `0`/`1` |
132
+ | Training | 20 epochs, batch 128, AdamW 3e-3 cosine, bf16 AMP, 15% typo augmentation, 2x augmented train data, distillation (alpha=0.65, T=4) from a 5-teacher ensemble |
133
+ | Checkpoint size | 3.6 MB (fp32) |
134
+
135
+ ## Prompt format
136
+
137
+ The tokenizer bakes the template into its post-processor, so plain
138
+ `tokenizer(text)` already produces:
139
+
140
+ ```
141
+ [BOS][USER] <text> [ASSISTANT] -> answer logits over tokens "0"/"1"
142
+ ```
143
+
144
+ Text is truncated at 500 characters before tokenisation.
145
+
146
+ ## Metrics
147
+
148
+ Evaluation on a held-out split of 6,496 rows (identical split across all Censer
149
+ versions), threshold 0.5:
150
+
151
+ | Metric | Censer.1 | Censer.2 | Censer.3 | Censer.4 | **Censer.5** |
152
+ | --- | --- | --- | --- | --- | --- |
153
+ | Accuracy | 0.865 | 0.863 | 0.857 | 0.874 | **0.876** |
154
+ | F1 | 0.835 | 0.829 | 0.822 | 0.845 | **0.849** |
155
+ | Precision | 0.848 | 0.861 | 0.855 | 0.868 | **0.864** |
156
+ | Recall | 0.823 | 0.799 | 0.792 | 0.824 | **0.834** |
157
+
158
+ Per-source F1: aegis (human-annotated) **0.912**, BeaverTails **0.883**, jigsaw
159
+ **0.880**, civil_comments **0.790** — up from 0.887 / 0.856 / 0.863 / 0.753 in
160
+ Censer.3. The remaining overall-test errors are largely label noise in the
161
+ scraped sources.
162
+
163
+ A hand-written probe set of 97 cases, none present in training data, scores
164
+ **97.9%** (95/97). Probe progression on the same 97 cases: Censer.1 75.3% ->
165
+ .2 91.8% -> .3 100% -> .4 99.0% -> **.5 97.9%**.
166
+
167
+ ## Benchmark
168
+
169
+ Measured on an AMD Radeon RX 9060 XT (ROCm 7.1, torch 2.13.0+rocm7.1):
170
+
171
+ | Metric | Censer.1 | Censer.4 | **Censer.5** |
172
+ | --- | --- | --- | --- |
173
+ | Parameters | 1,115,264 | 902,016 | **902,016** |
174
+ | Checkpoint size | 4.5 MB | 3.6 MB | **3.6 MB** |
175
+ | Single text latency | 1.58 ms | 1.16 ms | **1.21 ms** |
176
+ | Single text throughput | 633 texts/s | 864 texts/s | **823 texts/s** |
177
+ | Batched throughput (batch of 8) | 1,843 texts/s | 6,010 texts/s | **5,709 texts/s** |
178
+ | Peak VRAM (inference) | 101 MB | 84.1 MB | **84.1 MB** |
179
+
180
+ ## Comparison with other models
181
+
182
+ Benchmarked against public safety classifiers on the **same held-out test split**
183
+ (6,496 rows, threshold 0.5, AMD Radeon RX 9060 XT):
184
+
185
+ | Model | Params | Checkpoint | F1 | Latency | Throughput (b=8) | Peak VRAM |
186
+ | --- | --- | --- | --- | --- | --- | --- |
187
+ | **Censer.5 (ours)** | **0.9M** | **3.6 MB** | **0.849** | **1.43 ms** | **1,842 texts/s** | **105 MB** |
188
+ | KucLab/Censer.4 (prev) | 0.9M | 3.6 MB | 0.845 | 1.39 ms | 2,047 texts/s | 101 MB |
189
+ | KucLab/Censer.1 (old) | 1.1M | 4.5 MB | 0.835 | 1.66 ms | 1,719 texts/s | 102 MB |
190
+ | KucLab/Censer.2 (old) | 7.3M | 29.4 MB | 0.829 | 2.96 ms | 489 texts/s | 196 MB |
191
+ | KucLab/Censer.3 (old) | 14.4M | 57.7 MB | 0.822 | 4.60 ms | 256 texts/s | 267 MB |
192
+ | unitary/unbiased-toxic-roberta | 124.7M | 498.7 MB | 0.741 | 14.71 ms | 65 texts/s | 794 MB |
193
+ | unitary/toxic-bert | 109.5M | 876.0 MB | 0.681 | 14.10 ms | 67 texts/s | 731 MB |
194
+ | martin-ha/toxic-comment-model | 67.0M | 267.9 MB | 0.566 | 7.50 ms | 128 texts/s | 568 MB |
195
+ | Hate-speech-CNERG/bert-base-uncased-hatexplain | 109.5M | 438.0 MB | 0.543 | 14.25 ms | 67 texts/s | 704 MB |
196
+ | Hate-speech-CNERG/dehatebert-mono-english | 167.4M | 669.5 MB | 0.252 | 14.43 ms | 66 texts/s | 937 MB |
197
+ | keyword baseline | — | — | 0.094 | 0.03 ms | 34,516 texts/s | — |
198
+
199
+ ![Comparison overview](figs/comparison_overview.png)
200
+ ![F1 per megabyte](figs/f1_vs_size.png)
201
+ ![Metric bars](figs/metrics_bars.png)
202
+
203
+ Among all compared classifiers, **Censer.5 ranks #1 on F1 while being the
204
+ smallest and fastest**: ~75-190x smaller than the public encoder baselines,
205
+ ~5-10x faster per text, +0.11 F1 over the best of them.
206
+
207
+ ## Usage
208
+
209
+ Plain `transformers`, nothing else to install — `from_pretrained` downloads
210
+ everything by itself:
211
+
212
+ ```bash
213
+ pip install torch transformers
214
+ ```
215
+
216
+ ```python
217
+ import torch
218
+ from transformers import AutoTokenizer, AutoModelForSequenceClassification
219
+
220
+ repo = "KucLab/Censer.5"
221
+ tok = AutoTokenizer.from_pretrained(repo)
222
+ model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
223
+
224
+ text = "how to make a bomb at home"
225
+ with torch.no_grad():
226
+ probs = model(**tok(text, return_tensors="pt")).logits.softmax(-1)[0]
227
+
228
+ print(probs) # tensor([0.006, 0.994]) -> [safe, unsafe]
229
+ print(model.config.id2label) # {0: 'safe', 1: 'unsafe'}
230
+ ```
231
+
232
+ Or with the high-level pipeline:
233
+
234
+ ```python
235
+ from transformers import pipeline
236
+
237
+ clf = pipeline("text-classification", model="KucLab/Censer.5")
238
+ print(clf("how to make a bomb at home")) # [{'label': 'unsafe', 'score': 0.994}]
239
+ print(clf("Hello how are you?")) # [{'label': 'safe', 'score': 0.878}]
240
+ ```
241
+
242
+ ## Usage without transformers
243
+
244
+ The repository is also self-contained on its own: `model.safetensors`,
245
+ `config.json`, `tokenizer.json` and `censer.py`. The weights are stored in the
246
+ standard Llama layout; `censer.py` maps them back internally, so it only needs
247
+ `torch`, `tokenizers` and `safetensors`:
248
+
249
+ ```bash
250
+ hf download KucLab/Censer.5 --local-dir Censer.5
251
+ cd Censer.5
252
+ python censer.py "how to make a bomb at home" # -> 1
253
+ python censer.py "Hello how are you?" # -> 0
254
+ python censer.py "suck my dick" --json
255
+ # {"label": "1", "unsafe": true, "score": 0.9955}
256
+ chmod +x censer.5 && ./censer.5 # interactive UI (the `censer.5` command)
257
+ python censer.py # interactive UI as well, no text argument
258
+ ```
259
+
260
+ Or use the same file as a library:
261
+
262
+ ```python
263
+ from censer import CenserPipeline
264
+
265
+ pipe = CenserPipeline("Censer.5")
266
+ print(pipe.predict("how to make a bomb at home"))
267
+ # {"label": "1", "unsafe": True, "score": 0.994}
268
+
269
+ print(pipe.predict_batch(["Hello how are you?", "suck my dick"]))
270
+ ```
271
+
272
+ Files:
273
+
274
+ | File | Purpose |
275
+ | --- | --- |
276
+ | `model.safetensors` | weights in standard Llama layout, 3.6 MB |
277
+ | `config.json` | standard `LlamaForSequenceClassification` config |
278
+ | `tokenizer.json` / `tokenizer_config.json` | byte-level BPE, 2048 tokens, prompt template baked in |
279
+ | `censer.py` | standalone inference (torch + tokenizers only), no repo needed |
280
+ | `censer.5` | executable launcher for the interactive terminal UI |
281
+
282
+ `censer.py` was verified to produce predictions identical to the training code.
283
+ Runs on CPU or CUDA/ROCm; pass `device="cpu"` to force CPU.
284
+
285
+ ## Limitations
286
+
287
+ - **English only.** Other languages mostly score safe (common profanity from the
288
+ training data is still caught). Not a language detector.
289
+ - Implication and sarcasm beyond simple cases are out of reach at this size.
290
+ - The test split contains label noise from the scraped sources, which caps the
291
+ measurable overall F1 there.
292
+ - A moderation signal, not a safety guarantee.
censer.5 ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """censer.5 - run the Censer.5 safety classifier as an interactive terminal UI.
3
+
4
+ Usage:
5
+ censer.5 # interactive: type a phrase, Enter -> safe/unsafe + score
6
+ censer.5 "how to make a bomb" # one-shot
7
+ censer.5 --help
8
+
9
+ Expects censer.py, model.safetensors, config.json and tokenizer.json next to
10
+ this file (they are found automatically, whatever the current directory is).
11
+ Get them all with:
12
+
13
+ hf download KucLab/Censer.5 --local-dir .
14
+
15
+ Requires: torch, tokenizers, safetensors (pip install torch tokenizers safetensors)
16
+ """
17
+
18
+ import os
19
+ import sys
20
+
21
+ HERE = os.path.dirname(os.path.abspath(__file__))
22
+ sys.path.insert(0, HERE)
23
+
24
+ from censer import main
25
+
26
+ if __name__ == "__main__":
27
+ argv = sys.argv[1:]
28
+ if not any(a == "--model-dir" or a.startswith("--model-dir=") for a in argv):
29
+ argv = ["--model-dir", HERE] + argv
30
+ main(argv)
censer.py ADDED
@@ -0,0 +1,295 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Inference for Censer.5 — self-contained, no dependency on the training repo.
2
+
3
+ Copy this file next to the downloaded weights, install torch and tokenizers,
4
+ and you are ready:
5
+
6
+ pip install torch tokenizers safetensors
7
+ python censer.py "how to make a bomb at home"
8
+ python censer.py # or just run it: interactive UI, phrase in -> verdict out
9
+ ./censer.5 # same UI via the censer.5 launcher next to this file
10
+
11
+ It expects, in the same directory:
12
+ model.safetensors (standard LlamaForSequenceClassification layout)
13
+ tokenizer.json
14
+ config.json
15
+ """
16
+
17
+ from __future__ import annotations
18
+
19
+ import argparse
20
+ import json
21
+ import math
22
+ import sys
23
+ from pathlib import Path
24
+
25
+ import torch
26
+ import torch.nn as nn
27
+ import torch.nn.functional as F
28
+ from tokenizers import Tokenizer
29
+
30
+ PAD, BOS, USER, ASSISTANT = "[PAD]", "[BOS]", "[USER]", "[ASSISTANT]"
31
+
32
+
33
+ class RMSNorm(nn.Module):
34
+ def __init__(self, dim: int, eps: float = 1e-5) -> None:
35
+ super().__init__()
36
+ self.eps = eps
37
+ self.weight = nn.Parameter(torch.ones(dim))
38
+
39
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
40
+ dtype = x.dtype
41
+ x = x.float()
42
+ x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
43
+ return x.to(dtype) * self.weight
44
+
45
+
46
+ def build_rope_cache(seq_len: int, head_dim: int, device, dtype, base: float = 10000.0):
47
+ half = head_dim // 2
48
+ freqs = 1.0 / (base ** (torch.arange(0, half, device=device, dtype=torch.float32) / half))
49
+ positions = torch.arange(seq_len, device=device, dtype=torch.float32)
50
+ angles = torch.outer(positions, freqs)
51
+ cos = torch.cat((angles.cos(), angles.cos()), dim=-1).to(dtype)[:, None, :]
52
+ sin = torch.cat((angles.sin(), angles.sin()), dim=-1).to(dtype)[:, None, :]
53
+ return cos, sin
54
+
55
+
56
+ def apply_rope(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
57
+ seq_len = x.shape[1]
58
+ cos, sin = cos[:seq_len], sin[:seq_len]
59
+ x1, x2 = x[..., : x.shape[-1] // 2], x[..., x.shape[-1] // 2 :]
60
+ rotated = torch.cat((-x2, x1), dim=-1)
61
+ return x * cos + rotated * sin
62
+
63
+
64
+ class Attention(nn.Module):
65
+ def __init__(self, cfg: dict) -> None:
66
+ super().__init__()
67
+ self.n_heads = cfg["n_heads"]
68
+ self.head_dim = cfg["d_model"] // cfg["n_heads"]
69
+ self.dropout = cfg["dropout"]
70
+ self.qkv = nn.Linear(cfg["d_model"], 3 * cfg["d_model"], bias=False)
71
+ self.proj = nn.Linear(cfg["d_model"], cfg["d_model"], bias=False)
72
+
73
+ def forward(self, x, cos, sin, mask):
74
+ b, t, _ = x.shape
75
+ q, k, v = self.qkv(x).split(x.shape[-1], dim=-1)
76
+ q = q.view(b, t, self.n_heads, self.head_dim)
77
+ k = k.view(b, t, self.n_heads, self.head_dim)
78
+ v = v.view(b, t, self.n_heads, self.head_dim)
79
+ q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
80
+ out = F.scaled_dot_product_attention(
81
+ q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2),
82
+ attn_mask=mask,
83
+ dropout_p=self.dropout if self.training else 0.0,
84
+ is_causal=mask is None,
85
+ )
86
+ return self.proj(out.transpose(1, 2).reshape(b, t, -1))
87
+
88
+
89
+ class SwiGLU(nn.Module):
90
+ def __init__(self, cfg: dict) -> None:
91
+ super().__init__()
92
+ self.gate = nn.Linear(cfg["d_model"], cfg["d_ff"], bias=False)
93
+ self.up = nn.Linear(cfg["d_model"], cfg["d_ff"], bias=False)
94
+ self.down = nn.Linear(cfg["d_ff"], cfg["d_model"], bias=False)
95
+
96
+ def forward(self, x):
97
+ return self.down(F.silu(self.gate(x)) * self.up(x))
98
+
99
+
100
+ class Block(nn.Module):
101
+ def __init__(self, cfg: dict) -> None:
102
+ super().__init__()
103
+ self.norm1 = RMSNorm(cfg["d_model"])
104
+ self.attn = Attention(cfg)
105
+ self.norm2 = RMSNorm(cfg["d_model"])
106
+ self.mlp = SwiGLU(cfg)
107
+ self.dropout = nn.Dropout(cfg["dropout"])
108
+
109
+ def forward(self, x, cos, sin, mask):
110
+ x = x + self.dropout(self.attn(self.norm1(x), cos, sin, mask))
111
+ x = x + self.dropout(self.mlp(self.norm2(x)))
112
+ return x
113
+
114
+
115
+ class Censer(nn.Module):
116
+ def __init__(self, cfg: dict) -> None:
117
+ super().__init__()
118
+ self.cfg = cfg
119
+ self.register_buffer(
120
+ "answer_id_tensor", torch.tensor(cfg["answer_ids"], dtype=torch.long), persistent=False
121
+ )
122
+ self.embed = nn.Embedding(cfg["vocab_size"], cfg["d_model"], padding_idx=cfg["pad_id"])
123
+ self.blocks = nn.ModuleList([Block(cfg) for _ in range(cfg["n_layers"])])
124
+ self.norm = RMSNorm(cfg["d_model"])
125
+ self.lm_head = nn.Linear(cfg["d_model"], cfg["vocab_size"], bias=False)
126
+ if cfg["tie_embeddings"]:
127
+ self.lm_head.weight = self.embed.weight
128
+ self._rope_cache = None
129
+
130
+ def _rope(self, device, dtype):
131
+ key = f"{device}|{dtype}"
132
+ if self._rope_cache is None or self._rope_cache[2] != key:
133
+ head_dim = self.cfg["d_model"] // self.cfg["n_heads"]
134
+ cos, sin = build_rope_cache(self.cfg["max_seq_len"], head_dim, device, dtype)
135
+ self._rope_cache = (cos, sin, key)
136
+ return self._rope_cache[0], self._rope_cache[1]
137
+
138
+ def _attention_mask(self, input_ids):
139
+ b, t = input_ids.shape
140
+ pad = input_ids != self.cfg["pad_id"]
141
+ causal = torch.ones(t, t, dtype=torch.bool, device=input_ids.device).tril()
142
+ return pad[:, None, None, :] & causal[None, None, :, :]
143
+
144
+ def forward(self, input_ids):
145
+ cos, sin = self._rope(input_ids.device, self.embed.weight.dtype)
146
+ mask = self._attention_mask(input_ids)
147
+ x = self.embed(input_ids)
148
+ for block in self.blocks:
149
+ x = block(x, cos, sin, mask)
150
+ return self.lm_head(self.norm(x))
151
+
152
+ @torch.no_grad()
153
+ def classify(self, input_ids):
154
+ hidden = self.forward(input_ids)
155
+ non_pad = (input_ids != self.cfg["pad_id"]).sum(dim=1).clamp(min=1)
156
+ idx = (non_pad - 1)[:, None, None].expand(-1, 1, hidden.shape[-1])
157
+ logits = hidden.gather(1, idx)[:, 0, :].float()
158
+ logits = logits.index_select(-1, self.answer_id_tensor)
159
+ return logits.softmax(-1)[:, 1]
160
+
161
+
162
+ def remap_hf_state(state: dict) -> dict:
163
+ """Standard Llama-layout safetensors -> training-layout state dict."""
164
+ import re
165
+
166
+ out = {"embed.weight": state["model.embed_tokens.weight"]}
167
+ layer_ids = sorted({int(m.group(1)) for k in state if (m := re.match(r"model\.layers\.(\d+)\.", k))})
168
+ for i in layer_ids:
169
+ p, b = f"model.layers.{i}.", f"blocks.{i}."
170
+ qkv = torch.cat(
171
+ [state[p + "self_attn.q_proj.weight"], state[p + "self_attn.k_proj.weight"], state[p + "self_attn.v_proj.weight"]],
172
+ dim=0,
173
+ )
174
+ out[b + "attn.qkv.weight"] = qkv.contiguous()
175
+ out[b + "attn.proj.weight"] = state[p + "self_attn.o_proj.weight"]
176
+ out[b + "mlp.gate.weight"] = state[p + "mlp.gate_proj.weight"]
177
+ out[b + "mlp.up.weight"] = state[p + "mlp.up_proj.weight"]
178
+ out[b + "mlp.down.weight"] = state[p + "mlp.down_proj.weight"]
179
+ out[b + "norm1.weight"] = state[p + "input_layernorm.weight"]
180
+ out[b + "norm2.weight"] = state[p + "post_attention_layernorm.weight"]
181
+ out["norm.weight"] = state["model.norm.weight"]
182
+ return out
183
+
184
+
185
+ def internal_cfg(cfg: dict) -> dict:
186
+ """config.json (Llama layout) -> the dict the Censer module expects."""
187
+ return {
188
+ "vocab_size": cfg["vocab_size"],
189
+ "n_layers": cfg["num_hidden_layers"],
190
+ "n_heads": cfg["num_attention_heads"],
191
+ "d_model": cfg["hidden_size"],
192
+ "d_ff": cfg["intermediate_size"],
193
+ "max_seq_len": cfg["max_position_embeddings"],
194
+ "dropout": cfg.get("dropout", 0.0),
195
+ "tie_embeddings": True,
196
+ "pad_id": cfg.get("pad_token_id", 0),
197
+ "answer_ids": cfg["answer_ids"],
198
+ "max_chars": cfg.get("max_chars", 500),
199
+ }
200
+
201
+
202
+ class CenserPipeline:
203
+ """Text in, "1" (unsafe) or "0" (safe) out."""
204
+
205
+ def __init__(self, directory: str | Path = ".", threshold: float = 0.5, device: str | None = None) -> None:
206
+ from safetensors.torch import load_file
207
+
208
+ directory = Path(directory)
209
+ self.cfg = internal_cfg(json.loads((directory / "config.json").read_text(encoding="utf-8")))
210
+ self.tok = Tokenizer.from_file(str(directory / "tokenizer.json"))
211
+ self.pad_id = self.tok.token_to_id(PAD)
212
+ self.bos_id = self.tok.token_to_id(BOS)
213
+ self.user_id = self.tok.token_to_id(USER)
214
+ self.assistant_id = self.tok.token_to_id(ASSISTANT)
215
+ self.threshold = threshold
216
+ self.device = torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
217
+ self.model = Censer(self.cfg)
218
+ state = remap_hf_state(load_file(directory / "model.safetensors"))
219
+ missing, unexpected = self.model.load_state_dict(state, strict=False)
220
+ missing = [k for k in missing if k != "lm_head.weight"] # tied to embed.weight
221
+ if missing or unexpected:
222
+ raise RuntimeError(f"weight mismatch: missing={missing} unexpected={unexpected}")
223
+ self.model.to(self.device).eval()
224
+ self.max_chars = self.cfg["max_chars"]
225
+
226
+ def _encode(self, text: str) -> list[int]:
227
+ body = self.tok.encode(text[: self.max_chars], add_special_tokens=False).ids
228
+ ids = [self.bos_id, self.user_id, *body, self.assistant_id]
229
+ return ids[: self.cfg["max_seq_len"]]
230
+
231
+ def predict(self, text: str) -> dict:
232
+ return self.predict_batch([text])[0]
233
+
234
+ def predict_batch(self, texts: list[str]) -> list[dict]:
235
+ seqs = [self._encode(t) for t in texts]
236
+ width = max(len(s) for s in seqs)
237
+ ids = torch.full((len(seqs), width), self.pad_id, dtype=torch.long)
238
+ for i, seq in enumerate(seqs):
239
+ ids[i, : len(seq)] = torch.tensor(seq)
240
+ probs = self.model.classify(ids.to(self.device)).float().cpu().tolist()
241
+ return [
242
+ {"label": "1" if p >= self.threshold else "0", "unsafe": p >= self.threshold, "score": round(p, 4)}
243
+ for p in probs
244
+ ]
245
+
246
+
247
+ def _color(text: str, code: str) -> str:
248
+ if not sys.stdout.isatty():
249
+ return text
250
+ return f"\033[{code}m{text}\033[0m"
251
+
252
+
253
+ def repl(pipe: "CenserPipeline") -> int:
254
+ """Small interactive terminal UI: phrase in, safe/unsafe + score out."""
255
+ print("Censer.5 - interactive terminal UI")
256
+ print("Type a phrase, Enter to classify. 'exit' to quit.\n")
257
+ while True:
258
+ try:
259
+ line = input("> ")
260
+ except (EOFError, KeyboardInterrupt):
261
+ print()
262
+ return 0
263
+ text = line.strip()
264
+ if text.lower() in {"exit", "quit", ":q"}:
265
+ return 0
266
+ if not text:
267
+ continue
268
+ v = pipe.predict(text)
269
+ label = "unsafe" if v["unsafe"] else "safe"
270
+ colored = _color(f"{label:6s}", "31" if v["unsafe"] else "32")
271
+ print(f"{colored} {v['score']:.4f} | {text}")
272
+
273
+
274
+ def main(argv: list[str] | None = None) -> None:
275
+ parser = argparse.ArgumentParser(
276
+ prog="censer.5",
277
+ description="Censer.5 safety classifier (standalone). Run without text to open the interactive UI.",
278
+ )
279
+ parser.add_argument("text", nargs="*", help="phrase to classify once; omit to open the interactive UI")
280
+ parser.add_argument("--model-dir", default=".")
281
+ parser.add_argument("--threshold", type=float, default=0.5)
282
+ parser.add_argument("--json", action="store_true")
283
+ parser.add_argument("--repl", action="store_true", help="interactive UI (default when no text is given)")
284
+ args = parser.parse_args(argv)
285
+
286
+ pipe = CenserPipeline(args.model_dir, args.threshold)
287
+ text = " ".join(args.text).strip()
288
+ if args.repl or not text:
289
+ raise SystemExit(repl(pipe))
290
+ verdict = pipe.predict(text)
291
+ print(json.dumps(verdict) if args.json else verdict["label"])
292
+
293
+
294
+ if __name__ == "__main__":
295
+ main()
config.json ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForSequenceClassification"
4
+ ],
5
+ "model_type": "llama",
6
+ "num_labels": 2,
7
+ "id2label": {
8
+ "0": "safe",
9
+ "1": "unsafe"
10
+ },
11
+ "label2id": {
12
+ "safe": 0,
13
+ "unsafe": 1
14
+ },
15
+ "vocab_size": 2048,
16
+ "hidden_size": 128,
17
+ "intermediate_size": 384,
18
+ "num_hidden_layers": 3,
19
+ "num_attention_heads": 4,
20
+ "num_key_value_heads": 4,
21
+ "head_dim": 32,
22
+ "max_position_embeddings": 192,
23
+ "rms_norm_eps": 1e-05,
24
+ "rope_parameters": {
25
+ "rope_type": "default",
26
+ "rope_theta": 10000.0
27
+ },
28
+ "attention_bias": false,
29
+ "mlp_bias": false,
30
+ "tie_word_embeddings": true,
31
+ "pad_token_id": 0,
32
+ "bos_token_id": 1,
33
+ "eos_token_id": 3,
34
+ "name": "Censer.5-e7-soft5long",
35
+ "max_chars": 500,
36
+ "answer_ids": [
37
+ 19,
38
+ 20
39
+ ],
40
+ "special_tokens": [
41
+ "[PAD]",
42
+ "[BOS]",
43
+ "[USER]",
44
+ "[ASSISTANT]"
45
+ ],
46
+ "threshold": 0.5
47
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:34562a38ef5f4e53b5c08331cfe0168e58252d2028b108d95658825014594f6e
3
+ size 3612264
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "tokenizer_class": "PreTrainedTokenizerFast",
3
+ "pad_token": "[PAD]",
4
+ "bos_token": "[BOS]",
5
+ "eos_token": "[ASSISTANT]"
6
+ }