KucLab commited on
Commit
e18fa93
·
verified ·
1 Parent(s): 5d182f7

Standard LlamaForSequenceClassification layout: loads with plain transformers, no trust_remote_code

Browse files
Files changed (7) hide show
  1. README.md +290 -0
  2. censer.2 +30 -0
  3. censer.py +295 -0
  4. config.json +47 -0
  5. model.safetensors +3 -0
  6. tokenizer.json +0 -0
  7. tokenizer_config.json +6 -0
README.md ADDED
@@ -0,0 +1,290 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ pipeline_tag: text-classification
4
+ tags:
5
+ - text-classification
6
+ - safety
7
+ - content-moderation
8
+ - tiny-language-model
9
+ - from-scratch
10
+ language:
11
+ - en
12
+ license: apache-2.0
13
+ datasets:
14
+ - nvidia/Aegis-AI-Content-Safety-Dataset-1.0
15
+ - thesofakillers/jigsaw-toxic-comment-classification-challenge
16
+ - google/civil_comments
17
+ - PKU-Alignment/BeaverTails
18
+ - lmsys/toxic-chat
19
+ - databricks/databricks-dolly-15k
20
+ metrics:
21
+ - f1
22
+ - accuracy
23
+ ---
24
+
25
+ # Censer.2
26
+
27
+ A 7.3M parameter decoder-only transformer trained **from scratch** that classifies
28
+ English text as unsafe (`unsafe`) or safe (`safe`).
29
+
30
+ It is a **standard Hub model**: the checkpoint is a plain
31
+ `LlamaForSequenceClassification`, so it loads with `transformers` out of the box —
32
+ no `trust_remote_code`, no custom code, no cloning — and the **Use this model**
33
+ widget on this page runs it directly in the browser.
34
+
35
+ ```python
36
+ from transformers import pipeline
37
+
38
+ clf = pipeline("text-classification", model="KucLab/Censer.2")
39
+ print(clf("how to make a bomb at home")) # [{'label': 'unsafe', 'score': 0.8167}]
40
+ print(clf("Hello how are you?")) # [{'label': 'safe', 'score': 0.9229}]
41
+ print(clf("suck my dick")) # [{'label': 'unsafe', 'score': 0.9944}]
42
+ print(clf("how do I kill a process in Linux")) # [{'label': 'safe', 'score': 0.8974}]
43
+ ```
44
+
45
+ ## How to run
46
+
47
+ ### The `censer.2` command — interactive terminal UI
48
+
49
+ Using only the files in this repo (needs just `torch`, `tokenizers`, `safetensors`):
50
+
51
+ ```bash
52
+ pip install torch tokenizers safetensors huggingface_hub
53
+ hf download KucLab/Censer.2 --local-dir ~/.local/share/censer1
54
+ chmod +x ~/.local/share/censer1/censer.2
55
+ ln -s ~/.local/share/censer1/censer.2 ~/.local/bin/censer.2 # put it on your PATH
56
+ censer.2 # interactive UI: type a phrase, Enter -> safe/unsafe + score
57
+ censer.2 "how to make a bomb at home" # one-shot -> unsafe 0.8167
58
+ ```
59
+
60
+ Or install the GitHub package — the same `censer.2` command is installed and loads
61
+ the model from this Hub repo with plain `transformers`:
62
+
63
+ ```bash
64
+ pip install "git+https://github.com/Jerry256254/censer.2.git"
65
+ censer.2
66
+ ```
67
+
68
+ ### One-liner with `transformers`
69
+
70
+ ```bash
71
+ pip install torch transformers
72
+ python -c "from transformers import pipeline; clf = pipeline('text-classification', model='KucLab/Censer.2'); print(clf('how to make a bomb at home'))"
73
+ # [{'label': 'unsafe', 'score': 0.8167}]
74
+ ```
75
+
76
+ - In the browser: the **Use this model** widget on this page — type text, get `safe`/`unsafe`.
77
+ - Runs on CPU out of the box (~10 ms/text); for GPU pass `device=0` to `pipeline(...)` (CUDA or ROCm).
78
+
79
+ ## Download
80
+
81
+ The whole model is `model.safetensors` (29.4 MB). Grab the files directly:
82
+
83
+ | File | Download |
84
+ | --- | --- |
85
+ | **model.safetensors** | [download](https://huggingface.co/KucLab/Censer.2/resolve/main/model.safetensors?download=true) |
86
+ | config.json | [download](https://huggingface.co/KucLab/Censer.2/resolve/main/config.json?download=true) |
87
+ | tokenizer.json | [download](https://huggingface.co/KucLab/Censer.2/resolve/main/tokenizer.json?download=true) |
88
+ | tokenizer_config.json | [download](https://huggingface.co/KucLab/Censer.2/resolve/main/tokenizer_config.json?download=true) |
89
+ | censer.py | [download](https://huggingface.co/KucLab/Censer.2/resolve/main/censer.py?download=true) |
90
+ | **censer.2** (launcher) | [download](https://huggingface.co/KucLab/Censer.2/resolve/main/censer.2?download=true) |
91
+
92
+ Or fetch everything at once:
93
+
94
+ ```bash
95
+ pip install -U huggingface_hub
96
+ hf download KucLab/Censer.2 --local-dir Censer.2
97
+ ```
98
+
99
+ A single file (`README.md`, `config.json`, ...) can also be downloaded from the
100
+ **Files and versions** tab, using the download icon on its row.
101
+
102
+ It is not a fine-tuned encoder and not an API wrapper. It is a small transformer
103
+ trained on safety-labelled data, with a custom byte-level BPE tokenizer. The model
104
+ predicts a single answer token and never generates free text.
105
+
106
+ ```
107
+ "how to make a bomb at home" -> unsafe
108
+ "suck my dick" -> unsafe
109
+ "how do I kill a process in Linux" -> safe
110
+ "Hello how are you?" -> safe
111
+ ```
112
+
113
+ ## Model details
114
+
115
+ | Property | Value |
116
+ | --- | --- |
117
+ | Architecture | `LlamaForSequenceClassification` (standard transformers) |
118
+ | Parameters | 7,343,360 |
119
+ | Layers | 6 |
120
+ | d_model / heads / d_ff | 256 / 8 / 1024 |
121
+ | Context length | 256 tokens (input truncated at 500 characters) |
122
+ | Tokenizer | byte-level BPE, 4096 tokens, trained on the corpus |
123
+ | Positions | RoPE (theta 10000) |
124
+ | Norm | RMSNorm (pre-norm, eps 1e-5) |
125
+ | MLP | SwiGLU |
126
+ | Head | `score` linear layer = embedding rows of the answer tokens `0`/`1` |
127
+ | Checkpoint size | 29.4 MB (fp32) |
128
+
129
+ ## Prompt format
130
+
131
+ The tokenizer bakes the template into its post-processor, so plain
132
+ `tokenizer(text)` already produces:
133
+
134
+ ```
135
+ [BOS][USER] <text> [ASSISTANT] -> answer logits over tokens "0"/"1"
136
+ ```
137
+
138
+ Text is truncated at 500 characters before tokenisation.
139
+
140
+ ## Metrics
141
+
142
+ Evaluation on a held-out split of 6,496 rows:
143
+
144
+ | Metric | Value |
145
+ | --- | --- |
146
+ | Accuracy | 0.857 |
147
+ | F1 | 0.824 |
148
+ | Precision | 0.846 |
149
+ | Recall | 0.804 |
150
+
151
+ A hand-written probe set of 62 cases, none present in training data, scores
152
+ **98.4% accuracy** (recall 1.0 — every unsafe probe caught). The previous
153
+ Censer.1 scores 88.7% on the same probes.
154
+
155
+ Per-source F1 reflects label noise in the scraped sources:
156
+
157
+ | Source | F1 |
158
+ | --- | --- |
159
+ | aegis | 0.89 |
160
+ | beavertails | 0.88 |
161
+ | jigsaw | 0.87 |
162
+ | civil_comments | 0.77 |
163
+ | toxic_chat | 0.48 |
164
+
165
+ ## Benchmark
166
+
167
+ Measured on an AMD Radeon RX 9060 XT (ROCm 7.1, torch 2.13.0):
168
+
169
+ | Metric | Value |
170
+ | --- | --- |
171
+ | Parameters | 7,343,360 |
172
+ | Checkpoint size | 4.26 MB |
173
+ | Single text latency | 1.67 ms |
174
+ | Single text throughput | 599 texts/s |
175
+ | Batched latency (batch of 8) | 1.94 ms |
176
+ | Batched throughput | 4,119 texts/s |
177
+ | Peak VRAM (inference) | 85.7 MB |
178
+
179
+ Training the whole model from scratch takes about 3.5 minutes on the same GPU.
180
+
181
+ ## Comparison with other models
182
+
183
+ Benchmarked against public safety classifiers on the **same held-out test split**
184
+ (6,496 rows, threshold 0.5, AMD Radeon RX 9060 XT):
185
+
186
+ | Model | Params | Checkpoint | F1 | Latency | Throughput (b=32) | Peak VRAM |
187
+ | --- | --- | --- | --- | --- | --- | --- |
188
+ | **Censer.2 (ours)** | 7.3M | 28.0 MB | 0.824 | 3.14 ms | 497 texts/s | 177 MB |
189
+ | KucLab/Censer.1 (previous) | 1.1M | 4.5 MB | 0.836 | 1.79 ms | 1,561 texts/s | 101 MB |
190
+ | unitary/unbiased-toxic-roberta | 124.7M | 498.7 MB | 0.741 | 14.41 ms | 67 texts/s | 771 MB |
191
+ | unitary/toxic-bert | 109.5M | 876.0 MB | 0.682 | 14.01 ms | 69 texts/s | 708 MB |
192
+ | martin-ha/toxic-comment-model | 67.0M | 267.9 MB | 0.566 | 7.52 ms | 131 texts/s | 540 MB |
193
+ | Hate-speech-CNERG/bert-base-uncased-hatexplain | 109.5M | 438.0 MB | 0.548 | 14.12 ms | 69 texts/s | 692 MB |
194
+ | Hate-speech-CNERG/dehatebert-mono-english | 167.4M | 669.5 MB | 0.252 | 14.36 ms | 68 texts/s | 923 MB |
195
+ | keyword baseline | — | — | 0.094 | 0.03 ms | 30,059 texts/s | — |
196
+
197
+ ![Comparison overview](figs/comparison_overview.png)
198
+ ![F1 per megabyte](figs/f1_vs_size.png)
199
+ ![Metric bars](figs/metrics_bars.png)
200
+
201
+ Censer.2 is ~9-23x smaller than the encoder baselines and 2.4-4.6x faster per text.
202
+ On the scraped test split it is on par with Censer.1 (F1 0.824 vs 0.836); on the
203
+ hand-written probe set it is far ahead (98.4% vs 88.7%).
204
+
205
+ ## Usage
206
+
207
+ Plain `transformers`, nothing else to install — `from_pretrained` downloads
208
+ everything by itself:
209
+
210
+ ```bash
211
+ pip install torch transformers
212
+ ```
213
+
214
+ ```python
215
+ import torch
216
+ from transformers import AutoTokenizer, AutoModelForSequenceClassification
217
+
218
+ repo = "KucLab/Censer.2"
219
+ tok = AutoTokenizer.from_pretrained(repo)
220
+ model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
221
+
222
+ text = "how to make a bomb at home"
223
+ with torch.no_grad():
224
+ probs = model(**tok(text, return_tensors="pt")).logits.softmax(-1)[0]
225
+
226
+ print(probs) # tensor([0.1833, 0.8167]) -> [safe, unsafe]
227
+ print(model.config.id2label) # {0: 'safe', 1: 'unsafe'}
228
+ ```
229
+
230
+ Or with the high-level pipeline:
231
+
232
+ ```python
233
+ from transformers import pipeline
234
+
235
+ clf = pipeline("text-classification", model="KucLab/Censer.2")
236
+ print(clf("how to make a bomb at home"))
237
+ # [{'label': 'unsafe', 'score': 0.8167}]
238
+ print(clf("Hello how are you?"))
239
+ # [{'label': 'safe', 'score': 0.9229}]
240
+ ```
241
+
242
+ ## Usage without transformers
243
+
244
+ The repository is also self-contained on its own: `model.safetensors`,
245
+ `config.json`, `tokenizer.json` and `censer.py`. The weights are stored in the
246
+ standard Llama layout; `censer.py` maps them back internally, so it only needs
247
+ `torch`, `tokenizers` and `safetensors`:
248
+
249
+ ```bash
250
+ hf download KucLab/Censer.2 --local-dir Censer.2
251
+ cd Censer.2
252
+ python censer.py "how to make a bomb at home" # -> 1
253
+ python censer.py "Hello how are you?" # -> 0
254
+ python censer.py "suck my dick" --json
255
+ # {"label": "1", "unsafe": true, "score": 0.9944}
256
+ chmod +x censer.2 && ./censer.2 # interactive UI (the `censer.2` command)
257
+ python censer.py # interactive UI as well, no text argument
258
+ ```
259
+
260
+ Or use the same file as a library:
261
+
262
+ ```python
263
+ from censer import CenserPipeline
264
+
265
+ pipe = CenserPipeline("Censer.2")
266
+ print(pipe.predict("how to make a bomb at home"))
267
+ # {"label": "1", "unsafe": True, "score": 0.8167}
268
+
269
+ print(pipe.predict_batch(["Hello how are you?", "suck my dick"]))
270
+ ```
271
+
272
+ Files:
273
+
274
+ | File | Purpose |
275
+ | --- | --- |
276
+ | `model.safetensors` | weights in standard Llama layout, 29.4 MB |
277
+ | `config.json` | standard `LlamaForSequenceClassification` config |
278
+ | `tokenizer.json` / `tokenizer_config.json` | byte-level BPE, 2048 tokens, prompt template baked in |
279
+ | `censer.py` | standalone inference (torch + tokenizers only), no repo needed |
280
+ | `censer.2` | executable launcher for the interactive terminal UI |
281
+
282
+ `censer.py` was verified to produce predictions identical to the training code.
283
+ Runs on CPU or CUDA/ROCm; pass `device="cpu"` to force CPU.
284
+
285
+ ## Limitations
286
+
287
+ - **English only.** Other languages mostly score safe.
288
+ - Implication and sarcasm beyond simple cases are out of reach at this size.
289
+ - Weak against jailbreak-style prompts (lowest per-source score).
290
+ - A moderation signal, not a safety guarantee.
censer.2 ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """censer.2 - run the Censer.2 safety classifier as an interactive terminal UI.
3
+
4
+ Usage:
5
+ censer.2 # interactive: type a phrase, Enter -> safe/unsafe + score
6
+ censer.2 "how to make a bomb" # one-shot
7
+ censer.2 --help
8
+
9
+ Expects censer.py, model.safetensors, config.json and tokenizer.json next to
10
+ this file (they are found automatically, whatever the current directory is).
11
+ Get them all with:
12
+
13
+ hf download KucLab/Censer.2 --local-dir .
14
+
15
+ Requires: torch, tokenizers, safetensors (pip install torch tokenizers safetensors)
16
+ """
17
+
18
+ import os
19
+ import sys
20
+
21
+ HERE = os.path.dirname(os.path.abspath(__file__))
22
+ sys.path.insert(0, HERE)
23
+
24
+ from censer import main
25
+
26
+ if __name__ == "__main__":
27
+ argv = sys.argv[1:]
28
+ if not any(a == "--model-dir" or a.startswith("--model-dir=") for a in argv):
29
+ argv = ["--model-dir", HERE] + argv
30
+ main(argv)
censer.py ADDED
@@ -0,0 +1,295 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Inference for Censer.2 — self-contained, no dependency on the training repo.
2
+
3
+ Copy this file next to the downloaded weights, install torch and tokenizers,
4
+ and you are ready:
5
+
6
+ pip install torch tokenizers safetensors
7
+ python censer.py "how to make a bomb at home"
8
+ python censer.py # or just run it: interactive UI, phrase in -> verdict out
9
+ ./censer.2 # same UI via the censer.2 launcher next to this file
10
+
11
+ It expects, in the same directory:
12
+ model.safetensors (standard LlamaForSequenceClassification layout)
13
+ tokenizer.json
14
+ config.json
15
+ """
16
+
17
+ from __future__ import annotations
18
+
19
+ import argparse
20
+ import json
21
+ import math
22
+ import sys
23
+ from pathlib import Path
24
+
25
+ import torch
26
+ import torch.nn as nn
27
+ import torch.nn.functional as F
28
+ from tokenizers import Tokenizer
29
+
30
+ PAD, BOS, USER, ASSISTANT = "[PAD]", "[BOS]", "[USER]", "[ASSISTANT]"
31
+
32
+
33
+ class RMSNorm(nn.Module):
34
+ def __init__(self, dim: int, eps: float = 1e-5) -> None:
35
+ super().__init__()
36
+ self.eps = eps
37
+ self.weight = nn.Parameter(torch.ones(dim))
38
+
39
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
40
+ dtype = x.dtype
41
+ x = x.float()
42
+ x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
43
+ return x.to(dtype) * self.weight
44
+
45
+
46
+ def build_rope_cache(seq_len: int, head_dim: int, device, dtype, base: float = 10000.0):
47
+ half = head_dim // 2
48
+ freqs = 1.0 / (base ** (torch.arange(0, half, device=device, dtype=torch.float32) / half))
49
+ positions = torch.arange(seq_len, device=device, dtype=torch.float32)
50
+ angles = torch.outer(positions, freqs)
51
+ cos = torch.cat((angles.cos(), angles.cos()), dim=-1).to(dtype)[:, None, :]
52
+ sin = torch.cat((angles.sin(), angles.sin()), dim=-1).to(dtype)[:, None, :]
53
+ return cos, sin
54
+
55
+
56
+ def apply_rope(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
57
+ seq_len = x.shape[1]
58
+ cos, sin = cos[:seq_len], sin[:seq_len]
59
+ x1, x2 = x[..., : x.shape[-1] // 2], x[..., x.shape[-1] // 2 :]
60
+ rotated = torch.cat((-x2, x1), dim=-1)
61
+ return x * cos + rotated * sin
62
+
63
+
64
+ class Attention(nn.Module):
65
+ def __init__(self, cfg: dict) -> None:
66
+ super().__init__()
67
+ self.n_heads = cfg["n_heads"]
68
+ self.head_dim = cfg["d_model"] // cfg["n_heads"]
69
+ self.dropout = cfg["dropout"]
70
+ self.qkv = nn.Linear(cfg["d_model"], 3 * cfg["d_model"], bias=False)
71
+ self.proj = nn.Linear(cfg["d_model"], cfg["d_model"], bias=False)
72
+
73
+ def forward(self, x, cos, sin, mask):
74
+ b, t, _ = x.shape
75
+ q, k, v = self.qkv(x).split(x.shape[-1], dim=-1)
76
+ q = q.view(b, t, self.n_heads, self.head_dim)
77
+ k = k.view(b, t, self.n_heads, self.head_dim)
78
+ v = v.view(b, t, self.n_heads, self.head_dim)
79
+ q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
80
+ out = F.scaled_dot_product_attention(
81
+ q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2),
82
+ attn_mask=mask,
83
+ dropout_p=self.dropout if self.training else 0.0,
84
+ is_causal=mask is None,
85
+ )
86
+ return self.proj(out.transpose(1, 2).reshape(b, t, -1))
87
+
88
+
89
+ class SwiGLU(nn.Module):
90
+ def __init__(self, cfg: dict) -> None:
91
+ super().__init__()
92
+ self.gate = nn.Linear(cfg["d_model"], cfg["d_ff"], bias=False)
93
+ self.up = nn.Linear(cfg["d_model"], cfg["d_ff"], bias=False)
94
+ self.down = nn.Linear(cfg["d_ff"], cfg["d_model"], bias=False)
95
+
96
+ def forward(self, x):
97
+ return self.down(F.silu(self.gate(x)) * self.up(x))
98
+
99
+
100
+ class Block(nn.Module):
101
+ def __init__(self, cfg: dict) -> None:
102
+ super().__init__()
103
+ self.norm1 = RMSNorm(cfg["d_model"])
104
+ self.attn = Attention(cfg)
105
+ self.norm2 = RMSNorm(cfg["d_model"])
106
+ self.mlp = SwiGLU(cfg)
107
+ self.dropout = nn.Dropout(cfg["dropout"])
108
+
109
+ def forward(self, x, cos, sin, mask):
110
+ x = x + self.dropout(self.attn(self.norm1(x), cos, sin, mask))
111
+ x = x + self.dropout(self.mlp(self.norm2(x)))
112
+ return x
113
+
114
+
115
+ class Censer(nn.Module):
116
+ def __init__(self, cfg: dict) -> None:
117
+ super().__init__()
118
+ self.cfg = cfg
119
+ self.register_buffer(
120
+ "answer_id_tensor", torch.tensor(cfg["answer_ids"], dtype=torch.long), persistent=False
121
+ )
122
+ self.embed = nn.Embedding(cfg["vocab_size"], cfg["d_model"], padding_idx=cfg["pad_id"])
123
+ self.blocks = nn.ModuleList([Block(cfg) for _ in range(cfg["n_layers"])])
124
+ self.norm = RMSNorm(cfg["d_model"])
125
+ self.lm_head = nn.Linear(cfg["d_model"], cfg["vocab_size"], bias=False)
126
+ if cfg["tie_embeddings"]:
127
+ self.lm_head.weight = self.embed.weight
128
+ self._rope_cache = None
129
+
130
+ def _rope(self, device, dtype):
131
+ key = f"{device}|{dtype}"
132
+ if self._rope_cache is None or self._rope_cache[2] != key:
133
+ head_dim = self.cfg["d_model"] // self.cfg["n_heads"]
134
+ cos, sin = build_rope_cache(self.cfg["max_seq_len"], head_dim, device, dtype)
135
+ self._rope_cache = (cos, sin, key)
136
+ return self._rope_cache[0], self._rope_cache[1]
137
+
138
+ def _attention_mask(self, input_ids):
139
+ b, t = input_ids.shape
140
+ pad = input_ids != self.cfg["pad_id"]
141
+ causal = torch.ones(t, t, dtype=torch.bool, device=input_ids.device).tril()
142
+ return pad[:, None, None, :] & causal[None, None, :, :]
143
+
144
+ def forward(self, input_ids):
145
+ cos, sin = self._rope(input_ids.device, self.embed.weight.dtype)
146
+ mask = self._attention_mask(input_ids)
147
+ x = self.embed(input_ids)
148
+ for block in self.blocks:
149
+ x = block(x, cos, sin, mask)
150
+ return self.lm_head(self.norm(x))
151
+
152
+ @torch.no_grad()
153
+ def classify(self, input_ids):
154
+ hidden = self.forward(input_ids)
155
+ non_pad = (input_ids != self.cfg["pad_id"]).sum(dim=1).clamp(min=1)
156
+ idx = (non_pad - 1)[:, None, None].expand(-1, 1, hidden.shape[-1])
157
+ logits = hidden.gather(1, idx)[:, 0, :].float()
158
+ logits = logits.index_select(-1, self.answer_id_tensor)
159
+ return logits.softmax(-1)[:, 1]
160
+
161
+
162
+ def remap_hf_state(state: dict) -> dict:
163
+ """Standard Llama-layout safetensors -> training-layout state dict."""
164
+ import re
165
+
166
+ out = {"embed.weight": state["model.embed_tokens.weight"]}
167
+ layer_ids = sorted({int(m.group(1)) for k in state if (m := re.match(r"model\.layers\.(\d+)\.", k))})
168
+ for i in layer_ids:
169
+ p, b = f"model.layers.{i}.", f"blocks.{i}."
170
+ qkv = torch.cat(
171
+ [state[p + "self_attn.q_proj.weight"], state[p + "self_attn.k_proj.weight"], state[p + "self_attn.v_proj.weight"]],
172
+ dim=0,
173
+ )
174
+ out[b + "attn.qkv.weight"] = qkv.contiguous()
175
+ out[b + "attn.proj.weight"] = state[p + "self_attn.o_proj.weight"]
176
+ out[b + "mlp.gate.weight"] = state[p + "mlp.gate_proj.weight"]
177
+ out[b + "mlp.up.weight"] = state[p + "mlp.up_proj.weight"]
178
+ out[b + "mlp.down.weight"] = state[p + "mlp.down_proj.weight"]
179
+ out[b + "norm1.weight"] = state[p + "input_layernorm.weight"]
180
+ out[b + "norm2.weight"] = state[p + "post_attention_layernorm.weight"]
181
+ out["norm.weight"] = state["model.norm.weight"]
182
+ return out
183
+
184
+
185
+ def internal_cfg(cfg: dict) -> dict:
186
+ """config.json (Llama layout) -> the dict the Censer module expects."""
187
+ return {
188
+ "vocab_size": cfg["vocab_size"],
189
+ "n_layers": cfg["num_hidden_layers"],
190
+ "n_heads": cfg["num_attention_heads"],
191
+ "d_model": cfg["hidden_size"],
192
+ "d_ff": cfg["intermediate_size"],
193
+ "max_seq_len": cfg["max_position_embeddings"],
194
+ "dropout": cfg.get("dropout", 0.0),
195
+ "tie_embeddings": True,
196
+ "pad_id": cfg.get("pad_token_id", 0),
197
+ "answer_ids": cfg["answer_ids"],
198
+ "max_chars": cfg.get("max_chars", 500),
199
+ }
200
+
201
+
202
+ class CenserPipeline:
203
+ """Text in, "1" (unsafe) or "0" (safe) out."""
204
+
205
+ def __init__(self, directory: str | Path = ".", threshold: float = 0.5, device: str | None = None) -> None:
206
+ from safetensors.torch import load_file
207
+
208
+ directory = Path(directory)
209
+ self.cfg = internal_cfg(json.loads((directory / "config.json").read_text(encoding="utf-8")))
210
+ self.tok = Tokenizer.from_file(str(directory / "tokenizer.json"))
211
+ self.pad_id = self.tok.token_to_id(PAD)
212
+ self.bos_id = self.tok.token_to_id(BOS)
213
+ self.user_id = self.tok.token_to_id(USER)
214
+ self.assistant_id = self.tok.token_to_id(ASSISTANT)
215
+ self.threshold = threshold
216
+ self.device = torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
217
+ self.model = Censer(self.cfg)
218
+ state = remap_hf_state(load_file(directory / "model.safetensors"))
219
+ missing, unexpected = self.model.load_state_dict(state, strict=False)
220
+ missing = [k for k in missing if k != "lm_head.weight"] # tied to embed.weight
221
+ if missing or unexpected:
222
+ raise RuntimeError(f"weight mismatch: missing={missing} unexpected={unexpected}")
223
+ self.model.to(self.device).eval()
224
+ self.max_chars = self.cfg["max_chars"]
225
+
226
+ def _encode(self, text: str) -> list[int]:
227
+ body = self.tok.encode(text[: self.max_chars], add_special_tokens=False).ids
228
+ ids = [self.bos_id, self.user_id, *body, self.assistant_id]
229
+ return ids[: self.cfg["max_seq_len"]]
230
+
231
+ def predict(self, text: str) -> dict:
232
+ return self.predict_batch([text])[0]
233
+
234
+ def predict_batch(self, texts: list[str]) -> list[dict]:
235
+ seqs = [self._encode(t) for t in texts]
236
+ width = max(len(s) for s in seqs)
237
+ ids = torch.full((len(seqs), width), self.pad_id, dtype=torch.long)
238
+ for i, seq in enumerate(seqs):
239
+ ids[i, : len(seq)] = torch.tensor(seq)
240
+ probs = self.model.classify(ids.to(self.device)).float().cpu().tolist()
241
+ return [
242
+ {"label": "1" if p >= self.threshold else "0", "unsafe": p >= self.threshold, "score": round(p, 4)}
243
+ for p in probs
244
+ ]
245
+
246
+
247
+ def _color(text: str, code: str) -> str:
248
+ if not sys.stdout.isatty():
249
+ return text
250
+ return f"\033[{code}m{text}\033[0m"
251
+
252
+
253
+ def repl(pipe: "CenserPipeline") -> int:
254
+ """Small interactive terminal UI: phrase in, safe/unsafe + score out."""
255
+ print("Censer.2 - interactive terminal UI")
256
+ print("Type a phrase, Enter to classify. 'exit' to quit.\n")
257
+ while True:
258
+ try:
259
+ line = input("> ")
260
+ except (EOFError, KeyboardInterrupt):
261
+ print()
262
+ return 0
263
+ text = line.strip()
264
+ if text.lower() in {"exit", "quit", ":q"}:
265
+ return 0
266
+ if not text:
267
+ continue
268
+ v = pipe.predict(text)
269
+ label = "unsafe" if v["unsafe"] else "safe"
270
+ colored = _color(f"{label:6s}", "31" if v["unsafe"] else "32")
271
+ print(f"{colored} {v['score']:.4f} | {text}")
272
+
273
+
274
+ def main(argv: list[str] | None = None) -> None:
275
+ parser = argparse.ArgumentParser(
276
+ prog="censer.2",
277
+ description="Censer.2 safety classifier (standalone). Run without text to open the interactive UI.",
278
+ )
279
+ parser.add_argument("text", nargs="*", help="phrase to classify once; omit to open the interactive UI")
280
+ parser.add_argument("--model-dir", default=".")
281
+ parser.add_argument("--threshold", type=float, default=0.5)
282
+ parser.add_argument("--json", action="store_true")
283
+ parser.add_argument("--repl", action="store_true", help="interactive UI (default when no text is given)")
284
+ args = parser.parse_args(argv)
285
+
286
+ pipe = CenserPipeline(args.model_dir, args.threshold)
287
+ text = " ".join(args.text).strip()
288
+ if args.repl or not text:
289
+ raise SystemExit(repl(pipe))
290
+ verdict = pipe.predict(text)
291
+ print(json.dumps(verdict) if args.json else verdict["label"])
292
+
293
+
294
+ if __name__ == "__main__":
295
+ main()
config.json ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForSequenceClassification"
4
+ ],
5
+ "model_type": "llama",
6
+ "num_labels": 2,
7
+ "id2label": {
8
+ "0": "safe",
9
+ "1": "unsafe"
10
+ },
11
+ "label2id": {
12
+ "safe": 0,
13
+ "unsafe": 1
14
+ },
15
+ "vocab_size": 4096,
16
+ "hidden_size": 256,
17
+ "intermediate_size": 1024,
18
+ "num_hidden_layers": 6,
19
+ "num_attention_heads": 8,
20
+ "num_key_value_heads": 8,
21
+ "head_dim": 32,
22
+ "max_position_embeddings": 256,
23
+ "rms_norm_eps": 1e-05,
24
+ "rope_parameters": {
25
+ "rope_type": "default",
26
+ "rope_theta": 10000.0
27
+ },
28
+ "attention_bias": false,
29
+ "mlp_bias": false,
30
+ "tie_word_embeddings": true,
31
+ "pad_token_id": 0,
32
+ "bos_token_id": 1,
33
+ "eos_token_id": 3,
34
+ "name": "Censer.2",
35
+ "max_chars": 500,
36
+ "answer_ids": [
37
+ 19,
38
+ 20
39
+ ],
40
+ "special_tokens": [
41
+ "[PAD]",
42
+ "[BOS]",
43
+ "[USER]",
44
+ "[ASSISTANT]"
45
+ ],
46
+ "threshold": 0.5
47
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0f21ea032eb9c6a6fcbbeeb4cf7011a9664b69e343fdafcbffcf5640e026dd8d
3
+ size 29381648
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "tokenizer_class": "PreTrainedTokenizerFast",
3
+ "pad_token": "[PAD]",
4
+ "bos_token": "[BOS]",
5
+ "eos_token": "[ASSISTANT]"
6
+ }