Text Classification
Transformers
Safetensors
English
llama
safety
content-moderation
tiny-language-model
from-scratch
text-embeddings-inference
Instructions to use KucLab/Censer.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KucLab/Censer.1 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="KucLab/Censer.1")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("KucLab/Censer.1") model = AutoModelForSequenceClassification.from_pretrained("KucLab/Censer.1", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Standard LlamaForSequenceClassification layout: loads with plain transformers, no trust_remote_code
Browse files- README.md +44 -27
- censer.py +45 -7
- config.json +29 -23
- model.safetensors +2 -2
- modeling_censer.py +0 -192
- tokenizer_config.json +6 -0
README.md
CHANGED
|
@@ -7,7 +7,6 @@ tags:
|
|
| 7 |
- content-moderation
|
| 8 |
- tiny-language-model
|
| 9 |
- from-scratch
|
| 10 |
-
- custom-code
|
| 11 |
language:
|
| 12 |
- en
|
| 13 |
license: apache-2.0
|
|
@@ -26,7 +25,22 @@ metrics:
|
|
| 26 |
# Censer.1
|
| 27 |
|
| 28 |
A 1.1M parameter decoder-only transformer trained **from scratch** that classifies
|
| 29 |
-
English text as unsafe (`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
## Download
|
| 32 |
|
|
@@ -37,6 +51,7 @@ The whole model is `model.safetensors` (4.5 MB). Grab the files directly:
|
|
| 37 |
| **model.safetensors** | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/model.safetensors?download=true) |
|
| 38 |
| config.json | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/config.json?download=true) |
|
| 39 |
| tokenizer.json | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/tokenizer.json?download=true) |
|
|
|
|
| 40 |
| censer.py | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/censer.py?download=true) |
|
| 41 |
|
| 42 |
Or fetch everything at once:
|
|
@@ -54,32 +69,35 @@ trained on safety-labelled data, with a custom byte-level BPE tokenizer. The mod
|
|
| 54 |
predicts a single answer token and never generates free text.
|
| 55 |
|
| 56 |
```
|
| 57 |
-
"
|
| 58 |
-
"
|
| 59 |
-
"
|
| 60 |
-
"how
|
| 61 |
-
"Hello how are you?" -> 0
|
| 62 |
```
|
| 63 |
|
| 64 |
## Model details
|
| 65 |
|
| 66 |
| Property | Value |
|
| 67 |
| --- | --- |
|
|
|
|
| 68 |
| Parameters | 1,115,264 |
|
| 69 |
| Layers | 4 |
|
| 70 |
| d_model / heads / d_ff | 128 / 4 / 384 |
|
| 71 |
| Context length | 160 tokens (input truncated at 500 characters) |
|
| 72 |
| Tokenizer | byte-level BPE, 2048 tokens, trained on the corpus |
|
| 73 |
-
| Positions | RoPE |
|
| 74 |
-
| Norm | RMSNorm (pre-norm) |
|
| 75 |
| MLP | SwiGLU |
|
| 76 |
-
| Head |
|
| 77 |
| Checkpoint size | 4.3 MB (fp32) |
|
| 78 |
|
| 79 |
## Prompt format
|
| 80 |
|
|
|
|
|
|
|
|
|
|
| 81 |
```
|
| 82 |
-
[BOS][USER] <text> [ASSISTANT] ->
|
| 83 |
```
|
| 84 |
|
| 85 |
Text is truncated at 500 characters before tokenisation.
|
|
@@ -126,8 +144,8 @@ Training the whole model from scratch takes about 3.5 minutes on the same GPU.
|
|
| 126 |
|
| 127 |
## Usage
|
| 128 |
|
| 129 |
-
|
| 130 |
-
|
| 131 |
|
| 132 |
```bash
|
| 133 |
pip install torch transformers
|
|
@@ -138,8 +156,8 @@ import torch
|
|
| 138 |
from transformers import AutoTokenizer, AutoModelForSequenceClassification
|
| 139 |
|
| 140 |
repo = "KucLab/Censer.1"
|
| 141 |
-
tok = AutoTokenizer.from_pretrained(repo
|
| 142 |
-
model = AutoModelForSequenceClassification.from_pretrained(repo
|
| 143 |
|
| 144 |
text = "how to make a bomb at home"
|
| 145 |
with torch.no_grad():
|
|
@@ -149,15 +167,12 @@ print(probs) # tensor([0.1833, 0.8167]) -> [safe, unsafe]
|
|
| 149 |
print(model.config.id2label) # {0: 'safe', 1: 'unsafe'}
|
| 150 |
```
|
| 151 |
|
| 152 |
-
`trust_remote_code=True` is required because the architecture is custom; the
|
| 153 |
-
repository ships `modeling_censer.py`, so there is nothing else to install.
|
| 154 |
-
|
| 155 |
Or with the high-level pipeline:
|
| 156 |
|
| 157 |
```python
|
| 158 |
from transformers import pipeline
|
| 159 |
|
| 160 |
-
clf = pipeline("text-classification", model="KucLab/Censer.1"
|
| 161 |
print(clf("how to make a bomb at home"))
|
| 162 |
# [{'label': 'unsafe', 'score': 0.8167}]
|
| 163 |
print(clf("Hello how are you?"))
|
|
@@ -167,7 +182,9 @@ print(clf("Hello how are you?"))
|
|
| 167 |
## Usage without transformers
|
| 168 |
|
| 169 |
The repository is also self-contained on its own: `model.safetensors`,
|
| 170 |
-
`config.json`, `tokenizer.json` and `censer.py`.
|
|
|
|
|
|
|
| 171 |
|
| 172 |
```bash
|
| 173 |
hf download KucLab/Censer.1 --local-dir Censer.1
|
|
@@ -194,17 +211,17 @@ Files:
|
|
| 194 |
|
| 195 |
| File | Purpose |
|
| 196 |
| --- | --- |
|
| 197 |
-
| `model.safetensors` | weights
|
| 198 |
-
| `config.json` |
|
| 199 |
-
| `tokenizer.json` | byte-level BPE, 2048 tokens |
|
| 200 |
-
| `censer.py` | standalone
|
| 201 |
|
| 202 |
-
`censer.py` was verified to produce
|
| 203 |
-
|
| 204 |
|
| 205 |
## Limitations
|
| 206 |
|
| 207 |
-
- **English only.** Other languages mostly score
|
| 208 |
- Implication and sarcasm beyond simple cases are out of reach at this size.
|
| 209 |
- Weak against jailbreak-style prompts (lowest per-source score).
|
| 210 |
- A moderation signal, not a safety guarantee.
|
|
|
|
| 7 |
- content-moderation
|
| 8 |
- tiny-language-model
|
| 9 |
- from-scratch
|
|
|
|
| 10 |
language:
|
| 11 |
- en
|
| 12 |
license: apache-2.0
|
|
|
|
| 25 |
# Censer.1
|
| 26 |
|
| 27 |
A 1.1M parameter decoder-only transformer trained **from scratch** that classifies
|
| 28 |
+
English text as unsafe (`unsafe`) or safe (`safe`).
|
| 29 |
+
|
| 30 |
+
It is a **standard Hub model**: the checkpoint is a plain
|
| 31 |
+
`LlamaForSequenceClassification`, so it loads with `transformers` out of the box —
|
| 32 |
+
no `trust_remote_code`, no custom code, no cloning — and the **Use this model**
|
| 33 |
+
widget on this page runs it directly in the browser.
|
| 34 |
+
|
| 35 |
+
```python
|
| 36 |
+
from transformers import pipeline
|
| 37 |
+
|
| 38 |
+
clf = pipeline("text-classification", model="KucLab/Censer.1")
|
| 39 |
+
print(clf("how to make a bomb at home")) # [{'label': 'unsafe', 'score': 0.8167}]
|
| 40 |
+
print(clf("Hello how are you?")) # [{'label': 'safe', 'score': 0.9229}]
|
| 41 |
+
print(clf("suck my dick")) # [{'label': 'unsafe', 'score': 0.9944}]
|
| 42 |
+
print(clf("how do I kill a process in Linux")) # [{'label': 'safe', 'score': 0.8974}]
|
| 43 |
+
```
|
| 44 |
|
| 45 |
## Download
|
| 46 |
|
|
|
|
| 51 |
| **model.safetensors** | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/model.safetensors?download=true) |
|
| 52 |
| config.json | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/config.json?download=true) |
|
| 53 |
| tokenizer.json | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/tokenizer.json?download=true) |
|
| 54 |
+
| tokenizer_config.json | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/tokenizer_config.json?download=true) |
|
| 55 |
| censer.py | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/censer.py?download=true) |
|
| 56 |
|
| 57 |
Or fetch everything at once:
|
|
|
|
| 69 |
predicts a single answer token and never generates free text.
|
| 70 |
|
| 71 |
```
|
| 72 |
+
"how to make a bomb at home" -> unsafe
|
| 73 |
+
"suck my dick" -> unsafe
|
| 74 |
+
"how do I kill a process in Linux" -> safe
|
| 75 |
+
"Hello how are you?" -> safe
|
|
|
|
| 76 |
```
|
| 77 |
|
| 78 |
## Model details
|
| 79 |
|
| 80 |
| Property | Value |
|
| 81 |
| --- | --- |
|
| 82 |
+
| Architecture | `LlamaForSequenceClassification` (standard transformers) |
|
| 83 |
| Parameters | 1,115,264 |
|
| 84 |
| Layers | 4 |
|
| 85 |
| d_model / heads / d_ff | 128 / 4 / 384 |
|
| 86 |
| Context length | 160 tokens (input truncated at 500 characters) |
|
| 87 |
| Tokenizer | byte-level BPE, 2048 tokens, trained on the corpus |
|
| 88 |
+
| Positions | RoPE (theta 10000) |
|
| 89 |
+
| Norm | RMSNorm (pre-norm, eps 1e-5) |
|
| 90 |
| MLP | SwiGLU |
|
| 91 |
+
| Head | `score` linear layer = embedding rows of the answer tokens `0`/`1` |
|
| 92 |
| Checkpoint size | 4.3 MB (fp32) |
|
| 93 |
|
| 94 |
## Prompt format
|
| 95 |
|
| 96 |
+
The tokenizer bakes the template into its post-processor, so plain
|
| 97 |
+
`tokenizer(text)` already produces:
|
| 98 |
+
|
| 99 |
```
|
| 100 |
+
[BOS][USER] <text> [ASSISTANT] -> answer logits over tokens "0"/"1"
|
| 101 |
```
|
| 102 |
|
| 103 |
Text is truncated at 500 characters before tokenisation.
|
|
|
|
| 144 |
|
| 145 |
## Usage
|
| 146 |
|
| 147 |
+
Plain `transformers`, nothing else to install — `from_pretrained` downloads
|
| 148 |
+
everything by itself:
|
| 149 |
|
| 150 |
```bash
|
| 151 |
pip install torch transformers
|
|
|
|
| 156 |
from transformers import AutoTokenizer, AutoModelForSequenceClassification
|
| 157 |
|
| 158 |
repo = "KucLab/Censer.1"
|
| 159 |
+
tok = AutoTokenizer.from_pretrained(repo)
|
| 160 |
+
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
|
| 161 |
|
| 162 |
text = "how to make a bomb at home"
|
| 163 |
with torch.no_grad():
|
|
|
|
| 167 |
print(model.config.id2label) # {0: 'safe', 1: 'unsafe'}
|
| 168 |
```
|
| 169 |
|
|
|
|
|
|
|
|
|
|
| 170 |
Or with the high-level pipeline:
|
| 171 |
|
| 172 |
```python
|
| 173 |
from transformers import pipeline
|
| 174 |
|
| 175 |
+
clf = pipeline("text-classification", model="KucLab/Censer.1")
|
| 176 |
print(clf("how to make a bomb at home"))
|
| 177 |
# [{'label': 'unsafe', 'score': 0.8167}]
|
| 178 |
print(clf("Hello how are you?"))
|
|
|
|
| 182 |
## Usage without transformers
|
| 183 |
|
| 184 |
The repository is also self-contained on its own: `model.safetensors`,
|
| 185 |
+
`config.json`, `tokenizer.json` and `censer.py`. The weights are stored in the
|
| 186 |
+
standard Llama layout; `censer.py` maps them back internally, so it only needs
|
| 187 |
+
`torch`, `tokenizers` and `safetensors`:
|
| 188 |
|
| 189 |
```bash
|
| 190 |
hf download KucLab/Censer.1 --local-dir Censer.1
|
|
|
|
| 211 |
|
| 212 |
| File | Purpose |
|
| 213 |
| --- | --- |
|
| 214 |
+
| `model.safetensors` | weights in standard Llama layout, 4.3 MB |
|
| 215 |
+
| `config.json` | standard `LlamaForSequenceClassification` config |
|
| 216 |
+
| `tokenizer.json` / `tokenizer_config.json` | byte-level BPE, 2048 tokens, prompt template baked in |
|
| 217 |
+
| `censer.py` | standalone inference (torch + tokenizers only), no repo needed |
|
| 218 |
|
| 219 |
+
`censer.py` was verified to produce predictions identical to the training code.
|
| 220 |
+
Runs on CPU or CUDA/ROCm; pass `device="cpu"` to force CPU.
|
| 221 |
|
| 222 |
## Limitations
|
| 223 |
|
| 224 |
+
- **English only.** Other languages mostly score safe.
|
| 225 |
- Implication and sarcasm beyond simple cases are out of reach at this size.
|
| 226 |
- Weak against jailbreak-style prompts (lowest per-source score).
|
| 227 |
- A moderation signal, not a safety guarantee.
|
censer.py
CHANGED
|
@@ -3,11 +3,11 @@
|
|
| 3 |
Copy this file next to the downloaded weights, install torch and tokenizers,
|
| 4 |
and you are ready:
|
| 5 |
|
| 6 |
-
pip install torch tokenizers
|
| 7 |
python censer.py "how to make a bomb at home"
|
| 8 |
|
| 9 |
It expects, in the same directory:
|
| 10 |
-
model.safetensors
|
| 11 |
tokenizer.json
|
| 12 |
config.json
|
| 13 |
"""
|
|
@@ -155,6 +155,46 @@ class Censer(nn.Module):
|
|
| 155 |
return logits.softmax(-1)[:, 1]
|
| 156 |
|
| 157 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 158 |
class CenserPipeline:
|
| 159 |
"""Text in, "1" (unsafe) or "0" (safe) out."""
|
| 160 |
|
|
@@ -162,7 +202,7 @@ class CenserPipeline:
|
|
| 162 |
from safetensors.torch import load_file
|
| 163 |
|
| 164 |
directory = Path(directory)
|
| 165 |
-
self.cfg = json.loads((directory / "config.json").read_text(encoding="utf-8"))
|
| 166 |
self.tok = Tokenizer.from_file(str(directory / "tokenizer.json"))
|
| 167 |
self.pad_id = self.tok.token_to_id(PAD)
|
| 168 |
self.bos_id = self.tok.token_to_id(BOS)
|
|
@@ -171,11 +211,9 @@ class CenserPipeline:
|
|
| 171 |
self.threshold = threshold
|
| 172 |
self.device = torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
|
| 173 |
self.model = Censer(self.cfg)
|
| 174 |
-
state = load_file(directory / "model.safetensors")
|
| 175 |
-
if self.cfg["tie_embeddings"]:
|
| 176 |
-
state.pop("lm_head.weight", None)
|
| 177 |
missing, unexpected = self.model.load_state_dict(state, strict=False)
|
| 178 |
-
missing = [k for k in missing if
|
| 179 |
if missing or unexpected:
|
| 180 |
raise RuntimeError(f"weight mismatch: missing={missing} unexpected={unexpected}")
|
| 181 |
self.model.to(self.device).eval()
|
|
|
|
| 3 |
Copy this file next to the downloaded weights, install torch and tokenizers,
|
| 4 |
and you are ready:
|
| 5 |
|
| 6 |
+
pip install torch tokenizers safetensors
|
| 7 |
python censer.py "how to make a bomb at home"
|
| 8 |
|
| 9 |
It expects, in the same directory:
|
| 10 |
+
model.safetensors (standard LlamaForSequenceClassification layout)
|
| 11 |
tokenizer.json
|
| 12 |
config.json
|
| 13 |
"""
|
|
|
|
| 155 |
return logits.softmax(-1)[:, 1]
|
| 156 |
|
| 157 |
|
| 158 |
+
def remap_hf_state(state: dict) -> dict:
|
| 159 |
+
"""Standard Llama-layout safetensors -> training-layout state dict."""
|
| 160 |
+
import re
|
| 161 |
+
|
| 162 |
+
out = {"embed.weight": state["model.embed_tokens.weight"]}
|
| 163 |
+
layer_ids = sorted({int(m.group(1)) for k in state if (m := re.match(r"model\.layers\.(\d+)\.", k))})
|
| 164 |
+
for i in layer_ids:
|
| 165 |
+
p, b = f"model.layers.{i}.", f"blocks.{i}."
|
| 166 |
+
qkv = torch.cat(
|
| 167 |
+
[state[p + "self_attn.q_proj.weight"], state[p + "self_attn.k_proj.weight"], state[p + "self_attn.v_proj.weight"]],
|
| 168 |
+
dim=0,
|
| 169 |
+
)
|
| 170 |
+
out[b + "attn.qkv.weight"] = qkv.contiguous()
|
| 171 |
+
out[b + "attn.proj.weight"] = state[p + "self_attn.o_proj.weight"]
|
| 172 |
+
out[b + "mlp.gate.weight"] = state[p + "mlp.gate_proj.weight"]
|
| 173 |
+
out[b + "mlp.up.weight"] = state[p + "mlp.up_proj.weight"]
|
| 174 |
+
out[b + "mlp.down.weight"] = state[p + "mlp.down_proj.weight"]
|
| 175 |
+
out[b + "norm1.weight"] = state[p + "input_layernorm.weight"]
|
| 176 |
+
out[b + "norm2.weight"] = state[p + "post_attention_layernorm.weight"]
|
| 177 |
+
out["norm.weight"] = state["model.norm.weight"]
|
| 178 |
+
return out
|
| 179 |
+
|
| 180 |
+
|
| 181 |
+
def internal_cfg(cfg: dict) -> dict:
|
| 182 |
+
"""config.json (Llama layout) -> the dict the Censer module expects."""
|
| 183 |
+
return {
|
| 184 |
+
"vocab_size": cfg["vocab_size"],
|
| 185 |
+
"n_layers": cfg["num_hidden_layers"],
|
| 186 |
+
"n_heads": cfg["num_attention_heads"],
|
| 187 |
+
"d_model": cfg["hidden_size"],
|
| 188 |
+
"d_ff": cfg["intermediate_size"],
|
| 189 |
+
"max_seq_len": cfg["max_position_embeddings"],
|
| 190 |
+
"dropout": cfg.get("dropout", 0.0),
|
| 191 |
+
"tie_embeddings": True,
|
| 192 |
+
"pad_id": cfg.get("pad_token_id", 0),
|
| 193 |
+
"answer_ids": cfg["answer_ids"],
|
| 194 |
+
"max_chars": cfg.get("max_chars", 500),
|
| 195 |
+
}
|
| 196 |
+
|
| 197 |
+
|
| 198 |
class CenserPipeline:
|
| 199 |
"""Text in, "1" (unsafe) or "0" (safe) out."""
|
| 200 |
|
|
|
|
| 202 |
from safetensors.torch import load_file
|
| 203 |
|
| 204 |
directory = Path(directory)
|
| 205 |
+
self.cfg = internal_cfg(json.loads((directory / "config.json").read_text(encoding="utf-8")))
|
| 206 |
self.tok = Tokenizer.from_file(str(directory / "tokenizer.json"))
|
| 207 |
self.pad_id = self.tok.token_to_id(PAD)
|
| 208 |
self.bos_id = self.tok.token_to_id(BOS)
|
|
|
|
| 211 |
self.threshold = threshold
|
| 212 |
self.device = torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
|
| 213 |
self.model = Censer(self.cfg)
|
| 214 |
+
state = remap_hf_state(load_file(directory / "model.safetensors"))
|
|
|
|
|
|
|
| 215 |
missing, unexpected = self.model.load_state_dict(state, strict=False)
|
| 216 |
+
missing = [k for k in missing if k != "lm_head.weight"] # tied to embed.weight
|
| 217 |
if missing or unexpected:
|
| 218 |
raise RuntimeError(f"weight mismatch: missing={missing} unexpected={unexpected}")
|
| 219 |
self.model.to(self.device).eval()
|
config.json
CHANGED
|
@@ -1,22 +1,37 @@
|
|
| 1 |
{
|
| 2 |
"architectures": [
|
| 3 |
-
"
|
| 4 |
],
|
| 5 |
-
"model_type": "
|
| 6 |
-
"
|
| 7 |
-
|
| 8 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
},
|
| 10 |
-
"name": "Censer.1",
|
| 11 |
"vocab_size": 2048,
|
| 12 |
-
"
|
| 13 |
-
"
|
| 14 |
-
"
|
| 15 |
-
"
|
| 16 |
-
"
|
| 17 |
-
"
|
| 18 |
-
"
|
| 19 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
"max_chars": 500,
|
| 21 |
"answer_ids": [
|
| 22 |
19,
|
|
@@ -28,14 +43,5 @@
|
|
| 28 |
"[USER]",
|
| 29 |
"[ASSISTANT]"
|
| 30 |
],
|
| 31 |
-
"num_labels": 2,
|
| 32 |
-
"id2label": {
|
| 33 |
-
"0": "safe",
|
| 34 |
-
"1": "unsafe"
|
| 35 |
-
},
|
| 36 |
-
"label2id": {
|
| 37 |
-
"safe": 0,
|
| 38 |
-
"unsafe": 1
|
| 39 |
-
},
|
| 40 |
"threshold": 0.5
|
| 41 |
}
|
|
|
|
| 1 |
{
|
| 2 |
"architectures": [
|
| 3 |
+
"LlamaForSequenceClassification"
|
| 4 |
],
|
| 5 |
+
"model_type": "llama",
|
| 6 |
+
"num_labels": 2,
|
| 7 |
+
"id2label": {
|
| 8 |
+
"0": "safe",
|
| 9 |
+
"1": "unsafe"
|
| 10 |
+
},
|
| 11 |
+
"label2id": {
|
| 12 |
+
"safe": 0,
|
| 13 |
+
"unsafe": 1
|
| 14 |
},
|
|
|
|
| 15 |
"vocab_size": 2048,
|
| 16 |
+
"hidden_size": 128,
|
| 17 |
+
"intermediate_size": 384,
|
| 18 |
+
"num_hidden_layers": 4,
|
| 19 |
+
"num_attention_heads": 4,
|
| 20 |
+
"num_key_value_heads": 4,
|
| 21 |
+
"head_dim": 32,
|
| 22 |
+
"max_position_embeddings": 160,
|
| 23 |
+
"rms_norm_eps": 1e-05,
|
| 24 |
+
"rope_parameters": {
|
| 25 |
+
"rope_type": "default",
|
| 26 |
+
"rope_theta": 10000.0
|
| 27 |
+
},
|
| 28 |
+
"attention_bias": false,
|
| 29 |
+
"mlp_bias": false,
|
| 30 |
+
"tie_word_embeddings": true,
|
| 31 |
+
"pad_token_id": 0,
|
| 32 |
+
"bos_token_id": 1,
|
| 33 |
+
"eos_token_id": 3,
|
| 34 |
+
"name": "KucLab Cneser",
|
| 35 |
"max_chars": 500,
|
| 36 |
"answer_ids": [
|
| 37 |
19,
|
|
|
|
| 43 |
"[USER]",
|
| 44 |
"[ASSISTANT]"
|
| 45 |
],
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
"threshold": 0.5
|
| 47 |
}
|
model.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4e6deaa1ab5603eb4ebfb644205d105f25fdc730b240df87912f1ceda70f2def
|
| 3 |
+
size 4466216
|
modeling_censer.py
DELETED
|
@@ -1,192 +0,0 @@
|
|
| 1 |
-
"""Censer.1 for Hugging Face transformers.
|
| 2 |
-
|
| 3 |
-
from transformers import AutoTokenizer, AutoModelForSequenceClassification
|
| 4 |
-
|
| 5 |
-
tok = AutoTokenizer.from_pretrained("KucLab/Censer.1", trust_remote_code=True)
|
| 6 |
-
model = AutoModelForSequenceClassification.from_pretrained(
|
| 7 |
-
"KucLab/Censer.1", trust_remote_code=True
|
| 8 |
-
)
|
| 9 |
-
out = model(**tok("how to make a bomb at home", return_tensors="pt"))
|
| 10 |
-
# out.logits -> [safe, unsafe]
|
| 11 |
-
"""
|
| 12 |
-
|
| 13 |
-
from __future__ import annotations
|
| 14 |
-
|
| 15 |
-
import torch
|
| 16 |
-
import torch.nn as nn
|
| 17 |
-
import torch.nn.functional as F
|
| 18 |
-
from transformers import PreTrainedModel
|
| 19 |
-
from transformers.configuration_utils import PretrainedConfig
|
| 20 |
-
from transformers.modeling_outputs import SequenceClassifierOutput
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
class CenserConfig(PretrainedConfig):
|
| 24 |
-
model_type = "censer"
|
| 25 |
-
auto_map = {
|
| 26 |
-
"AutoConfig": "modeling_censer.CenserConfig",
|
| 27 |
-
"AutoModelForSequenceClassification": "modeling_censer.CenserForSequenceClassification",
|
| 28 |
-
}
|
| 29 |
-
|
| 30 |
-
def __init__(
|
| 31 |
-
self,
|
| 32 |
-
vocab_size: int = 2048,
|
| 33 |
-
n_layers: int = 4,
|
| 34 |
-
n_heads: int = 4,
|
| 35 |
-
d_model: int = 128,
|
| 36 |
-
d_ff: int = 384,
|
| 37 |
-
max_seq_len: int = 160,
|
| 38 |
-
dropout: float = 0.1,
|
| 39 |
-
pad_id: int = 0,
|
| 40 |
-
answer_ids: list[int] | None = None,
|
| 41 |
-
max_chars: int = 500,
|
| 42 |
-
num_labels: int = 2,
|
| 43 |
-
**kwargs,
|
| 44 |
-
) -> None:
|
| 45 |
-
self.vocab_size = vocab_size
|
| 46 |
-
self.n_layers = n_layers
|
| 47 |
-
self.n_heads = n_heads
|
| 48 |
-
self.d_model = d_model
|
| 49 |
-
self.d_ff = d_ff
|
| 50 |
-
self.max_seq_len = max_seq_len
|
| 51 |
-
self.dropout = dropout
|
| 52 |
-
self.pad_id = pad_id
|
| 53 |
-
self.answer_ids = list(answer_ids or [0, 1])
|
| 54 |
-
self.max_chars = max_chars
|
| 55 |
-
kwargs.setdefault("id2label", {0: "safe", 1: "unsafe"})
|
| 56 |
-
kwargs.setdefault("label2id", {"safe": 0, "unsafe": 1})
|
| 57 |
-
super().__init__(pad_token_id=pad_id, num_labels=num_labels, **kwargs)
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
class RMSNorm(nn.Module):
|
| 61 |
-
def __init__(self, dim: int, eps: float = 1e-5) -> None:
|
| 62 |
-
super().__init__()
|
| 63 |
-
self.eps = eps
|
| 64 |
-
self.weight = nn.Parameter(torch.ones(dim))
|
| 65 |
-
|
| 66 |
-
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 67 |
-
dtype = x.dtype
|
| 68 |
-
x = x.float()
|
| 69 |
-
x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
|
| 70 |
-
return x.to(dtype) * self.weight
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
def build_rope_cache(seq_len: int, head_dim: int, device, dtype, base: float = 10000.0):
|
| 74 |
-
half = head_dim // 2
|
| 75 |
-
freqs = 1.0 / (base ** (torch.arange(0, half, device=device, dtype=torch.float32) / half))
|
| 76 |
-
positions = torch.arange(seq_len, device=device, dtype=torch.float32)
|
| 77 |
-
angles = torch.outer(positions, freqs)
|
| 78 |
-
cos = torch.cat((angles.cos(), angles.cos()), dim=-1).to(dtype)[:, None, :]
|
| 79 |
-
sin = torch.cat((angles.sin(), angles.sin()), dim=-1).to(dtype)[:, None, :]
|
| 80 |
-
return cos, sin
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
def apply_rope(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
|
| 84 |
-
seq_len = x.shape[1]
|
| 85 |
-
cos, sin = cos[:seq_len], sin[:seq_len]
|
| 86 |
-
x1, x2 = x[..., : x.shape[-1] // 2], x[..., x.shape[-1] // 2 :]
|
| 87 |
-
rotated = torch.cat((-x2, x1), dim=-1)
|
| 88 |
-
return x * cos + rotated * sin
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
class Attention(nn.Module):
|
| 92 |
-
def __init__(self, cfg: CenserConfig) -> None:
|
| 93 |
-
super().__init__()
|
| 94 |
-
self.n_heads = cfg.n_heads
|
| 95 |
-
self.head_dim = cfg.d_model // cfg.n_heads
|
| 96 |
-
self.dropout = cfg.dropout
|
| 97 |
-
self.qkv = nn.Linear(cfg.d_model, 3 * cfg.d_model, bias=False)
|
| 98 |
-
self.proj = nn.Linear(cfg.d_model, cfg.d_model, bias=False)
|
| 99 |
-
|
| 100 |
-
def forward(self, x, cos, sin, mask):
|
| 101 |
-
b, t, _ = x.shape
|
| 102 |
-
q, k, v = self.qkv(x).split(x.shape[-1], dim=-1)
|
| 103 |
-
q = q.view(b, t, self.n_heads, self.head_dim)
|
| 104 |
-
k = k.view(b, t, self.n_heads, self.head_dim)
|
| 105 |
-
v = v.view(b, t, self.n_heads, self.head_dim)
|
| 106 |
-
q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
|
| 107 |
-
out = F.scaled_dot_product_attention(
|
| 108 |
-
q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2),
|
| 109 |
-
attn_mask=mask,
|
| 110 |
-
dropout_p=self.dropout if self.training else 0.0,
|
| 111 |
-
is_causal=mask is None,
|
| 112 |
-
)
|
| 113 |
-
return self.proj(out.transpose(1, 2).reshape(b, t, -1))
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
class SwiGLU(nn.Module):
|
| 117 |
-
def __init__(self, cfg: CenserConfig) -> None:
|
| 118 |
-
super().__init__()
|
| 119 |
-
self.gate = nn.Linear(cfg.d_model, cfg.d_ff, bias=False)
|
| 120 |
-
self.up = nn.Linear(cfg.d_model, cfg.d_ff, bias=False)
|
| 121 |
-
self.down = nn.Linear(cfg.d_ff, cfg.d_model, bias=False)
|
| 122 |
-
|
| 123 |
-
def forward(self, x):
|
| 124 |
-
return self.down(F.silu(self.gate(x)) * self.up(x))
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
class Block(nn.Module):
|
| 128 |
-
def __init__(self, cfg: CenserConfig) -> None:
|
| 129 |
-
super().__init__()
|
| 130 |
-
self.norm1 = RMSNorm(cfg.d_model)
|
| 131 |
-
self.attn = Attention(cfg)
|
| 132 |
-
self.norm2 = RMSNorm(cfg.d_model)
|
| 133 |
-
self.mlp = SwiGLU(cfg)
|
| 134 |
-
self.dropout = nn.Dropout(cfg.dropout)
|
| 135 |
-
|
| 136 |
-
def forward(self, x, cos, sin, mask):
|
| 137 |
-
x = x + self.dropout(self.attn(self.norm1(x), cos, sin, mask))
|
| 138 |
-
x = x + self.dropout(self.mlp(self.norm2(x)))
|
| 139 |
-
return x
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
class CenserForSequenceClassification(PreTrainedModel):
|
| 143 |
-
"""Transformer trunk plus a two-way answer head: logits [safe, unsafe]."""
|
| 144 |
-
|
| 145 |
-
config_class = CenserConfig
|
| 146 |
-
base_model_prefix = ""
|
| 147 |
-
main_input_name = "input_ids"
|
| 148 |
-
|
| 149 |
-
def __init__(self, cfg: CenserConfig) -> None:
|
| 150 |
-
super().__init__(cfg)
|
| 151 |
-
self.cfg = cfg
|
| 152 |
-
self.embed = nn.Embedding(cfg.vocab_size, cfg.d_model, padding_idx=cfg.pad_id)
|
| 153 |
-
self.blocks = nn.ModuleList([Block(cfg) for _ in range(cfg.n_layers)])
|
| 154 |
-
self.norm = RMSNorm(cfg.d_model)
|
| 155 |
-
self.lm_head = nn.Linear(cfg.d_model, cfg.vocab_size, bias=False)
|
| 156 |
-
self._rope_cache = None
|
| 157 |
-
self.post_init()
|
| 158 |
-
|
| 159 |
-
def _rope(self, device, dtype):
|
| 160 |
-
key = f"{device}|{dtype}"
|
| 161 |
-
if self._rope_cache is None or self._rope_cache[2] != key:
|
| 162 |
-
head_dim = self.cfg.d_model // self.cfg.n_heads
|
| 163 |
-
cos, sin = build_rope_cache(self.cfg.max_seq_len, head_dim, device, dtype)
|
| 164 |
-
self._rope_cache = (cos, sin, key)
|
| 165 |
-
return self._rope_cache[0], self._rope_cache[1]
|
| 166 |
-
|
| 167 |
-
def _attention_mask(self, input_ids):
|
| 168 |
-
t = input_ids.shape[1]
|
| 169 |
-
pad = input_ids != self.cfg.pad_id
|
| 170 |
-
causal = torch.ones(t, t, dtype=torch.bool, device=input_ids.device).tril()
|
| 171 |
-
return pad[:, None, None, :] & causal[None, None, :, :]
|
| 172 |
-
|
| 173 |
-
def forward(self, input_ids=None, attention_mask=None, labels=None, **kwargs):
|
| 174 |
-
if input_ids is None:
|
| 175 |
-
raise ValueError("input_ids is required")
|
| 176 |
-
cos, sin = self._rope(input_ids.device, self.embed.weight.dtype)
|
| 177 |
-
mask = self._attention_mask(input_ids)
|
| 178 |
-
x = self.embed(input_ids)
|
| 179 |
-
for block in self.blocks:
|
| 180 |
-
x = block(x, cos, sin, mask)
|
| 181 |
-
hidden = self.norm(x)
|
| 182 |
-
non_pad = (input_ids != self.cfg.pad_id).sum(dim=1).clamp(min=1)
|
| 183 |
-
idx = (non_pad - 1)[:, None, None].expand(-1, 1, hidden.shape[-1])
|
| 184 |
-
pooled = hidden.gather(1, idx)[:, 0, :]
|
| 185 |
-
logits = self.lm_head(pooled).float()
|
| 186 |
-
answer_ids = torch.tensor(self.cfg.answer_ids, device=logits.device)
|
| 187 |
-
logits = logits.index_select(-1, answer_ids)
|
| 188 |
-
|
| 189 |
-
loss = None
|
| 190 |
-
if labels is not None:
|
| 191 |
-
loss = F.cross_entropy(logits, labels.to(logits.device))
|
| 192 |
-
return SequenceClassifierOutput(logits=logits, loss=loss)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"tokenizer_class": "PreTrainedTokenizerFast",
|
| 3 |
+
"pad_token": "[PAD]",
|
| 4 |
+
"bos_token": "[BOS]",
|
| 5 |
+
"eos_token": "[ASSISTANT]"
|
| 6 |
+
}
|