KucLab commited on
Commit
2c669f3
·
verified ·
1 Parent(s): f08c2b8

Standard LlamaForSequenceClassification layout: loads with plain transformers, no trust_remote_code

Browse files
Files changed (6) hide show
  1. README.md +44 -27
  2. censer.py +45 -7
  3. config.json +29 -23
  4. model.safetensors +2 -2
  5. modeling_censer.py +0 -192
  6. tokenizer_config.json +6 -0
README.md CHANGED
@@ -7,7 +7,6 @@ tags:
7
  - content-moderation
8
  - tiny-language-model
9
  - from-scratch
10
- - custom-code
11
  language:
12
  - en
13
  license: apache-2.0
@@ -26,7 +25,22 @@ metrics:
26
  # Censer.1
27
 
28
  A 1.1M parameter decoder-only transformer trained **from scratch** that classifies
29
- English text as unsafe (`1`) or safe (`0`).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30
 
31
  ## Download
32
 
@@ -37,6 +51,7 @@ The whole model is `model.safetensors` (4.5 MB). Grab the files directly:
37
  | **model.safetensors** | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/model.safetensors?download=true) |
38
  | config.json | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/config.json?download=true) |
39
  | tokenizer.json | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/tokenizer.json?download=true) |
 
40
  | censer.py | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/censer.py?download=true) |
41
 
42
  Or fetch everything at once:
@@ -54,32 +69,35 @@ trained on safety-labelled data, with a custom byte-level BPE tokenizer. The mod
54
  predicts a single answer token and never generates free text.
55
 
56
  ```
57
- "návod na výrobu bomby" -> not English, not the target language
58
- "how to make a bomb at home" -> 1
59
- "suck my dick" -> 1
60
- "how do I kill a process in Linux" -> 0
61
- "Hello how are you?" -> 0
62
  ```
63
 
64
  ## Model details
65
 
66
  | Property | Value |
67
  | --- | --- |
 
68
  | Parameters | 1,115,264 |
69
  | Layers | 4 |
70
  | d_model / heads / d_ff | 128 / 4 / 384 |
71
  | Context length | 160 tokens (input truncated at 500 characters) |
72
  | Tokenizer | byte-level BPE, 2048 tokens, trained on the corpus |
73
- | Positions | RoPE |
74
- | Norm | RMSNorm (pre-norm) |
75
  | MLP | SwiGLU |
76
- | Head | tied embeddings, two answer logits |
77
  | Checkpoint size | 4.3 MB (fp32) |
78
 
79
  ## Prompt format
80
 
 
 
 
81
  ```
82
- [BOS][USER] <text> [ASSISTANT] -> 0 or 1
83
  ```
84
 
85
  Text is truncated at 500 characters before tokenisation.
@@ -126,8 +144,8 @@ Training the whole model from scratch takes about 3.5 minutes on the same GPU.
126
 
127
  ## Usage
128
 
129
- Works like any other text-classification model on the Hub — `from_pretrained`
130
- downloads everything by itself, no cloning and no manual files.
131
 
132
  ```bash
133
  pip install torch transformers
@@ -138,8 +156,8 @@ import torch
138
  from transformers import AutoTokenizer, AutoModelForSequenceClassification
139
 
140
  repo = "KucLab/Censer.1"
141
- tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
142
- model = AutoModelForSequenceClassification.from_pretrained(repo, trust_remote_code=True).eval()
143
 
144
  text = "how to make a bomb at home"
145
  with torch.no_grad():
@@ -149,15 +167,12 @@ print(probs) # tensor([0.1833, 0.8167]) -> [safe, unsafe]
149
  print(model.config.id2label) # {0: 'safe', 1: 'unsafe'}
150
  ```
151
 
152
- `trust_remote_code=True` is required because the architecture is custom; the
153
- repository ships `modeling_censer.py`, so there is nothing else to install.
154
-
155
  Or with the high-level pipeline:
156
 
157
  ```python
158
  from transformers import pipeline
159
 
160
- clf = pipeline("text-classification", model="KucLab/Censer.1", trust_remote_code=True)
161
  print(clf("how to make a bomb at home"))
162
  # [{'label': 'unsafe', 'score': 0.8167}]
163
  print(clf("Hello how are you?"))
@@ -167,7 +182,9 @@ print(clf("Hello how are you?"))
167
  ## Usage without transformers
168
 
169
  The repository is also self-contained on its own: `model.safetensors`,
170
- `config.json`, `tokenizer.json` and `censer.py`.
 
 
171
 
172
  ```bash
173
  hf download KucLab/Censer.1 --local-dir Censer.1
@@ -194,17 +211,17 @@ Files:
194
 
195
  | File | Purpose |
196
  | --- | --- |
197
- | `model.safetensors` | weights only, 4.3 MB |
198
- | `config.json` | architecture and inference settings |
199
- | `tokenizer.json` | byte-level BPE, 2048 tokens |
200
- | `censer.py` | standalone model + inference, no repo needed |
201
 
202
- `censer.py` was verified to produce byte-identical predictions to the training
203
- code. Runs on CPU or CUDA/ROCm; pass `device="cpu"` to force CPU.
204
 
205
  ## Limitations
206
 
207
- - **English only.** Other languages mostly score `0`.
208
  - Implication and sarcasm beyond simple cases are out of reach at this size.
209
  - Weak against jailbreak-style prompts (lowest per-source score).
210
  - A moderation signal, not a safety guarantee.
 
7
  - content-moderation
8
  - tiny-language-model
9
  - from-scratch
 
10
  language:
11
  - en
12
  license: apache-2.0
 
25
  # Censer.1
26
 
27
  A 1.1M parameter decoder-only transformer trained **from scratch** that classifies
28
+ English text as unsafe (`unsafe`) or safe (`safe`).
29
+
30
+ It is a **standard Hub model**: the checkpoint is a plain
31
+ `LlamaForSequenceClassification`, so it loads with `transformers` out of the box —
32
+ no `trust_remote_code`, no custom code, no cloning — and the **Use this model**
33
+ widget on this page runs it directly in the browser.
34
+
35
+ ```python
36
+ from transformers import pipeline
37
+
38
+ clf = pipeline("text-classification", model="KucLab/Censer.1")
39
+ print(clf("how to make a bomb at home")) # [{'label': 'unsafe', 'score': 0.8167}]
40
+ print(clf("Hello how are you?")) # [{'label': 'safe', 'score': 0.9229}]
41
+ print(clf("suck my dick")) # [{'label': 'unsafe', 'score': 0.9944}]
42
+ print(clf("how do I kill a process in Linux")) # [{'label': 'safe', 'score': 0.8974}]
43
+ ```
44
 
45
  ## Download
46
 
 
51
  | **model.safetensors** | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/model.safetensors?download=true) |
52
  | config.json | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/config.json?download=true) |
53
  | tokenizer.json | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/tokenizer.json?download=true) |
54
+ | tokenizer_config.json | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/tokenizer_config.json?download=true) |
55
  | censer.py | [download](https://huggingface.co/KucLab/Censer.1/resolve/main/censer.py?download=true) |
56
 
57
  Or fetch everything at once:
 
69
  predicts a single answer token and never generates free text.
70
 
71
  ```
72
+ "how to make a bomb at home" -> unsafe
73
+ "suck my dick" -> unsafe
74
+ "how do I kill a process in Linux" -> safe
75
+ "Hello how are you?" -> safe
 
76
  ```
77
 
78
  ## Model details
79
 
80
  | Property | Value |
81
  | --- | --- |
82
+ | Architecture | `LlamaForSequenceClassification` (standard transformers) |
83
  | Parameters | 1,115,264 |
84
  | Layers | 4 |
85
  | d_model / heads / d_ff | 128 / 4 / 384 |
86
  | Context length | 160 tokens (input truncated at 500 characters) |
87
  | Tokenizer | byte-level BPE, 2048 tokens, trained on the corpus |
88
+ | Positions | RoPE (theta 10000) |
89
+ | Norm | RMSNorm (pre-norm, eps 1e-5) |
90
  | MLP | SwiGLU |
91
+ | Head | `score` linear layer = embedding rows of the answer tokens `0`/`1` |
92
  | Checkpoint size | 4.3 MB (fp32) |
93
 
94
  ## Prompt format
95
 
96
+ The tokenizer bakes the template into its post-processor, so plain
97
+ `tokenizer(text)` already produces:
98
+
99
  ```
100
+ [BOS][USER] <text> [ASSISTANT] -> answer logits over tokens "0"/"1"
101
  ```
102
 
103
  Text is truncated at 500 characters before tokenisation.
 
144
 
145
  ## Usage
146
 
147
+ Plain `transformers`, nothing else to install — `from_pretrained` downloads
148
+ everything by itself:
149
 
150
  ```bash
151
  pip install torch transformers
 
156
  from transformers import AutoTokenizer, AutoModelForSequenceClassification
157
 
158
  repo = "KucLab/Censer.1"
159
+ tok = AutoTokenizer.from_pretrained(repo)
160
+ model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
161
 
162
  text = "how to make a bomb at home"
163
  with torch.no_grad():
 
167
  print(model.config.id2label) # {0: 'safe', 1: 'unsafe'}
168
  ```
169
 
 
 
 
170
  Or with the high-level pipeline:
171
 
172
  ```python
173
  from transformers import pipeline
174
 
175
+ clf = pipeline("text-classification", model="KucLab/Censer.1")
176
  print(clf("how to make a bomb at home"))
177
  # [{'label': 'unsafe', 'score': 0.8167}]
178
  print(clf("Hello how are you?"))
 
182
  ## Usage without transformers
183
 
184
  The repository is also self-contained on its own: `model.safetensors`,
185
+ `config.json`, `tokenizer.json` and `censer.py`. The weights are stored in the
186
+ standard Llama layout; `censer.py` maps them back internally, so it only needs
187
+ `torch`, `tokenizers` and `safetensors`:
188
 
189
  ```bash
190
  hf download KucLab/Censer.1 --local-dir Censer.1
 
211
 
212
  | File | Purpose |
213
  | --- | --- |
214
+ | `model.safetensors` | weights in standard Llama layout, 4.3 MB |
215
+ | `config.json` | standard `LlamaForSequenceClassification` config |
216
+ | `tokenizer.json` / `tokenizer_config.json` | byte-level BPE, 2048 tokens, prompt template baked in |
217
+ | `censer.py` | standalone inference (torch + tokenizers only), no repo needed |
218
 
219
+ `censer.py` was verified to produce predictions identical to the training code.
220
+ Runs on CPU or CUDA/ROCm; pass `device="cpu"` to force CPU.
221
 
222
  ## Limitations
223
 
224
+ - **English only.** Other languages mostly score safe.
225
  - Implication and sarcasm beyond simple cases are out of reach at this size.
226
  - Weak against jailbreak-style prompts (lowest per-source score).
227
  - A moderation signal, not a safety guarantee.
censer.py CHANGED
@@ -3,11 +3,11 @@
3
  Copy this file next to the downloaded weights, install torch and tokenizers,
4
  and you are ready:
5
 
6
- pip install torch tokenizers huggingface_hub
7
  python censer.py "how to make a bomb at home"
8
 
9
  It expects, in the same directory:
10
- model.safetensors
11
  tokenizer.json
12
  config.json
13
  """
@@ -155,6 +155,46 @@ class Censer(nn.Module):
155
  return logits.softmax(-1)[:, 1]
156
 
157
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
158
  class CenserPipeline:
159
  """Text in, "1" (unsafe) or "0" (safe) out."""
160
 
@@ -162,7 +202,7 @@ class CenserPipeline:
162
  from safetensors.torch import load_file
163
 
164
  directory = Path(directory)
165
- self.cfg = json.loads((directory / "config.json").read_text(encoding="utf-8"))
166
  self.tok = Tokenizer.from_file(str(directory / "tokenizer.json"))
167
  self.pad_id = self.tok.token_to_id(PAD)
168
  self.bos_id = self.tok.token_to_id(BOS)
@@ -171,11 +211,9 @@ class CenserPipeline:
171
  self.threshold = threshold
172
  self.device = torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
173
  self.model = Censer(self.cfg)
174
- state = load_file(directory / "model.safetensors")
175
- if self.cfg["tie_embeddings"]:
176
- state.pop("lm_head.weight", None)
177
  missing, unexpected = self.model.load_state_dict(state, strict=False)
178
- missing = [k for k in missing if not (self.cfg["tie_embeddings"] and k == "lm_head.weight")]
179
  if missing or unexpected:
180
  raise RuntimeError(f"weight mismatch: missing={missing} unexpected={unexpected}")
181
  self.model.to(self.device).eval()
 
3
  Copy this file next to the downloaded weights, install torch and tokenizers,
4
  and you are ready:
5
 
6
+ pip install torch tokenizers safetensors
7
  python censer.py "how to make a bomb at home"
8
 
9
  It expects, in the same directory:
10
+ model.safetensors (standard LlamaForSequenceClassification layout)
11
  tokenizer.json
12
  config.json
13
  """
 
155
  return logits.softmax(-1)[:, 1]
156
 
157
 
158
+ def remap_hf_state(state: dict) -> dict:
159
+ """Standard Llama-layout safetensors -> training-layout state dict."""
160
+ import re
161
+
162
+ out = {"embed.weight": state["model.embed_tokens.weight"]}
163
+ layer_ids = sorted({int(m.group(1)) for k in state if (m := re.match(r"model\.layers\.(\d+)\.", k))})
164
+ for i in layer_ids:
165
+ p, b = f"model.layers.{i}.", f"blocks.{i}."
166
+ qkv = torch.cat(
167
+ [state[p + "self_attn.q_proj.weight"], state[p + "self_attn.k_proj.weight"], state[p + "self_attn.v_proj.weight"]],
168
+ dim=0,
169
+ )
170
+ out[b + "attn.qkv.weight"] = qkv.contiguous()
171
+ out[b + "attn.proj.weight"] = state[p + "self_attn.o_proj.weight"]
172
+ out[b + "mlp.gate.weight"] = state[p + "mlp.gate_proj.weight"]
173
+ out[b + "mlp.up.weight"] = state[p + "mlp.up_proj.weight"]
174
+ out[b + "mlp.down.weight"] = state[p + "mlp.down_proj.weight"]
175
+ out[b + "norm1.weight"] = state[p + "input_layernorm.weight"]
176
+ out[b + "norm2.weight"] = state[p + "post_attention_layernorm.weight"]
177
+ out["norm.weight"] = state["model.norm.weight"]
178
+ return out
179
+
180
+
181
+ def internal_cfg(cfg: dict) -> dict:
182
+ """config.json (Llama layout) -> the dict the Censer module expects."""
183
+ return {
184
+ "vocab_size": cfg["vocab_size"],
185
+ "n_layers": cfg["num_hidden_layers"],
186
+ "n_heads": cfg["num_attention_heads"],
187
+ "d_model": cfg["hidden_size"],
188
+ "d_ff": cfg["intermediate_size"],
189
+ "max_seq_len": cfg["max_position_embeddings"],
190
+ "dropout": cfg.get("dropout", 0.0),
191
+ "tie_embeddings": True,
192
+ "pad_id": cfg.get("pad_token_id", 0),
193
+ "answer_ids": cfg["answer_ids"],
194
+ "max_chars": cfg.get("max_chars", 500),
195
+ }
196
+
197
+
198
  class CenserPipeline:
199
  """Text in, "1" (unsafe) or "0" (safe) out."""
200
 
 
202
  from safetensors.torch import load_file
203
 
204
  directory = Path(directory)
205
+ self.cfg = internal_cfg(json.loads((directory / "config.json").read_text(encoding="utf-8")))
206
  self.tok = Tokenizer.from_file(str(directory / "tokenizer.json"))
207
  self.pad_id = self.tok.token_to_id(PAD)
208
  self.bos_id = self.tok.token_to_id(BOS)
 
211
  self.threshold = threshold
212
  self.device = torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
213
  self.model = Censer(self.cfg)
214
+ state = remap_hf_state(load_file(directory / "model.safetensors"))
 
 
215
  missing, unexpected = self.model.load_state_dict(state, strict=False)
216
+ missing = [k for k in missing if k != "lm_head.weight"] # tied to embed.weight
217
  if missing or unexpected:
218
  raise RuntimeError(f"weight mismatch: missing={missing} unexpected={unexpected}")
219
  self.model.to(self.device).eval()
config.json CHANGED
@@ -1,22 +1,37 @@
1
  {
2
  "architectures": [
3
- "CenserForSequenceClassification"
4
  ],
5
- "model_type": "censer",
6
- "auto_map": {
7
- "AutoConfig": "modeling_censer.CenserConfig",
8
- "AutoModelForSequenceClassification": "modeling_censer.CenserForSequenceClassification"
 
 
 
 
 
9
  },
10
- "name": "Censer.1",
11
  "vocab_size": 2048,
12
- "n_layers": 4,
13
- "n_heads": 4,
14
- "d_model": 128,
15
- "d_ff": 384,
16
- "max_seq_len": 160,
17
- "dropout": 0.1,
18
- "tie_embeddings": true,
19
- "pad_id": 0,
 
 
 
 
 
 
 
 
 
 
 
20
  "max_chars": 500,
21
  "answer_ids": [
22
  19,
@@ -28,14 +43,5 @@
28
  "[USER]",
29
  "[ASSISTANT]"
30
  ],
31
- "num_labels": 2,
32
- "id2label": {
33
- "0": "safe",
34
- "1": "unsafe"
35
- },
36
- "label2id": {
37
- "safe": 0,
38
- "unsafe": 1
39
- },
40
  "threshold": 0.5
41
  }
 
1
  {
2
  "architectures": [
3
+ "LlamaForSequenceClassification"
4
  ],
5
+ "model_type": "llama",
6
+ "num_labels": 2,
7
+ "id2label": {
8
+ "0": "safe",
9
+ "1": "unsafe"
10
+ },
11
+ "label2id": {
12
+ "safe": 0,
13
+ "unsafe": 1
14
  },
 
15
  "vocab_size": 2048,
16
+ "hidden_size": 128,
17
+ "intermediate_size": 384,
18
+ "num_hidden_layers": 4,
19
+ "num_attention_heads": 4,
20
+ "num_key_value_heads": 4,
21
+ "head_dim": 32,
22
+ "max_position_embeddings": 160,
23
+ "rms_norm_eps": 1e-05,
24
+ "rope_parameters": {
25
+ "rope_type": "default",
26
+ "rope_theta": 10000.0
27
+ },
28
+ "attention_bias": false,
29
+ "mlp_bias": false,
30
+ "tie_word_embeddings": true,
31
+ "pad_token_id": 0,
32
+ "bos_token_id": 1,
33
+ "eos_token_id": 3,
34
+ "name": "KucLab Cneser",
35
  "max_chars": 500,
36
  "answer_ids": [
37
  19,
 
43
  "[USER]",
44
  "[ASSISTANT]"
45
  ],
 
 
 
 
 
 
 
 
 
46
  "threshold": 0.5
47
  }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:3e7279d3542ad7a1e09c75cdb22daeb6042b26879dc468be2bbb0b6a0ac22587
3
- size 5512472
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4e6deaa1ab5603eb4ebfb644205d105f25fdc730b240df87912f1ceda70f2def
3
+ size 4466216
modeling_censer.py DELETED
@@ -1,192 +0,0 @@
1
- """Censer.1 for Hugging Face transformers.
2
-
3
- from transformers import AutoTokenizer, AutoModelForSequenceClassification
4
-
5
- tok = AutoTokenizer.from_pretrained("KucLab/Censer.1", trust_remote_code=True)
6
- model = AutoModelForSequenceClassification.from_pretrained(
7
- "KucLab/Censer.1", trust_remote_code=True
8
- )
9
- out = model(**tok("how to make a bomb at home", return_tensors="pt"))
10
- # out.logits -> [safe, unsafe]
11
- """
12
-
13
- from __future__ import annotations
14
-
15
- import torch
16
- import torch.nn as nn
17
- import torch.nn.functional as F
18
- from transformers import PreTrainedModel
19
- from transformers.configuration_utils import PretrainedConfig
20
- from transformers.modeling_outputs import SequenceClassifierOutput
21
-
22
-
23
- class CenserConfig(PretrainedConfig):
24
- model_type = "censer"
25
- auto_map = {
26
- "AutoConfig": "modeling_censer.CenserConfig",
27
- "AutoModelForSequenceClassification": "modeling_censer.CenserForSequenceClassification",
28
- }
29
-
30
- def __init__(
31
- self,
32
- vocab_size: int = 2048,
33
- n_layers: int = 4,
34
- n_heads: int = 4,
35
- d_model: int = 128,
36
- d_ff: int = 384,
37
- max_seq_len: int = 160,
38
- dropout: float = 0.1,
39
- pad_id: int = 0,
40
- answer_ids: list[int] | None = None,
41
- max_chars: int = 500,
42
- num_labels: int = 2,
43
- **kwargs,
44
- ) -> None:
45
- self.vocab_size = vocab_size
46
- self.n_layers = n_layers
47
- self.n_heads = n_heads
48
- self.d_model = d_model
49
- self.d_ff = d_ff
50
- self.max_seq_len = max_seq_len
51
- self.dropout = dropout
52
- self.pad_id = pad_id
53
- self.answer_ids = list(answer_ids or [0, 1])
54
- self.max_chars = max_chars
55
- kwargs.setdefault("id2label", {0: "safe", 1: "unsafe"})
56
- kwargs.setdefault("label2id", {"safe": 0, "unsafe": 1})
57
- super().__init__(pad_token_id=pad_id, num_labels=num_labels, **kwargs)
58
-
59
-
60
- class RMSNorm(nn.Module):
61
- def __init__(self, dim: int, eps: float = 1e-5) -> None:
62
- super().__init__()
63
- self.eps = eps
64
- self.weight = nn.Parameter(torch.ones(dim))
65
-
66
- def forward(self, x: torch.Tensor) -> torch.Tensor:
67
- dtype = x.dtype
68
- x = x.float()
69
- x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
70
- return x.to(dtype) * self.weight
71
-
72
-
73
- def build_rope_cache(seq_len: int, head_dim: int, device, dtype, base: float = 10000.0):
74
- half = head_dim // 2
75
- freqs = 1.0 / (base ** (torch.arange(0, half, device=device, dtype=torch.float32) / half))
76
- positions = torch.arange(seq_len, device=device, dtype=torch.float32)
77
- angles = torch.outer(positions, freqs)
78
- cos = torch.cat((angles.cos(), angles.cos()), dim=-1).to(dtype)[:, None, :]
79
- sin = torch.cat((angles.sin(), angles.sin()), dim=-1).to(dtype)[:, None, :]
80
- return cos, sin
81
-
82
-
83
- def apply_rope(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
84
- seq_len = x.shape[1]
85
- cos, sin = cos[:seq_len], sin[:seq_len]
86
- x1, x2 = x[..., : x.shape[-1] // 2], x[..., x.shape[-1] // 2 :]
87
- rotated = torch.cat((-x2, x1), dim=-1)
88
- return x * cos + rotated * sin
89
-
90
-
91
- class Attention(nn.Module):
92
- def __init__(self, cfg: CenserConfig) -> None:
93
- super().__init__()
94
- self.n_heads = cfg.n_heads
95
- self.head_dim = cfg.d_model // cfg.n_heads
96
- self.dropout = cfg.dropout
97
- self.qkv = nn.Linear(cfg.d_model, 3 * cfg.d_model, bias=False)
98
- self.proj = nn.Linear(cfg.d_model, cfg.d_model, bias=False)
99
-
100
- def forward(self, x, cos, sin, mask):
101
- b, t, _ = x.shape
102
- q, k, v = self.qkv(x).split(x.shape[-1], dim=-1)
103
- q = q.view(b, t, self.n_heads, self.head_dim)
104
- k = k.view(b, t, self.n_heads, self.head_dim)
105
- v = v.view(b, t, self.n_heads, self.head_dim)
106
- q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
107
- out = F.scaled_dot_product_attention(
108
- q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2),
109
- attn_mask=mask,
110
- dropout_p=self.dropout if self.training else 0.0,
111
- is_causal=mask is None,
112
- )
113
- return self.proj(out.transpose(1, 2).reshape(b, t, -1))
114
-
115
-
116
- class SwiGLU(nn.Module):
117
- def __init__(self, cfg: CenserConfig) -> None:
118
- super().__init__()
119
- self.gate = nn.Linear(cfg.d_model, cfg.d_ff, bias=False)
120
- self.up = nn.Linear(cfg.d_model, cfg.d_ff, bias=False)
121
- self.down = nn.Linear(cfg.d_ff, cfg.d_model, bias=False)
122
-
123
- def forward(self, x):
124
- return self.down(F.silu(self.gate(x)) * self.up(x))
125
-
126
-
127
- class Block(nn.Module):
128
- def __init__(self, cfg: CenserConfig) -> None:
129
- super().__init__()
130
- self.norm1 = RMSNorm(cfg.d_model)
131
- self.attn = Attention(cfg)
132
- self.norm2 = RMSNorm(cfg.d_model)
133
- self.mlp = SwiGLU(cfg)
134
- self.dropout = nn.Dropout(cfg.dropout)
135
-
136
- def forward(self, x, cos, sin, mask):
137
- x = x + self.dropout(self.attn(self.norm1(x), cos, sin, mask))
138
- x = x + self.dropout(self.mlp(self.norm2(x)))
139
- return x
140
-
141
-
142
- class CenserForSequenceClassification(PreTrainedModel):
143
- """Transformer trunk plus a two-way answer head: logits [safe, unsafe]."""
144
-
145
- config_class = CenserConfig
146
- base_model_prefix = ""
147
- main_input_name = "input_ids"
148
-
149
- def __init__(self, cfg: CenserConfig) -> None:
150
- super().__init__(cfg)
151
- self.cfg = cfg
152
- self.embed = nn.Embedding(cfg.vocab_size, cfg.d_model, padding_idx=cfg.pad_id)
153
- self.blocks = nn.ModuleList([Block(cfg) for _ in range(cfg.n_layers)])
154
- self.norm = RMSNorm(cfg.d_model)
155
- self.lm_head = nn.Linear(cfg.d_model, cfg.vocab_size, bias=False)
156
- self._rope_cache = None
157
- self.post_init()
158
-
159
- def _rope(self, device, dtype):
160
- key = f"{device}|{dtype}"
161
- if self._rope_cache is None or self._rope_cache[2] != key:
162
- head_dim = self.cfg.d_model // self.cfg.n_heads
163
- cos, sin = build_rope_cache(self.cfg.max_seq_len, head_dim, device, dtype)
164
- self._rope_cache = (cos, sin, key)
165
- return self._rope_cache[0], self._rope_cache[1]
166
-
167
- def _attention_mask(self, input_ids):
168
- t = input_ids.shape[1]
169
- pad = input_ids != self.cfg.pad_id
170
- causal = torch.ones(t, t, dtype=torch.bool, device=input_ids.device).tril()
171
- return pad[:, None, None, :] & causal[None, None, :, :]
172
-
173
- def forward(self, input_ids=None, attention_mask=None, labels=None, **kwargs):
174
- if input_ids is None:
175
- raise ValueError("input_ids is required")
176
- cos, sin = self._rope(input_ids.device, self.embed.weight.dtype)
177
- mask = self._attention_mask(input_ids)
178
- x = self.embed(input_ids)
179
- for block in self.blocks:
180
- x = block(x, cos, sin, mask)
181
- hidden = self.norm(x)
182
- non_pad = (input_ids != self.cfg.pad_id).sum(dim=1).clamp(min=1)
183
- idx = (non_pad - 1)[:, None, None].expand(-1, 1, hidden.shape[-1])
184
- pooled = hidden.gather(1, idx)[:, 0, :]
185
- logits = self.lm_head(pooled).float()
186
- answer_ids = torch.tensor(self.cfg.answer_ids, device=logits.device)
187
- logits = logits.index_select(-1, answer_ids)
188
-
189
- loss = None
190
- if labels is not None:
191
- loss = F.cross_entropy(logits, labels.to(logits.device))
192
- return SequenceClassifierOutput(logits=logits, loss=loss)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tokenizer_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "tokenizer_class": "PreTrainedTokenizerFast",
3
+ "pad_token": "[PAD]",
4
+ "bos_token": "[BOS]",
5
+ "eos_token": "[ASSISTANT]"
6
+ }