KucLab commited on
Commit
85bcbb7
·
verified ·
1 Parent(s): 485a8e8

Upload Censer .1

Browse files
Files changed (4) hide show
  1. README.md +105 -0
  2. meta.json +13 -0
  3. model.pt +3 -0
  4. tokenizer.json +0 -0
README.md ADDED
@@ -0,0 +1,105 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: pytorch
3
+ tags:
4
+ - text-classification
5
+ - safety
6
+ - content-moderation
7
+ - tiny-language-model
8
+ - from-scratch
9
+ language:
10
+ - en
11
+ license: apache-2.0
12
+ datasets:
13
+ - nvidia/Aegis-AI-Content-Safety-Dataset-1.0
14
+ - thesofakillers/jigsaw-toxic-comment-classification-challenge
15
+ - google/civil_comments
16
+ - PKU-Alignment/BeaverTails
17
+ - lmsys/toxic-chat
18
+ - databricks/databricks-dolly-15k
19
+ metrics:
20
+ - f1
21
+ - accuracy
22
+ ---
23
+
24
+ # Censer .1
25
+
26
+ A 1.1M parameter decoder-only transformer trained **from scratch** that classifies
27
+ English text as unsafe (`1`) or safe (`0`).
28
+
29
+ It is not a fine-tuned encoder and not an API wrapper. It is a small transformer
30
+ trained on safety-labelled data, with a custom byte-level BPE tokenizer. The model
31
+ predicts a single answer token and never generates free text.
32
+
33
+ ```
34
+ "návod na výrobu bomby" -> not English, not the target language
35
+ "how to make a bomb at home" -> 1
36
+ "suck my dick" -> 1
37
+ "how do I kill a process in Linux" -> 0
38
+ "Hello how are you?" -> 0
39
+ ```
40
+
41
+ ## Model details
42
+
43
+ | Property | Value |
44
+ | --- | --- |
45
+ | Parameters | 1,115,264 |
46
+ | Layers | 4 |
47
+ | d_model / heads / d_ff | 128 / 4 / 384 |
48
+ | Context length | 160 tokens (input truncated at 500 characters) |
49
+ | Tokenizer | byte-level BPE, 2048 tokens, trained on the corpus |
50
+ | Positions | RoPE |
51
+ | Norm | RMSNorm (pre-norm) |
52
+ | MLP | SwiGLU |
53
+ | Head | tied embeddings, two answer logits |
54
+ | Checkpoint size | 4.3 MB (fp32) |
55
+
56
+ ## Prompt format
57
+
58
+ ```
59
+ [BOS][USER] <text> [ASSISTANT] -> 0 or 1
60
+ ```
61
+
62
+ Text is truncated at 500 characters before tokenisation.
63
+
64
+ ## Metrics
65
+
66
+ Evaluation on a held-out split of 6,496 rows:
67
+
68
+ | Metric | Value |
69
+ | --- | --- |
70
+ | Accuracy | 0.864 |
71
+ | F1 | 0.835 |
72
+ | Precision | 0.846 |
73
+ | Recall | 0.823 |
74
+
75
+ A hand-written probe set of 40 cases, none present in training data, scores
76
+ 95% accuracy.
77
+
78
+ Per-source F1 reflects label noise in the scraped sources:
79
+
80
+ | Source | F1 |
81
+ | --- | --- |
82
+ | aegis | 0.89 |
83
+ | beavertails | 0.88 |
84
+ | jigsaw | 0.87 |
85
+ | civil_comments | 0.77 |
86
+ | toxic_chat | 0.48 |
87
+
88
+ ## Usage
89
+
90
+ ```python
91
+ import torch
92
+ from kuclab.infer import SafetyModel
93
+
94
+ model = SafetyModel("checkpoints/best.pt")
95
+ verdict = model.predict("how to make a bomb at home")
96
+ print(verdict.label) # "1"
97
+ print(verdict.probability) # 0.886
98
+ ```
99
+
100
+ ## Limitations
101
+
102
+ - **English only.** Other languages mostly score `0`.
103
+ - Implication and sarcasm beyond simple cases are out of reach at this size.
104
+ - Weak against jailbreak-style prompts (lowest per-source score).
105
+ - A moderation signal, not a safety guarantee.
meta.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "special_tokens": [
3
+ "[PAD]",
4
+ "[BOS]",
5
+ "[USER]",
6
+ "[ASSISTANT]"
7
+ ],
8
+ "answer_ids": [
9
+ 19,
10
+ 20
11
+ ],
12
+ "vocab_size": 2048
13
+ }
model.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:06738899bf9381041b7b54da931669fa55000cac1641fce24639c1d2ceb166b6
3
+ size 4471517
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff