v2 models: 8-layer (apps) and 5-layer (low memory) students; README; v1 manifest kept

README.md

@@ -6,23 +6,28 @@ tags:
   - toxic
   - onnx
   - int8
-language: [en, de, fr, es, it, ru, uk, ar, hi, ja, zh, he, am, tt]
+language: [en, de, fr, es, it, ru, uk]
 ---
 
 # Ordicio toxicity classifier (ONNX, int8)
 
-An int8 ONNX export of
+Students of
 [textdetox/bert-multilingual-toxicity-classifier](https://huggingface.co/textdetox/bert-multilingual-toxicity-classifier)
-for on-device inference in the Ordicio apps (ONNX Runtime on Android / iOS, onnxruntime-web in browsers).
+for on-device inference in the Ordicio apps (ONNX Runtime on Android / iOS, onnxruntime-web in browsers): layers
+taken from the base model, distilled from XLM-R large teachers and fine-tuned on textdetox, GermEval 2018/2019/2021 and
+hand-labelled comments. Trained for English, German, French, Spanish, Italian, Russian and Ukrainian; texts in other
+scripts score as harmless.
 
-- `manifest.json`: the current version, file hashes, tokenizer settings, labels and threshold.
-- `v1/model.onnx`: weights quantized to int8 (dynamic quantization), inputs `input_ids`,
-  `attention_mask`, `token_type_ids` as int32 (cast to int64 inside the graph), output `logits`
-  (index 0 neutral, 1 toxic).
-- `v1/vocab.txt`: the WordPiece vocabulary of the base model (cased, no lowercasing).
+- `manifest.json`: the model the apps load (8 layers, `v2/L8/model.onnx`, 82 MB), with file hashes, tokenizer
+  settings, labels and threshold. `lowMemory` names the manifest for devices with little memory.
+- `manifest-low-memory.json`: the 5-layer model (`v2/L5/model.onnx`, 52 MB) for iPhone and iPad browsers.
+- `v2/vocab.txt`: the WordPiece vocabulary of both models (cased, no lowercasing).
+- Models: weights quantized to int8, inputs `input_ids`, `attention_mask`, `token_type_ids` as int32, output `logits`
+  (index 0 harmless, 1 toxic).
+- `manifest-v1.json`, `v1/`: the previous model (int8 export of the base model, 179 MB).
 
-The int8 model agrees with the fp32 model on 98.5 % of decisions (3600 samples of
-textdetox/multilingual_toxicity_dataset in 12 languages). For accuracy per language, see the base
-model's card: German results are weak (F1 0.52).
+On a hand-checked test of 3,664 real comments, at threshold 0.5 the 8-layer model finds 41 % of the attacks and flags
+4 of 1,000 harmless comments (5-layer: 38 % and 5.4); ROC-AUC 96 (German 93). In ordinary traffic both flag fewer than
+0.4 of 1,000 harmless comments.
 
-The use restrictions of the openrail++ license of the base model apply to this export.
+The use restrictions of the openrail++ license of the base model apply to these models.

manifest-low-memory.json

@@ -0,0 +1,40 @@
+{
+  "id": "ordicio-mbert-eu-L5-r256-int8-v2",
+  "source": "5 of the 12 layers of textdetox/bert-multilingual-toxicity-classifier, distilled from XLM-R large teachers and fine-tuned on textdetox, GermEval 2018/2019/2021 and hand-labelled comments (en, de, fr, es, it, ru, uk; abstains on other scripts)",
+  "license": "openrail++",
+  "files": {
+    "model": {
+      "path": "v2/L5/model.onnx",
+      "sha256": "92fe15d8dc2bd81ef999b5d0b3b4992e8fa58091cafdce871b40a15160fd6a5b",
+      "size": 52346473
+    },
+    "vocab": {
+      "path": "v2/vocab.txt",
+      "sha256": "8757cbccfe7417468b67f7543996c5a457520ffe00a7839def6177d09ee90963",
+      "size": 444290
+    }
+  },
+  "tokenizer": {
+    "type": "bert-wordpiece",
+    "lowercase": false,
+    "unk": "[UNK]",
+    "cls": "[CLS]",
+    "sep": "[SEP]",
+    "continuingPrefix": "##",
+    "maxCharsPerWord": 100
+  },
+  "maxLength": 128,
+  "maxWindows": 8,
+  "inputs": {
+    "ids": "input_ids",
+    "mask": "attention_mask",
+    "types": "token_type_ids",
+    "dtype": "int32"
+  },
+  "output": "logits",
+  "labels": [
+    null,
+    "toxic"
+  ],
+  "threshold": 0.5
+}

manifest-v1.json

@@ -0,0 +1,40 @@
+{
+  "id": "textdetox-bert-multilingual-int8-v1",
+  "source": "https://huggingface.co/textdetox/bert-multilingual-toxicity-classifier",
+  "license": "openrail++",
+  "files": {
+    "model": {
+      "path": "v1/model.onnx",
+      "sha256": "7848310732d59be089d16c0b680efc7231415d9d758b8799dbee65205454fc4a",
+      "size": 178612680
+    },
+    "vocab": {
+      "path": "v1/vocab.txt",
+      "sha256": "fe0fda7c425b48c516fc8f160d594c8022a0808447475c1a7c6d6479763f310c",
+      "size": 995526
+    }
+  },
+  "tokenizer": {
+    "type": "bert-wordpiece",
+    "lowercase": false,
+    "unk": "[UNK]",
+    "cls": "[CLS]",
+    "sep": "[SEP]",
+    "continuingPrefix": "##",
+    "maxCharsPerWord": 100
+  },
+  "maxLength": 128,
+  "maxWindows": 8,
+  "inputs": {
+    "ids": "input_ids",
+    "mask": "attention_mask",
+    "types": "token_type_ids",
+    "dtype": "int32"
+  },
+  "output": "logits",
+  "labels": [
+    null,
+    "toxic"
+  ],
+  "threshold": 0.5
+}

v2/L5/model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:92fe15d8dc2bd81ef999b5d0b3b4992e8fa58091cafdce871b40a15160fd6a5b
+size 52346473

v2/L8/model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61bf1b03add4172446972b27059d50c6475958fbb5bfd063d2a920a12dfe4c94
+size 81597346