| --- |
| license: mit |
| language: de |
| tags: |
| - bert |
| - ner |
| metrics: |
| - type: accuracy |
| value: 0.922 |
| base_model: "deepset/gbert-base" |
| --- |
| |
| # NERToxicBERT |
|
|
| This model was trained to do a token classification of online comments to determine |
| whether the token contains a vulgarity or not (swear words, insult, ...). |
|
|
| This model is based don GBERT from deepset (https://huggingface.co/deepset/gbert-base) which was mainly trained on wikipedia. |
| To this model we added a freshly initialized token classification header, which had to be trained on our labeled data. |
|
|
| # Training |
|
|
| For the training a dataset of 4500 comments german comments label on toxicity was used. |
| This dataset is not publicly available, but can be requested form TU-Wien (https://doi.org/10.5281/zenodo.10996203). |
|
|
|
|
| ## Data preparation |
|
|
| The dataset contains additional tags, which are |
| * Target_Group |
| * Target_Individual |
| * Target_Other |
| * Vulgarity |
| |
| We decided to use the Vulgarity tag to mark the words which are considered to be an insult. |
| 1306 Comments contained a Vulgarity, but 452 did not belong to a toxic considered comment. |
| These comments are split into 1484 number of sentences containing vulgarities. |
| Data prepared to have sentence by sentence data set tagged with vulgarity token. [‘O’,’Vul’] (1484 sentences). |
| A 80/10/10 train/validation/test split was used. |
| |
| ### Training Setup |
| |
| Out of 4500 comments 1306 contained a vulgarity tags. |
| In order to identify an optimally performing model for classifying toxic speech, a large set of models was trained and evaluated. |
| Hyperparameter: |
| - Layer 2 and 6 layers frozen |
| - 5 and 10 epochs, with a batch size of 8 |
| |
| |
| ### Model Evaluation |
| |
| |
| The best model used 2 frozen layers and was evaluated on the training set with the following metrics: |
| |
| | accuracy | f1 | precision | recall | |
| |----------|----|-----------|--------| |
| | 0.922 | 0.761 | 0.815 | 0.764 | |
| |
| |
| ## Usage |
| |
| Here is how to use this model to get the features of a given text in PyTorch: |
| |
| ```python |
| from transformers import AutoModelForSequenceClassification, AutoTokenizer |
| import numpy as np |
| |
| from transformers import pipeline |
| |
| # Replace this with your own checkpoint |
| model_checkpoint = "./saved_model" |
| token_classifier = pipeline( |
| "token-classification", model=model_checkpoint, aggregation_strategy="simple" |
| ) |
| |
| print(token_classifier("Die Fpö hat also auch ein Bescheuert-Gen in ihrer politischen DNA.")) |
| ``` |
| [{'entity_group': 'Vul', 'score': 0.9548946, 'word': 'Bescheuert - Gen', 'start': 26, 'end': 40}] |
