| --- |
| license: mit |
| datasets: |
| - thesofakillers/jigsaw-toxic-comment-classification-challenge |
| language: |
| - en |
| metrics: |
| - accuracy |
| - f1 |
| base_model: |
| - distilbert/distilbert-base-uncased |
| pipeline_tag: text-classification |
| library_name: transformers |
| tags: |
| - social |
| --- |
| # **Toxic Comment Classification with Transformer Optimization** |
| This project demonstrates a high-performance pipeline for classifying toxic comments using a **binary classification** approach. The models were trained and evaluated using the **Jigsaw Toxic Comment Classification** dataset, specifically leveraging the domain-specific **Toxic-BERT** model as a primary architecture. |
| ## **Project Overview** |
| * **Objective**: To build an efficient **binary toxicity classifier** using state-of-the-art NLP models. |
| * **Model Type**: Binary classification (Toxic vs. Non-Toxic). |
| * **Dataset**: Jigsaw Toxic Comment Classification Challenge. |
| * **Scope**: Includes data visualization, model benchmarking, and size reduction for deployment. |
| ## **Technical Workflow** |
| ### **1. Data Preprocessing & EDA** |
| * **Labeling**: Multi-label categories (toxic, severe_toxic, obscene, threat, insult, identity_hate) were condensed into a single **binary 'is_toxic' label**. |
| * **Balancing**: The dataset was sampled to include 16,000 toxic and 16,000 non-toxic comments to ensure a balanced 32,000-sample training set. |
| * **Cleaning**: Newline characters were removed to standardize the text input for transformer tokenizers. |
| * **Visualization**: Word clouds were generated for both classes to identify the most frequent terms associated with toxic and non-toxic speech. |
| ### **2. Embedding Benchmarking** |
| The project evaluated 15 different embedding sets across two categories: |
| * **Light Models**: Includes DistilBERT, MiniLM, ALBERT, and ELECTRA-Small. |
| * **Heavy Models**: Includes BERT, RoBERTa, DeBERTa, XLNet, and domain-specific models like **Toxic-BERT** and HateBERT. |
| ### **3. Model Performance Results** |
| Models were evaluated using Logistic Regression (LR), Support Vector Machines (SVM), and Random Forest (RF) |
| | Embedding | LR_AUC | LinearSVM_ACC | RBF_AUC | RF_ACC | |
| | :--- | :--- | :--- | :--- | :--- | |
| | **Toxic-BERT_transformer_emb** | 0.997022 | 0.979531 | 0.991532 | 0.979375 | |
| | **HateBERT_transformer_emb** | 0.967701 | 0.901875 | 0.965530 | 0.852344 | |
| | **DistilBERT_transformer_emb** | 0.967614 | 0.898906 | 0.967362 | 0.878125 | |
| ## **Optimization Techniques** |
| ### **4. Dynamic Quantization** |
| To optimize the teacher model (Toxic-BERT) for CPU inference, dynamic quantization was applied to convert weights from FP32 to INT8. |
| * **Size Reduction**: The model size decreased from **438.01 MB** to **181.49 MB**. |
| * **Accuracy Retention**: The quantized model maintained a high **Test AUC of 0.9966**, showing negligible performance loss despite the 58% reduction in size. |
| ### **5. Knowledge Distillation** |
| A smaller student model (**DistilBERT**) was trained to mimic the behavior of the **Toxic-BERT** teacher. |
| * **Loss Function**: A custom **Binary Knowledge Distillation** loss was used, combining Kullback-Leibler (KL) divergence for soft teacher probabilities and Cross-Entropy for hard labels. |
| * **Student Performance**: Reached a **Validation AUC of 0.9866** after 5 training epochs. |
| * **Final Footprint**: The student model is **267.86 MB**, significantly more portable than the original **438.03 MB** teacher model. |
| ## **Requirements** |
| * `torch` |
| * `transformers` |
| * `sentence-transformers` |
| * `pandas`, `numpy` |
| * `matplotlib`, `wordcloud` |
| * `scikit-learn` |
