| # Model results and comparison |
|
|
| Canonical data: [`reports/summary.csv`](../reports/summary.csv) |
| Tuned hyperparameters: [`configs/best_params.yaml`](../configs/best_params.yaml) |
| **Full technical report:** [`reports/final_report.md`](../reports/final_report.md) · [ES](../reports/final_report.es.md) |
|
|
| ## Best sklearn model (production) |
|
|
| **Winner:** Logistic Regression + TF-IDF (Optuna-tuned), exported as `models/final_model.joblib`. |
|
|
| | Metric | Test value | Notes | |
| |--------|------------|-------| |
| | F1 (weighted) | **0.7579** | Primary project metric | |
| | ROC-AUC | **0.81** | Ranking quality | |
| | False positives | **18** | Safe comments marked toxic | |
| | False negatives | **30** | Toxic comments missed | |
| | F1 (train) | 0.8987 | In-sample | |
| | Train–test gap | 14.07 pp | High; prefer CV gap for generalization | |
| | CV–test gap | **4.76 pp** | Meets < 5 pp rubric | |
| | Test size | ~20% stratified | See `configs/pipeline.yaml` | |
|
|
| **Optuna hyperparameters (LR):** `C≈0.32`, `max_features=4045`, bigrams `(1,2)`, `min_df=2`. |
|
|
| ## Comparison table |
|
|
| | Model | Family | F1 (test) | ROC-AUC | FP | FN | Default in API/UI | |
| |-------|--------|-----------|---------|----|----|-------------------| |
| | LR + TF-IDF (tuned) | sklearn | 0.7579 | 0.81 | 18 | 30 | Yes | |
| | LR + TF-IDF (local) | sklearn | 0.7579 | 0.81 | 18 | 30 | Yes (`final_model.joblib`) | |
| | Random Forest | sklearn | — | — | — | — | Run pipeline `--model rf` | |
| | XGBoost | sklearn | — | — | — | — | Run pipeline `--model xgboost` | |
| | DistilBERT Toxicity | Hugging Face | — | — | — | — | Optional (`PUT /model/...`) | |
| | toxic-bert (multilabel) | Hugging Face | — | — | — | — | Optional | |
| | RoBERTa Toxicity | Hugging Face | — | — | — | — | Optional | |
|
|
| Rows with empty metrics are placeholders until you run the pipeline or evaluate HF models on the same test split. |
|
|
| ## How to refresh metrics |
|
|
| ```bash |
| python -m src.pipeline.run_pipeline --model lr |
| python -m src.pipeline.run_pipeline --model rf |
| python -m src.pipeline.run_pipeline --model xgboost |
| ``` |
|
|
| Each run appends/updates [`reports/summary.csv`](../reports/summary.csv) and writes: |
|
|
| - `reports/pipeline/{model}/cm_{model}.png` |
| - `reports/pipeline/{model}/roc_{model}.png` |
| - `reports/pipeline/{model}/errors_{model}.csv` |
|
|
| ## EDA and experiments |
|
|
| Additional figures (notebooks): `reports/v2/` — label distribution, TF-IDF features, ensemble charts, transformer confusion matrices (`nb08_*`). |
|
|
| ## Error analysis |
|
|
| The evaluator prints and saves: |
|
|
| - **Most common terms** in false positives and false negatives |
| - Example comments with highest/lowest toxic probability among errors |
|
|
| See `reports/pipeline/*/errors_*.csv` after a pipeline run. |
|
|
