Hybrid-Comment-Moderation / precomputed.py
Karl Qiu
Switch to Qwen2.5-14B-Instruct from Qwen2.5-3B-Instruct
7912365
Raw History Blame Contribute Delete
5.28 kB
"""
precomputed.py β€” Evaluation results computed on Google Colab (T4 GPU).
All three strategies were evaluated on the same 100 balanced samples
(50 toxic + 50 non-toxic) from google/civil_comments test split.
ML Classifier : martin-ha/toxic-comment-model (DistilBERT)
LLM : Qwen/Qwen2.5-14B-Instruct (NF4 quantization)
Grey Zone : [0.15, 0.85]
"""
# ══════════════════════════════════════════════════════════════
# System configuration (shown at top of Evaluation tab)
# ══════════════════════════════════════════════════════════════
SYSTEM_CONFIG_MD = """\
πŸ“¦ **Evaluation Environment**
| Setting | Value |
|:--------|:------|
| **Dataset** | google/civil_comments (test split) |
| **Eval Samples** | 100 (50 toxic + 50 non-toxic) |
| **ML Classifier** | martin-ha/toxic-comment-model (DistilBERT) |
| **LLM** | Qwen/Qwen2.5-14B-Instruct (NF4, ~8 GB VRAM) |
| **Grey Zone** | [0.15, 0.85] β€” ML uncertain β†’ escalate to LLM |
| **Runtime** | Google Colab, NVIDIA T4 (15 GB VRAM) |
"""
# ══════════════════════════════════════════════════════════════
# Strategy comparison table
# ══════════════════════════════════════════════════════════════
RESULTS_TABLE_MD = """\
| Metric | ML-Only (DistilBERT) | LLM-Only (Qwen2.5-14B) | Hybrid (ML→Qwen2.5-14B) |
|:-------|---------------------:|------------------------:|-------------------------:|
| **F1 (macro)** | 0.7803 | 0.7898 | **0.8368** |
| **F1 (toxic)** | 0.7342 | 0.7961 | **0.8140** |
| **Precision** | **1.0000** | 0.7736 | 0.9722 |
| **Recall** | 0.5800 | **0.8200** | 0.7000 |
| **True Positives** | 29 | **41** | 35 |
| **True Negatives** | **50** | 38 | 49 |
| **False Positives** | **0** | 12 | 1 |
| **False Negatives** | 21 | **9** | 15 |
| **LLM Calls** | 0 | 100 | 12 |
| **LLM Call %** | 0% | 100% | 12% |
| **Est. Cost (USD)** | $0.0000 | $0.0366 | $0.0050 |
"""
# ══════════════════════════════════════════════════════════════
# Key findings and per-strategy analysis
# ══════════════════════════════════════════════════════════════
KEY_FINDINGS_MD = """\
#### πŸ“‰ LLM Call Reduction
> Hybrid used **12** LLM calls (12%) vs LLM-Only's 100 calls (100%).
#### πŸ’° Cost Savings
> Hybrid: **$0.0050** vs LLM-Only: **$0.0366** β†’ **7.3Γ— cheaper**
#### 🎯 Best F1 (macro)
> **Hybrid (ML→Qwen2.5-14B-Instruct)** achieved the highest F1 of **0.8368**
---
#### Per-Strategy Analysis
**1. ML-Only (DistilBERT)**
> F1 = 0.7803 Β· Precision = 1.00 Β· Recall = 0.58
>
> DistilBERT was fine-tuned on Jigsaw toxic-comment data and has
> learned a strong decision boundary for patterns present in its
> training distribution. Comments that clearly match (or clearly
> don't match) those patterns are classified confidently, while
> subtler or out-of-distribution toxicity may be harder to
> catch. Zero LLM cost makes this the cheapest strategy.
**2. LLM-Only (Qwen2.5-14B-Instruct)**
> F1 = 0.7898 Β· Precision = 0.77 Β· Recall = 0.82
>
> The instruction-tuned LLM applies broad world knowledge and
> pragmatic understanding to every comment, producing a
> precision–recall trade-off that differs from the fine-tuned
> classifier. This generality comes at the highest operational
> cost: every comment requires a full autoregressive generation
> pass, making this the most expensive strategy.
**3. Hybrid (ML β†’ Qwen2.5-14B-Instruct)** ⭐
> F1 = 0.8368 Β· Precision = 0.97 Β· Recall = 0.70
>
> The hybrid routes only grey-zone comments β€” those where the
> ML classifier's confidence falls in [0.15, 0.85] β€” to
> the LLM (12% of the evaluation set). Confident ML
> predictions are accepted directly; uncertain ones get a second
> opinion from the LLM. This architectural separation yields
> 7.3Γ— lower cost than LLM-Only while using 12 LLM
> calls instead of 100.
---
#### Conclusion
The hybrid architecture validates the core thesis: intelligent routing
dramatically reduces LLM calls and cost while maintaining competitive
classification quality. The ML classifier handles confident predictions
efficiently, and the LLM reviews only the ambiguous ones β€” combining
both strengths at a fraction of the LLM-Only cost.
All three strategies were evaluated on identical footing β€” the only
variable is the routing logic (ML-only vs LLM-only vs ML→LLM
escalation).
For production deployment, the system could be improved by:
- Using few-shot examples in the LLM prompt
- Fine-tuning the LLM on toxic-comment data
- Using a chain-of-thought prompt for nuanced cases
- Tuning the confidence thresholds on a validation set
"""