""" precomputed.py — Evaluation results computed on Google Colab (T4 GPU). All three strategies were evaluated on the same 100 balanced samples (50 toxic + 50 non-toxic) from google/civil_comments test split. ML Classifier : martin-ha/toxic-comment-model (DistilBERT) LLM : Qwen/Qwen2.5-14B-Instruct (NF4 quantization) Grey Zone : [0.15, 0.85] """ # ══════════════════════════════════════════════════════════════ # System configuration (shown at top of Evaluation tab) # ══════════════════════════════════════════════════════════════ SYSTEM_CONFIG_MD = """\ 📦 **Evaluation Environment** | Setting | Value | |:--------|:------| | **Dataset** | google/civil_comments (test split) | | **Eval Samples** | 100 (50 toxic + 50 non-toxic) | | **ML Classifier** | martin-ha/toxic-comment-model (DistilBERT) | | **LLM** | Qwen/Qwen2.5-14B-Instruct (NF4, ~8 GB VRAM) | | **Grey Zone** | [0.15, 0.85] — ML uncertain → escalate to LLM | | **Runtime** | Google Colab, NVIDIA T4 (15 GB VRAM) | """ # ══════════════════════════════════════════════════════════════ # Strategy comparison table # ══════════════════════════════════════════════════════════════ RESULTS_TABLE_MD = """\ | Metric | ML-Only (DistilBERT) | LLM-Only (Qwen2.5-14B) | Hybrid (ML→Qwen2.5-14B) | |:-------|---------------------:|------------------------:|-------------------------:| | **F1 (macro)** | 0.7803 | 0.7898 | **0.8368** | | **F1 (toxic)** | 0.7342 | 0.7961 | **0.8140** | | **Precision** | **1.0000** | 0.7736 | 0.9722 | | **Recall** | 0.5800 | **0.8200** | 0.7000 | | **True Positives** | 29 | **41** | 35 | | **True Negatives** | **50** | 38 | 49 | | **False Positives** | **0** | 12 | 1 | | **False Negatives** | 21 | **9** | 15 | | **LLM Calls** | 0 | 100 | 12 | | **LLM Call %** | 0% | 100% | 12% | | **Est. Cost (USD)** | $0.0000 | $0.0366 | $0.0050 | """ # ══════════════════════════════════════════════════════════════ # Key findings and per-strategy analysis # ══════════════════════════════════════════════════════════════ KEY_FINDINGS_MD = """\ #### 📉 LLM Call Reduction > Hybrid used **12** LLM calls (12%) vs LLM-Only's 100 calls (100%). #### 💰 Cost Savings > Hybrid: **$0.0050** vs LLM-Only: **$0.0366** → **7.3× cheaper** #### 🎯 Best F1 (macro) > **Hybrid (ML→Qwen2.5-14B-Instruct)** achieved the highest F1 of **0.8368** --- #### Per-Strategy Analysis **1. ML-Only (DistilBERT)** > F1 = 0.7803 · Precision = 1.00 · Recall = 0.58 > > DistilBERT was fine-tuned on Jigsaw toxic-comment data and has > learned a strong decision boundary for patterns present in its > training distribution. Comments that clearly match (or clearly > don't match) those patterns are classified confidently, while > subtler or out-of-distribution toxicity may be harder to > catch. Zero LLM cost makes this the cheapest strategy. **2. LLM-Only (Qwen2.5-14B-Instruct)** > F1 = 0.7898 · Precision = 0.77 · Recall = 0.82 > > The instruction-tuned LLM applies broad world knowledge and > pragmatic understanding to every comment, producing a > precision–recall trade-off that differs from the fine-tuned > classifier. This generality comes at the highest operational > cost: every comment requires a full autoregressive generation > pass, making this the most expensive strategy. **3. Hybrid (ML → Qwen2.5-14B-Instruct)** ⭐ > F1 = 0.8368 · Precision = 0.97 · Recall = 0.70 > > The hybrid routes only grey-zone comments — those where the > ML classifier's confidence falls in [0.15, 0.85] — to > the LLM (12% of the evaluation set). Confident ML > predictions are accepted directly; uncertain ones get a second > opinion from the LLM. This architectural separation yields > 7.3× lower cost than LLM-Only while using 12 LLM > calls instead of 100. --- #### Conclusion The hybrid architecture validates the core thesis: intelligent routing dramatically reduces LLM calls and cost while maintaining competitive classification quality. The ML classifier handles confident predictions efficiently, and the LLM reviews only the ambiguous ones — combining both strengths at a fraction of the LLM-Only cost. All three strategies were evaluated on identical footing — the only variable is the routing logic (ML-only vs LLM-only vs ML→LLM escalation). For production deployment, the system could be improved by: - Using few-shot examples in the LLM prompt - Fine-tuning the LLM on toxic-comment data - Using a chain-of-thought prompt for nuanced cases - Tuning the confidence thresholds on a validation set """