File size: 5,276 Bytes
fb62a11
950f42d
 
 
 
 
 
7912365
950f42d
fb62a11
 
 
 
 
 
 
 
 
 
 
 
 
7912365
fb62a11
 
 
 
 
950f42d
fb62a11
 
7912365
 
 
 
 
 
 
 
 
 
950f42d
 
 
fb62a11
 
 
950f42d
fb62a11
 
 
950f42d
fb62a11
 
950f42d
fb62a11
 
7912365
fb62a11
 
 
 
 
 
950f42d
fb62a11
950f42d
 
 
 
 
 
fb62a11
7912365
 
fb62a11
950f42d
 
 
 
 
 
fb62a11
7912365
 
fb62a11
950f42d
 
 
 
 
 
 
fb62a11
 
 
 
 
 
 
 
 
 
950f42d
 
 
 
 
 
 
 
 
 
fb62a11
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
"""
precomputed.py β€” Evaluation results computed on Google Colab (T4 GPU).

All three strategies were evaluated on the same 100 balanced samples
(50 toxic + 50 non-toxic) from google/civil_comments test split.

ML Classifier : martin-ha/toxic-comment-model (DistilBERT)
LLM           : Qwen/Qwen2.5-14B-Instruct (NF4 quantization)
Grey Zone     : [0.15, 0.85]
"""

# ══════════════════════════════════════════════════════════════
# System configuration (shown at top of Evaluation tab)
# ══════════════════════════════════════════════════════════════
SYSTEM_CONFIG_MD = """\
πŸ“¦ **Evaluation Environment**

| Setting | Value |
|:--------|:------|
| **Dataset** | google/civil_comments (test split) |
| **Eval Samples** | 100 (50 toxic + 50 non-toxic) |
| **ML Classifier** | martin-ha/toxic-comment-model (DistilBERT) |
| **LLM** | Qwen/Qwen2.5-14B-Instruct (NF4, ~8 GB VRAM) |
| **Grey Zone** | [0.15, 0.85] β€” ML uncertain β†’ escalate to LLM |
| **Runtime** | Google Colab, NVIDIA T4 (15 GB VRAM) |
"""

# ══════════════════════════════════════════════════════════════
# Strategy comparison table
# ══════════════════════════════════════════════════════════════
RESULTS_TABLE_MD = """\
| Metric | ML-Only (DistilBERT) | LLM-Only (Qwen2.5-14B) | Hybrid (ML→Qwen2.5-14B) |
|:-------|---------------------:|------------------------:|-------------------------:|
| **F1 (macro)** | 0.7803 | 0.7898 | **0.8368** |
| **F1 (toxic)** | 0.7342 | 0.7961 | **0.8140** |
| **Precision** | **1.0000** | 0.7736 | 0.9722 |
| **Recall** | 0.5800 | **0.8200** | 0.7000 |
| **True Positives** | 29 | **41** | 35 |
| **True Negatives** | **50** | 38 | 49 |
| **False Positives** | **0** | 12 | 1 |
| **False Negatives** | 21 | **9** | 15 |
| **LLM Calls** | 0 | 100 | 12 |
| **LLM Call %** | 0% | 100% | 12% |
| **Est. Cost (USD)** | $0.0000 | $0.0366 | $0.0050 |
"""

# ══════════════════════════════════════════════════════════════
# Key findings and per-strategy analysis
# ══════════════════════════════════════════════════════════════
KEY_FINDINGS_MD = """\
#### πŸ“‰ LLM Call Reduction
> Hybrid used **12** LLM calls (12%) vs LLM-Only's 100 calls (100%).

#### πŸ’° Cost Savings
> Hybrid: **$0.0050** vs LLM-Only: **$0.0366** β†’ **7.3Γ— cheaper**

#### 🎯 Best F1 (macro)
> **Hybrid (ML→Qwen2.5-14B-Instruct)** achieved the highest F1 of **0.8368**

---

#### Per-Strategy Analysis

**1. ML-Only (DistilBERT)**
> F1 = 0.7803 Β· Precision = 1.00 Β· Recall = 0.58
>
> DistilBERT was fine-tuned on Jigsaw toxic-comment data and has
> learned a strong decision boundary for patterns present in its
> training distribution. Comments that clearly match (or clearly
> don't match) those patterns are classified confidently, while
> subtler or out-of-distribution toxicity may be harder to
> catch. Zero LLM cost makes this the cheapest strategy.

**2. LLM-Only (Qwen2.5-14B-Instruct)**
> F1 = 0.7898 Β· Precision = 0.77 Β· Recall = 0.82
>
> The instruction-tuned LLM applies broad world knowledge and
> pragmatic understanding to every comment, producing a
> precision–recall trade-off that differs from the fine-tuned
> classifier. This generality comes at the highest operational
> cost: every comment requires a full autoregressive generation
> pass, making this the most expensive strategy.

**3. Hybrid (ML β†’ Qwen2.5-14B-Instruct)** ⭐
> F1 = 0.8368 Β· Precision = 0.97 Β· Recall = 0.70
>
> The hybrid routes only grey-zone comments β€” those where the
> ML classifier's confidence falls in [0.15, 0.85] β€” to
> the LLM (12% of the evaluation set). Confident ML
> predictions are accepted directly; uncertain ones get a second
> opinion from the LLM. This architectural separation yields
> 7.3Γ— lower cost than LLM-Only while using 12 LLM
> calls instead of 100.

---

#### Conclusion

The hybrid architecture validates the core thesis: intelligent routing
dramatically reduces LLM calls and cost while maintaining competitive
classification quality. The ML classifier handles confident predictions
efficiently, and the LLM reviews only the ambiguous ones β€” combining
both strengths at a fraction of the LLM-Only cost.

All three strategies were evaluated on identical footing β€” the only
variable is the routing logic (ML-only vs LLM-only vs ML→LLM
escalation).

For production deployment, the system could be improved by:
- Using few-shot examples in the LLM prompt
- Fine-tuning the LLM on toxic-comment data
- Using a chain-of-thought prompt for nuanced cases
- Tuning the confidence thresholds on a validation set
"""