Spaces:
Running on Zero
Running on Zero
File size: 5,276 Bytes
fb62a11 950f42d 7912365 950f42d fb62a11 7912365 fb62a11 950f42d fb62a11 7912365 950f42d fb62a11 950f42d fb62a11 950f42d fb62a11 950f42d fb62a11 7912365 fb62a11 950f42d fb62a11 950f42d fb62a11 7912365 fb62a11 950f42d fb62a11 7912365 fb62a11 950f42d fb62a11 950f42d fb62a11 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 | """
precomputed.py β Evaluation results computed on Google Colab (T4 GPU).
All three strategies were evaluated on the same 100 balanced samples
(50 toxic + 50 non-toxic) from google/civil_comments test split.
ML Classifier : martin-ha/toxic-comment-model (DistilBERT)
LLM : Qwen/Qwen2.5-14B-Instruct (NF4 quantization)
Grey Zone : [0.15, 0.85]
"""
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
# System configuration (shown at top of Evaluation tab)
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
SYSTEM_CONFIG_MD = """\
π¦ **Evaluation Environment**
| Setting | Value |
|:--------|:------|
| **Dataset** | google/civil_comments (test split) |
| **Eval Samples** | 100 (50 toxic + 50 non-toxic) |
| **ML Classifier** | martin-ha/toxic-comment-model (DistilBERT) |
| **LLM** | Qwen/Qwen2.5-14B-Instruct (NF4, ~8 GB VRAM) |
| **Grey Zone** | [0.15, 0.85] β ML uncertain β escalate to LLM |
| **Runtime** | Google Colab, NVIDIA T4 (15 GB VRAM) |
"""
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
# Strategy comparison table
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
RESULTS_TABLE_MD = """\
| Metric | ML-Only (DistilBERT) | LLM-Only (Qwen2.5-14B) | Hybrid (MLβQwen2.5-14B) |
|:-------|---------------------:|------------------------:|-------------------------:|
| **F1 (macro)** | 0.7803 | 0.7898 | **0.8368** |
| **F1 (toxic)** | 0.7342 | 0.7961 | **0.8140** |
| **Precision** | **1.0000** | 0.7736 | 0.9722 |
| **Recall** | 0.5800 | **0.8200** | 0.7000 |
| **True Positives** | 29 | **41** | 35 |
| **True Negatives** | **50** | 38 | 49 |
| **False Positives** | **0** | 12 | 1 |
| **False Negatives** | 21 | **9** | 15 |
| **LLM Calls** | 0 | 100 | 12 |
| **LLM Call %** | 0% | 100% | 12% |
| **Est. Cost (USD)** | $0.0000 | $0.0366 | $0.0050 |
"""
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
# Key findings and per-strategy analysis
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
KEY_FINDINGS_MD = """\
#### π LLM Call Reduction
> Hybrid used **12** LLM calls (12%) vs LLM-Only's 100 calls (100%).
#### π° Cost Savings
> Hybrid: **$0.0050** vs LLM-Only: **$0.0366** β **7.3Γ cheaper**
#### π― Best F1 (macro)
> **Hybrid (MLβQwen2.5-14B-Instruct)** achieved the highest F1 of **0.8368**
---
#### Per-Strategy Analysis
**1. ML-Only (DistilBERT)**
> F1 = 0.7803 Β· Precision = 1.00 Β· Recall = 0.58
>
> DistilBERT was fine-tuned on Jigsaw toxic-comment data and has
> learned a strong decision boundary for patterns present in its
> training distribution. Comments that clearly match (or clearly
> don't match) those patterns are classified confidently, while
> subtler or out-of-distribution toxicity may be harder to
> catch. Zero LLM cost makes this the cheapest strategy.
**2. LLM-Only (Qwen2.5-14B-Instruct)**
> F1 = 0.7898 Β· Precision = 0.77 Β· Recall = 0.82
>
> The instruction-tuned LLM applies broad world knowledge and
> pragmatic understanding to every comment, producing a
> precisionβrecall trade-off that differs from the fine-tuned
> classifier. This generality comes at the highest operational
> cost: every comment requires a full autoregressive generation
> pass, making this the most expensive strategy.
**3. Hybrid (ML β Qwen2.5-14B-Instruct)** β
> F1 = 0.8368 Β· Precision = 0.97 Β· Recall = 0.70
>
> The hybrid routes only grey-zone comments β those where the
> ML classifier's confidence falls in [0.15, 0.85] β to
> the LLM (12% of the evaluation set). Confident ML
> predictions are accepted directly; uncertain ones get a second
> opinion from the LLM. This architectural separation yields
> 7.3Γ lower cost than LLM-Only while using 12 LLM
> calls instead of 100.
---
#### Conclusion
The hybrid architecture validates the core thesis: intelligent routing
dramatically reduces LLM calls and cost while maintaining competitive
classification quality. The ML classifier handles confident predictions
efficiently, and the LLM reviews only the ambiguous ones β combining
both strengths at a fraction of the LLM-Only cost.
All three strategies were evaluated on identical footing β the only
variable is the routing logic (ML-only vs LLM-only vs MLβLLM
escalation).
For production deployment, the system could be improved by:
- Using few-shot examples in the LLM prompt
- Fine-tuning the LLM on toxic-comment data
- Using a chain-of-thought prompt for nuanced cases
- Tuning the confidence thresholds on a validation set
""" |