Spaces:
Running on Zero
Running on Zero
Download precomputed.py from VanillaVector/Hybrid-Comment-Moderation: direct link, hf CLI and curl.
- Browser
- Download file 5.28 kB
-
https://huggingface.co/spaces/VanillaVector/Hybrid-Comment-Moderation/resolve/main/precomputed.py
- Command line
-
hf download hf://spaces/VanillaVector/Hybrid-Comment-Moderation/precomputed.py
-
curl -L -o precomputed.py https://huggingface.co/spaces/VanillaVector/Hybrid-Comment-Moderation/resolve/main/precomputed.py
5.28 kB
| """ | |
| precomputed.py β Evaluation results computed on Google Colab (T4 GPU). | |
| All three strategies were evaluated on the same 100 balanced samples | |
| (50 toxic + 50 non-toxic) from google/civil_comments test split. | |
| ML Classifier : martin-ha/toxic-comment-model (DistilBERT) | |
| LLM : Qwen/Qwen2.5-14B-Instruct (NF4 quantization) | |
| Grey Zone : [0.15, 0.85] | |
| """ | |
| # ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| # System configuration (shown at top of Evaluation tab) | |
| # ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| SYSTEM_CONFIG_MD = """\ | |
| π¦ **Evaluation Environment** | |
| | Setting | Value | | |
| |:--------|:------| | |
| | **Dataset** | google/civil_comments (test split) | | |
| | **Eval Samples** | 100 (50 toxic + 50 non-toxic) | | |
| | **ML Classifier** | martin-ha/toxic-comment-model (DistilBERT) | | |
| | **LLM** | Qwen/Qwen2.5-14B-Instruct (NF4, ~8 GB VRAM) | | |
| | **Grey Zone** | [0.15, 0.85] β ML uncertain β escalate to LLM | | |
| | **Runtime** | Google Colab, NVIDIA T4 (15 GB VRAM) | | |
| """ | |
| # ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| # Strategy comparison table | |
| # ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| RESULTS_TABLE_MD = """\ | |
| | Metric | ML-Only (DistilBERT) | LLM-Only (Qwen2.5-14B) | Hybrid (MLβQwen2.5-14B) | | |
| |:-------|---------------------:|------------------------:|-------------------------:| | |
| | **F1 (macro)** | 0.7803 | 0.7898 | **0.8368** | | |
| | **F1 (toxic)** | 0.7342 | 0.7961 | **0.8140** | | |
| | **Precision** | **1.0000** | 0.7736 | 0.9722 | | |
| | **Recall** | 0.5800 | **0.8200** | 0.7000 | | |
| | **True Positives** | 29 | **41** | 35 | | |
| | **True Negatives** | **50** | 38 | 49 | | |
| | **False Positives** | **0** | 12 | 1 | | |
| | **False Negatives** | 21 | **9** | 15 | | |
| | **LLM Calls** | 0 | 100 | 12 | | |
| | **LLM Call %** | 0% | 100% | 12% | | |
| | **Est. Cost (USD)** | $0.0000 | $0.0366 | $0.0050 | | |
| """ | |
| # ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| # Key findings and per-strategy analysis | |
| # ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| KEY_FINDINGS_MD = """\ | |
| #### π LLM Call Reduction | |
| > Hybrid used **12** LLM calls (12%) vs LLM-Only's 100 calls (100%). | |
| #### π° Cost Savings | |
| > Hybrid: **$0.0050** vs LLM-Only: **$0.0366** β **7.3Γ cheaper** | |
| #### π― Best F1 (macro) | |
| > **Hybrid (MLβQwen2.5-14B-Instruct)** achieved the highest F1 of **0.8368** | |
| --- | |
| #### Per-Strategy Analysis | |
| **1. ML-Only (DistilBERT)** | |
| > F1 = 0.7803 Β· Precision = 1.00 Β· Recall = 0.58 | |
| > | |
| > DistilBERT was fine-tuned on Jigsaw toxic-comment data and has | |
| > learned a strong decision boundary for patterns present in its | |
| > training distribution. Comments that clearly match (or clearly | |
| > don't match) those patterns are classified confidently, while | |
| > subtler or out-of-distribution toxicity may be harder to | |
| > catch. Zero LLM cost makes this the cheapest strategy. | |
| **2. LLM-Only (Qwen2.5-14B-Instruct)** | |
| > F1 = 0.7898 Β· Precision = 0.77 Β· Recall = 0.82 | |
| > | |
| > The instruction-tuned LLM applies broad world knowledge and | |
| > pragmatic understanding to every comment, producing a | |
| > precisionβrecall trade-off that differs from the fine-tuned | |
| > classifier. This generality comes at the highest operational | |
| > cost: every comment requires a full autoregressive generation | |
| > pass, making this the most expensive strategy. | |
| **3. Hybrid (ML β Qwen2.5-14B-Instruct)** β | |
| > F1 = 0.8368 Β· Precision = 0.97 Β· Recall = 0.70 | |
| > | |
| > The hybrid routes only grey-zone comments β those where the | |
| > ML classifier's confidence falls in [0.15, 0.85] β to | |
| > the LLM (12% of the evaluation set). Confident ML | |
| > predictions are accepted directly; uncertain ones get a second | |
| > opinion from the LLM. This architectural separation yields | |
| > 7.3Γ lower cost than LLM-Only while using 12 LLM | |
| > calls instead of 100. | |
| --- | |
| #### Conclusion | |
| The hybrid architecture validates the core thesis: intelligent routing | |
| dramatically reduces LLM calls and cost while maintaining competitive | |
| classification quality. The ML classifier handles confident predictions | |
| efficiently, and the LLM reviews only the ambiguous ones β combining | |
| both strengths at a fraction of the LLM-Only cost. | |
| All three strategies were evaluated on identical footing β the only | |
| variable is the routing logic (ML-only vs LLM-only vs MLβLLM | |
| escalation). | |
| For production deployment, the system could be improved by: | |
| - Using few-shot examples in the LLM prompt | |
| - Fine-tuning the LLM on toxic-comment data | |
| - Using a chain-of-thought prompt for nuanced cases | |
| - Tuning the confidence thresholds on a validation set | |
| """ |