Spaces:
Running on Zero
Running on Zero
Upload precomputed.py
Browse files- precomputed.py +57 -36
precomputed.py
CHANGED
|
@@ -1,17 +1,12 @@
|
|
| 1 |
"""
|
| 2 |
-
precomputed.py β
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
β 2. Replace every _TBD_ below with the actual numbers β
|
| 11 |
-
β from your df_results / Cell 15 output. β
|
| 12 |
-
β β
|
| 13 |
-
β 3. Place the two PNG files in assets/ β
|
| 14 |
-
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 15 |
"""
|
| 16 |
|
| 17 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
|
@@ -31,55 +26,71 @@ SYSTEM_CONFIG_MD = """\
|
|
| 31 |
"""
|
| 32 |
|
| 33 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 34 |
-
#
|
| 35 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 36 |
RESULTS_TABLE_MD = """\
|
| 37 |
| Metric | ML-Only (DistilBERT) | LLM-Only (Qwen2.5-3B) | Hybrid (MLβQwen2.5-3B) |
|
| 38 |
|:-------|---------------------:|-----------------------:|------------------------:|
|
| 39 |
-
| **F1 (macro)** |
|
| 40 |
-
| **F1 (toxic)** |
|
| 41 |
-
| **Precision** |
|
| 42 |
-
| **Recall** |
|
| 43 |
-
| **True Positives** |
|
| 44 |
-
| **True Negatives** |
|
| 45 |
-
| **False Positives** |
|
| 46 |
-
| **False Negatives** |
|
| 47 |
-
| **LLM Calls** | 0 | 100 |
|
| 48 |
-
| **LLM Call %** | 0% | 100% |
|
| 49 |
-
| **Est. Cost (USD)** | $0.0000 | $
|
| 50 |
"""
|
| 51 |
|
| 52 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 53 |
-
# Key findings
|
| 54 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 55 |
KEY_FINDINGS_MD = """\
|
| 56 |
#### π LLM Call Reduction
|
| 57 |
-
> Hybrid used **
|
| 58 |
|
| 59 |
#### π° Cost Savings
|
| 60 |
-
> Hybrid: **$
|
| 61 |
|
| 62 |
#### π― Best F1 (macro)
|
| 63 |
-
> **
|
| 64 |
|
| 65 |
---
|
| 66 |
|
| 67 |
#### Per-Strategy Analysis
|
| 68 |
|
| 69 |
**1. ML-Only (DistilBERT)**
|
| 70 |
-
> F1 =
|
| 71 |
>
|
| 72 |
-
>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
|
| 74 |
**2. LLM-Only (Qwen2.5-3B-Instruct)**
|
| 75 |
-
> F1 =
|
| 76 |
>
|
| 77 |
-
>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
|
| 79 |
-
**3. Hybrid (ML β Qwen2.5-3B-Instruct)**
|
| 80 |
-
> F1 =
|
| 81 |
>
|
| 82 |
-
>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 83 |
|
| 84 |
---
|
| 85 |
|
|
@@ -90,4 +101,14 @@ dramatically reduces LLM calls and cost while maintaining competitive
|
|
| 90 |
classification quality. The ML classifier handles confident predictions
|
| 91 |
efficiently, and the LLM reviews only the ambiguous ones β combining
|
| 92 |
both strengths at a fraction of the LLM-Only cost.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 93 |
"""
|
|
|
|
| 1 |
"""
|
| 2 |
+
precomputed.py β Evaluation results computed on Google Colab (T4 GPU).
|
| 3 |
+
|
| 4 |
+
All three strategies were evaluated on the same 100 balanced samples
|
| 5 |
+
(50 toxic + 50 non-toxic) from google/civil_comments test split.
|
| 6 |
+
|
| 7 |
+
ML Classifier : martin-ha/toxic-comment-model (DistilBERT)
|
| 8 |
+
LLM : Qwen/Qwen2.5-3B-Instruct (NF4 quantization)
|
| 9 |
+
Grey Zone : [0.15, 0.85]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
"""
|
| 11 |
|
| 12 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
|
|
|
| 26 |
"""
|
| 27 |
|
| 28 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 29 |
+
# Strategy comparison table
|
| 30 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 31 |
RESULTS_TABLE_MD = """\
|
| 32 |
| Metric | ML-Only (DistilBERT) | LLM-Only (Qwen2.5-3B) | Hybrid (MLβQwen2.5-3B) |
|
| 33 |
|:-------|---------------------:|-----------------------:|------------------------:|
|
| 34 |
+
| **F1 (macro)** | 0.7803 | 0.7064 | **0.7933** |
|
| 35 |
+
| **F1 (toxic)** | 0.7342 | 0.6742 | **0.7561** |
|
| 36 |
+
| **Precision** | **1.0000** | 0.7692 | 0.9687 |
|
| 37 |
+
| **Recall** | 0.5800 | 0.6000 | **0.6200** |
|
| 38 |
+
| **True Positives** | 29 | 30 | **31** |
|
| 39 |
+
| **True Negatives** | **50** | 41 | 49 |
|
| 40 |
+
| **False Positives** | **0** | 9 | 1 |
|
| 41 |
+
| **False Negatives** | 21 | 20 | **19** |
|
| 42 |
+
| **LLM Calls** | 0 | 100 | 12 |
|
| 43 |
+
| **LLM Call %** | 0% | 100% | 12% |
|
| 44 |
+
| **Est. Cost (USD)** | $0.0000 | $0.0366 | $0.0050 |
|
| 45 |
"""
|
| 46 |
|
| 47 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 48 |
+
# Key findings and per-strategy analysis
|
| 49 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 50 |
KEY_FINDINGS_MD = """\
|
| 51 |
#### π LLM Call Reduction
|
| 52 |
+
> Hybrid used **12** LLM calls (12%) vs LLM-Only's 100 calls (100%).
|
| 53 |
|
| 54 |
#### π° Cost Savings
|
| 55 |
+
> Hybrid: **$0.0050** vs LLM-Only: **$0.0366** β **7.3Γ cheaper**
|
| 56 |
|
| 57 |
#### π― Best F1 (macro)
|
| 58 |
+
> **Hybrid (MLβQwen2.5-3B-Instruct)** achieved the highest F1 of **0.7933**
|
| 59 |
|
| 60 |
---
|
| 61 |
|
| 62 |
#### Per-Strategy Analysis
|
| 63 |
|
| 64 |
**1. ML-Only (DistilBERT)**
|
| 65 |
+
> F1 = 0.7803 Β· Precision = 1.00 Β· Recall = 0.58
|
| 66 |
>
|
| 67 |
+
> DistilBERT was fine-tuned on Jigsaw toxic-comment data and has
|
| 68 |
+
> learned a strong decision boundary for patterns present in its
|
| 69 |
+
> training distribution. Comments that clearly match (or clearly
|
| 70 |
+
> don't match) those patterns are classified confidently, while
|
| 71 |
+
> subtler or out-of-distribution toxicity may be harder to
|
| 72 |
+
> catch. Zero LLM cost makes this the cheapest strategy.
|
| 73 |
|
| 74 |
**2. LLM-Only (Qwen2.5-3B-Instruct)**
|
| 75 |
+
> F1 = 0.7064 Β· Precision = 0.77 Β· Recall = 0.60
|
| 76 |
>
|
| 77 |
+
> The instruction-tuned LLM applies broad world knowledge and
|
| 78 |
+
> pragmatic understanding to every comment, producing a
|
| 79 |
+
> precisionβrecall trade-off that differs from the fine-tuned
|
| 80 |
+
> classifier. This generality comes at the highest operational
|
| 81 |
+
> cost: every comment requires a full autoregressive generation
|
| 82 |
+
> pass, making this the most expensive strategy.
|
| 83 |
|
| 84 |
+
**3. Hybrid (ML β Qwen2.5-3B-Instruct)** β
|
| 85 |
+
> F1 = 0.7933 Β· Precision = 0.97 Β· Recall = 0.62
|
| 86 |
>
|
| 87 |
+
> The hybrid routes only grey-zone comments β those where the
|
| 88 |
+
> ML classifier's confidence falls in [0.15, 0.85] β to
|
| 89 |
+
> the LLM (12% of the evaluation set). Confident ML
|
| 90 |
+
> predictions are accepted directly; uncertain ones get a second
|
| 91 |
+
> opinion from the LLM. This architectural separation yields
|
| 92 |
+
> 7.3Γ lower cost than LLM-Only while using 12 LLM
|
| 93 |
+
> calls instead of 100.
|
| 94 |
|
| 95 |
---
|
| 96 |
|
|
|
|
| 101 |
classification quality. The ML classifier handles confident predictions
|
| 102 |
efficiently, and the LLM reviews only the ambiguous ones β combining
|
| 103 |
both strengths at a fraction of the LLM-Only cost.
|
| 104 |
+
|
| 105 |
+
All three strategies were evaluated on identical footing β the only
|
| 106 |
+
variable is the routing logic (ML-only vs LLM-only vs MLβLLM
|
| 107 |
+
escalation).
|
| 108 |
+
|
| 109 |
+
For production deployment, the system could be improved by:
|
| 110 |
+
- Using few-shot examples in the LLM prompt
|
| 111 |
+
- Fine-tuning the LLM on toxic-comment data
|
| 112 |
+
- Using a chain-of-thought prompt for nuanced cases
|
| 113 |
+
- Tuning the confidence thresholds on a validation set
|
| 114 |
"""
|