VanillaVector commited on
Commit
950f42d
Β·
verified Β·
1 Parent(s): fb62a11

Upload precomputed.py

Browse files
Files changed (1) hide show
  1. precomputed.py +57 -36
precomputed.py CHANGED
@@ -1,17 +1,12 @@
1
  """
2
- precomputed.py β€” Pre-computed evaluation results from Colab.
3
-
4
- ╔══════════════════════════════════════════════════════════════╗
5
- β•‘ INSTRUCTIONS β•‘
6
- β•‘ β•‘
7
- β•‘ 1. Run the Colab notebook end-to-end with β•‘
8
- β•‘ LLM_MODEL_NAME = "Qwen/Qwen2.5-3B-Instruct" β•‘
9
- β•‘ β•‘
10
- β•‘ 2. Replace every _TBD_ below with the actual numbers β•‘
11
- β•‘ from your df_results / Cell 15 output. β•‘
12
- β•‘ β•‘
13
- β•‘ 3. Place the two PNG files in assets/ β•‘
14
- β•šβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•
15
  """
16
 
17
  # ══════════════════════════════════════════════════════════════
@@ -31,55 +26,71 @@ SYSTEM_CONFIG_MD = """\
31
  """
32
 
33
  # ══════════════════════════════════════════════════════════════
34
- # Results table β€” replace _TBD_ with actual values
35
  # ══════════════════════════════════════════════════════════════
36
  RESULTS_TABLE_MD = """\
37
  | Metric | ML-Only (DistilBERT) | LLM-Only (Qwen2.5-3B) | Hybrid (MLβ†’Qwen2.5-3B) |
38
  |:-------|---------------------:|-----------------------:|------------------------:|
39
- | **F1 (macro)** | _TBD_ | _TBD_ | _TBD_ |
40
- | **F1 (toxic)** | _TBD_ | _TBD_ | _TBD_ |
41
- | **Precision** | _TBD_ | _TBD_ | _TBD_ |
42
- | **Recall** | _TBD_ | _TBD_ | _TBD_ |
43
- | **True Positives** | _TBD_ | _TBD_ | _TBD_ |
44
- | **True Negatives** | _TBD_ | _TBD_ | _TBD_ |
45
- | **False Positives** | _TBD_ | _TBD_ | _TBD_ |
46
- | **False Negatives** | _TBD_ | _TBD_ | _TBD_ |
47
- | **LLM Calls** | 0 | 100 | _TBD_ |
48
- | **LLM Call %** | 0% | 100% | _TBD_% |
49
- | **Est. Cost (USD)** | $0.0000 | $_TBD_ | $_TBD_ |
50
  """
51
 
52
  # ══════════════════════════════════════════════════════════════
53
- # Key findings β€” replace with Cell 15 output
54
  # ══════════════════════════════════════════════════════════════
55
  KEY_FINDINGS_MD = """\
56
  #### πŸ“‰ LLM Call Reduction
57
- > Hybrid used **_TBD_** LLM calls (_TBD_%) vs LLM-Only's 100 calls (100%).
58
 
59
  #### πŸ’° Cost Savings
60
- > Hybrid: **$_TBD_** vs LLM-Only: **$_TBD_** β†’ **_TBD_Γ— cheaper**
61
 
62
  #### 🎯 Best F1 (macro)
63
- > **_TBD_** strategy achieved the highest F1 of **_TBD_**
64
 
65
  ---
66
 
67
  #### Per-Strategy Analysis
68
 
69
  **1. ML-Only (DistilBERT)**
70
- > F1 = _TBD_ Β· Precision = _TBD_ Β· Recall = _TBD_
71
  >
72
- > _Paste your Cell 15 ML-Only interpretation paragraph here._
 
 
 
 
 
73
 
74
  **2. LLM-Only (Qwen2.5-3B-Instruct)**
75
- > F1 = _TBD_ Β· Precision = _TBD_ Β· Recall = _TBD_
76
  >
77
- > _Paste your Cell 15 LLM-Only interpretation paragraph here._
 
 
 
 
 
78
 
79
- **3. Hybrid (ML β†’ Qwen2.5-3B-Instruct)**
80
- > F1 = _TBD_ Β· Precision = _TBD_ Β· Recall = _TBD_
81
  >
82
- > _Paste your Cell 15 Hybrid interpretation paragraph here._
 
 
 
 
 
 
83
 
84
  ---
85
 
@@ -90,4 +101,14 @@ dramatically reduces LLM calls and cost while maintaining competitive
90
  classification quality. The ML classifier handles confident predictions
91
  efficiently, and the LLM reviews only the ambiguous ones β€” combining
92
  both strengths at a fraction of the LLM-Only cost.
 
 
 
 
 
 
 
 
 
 
93
  """
 
1
  """
2
+ precomputed.py β€” Evaluation results computed on Google Colab (T4 GPU).
3
+
4
+ All three strategies were evaluated on the same 100 balanced samples
5
+ (50 toxic + 50 non-toxic) from google/civil_comments test split.
6
+
7
+ ML Classifier : martin-ha/toxic-comment-model (DistilBERT)
8
+ LLM : Qwen/Qwen2.5-3B-Instruct (NF4 quantization)
9
+ Grey Zone : [0.15, 0.85]
 
 
 
 
 
10
  """
11
 
12
  # ══════════════════════════════════════════════════════════════
 
26
  """
27
 
28
  # ══════════════════════════════════════════════════════════════
29
+ # Strategy comparison table
30
  # ══════════════════════════════════════════════════════════════
31
  RESULTS_TABLE_MD = """\
32
  | Metric | ML-Only (DistilBERT) | LLM-Only (Qwen2.5-3B) | Hybrid (MLβ†’Qwen2.5-3B) |
33
  |:-------|---------------------:|-----------------------:|------------------------:|
34
+ | **F1 (macro)** | 0.7803 | 0.7064 | **0.7933** |
35
+ | **F1 (toxic)** | 0.7342 | 0.6742 | **0.7561** |
36
+ | **Precision** | **1.0000** | 0.7692 | 0.9687 |
37
+ | **Recall** | 0.5800 | 0.6000 | **0.6200** |
38
+ | **True Positives** | 29 | 30 | **31** |
39
+ | **True Negatives** | **50** | 41 | 49 |
40
+ | **False Positives** | **0** | 9 | 1 |
41
+ | **False Negatives** | 21 | 20 | **19** |
42
+ | **LLM Calls** | 0 | 100 | 12 |
43
+ | **LLM Call %** | 0% | 100% | 12% |
44
+ | **Est. Cost (USD)** | $0.0000 | $0.0366 | $0.0050 |
45
  """
46
 
47
  # ══════════════════════════════════════════════════════════════
48
+ # Key findings and per-strategy analysis
49
  # ══════════════════════════════════════════════════════════════
50
  KEY_FINDINGS_MD = """\
51
  #### πŸ“‰ LLM Call Reduction
52
+ > Hybrid used **12** LLM calls (12%) vs LLM-Only's 100 calls (100%).
53
 
54
  #### πŸ’° Cost Savings
55
+ > Hybrid: **$0.0050** vs LLM-Only: **$0.0366** β†’ **7.3Γ— cheaper**
56
 
57
  #### 🎯 Best F1 (macro)
58
+ > **Hybrid (ML→Qwen2.5-3B-Instruct)** achieved the highest F1 of **0.7933**
59
 
60
  ---
61
 
62
  #### Per-Strategy Analysis
63
 
64
  **1. ML-Only (DistilBERT)**
65
+ > F1 = 0.7803 Β· Precision = 1.00 Β· Recall = 0.58
66
  >
67
+ > DistilBERT was fine-tuned on Jigsaw toxic-comment data and has
68
+ > learned a strong decision boundary for patterns present in its
69
+ > training distribution. Comments that clearly match (or clearly
70
+ > don't match) those patterns are classified confidently, while
71
+ > subtler or out-of-distribution toxicity may be harder to
72
+ > catch. Zero LLM cost makes this the cheapest strategy.
73
 
74
  **2. LLM-Only (Qwen2.5-3B-Instruct)**
75
+ > F1 = 0.7064 Β· Precision = 0.77 Β· Recall = 0.60
76
  >
77
+ > The instruction-tuned LLM applies broad world knowledge and
78
+ > pragmatic understanding to every comment, producing a
79
+ > precision–recall trade-off that differs from the fine-tuned
80
+ > classifier. This generality comes at the highest operational
81
+ > cost: every comment requires a full autoregressive generation
82
+ > pass, making this the most expensive strategy.
83
 
84
+ **3. Hybrid (ML β†’ Qwen2.5-3B-Instruct)** ⭐
85
+ > F1 = 0.7933 Β· Precision = 0.97 Β· Recall = 0.62
86
  >
87
+ > The hybrid routes only grey-zone comments β€” those where the
88
+ > ML classifier's confidence falls in [0.15, 0.85] β€” to
89
+ > the LLM (12% of the evaluation set). Confident ML
90
+ > predictions are accepted directly; uncertain ones get a second
91
+ > opinion from the LLM. This architectural separation yields
92
+ > 7.3Γ— lower cost than LLM-Only while using 12 LLM
93
+ > calls instead of 100.
94
 
95
  ---
96
 
 
101
  classification quality. The ML classifier handles confident predictions
102
  efficiently, and the LLM reviews only the ambiguous ones β€” combining
103
  both strengths at a fraction of the LLM-Only cost.
104
+
105
+ All three strategies were evaluated on identical footing β€” the only
106
+ variable is the routing logic (ML-only vs LLM-only vs ML→LLM
107
+ escalation).
108
+
109
+ For production deployment, the system could be improved by:
110
+ - Using few-shot examples in the LLM prompt
111
+ - Fine-tuning the LLM on toxic-comment data
112
+ - Using a chain-of-thought prompt for nuanced cases
113
+ - Tuning the confidence thresholds on a validation set
114
  """