Karl Qiu commited on
Commit
7912365
Β·
1 Parent(s): 4cf849c

Switch to Qwen2.5-14B-Instruct from Qwen2.5-3B-Instruct

Browse files
README.md CHANGED
@@ -23,7 +23,7 @@ tags:
23
  A two-stage content moderation pipeline that routes comments intelligently:
24
 
25
  1. **Stage 1 β€” ML Classifier** (DistilBERT, CPU, ~7 ms): handles confident predictions instantly.
26
- 2. **Stage 2 β€” LLM Reviewer** (Qwen2.5-3B-Instruct, GPU, ~1–2 s): reviews only ambiguous comments.
27
 
28
  ## Tabs
29
 
@@ -37,10 +37,11 @@ A two-stage content moderation pipeline that routes comments intelligently:
37
 
38
  - **ML-only** works for everyone (CPU, no sign-in needed).
39
  - **Hybrid** and **LLM-only** require a free HuggingFace account for ZeroGPU access.
 
40
 
41
  ## Reference
42
 
43
  - Textbook: *Understanding Deep Learning* by Simon J.D. Prince
44
  - Dataset: [google/civil_comments](https://huggingface.co/datasets/google/civil_comments)
45
  - ML model: [martin-ha/toxic-comment-model](https://huggingface.co/martin-ha/toxic-comment-model)
46
- - LLM: [Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct)
 
23
  A two-stage content moderation pipeline that routes comments intelligently:
24
 
25
  1. **Stage 1 β€” ML Classifier** (DistilBERT, CPU, ~7 ms): handles confident predictions instantly.
26
+ 2. **Stage 2 β€” LLM Reviewer** (Qwen2.5-14B-Instruct, NF4, GPU, ~1–2 s): reviews only ambiguous comments.
27
 
28
  ## Tabs
29
 
 
37
 
38
  - **ML-only** works for everyone (CPU, no sign-in needed).
39
  - **Hybrid** and **LLM-only** require a free HuggingFace account for ZeroGPU access.
40
+ - First LLM call loads the 14B model (~30–40 s); subsequent calls take ~1–2 s.
41
 
42
  ## Reference
43
 
44
  - Textbook: *Understanding Deep Learning* by Simon J.D. Prince
45
  - Dataset: [google/civil_comments](https://huggingface.co/datasets/google/civil_comments)
46
  - ML model: [martin-ha/toxic-comment-model](https://huggingface.co/martin-ha/toxic-comment-model)
47
+ - LLM: [Qwen/Qwen2.5-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct)
app.py CHANGED
@@ -1,13 +1,24 @@
1
  """
2
  app.py β€” HuggingFace Spaces entry point for Hybrid Comment Moderation.
3
 
 
 
 
 
4
  Three tabs:
5
- πŸ” Live Demo β€” interactive classification (ML/LLM/Hybrid)
6
- πŸ“Š Evaluation β€” pre-computed metrics and charts from Colab
7
- πŸ—οΈ Architecture β€” pipeline diagram and design rationale
8
  """
9
 
 
 
 
 
 
 
10
  import time
 
11
  import gradio as gr
12
  from config import (
13
  CONFIDENCE_LOW,
@@ -93,16 +104,28 @@ def classify_comment(comment: str, strategy: str):
93
  return verdict, details
94
 
95
  except Exception as e:
 
96
  err = str(e).lower()
97
- if any(k in err for k in ["gpu", "quota", "zero", "login", "sign"]):
 
 
 
 
98
  return (
99
  "## ⚠️ GPU Unavailable",
100
- "**Please [sign in to HuggingFace](https://huggingface.co/login)** "
101
- "(free account) to use LLM features.\n\n"
102
- "The **ML-only** strategy works without GPU β€” "
103
- "select it above to try it without signing in."
 
 
 
 
 
104
  )
105
- return "## ❌ Error", f"```\n{e}\n```"
 
 
106
 
107
 
108
  # ══════════════════════════════════════════════════════════════
@@ -129,7 +152,9 @@ with gr.Blocks(
129
  gr.Markdown(
130
  "> πŸ’‘ **ML-only** works for everyone (CPU). \n"
131
  "> **Hybrid** and **LLM-only** need a free HuggingFace "
132
- "account for GPU access.\n"
 
 
133
  )
134
 
135
  with gr.Row():
@@ -257,33 +282,33 @@ with gr.Blocks(
257
  # TAB 3: ARCHITECTURE
258
  # ══════════════════════════════════════════════════════
259
  with gr.Tab("πŸ—οΈ Architecture"):
 
260
  gr.Markdown(
261
  "### Two-Stage Pipeline\n"
262
  "\n"
263
  "```\n"
264
  "User Comment\n"
265
- " β”‚\n"
266
- " β–Ό\n"
267
- "β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”\n"
268
- "β”‚ Stage 1: ML Classifier β”‚\n"
269
- "β”‚ DistilBERT (CPU, ~7 ms per comment) β”‚\n"
270
- "β”‚ Fine-tuned on Jigsaw toxic comments β”‚\n"
271
- "β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜\n"
272
- " β”‚\n"
273
- " β”Œβ”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”\n"
274
- " β”‚ Confident? β”‚\n"
275
- " β””β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”˜\n"
276
- " Yes β”‚ β”‚ No (grey zone)\n"
277
- " β”‚ β”‚\n"
278
- " β–Ό β–Ό\n"
279
- " DONE β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”\n"
280
- f" β”‚ Stage 2: LLM Reviewer β”‚\n"
281
- f" β”‚ {_LLM_SHORT} (GPU, ~1-2 s) β”‚\n"
282
- " β”‚ Instruction-tuned, world knowledgeβ”‚\n"
283
- " β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜\n"
284
- " β”‚\n"
285
- " β–Ό\n"
286
- " DONE\n"
287
  "```\n"
288
  )
289
 
@@ -291,21 +316,21 @@ with gr.Blocks(
291
  "---\n"
292
  "### Confidence Routing Logic\n"
293
  "\n"
294
- "The ML classifier outputs a toxic probability in "
295
- "[0, 1]. Two thresholds define three zones:\n"
296
  "\n"
297
  "| Toxic Probability | Zone | Action |\n"
298
  "|:------------------|:-----|:-------|\n"
299
- f"| **β‰₯ {CONFIDENCE_HIGH}** | High confidence toxic "
300
- "| β†’ **BLOCK** (ML decides) |\n"
301
- f"| **≀ {CONFIDENCE_LOW}** | High confidence safe "
302
- "| β†’ **ALLOW** (ML decides) |\n"
303
- f"| **{CONFIDENCE_LOW} – {CONFIDENCE_HIGH}** | "
304
- "Grey zone (uncertain) | β†’ **Escalate to LLM** |\n"
305
  "\n"
306
  "Only grey-zone comments cost GPU time. On the "
307
- "evaluation set, typically **75–85%** of comments "
308
- "are handled by the ML classifier alone.\n"
309
  )
310
 
311
  gr.Markdown(
@@ -314,10 +339,10 @@ with gr.Blocks(
314
  "\n"
315
  "| Property | ML-Only | LLM-Only | Hybrid |\n"
316
  "|:---------|:--------|:---------|:-------|\n"
317
- "| **Speed** | ~7 ms | ~1–3 s | ~7 ms (most) |\n"
318
  "| **Cost** | $0 (CPU) | High (every comment) "
319
- "| Low (~15–25% reach LLM) |\n"
320
- "| **Nuance** | Limited to training dist. "
321
  "| Broad world knowledge | Best of both |\n"
322
  "| **Scalability** | Excellent | Poor | Good |\n"
323
  "\n"
@@ -330,12 +355,12 @@ with gr.Blocks(
330
  "---\n"
331
  "### Models\n"
332
  "\n"
333
- "| Component | Model | Parameters | Serving |\n"
334
- "|:----------|:------|:-----------|:--------|\n"
335
  f"| ML Classifier | `{CLASSIFIER_MODEL_NAME}` "
336
- "| ~67 M (DistilBERT) | CPU |\n"
337
  f"| LLM Reviewer | `{LLM_MODEL_NAME}` "
338
- "| ~3 B (float16) | ZeroGPU (A100) |\n"
339
  )
340
 
341
  gr.Markdown(
@@ -344,12 +369,12 @@ with gr.Blocks(
344
  "\n"
345
  "- **Course:** CS541 Artificial Intelligence, "
346
  "Stevens Institute of Technology\n"
347
- "- **Textbook:** *Understanding Deep Learning* β€” "
348
- "Simon J.D. Prince\n"
349
  "- **Dataset:** "
350
  "[google/civil_comments]"
351
- "(https://huggingface.co/datasets/google/"
352
- "civil_comments)\n"
353
  )
354
 
355
  # ── Footer ────────────────────────────────────────────────
 
1
  """
2
  app.py β€” HuggingFace Spaces entry point for Hybrid Comment Moderation.
3
 
4
+ `import spaces` MUST be the first import so that ZeroGPU can patch
5
+ torch before any other module loads CUDA. All subsequent imports
6
+ (config, ml_classifier, llm_reviewer) inherit the patched torch.
7
+
8
  Three tabs:
9
+ πŸ” Live Demo β€” interactive classification (ML / LLM / Hybrid)
10
+ πŸ“Š Evaluation β€” pre-computed metrics and charts from Colab
11
+ πŸ—οΈ Architecture β€” pipeline diagram and design rationale
12
  """
13
 
14
+ # ── ZeroGPU patching (must precede every other import) ────────
15
+ try:
16
+ import spaces # noqa: F401
17
+ except ImportError:
18
+ pass
19
+
20
  import time
21
+ import traceback
22
  import gradio as gr
23
  from config import (
24
  CONFIDENCE_LOW,
 
104
  return verdict, details
105
 
106
  except Exception as e:
107
+ tb = traceback.format_exc()
108
  err = str(e).lower()
109
+
110
+ # ── GPU / quota / auth errors β†’ helpful message ───────
111
+ if any(k in err for k in [
112
+ "gpu", "quota", "zero", "login", "sign", "unauthorized",
113
+ ]) or (isinstance(e, RuntimeError) and "gpu" not in err):
114
  return (
115
  "## ⚠️ GPU Unavailable",
116
+ "This can happen if:\n"
117
+ "- You are not signed in to HuggingFace "
118
+ "([sign in here](https://huggingface.co/login) β€” free)\n"
119
+ "- Your daily ZeroGPU quota is exhausted\n"
120
+ "- The GPU is temporarily busy\n\n"
121
+ "**ML-only works without GPU** β€” select "
122
+ "**ML-only (CPU, no login needed)** above.\n\n"
123
+ f"<details><summary>Technical details</summary>\n\n"
124
+ f"```\n{tb}\n```\n</details>",
125
  )
126
+
127
+ # ── All other errors β†’ show full traceback ────────────
128
+ return "## ❌ Error", f"```\n{tb}\n```"
129
 
130
 
131
  # ══════════════════════════════════════════════════════════════
 
152
  gr.Markdown(
153
  "> πŸ’‘ **ML-only** works for everyone (CPU). \n"
154
  "> **Hybrid** and **LLM-only** need a free HuggingFace "
155
+ "account for GPU access. \n"
156
+ f"> First LLM call loads the 14B model (~30–40 s); "
157
+ "subsequent calls take ~1–2 s.\n"
158
  )
159
 
160
  with gr.Row():
 
282
  # TAB 3: ARCHITECTURE
283
  # ══════════════════════════════════════════════════════
284
  with gr.Tab("πŸ—οΈ Architecture"):
285
+
286
  gr.Markdown(
287
  "### Two-Stage Pipeline\n"
288
  "\n"
289
  "```\n"
290
  "User Comment\n"
291
+ " |\n"
292
+ " v\n"
293
+ "+----------------------------------------+\n"
294
+ "| Stage 1: ML Classifier |\n"
295
+ "| DistilBERT (CPU, ~7 ms per comment) |\n"
296
+ "| Fine-tuned on Jigsaw toxic comments |\n"
297
+ "+-------------------+--------------------+\n"
298
+ " |\n"
299
+ " Confident?\n"
300
+ " / \\\n"
301
+ " Yes No (grey zone)\n"
302
+ " | |\n"
303
+ " v v\n"
304
+ " DONE +----------------------------------+\n"
305
+ f" | Stage 2: LLM Reviewer |\n"
306
+ f" | {_LLM_SHORT} (GPU, ~1-2 s) |\n"
307
+ " | 14B params, NF4 quantized |\n"
308
+ " +-----------------+----------------+\n"
309
+ " |\n"
310
+ " v\n"
311
+ " DONE\n"
 
312
  "```\n"
313
  )
314
 
 
316
  "---\n"
317
  "### Confidence Routing Logic\n"
318
  "\n"
319
+ "The ML classifier outputs a toxic probability "
320
+ "in [0, 1]. Two thresholds define three zones:\n"
321
  "\n"
322
  "| Toxic Probability | Zone | Action |\n"
323
  "|:------------------|:-----|:-------|\n"
324
+ f"| **>= {CONFIDENCE_HIGH}** | High confidence "
325
+ "toxic | BLOCK (ML decides) |\n"
326
+ f"| **<= {CONFIDENCE_LOW}** | High confidence "
327
+ "safe | ALLOW (ML decides) |\n"
328
+ f"| **{CONFIDENCE_LOW} - {CONFIDENCE_HIGH}** | "
329
+ "Grey zone | Escalate to LLM |\n"
330
  "\n"
331
  "Only grey-zone comments cost GPU time. On the "
332
+ "evaluation set, **88%** of comments were handled "
333
+ "by the ML classifier alone.\n"
334
  )
335
 
336
  gr.Markdown(
 
339
  "\n"
340
  "| Property | ML-Only | LLM-Only | Hybrid |\n"
341
  "|:---------|:--------|:---------|:-------|\n"
342
+ "| **Speed** | ~7 ms | ~1-2 s | ~7 ms (most) |\n"
343
  "| **Cost** | $0 (CPU) | High (every comment) "
344
+ "| Low (~12% reach LLM) |\n"
345
+ "| **Nuance** | Training dist. only "
346
  "| Broad world knowledge | Best of both |\n"
347
  "| **Scalability** | Excellent | Poor | Good |\n"
348
  "\n"
 
355
  "---\n"
356
  "### Models\n"
357
  "\n"
358
+ "| Component | Model | Parameters | Quantization | Serving |\n"
359
+ "|:----------|:------|:-----------|:-------------|:--------|\n"
360
  f"| ML Classifier | `{CLASSIFIER_MODEL_NAME}` "
361
+ "| ~67 M | None (full precision) | CPU |\n"
362
  f"| LLM Reviewer | `{LLM_MODEL_NAME}` "
363
+ "| ~14 B | NF4 (~8 GB) | ZeroGPU A100 |\n"
364
  )
365
 
366
  gr.Markdown(
 
369
  "\n"
370
  "- **Course:** CS541 Artificial Intelligence, "
371
  "Stevens Institute of Technology\n"
372
+ "- **Textbook:** *Understanding Deep Learning* "
373
+ "by Simon J.D. Prince\n"
374
  "- **Dataset:** "
375
  "[google/civil_comments]"
376
+ "(https://huggingface.co/datasets/"
377
+ "google/civil_comments)\n"
378
  )
379
 
380
  # ── Footer ────────────────────────────────────────────────
assets/stage_distribution.png CHANGED

Git LFS Details

  • SHA256: 0b7681d335380bda11e83d9e2870e2d3896e22e4bcdf6e9c405a1146088605ed
  • Pointer size: 130 Bytes
  • Size of remote file: 44 kB

Git LFS Details

  • SHA256: 0c3538361416f65008dc51d5160ce1565a71ebaa993b8f7bf97c2b9561993f47
  • Pointer size: 130 Bytes
  • Size of remote file: 44 kB
assets/strategy_comparison.png CHANGED

Git LFS Details

  • SHA256: 68a98993d9fe4c42c8b4d8bd44f6956c047fb4c78b6c1c38649720e5b86682c8
  • Pointer size: 131 Bytes
  • Size of remote file: 154 kB

Git LFS Details

  • SHA256: 556e690fada0fa023fb33ae42436d77a21abd119a568f8e77502202399d7ad66
  • Pointer size: 131 Bytes
  • Size of remote file: 156 kB
config.py CHANGED
@@ -8,7 +8,12 @@ DECISION_ALLOW = "ALLOW"
8
 
9
  # ── Model identifiers ────────────────────────────────────────
10
  CLASSIFIER_MODEL_NAME = "martin-ha/toxic-comment-model" # DistilBERT, ~270 MB
11
- LLM_MODEL_NAME = "Qwen/Qwen2.5-3B-Instruct" # fp16, ~6 GB
 
 
 
 
 
12
 
13
  # ── Generation parameters ─────────────────────────────────────
14
  LLM_MAX_NEW_TOKENS = 5
 
8
 
9
  # ── Model identifiers ────────────────────────────────────────
10
  CLASSIFIER_MODEL_NAME = "martin-ha/toxic-comment-model" # DistilBERT, ~270 MB
11
+ LLM_MODEL_NAME = "Qwen/Qwen2.5-14B-Instruct" # NF4, ~8 GB
12
+
13
+ # ── Quantization ──────────────────────────────────────────────
14
+ # NF4 via bitsandbytes compresses 14B params from ~28 GB (bf16)
15
+ # to ~8 GB, fitting comfortably on a ZeroGPU A100 slice.
16
+ LLM_QUANTIZATION = "4bit"
17
 
18
  # ── Generation parameters ─────────────────────────────────────
19
  LLM_MAX_NEW_TOKENS = 5
llm_reviewer.py CHANGED
@@ -1,24 +1,18 @@
1
  """
2
  llm_reviewer.py β€” LLM reviewer for ambiguous comments (Stage 2).
3
 
4
- Qwen2.5-3B-Instruct is loaded in float16 (~6 GB) at module level.
5
- ZeroGPU allocates an A100 slice only when @spaces.GPU functions run.
 
 
 
 
6
 
7
  If the `spaces` package is unavailable (local testing), the GPU
8
  decorator becomes a no-op so the code still works on any machine.
9
  """
10
 
11
- import torch
12
- from typing import Tuple
13
- from transformers import AutoModelForCausalLM, AutoTokenizer
14
- from config import (
15
- LLM_MODEL_NAME,
16
- LLM_MAX_NEW_TOKENS,
17
- DECISION_BLOCK,
18
- DECISION_ALLOW,
19
- )
20
-
21
- # ── ZeroGPU decorator (graceful fallback for local runs) ──────
22
  try:
23
  import spaces
24
  ZERO_GPU = True
@@ -35,6 +29,16 @@ except ImportError:
35
 
36
  spaces = _SpacesFallback()
37
 
 
 
 
 
 
 
 
 
 
 
38
 
39
  # ── Prompt ────────────────────────────────────────────────────
40
  _SYSTEM_PROMPT = (
@@ -46,29 +50,71 @@ _SYSTEM_PROMPT = (
46
  "Do not explain."
47
  )
48
 
49
- # ── Load model at module level ────────────────────────────────
50
- # ZeroGPU patches torch so that device_map="auto" works even
51
- # before GPU is allocated. The actual CUDA tensors are created
52
- # when the @spaces.GPU-decorated function is first called.
53
- print(f"⏳ Loading {LLM_MODEL_NAME} …")
 
 
54
 
55
- _tokenizer = AutoTokenizer.from_pretrained(
56
- LLM_MODEL_NAME, trust_remote_code=True
57
- )
58
- if _tokenizer.pad_token is None:
59
- _tokenizer.pad_token = _tokenizer.eos_token
60
 
61
- _model = AutoModelForCausalLM.from_pretrained(
62
- LLM_MODEL_NAME,
63
- torch_dtype=torch.float16,
64
- device_map="auto",
65
- trust_remote_code=True,
66
- )
67
- print(f"βœ… {LLM_MODEL_NAME} loaded")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
68
 
69
 
70
  # ── Answer parser ─────────────────────────────────────────────
71
  def _parse_answer(answer: str) -> Tuple[str, float]:
 
72
  a = answer.lower().strip().rstrip(".")
73
  if "non-toxic" in a or "non toxic" in a or "not toxic" in a:
74
  return (DECISION_ALLOW, 0.85)
@@ -79,14 +125,22 @@ def _parse_answer(answer: str) -> Tuple[str, float]:
79
 
80
 
81
  # ── GPU-decorated classification function ─────────────────────
82
- @spaces.GPU(duration=30)
83
  def gpu_classify(text: str) -> Tuple[str, float, str]:
84
  """
85
  Classify one comment with the LLM.
86
 
87
  On HuggingFace Spaces this runs on a ZeroGPU A100 slice.
88
- The caller's free-tier quota is consumed (~1–3 GPU-seconds).
 
89
  """
 
 
 
 
 
 
 
90
  messages = [
91
  {"role": "system", "content": _SYSTEM_PROMPT},
92
  {"role": "user", "content": f'Classify this comment: "{text[:1024]}"'},
@@ -104,8 +158,9 @@ def gpu_classify(text: str) -> Tuple[str, float, str]:
104
  )
105
 
106
  new_tokens = output_ids[0][input_ids.shape[-1]:]
107
- answer = _tokenizer.decode(new_tokens, skip_special_tokens=True)
108
- answer = answer.strip().lower()
 
109
 
110
  decision, confidence = _parse_answer(answer)
111
  return decision, confidence, f"LLM said: '{answer}'"
 
1
  """
2
  llm_reviewer.py β€” LLM reviewer for ambiguous comments (Stage 2).
3
 
4
+ Qwen2.5-14B-Instruct is loaded lazily in NF4 quantization via
5
+ bitsandbytes (~8 GB VRAM) on the first ZeroGPU call. The A100
6
+ handles this comfortably alongside DistilBERT on CPU.
7
+
8
+ The model stays in memory after first load, so subsequent calls
9
+ skip the loading step and run inference directly (~1–2 s).
10
 
11
  If the `spaces` package is unavailable (local testing), the GPU
12
  decorator becomes a no-op so the code still works on any machine.
13
  """
14
 
15
+ # ── ZeroGPU must be imported before torch ─────────────────────
 
 
 
 
 
 
 
 
 
 
16
  try:
17
  import spaces
18
  ZERO_GPU = True
 
29
 
30
  spaces = _SpacesFallback()
31
 
32
+ import torch
33
+ from typing import Tuple
34
+ from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
35
+ from config import (
36
+ LLM_MODEL_NAME,
37
+ LLM_QUANTIZATION,
38
+ LLM_MAX_NEW_TOKENS,
39
+ DECISION_BLOCK,
40
+ DECISION_ALLOW,
41
+ )
42
 
43
  # ── Prompt ────────────────────────────────────────────────────
44
  _SYSTEM_PROMPT = (
 
50
  "Do not explain."
51
  )
52
 
53
+ # ── Lazy-loaded model state ───────────────────────────────────
54
+ # The model is loaded on the first @spaces.GPU call, not at
55
+ # import time. This avoids startup failures when GPU is not
56
+ # yet allocated and keeps Space boot time fast.
57
+ _model = None
58
+ _tokenizer = None
59
+ _load_error = None
60
 
 
 
 
 
 
61
 
62
+ def _ensure_model_loaded():
63
+ """Load model and tokenizer once, on first use inside a GPU context."""
64
+ global _model, _tokenizer, _load_error
65
+
66
+ if _model is not None:
67
+ return True
68
+ if _load_error is not None:
69
+ return False
70
+
71
+ try:
72
+ print(f"⏳ Loading {LLM_MODEL_NAME} ({LLM_QUANTIZATION}) …")
73
+
74
+ _tokenizer = AutoTokenizer.from_pretrained(
75
+ LLM_MODEL_NAME, trust_remote_code=True
76
+ )
77
+ if _tokenizer.pad_token is None:
78
+ _tokenizer.pad_token = _tokenizer.eos_token
79
+
80
+ # ── Build quantization config ─────────────────────────
81
+ load_kwargs = {
82
+ "trust_remote_code": True,
83
+ "device_map": "auto",
84
+ }
85
+
86
+ if LLM_QUANTIZATION == "4bit" and torch.cuda.is_available():
87
+ load_kwargs["quantization_config"] = BitsAndBytesConfig(
88
+ load_in_4bit=True,
89
+ bnb_4bit_quant_type="nf4",
90
+ bnb_4bit_compute_dtype=torch.float16,
91
+ bnb_4bit_use_double_quant=True,
92
+ )
93
+ print(" NF4 quantization via bitsandbytes")
94
+ else:
95
+ load_kwargs["torch_dtype"] = torch.float16
96
+ print(" float16 (no quantization)")
97
+
98
+ _model = AutoModelForCausalLM.from_pretrained(
99
+ LLM_MODEL_NAME, **load_kwargs
100
+ )
101
+
102
+ if torch.cuda.is_available():
103
+ allocated = torch.cuda.memory_allocated() / 1e9
104
+ print(f" GPU memory allocated: {allocated:.1f} GB")
105
+
106
+ print(f"βœ… {LLM_MODEL_NAME} loaded successfully")
107
+ return True
108
+
109
+ except Exception as e:
110
+ _load_error = str(e)
111
+ print(f"❌ Failed to load {LLM_MODEL_NAME}: {_load_error}")
112
+ return False
113
 
114
 
115
  # ── Answer parser ─────────────────────────────────────────────
116
  def _parse_answer(answer: str) -> Tuple[str, float]:
117
+ """Map free-text LLM output to a moderation decision."""
118
  a = answer.lower().strip().rstrip(".")
119
  if "non-toxic" in a or "non toxic" in a or "not toxic" in a:
120
  return (DECISION_ALLOW, 0.85)
 
125
 
126
 
127
  # ── GPU-decorated classification function ─────────────────────
128
+ @spaces.GPU(duration=120)
129
  def gpu_classify(text: str) -> Tuple[str, float, str]:
130
  """
131
  Classify one comment with the LLM.
132
 
133
  On HuggingFace Spaces this runs on a ZeroGPU A100 slice.
134
+ The first call includes model loading (~30–40 s for 14B NF4);
135
+ subsequent calls run inference only (~1–2 s).
136
  """
137
+ if not _ensure_model_loaded():
138
+ return (
139
+ DECISION_ALLOW,
140
+ 0.5,
141
+ f"LLM unavailable: {_load_error}",
142
+ )
143
+
144
  messages = [
145
  {"role": "system", "content": _SYSTEM_PROMPT},
146
  {"role": "user", "content": f'Classify this comment: "{text[:1024]}"'},
 
158
  )
159
 
160
  new_tokens = output_ids[0][input_ids.shape[-1]:]
161
+ answer = _tokenizer.decode(
162
+ new_tokens, skip_special_tokens=True
163
+ ).strip().lower()
164
 
165
  decision, confidence = _parse_answer(answer)
166
  return decision, confidence, f"LLM said: '{answer}'"
precomputed.py CHANGED
@@ -5,7 +5,7 @@ All three strategies were evaluated on the same 100 balanced samples
5
  (50 toxic + 50 non-toxic) from google/civil_comments test split.
6
 
7
  ML Classifier : martin-ha/toxic-comment-model (DistilBERT)
8
- LLM : Qwen/Qwen2.5-3B-Instruct (NF4 quantization)
9
  Grey Zone : [0.15, 0.85]
10
  """
11
 
@@ -20,7 +20,7 @@ SYSTEM_CONFIG_MD = """\
20
  | **Dataset** | google/civil_comments (test split) |
21
  | **Eval Samples** | 100 (50 toxic + 50 non-toxic) |
22
  | **ML Classifier** | martin-ha/toxic-comment-model (DistilBERT) |
23
- | **LLM** | Qwen/Qwen2.5-3B-Instruct (NF4, ~2 GB VRAM) |
24
  | **Grey Zone** | [0.15, 0.85] β€” ML uncertain β†’ escalate to LLM |
25
  | **Runtime** | Google Colab, NVIDIA T4 (15 GB VRAM) |
26
  """
@@ -29,16 +29,16 @@ SYSTEM_CONFIG_MD = """\
29
  # Strategy comparison table
30
  # ══════════════════════════════════════════════════════════════
31
  RESULTS_TABLE_MD = """\
32
- | Metric | ML-Only (DistilBERT) | LLM-Only (Qwen2.5-3B) | Hybrid (ML→Qwen2.5-3B) |
33
- |:-------|---------------------:|-----------------------:|------------------------:|
34
- | **F1 (macro)** | 0.7803 | 0.7064 | **0.7933** |
35
- | **F1 (toxic)** | 0.7342 | 0.6742 | **0.7561** |
36
- | **Precision** | **1.0000** | 0.7692 | 0.9687 |
37
- | **Recall** | 0.5800 | 0.6000 | **0.6200** |
38
- | **True Positives** | 29 | 30 | **31** |
39
- | **True Negatives** | **50** | 41 | 49 |
40
- | **False Positives** | **0** | 9 | 1 |
41
- | **False Negatives** | 21 | 20 | **19** |
42
  | **LLM Calls** | 0 | 100 | 12 |
43
  | **LLM Call %** | 0% | 100% | 12% |
44
  | **Est. Cost (USD)** | $0.0000 | $0.0366 | $0.0050 |
@@ -55,7 +55,7 @@ KEY_FINDINGS_MD = """\
55
  > Hybrid: **$0.0050** vs LLM-Only: **$0.0366** β†’ **7.3Γ— cheaper**
56
 
57
  #### 🎯 Best F1 (macro)
58
- > **Hybrid (ML→Qwen2.5-3B-Instruct)** achieved the highest F1 of **0.7933**
59
 
60
  ---
61
 
@@ -71,8 +71,8 @@ KEY_FINDINGS_MD = """\
71
  > subtler or out-of-distribution toxicity may be harder to
72
  > catch. Zero LLM cost makes this the cheapest strategy.
73
 
74
- **2. LLM-Only (Qwen2.5-3B-Instruct)**
75
- > F1 = 0.7064 Β· Precision = 0.77 Β· Recall = 0.60
76
  >
77
  > The instruction-tuned LLM applies broad world knowledge and
78
  > pragmatic understanding to every comment, producing a
@@ -81,8 +81,8 @@ KEY_FINDINGS_MD = """\
81
  > cost: every comment requires a full autoregressive generation
82
  > pass, making this the most expensive strategy.
83
 
84
- **3. Hybrid (ML β†’ Qwen2.5-3B-Instruct)** ⭐
85
- > F1 = 0.7933 Β· Precision = 0.97 Β· Recall = 0.62
86
  >
87
  > The hybrid routes only grey-zone comments β€” those where the
88
  > ML classifier's confidence falls in [0.15, 0.85] β€” to
 
5
  (50 toxic + 50 non-toxic) from google/civil_comments test split.
6
 
7
  ML Classifier : martin-ha/toxic-comment-model (DistilBERT)
8
+ LLM : Qwen/Qwen2.5-14B-Instruct (NF4 quantization)
9
  Grey Zone : [0.15, 0.85]
10
  """
11
 
 
20
  | **Dataset** | google/civil_comments (test split) |
21
  | **Eval Samples** | 100 (50 toxic + 50 non-toxic) |
22
  | **ML Classifier** | martin-ha/toxic-comment-model (DistilBERT) |
23
+ | **LLM** | Qwen/Qwen2.5-14B-Instruct (NF4, ~8 GB VRAM) |
24
  | **Grey Zone** | [0.15, 0.85] β€” ML uncertain β†’ escalate to LLM |
25
  | **Runtime** | Google Colab, NVIDIA T4 (15 GB VRAM) |
26
  """
 
29
  # Strategy comparison table
30
  # ══════════════════════════════════════════════════════════════
31
  RESULTS_TABLE_MD = """\
32
+ | Metric | ML-Only (DistilBERT) | LLM-Only (Qwen2.5-14B) | Hybrid (ML→Qwen2.5-14B) |
33
+ |:-------|---------------------:|------------------------:|-------------------------:|
34
+ | **F1 (macro)** | 0.7803 | 0.7898 | **0.8368** |
35
+ | **F1 (toxic)** | 0.7342 | 0.7961 | **0.8140** |
36
+ | **Precision** | **1.0000** | 0.7736 | 0.9722 |
37
+ | **Recall** | 0.5800 | **0.8200** | 0.7000 |
38
+ | **True Positives** | 29 | **41** | 35 |
39
+ | **True Negatives** | **50** | 38 | 49 |
40
+ | **False Positives** | **0** | 12 | 1 |
41
+ | **False Negatives** | 21 | **9** | 15 |
42
  | **LLM Calls** | 0 | 100 | 12 |
43
  | **LLM Call %** | 0% | 100% | 12% |
44
  | **Est. Cost (USD)** | $0.0000 | $0.0366 | $0.0050 |
 
55
  > Hybrid: **$0.0050** vs LLM-Only: **$0.0366** β†’ **7.3Γ— cheaper**
56
 
57
  #### 🎯 Best F1 (macro)
58
+ > **Hybrid (ML→Qwen2.5-14B-Instruct)** achieved the highest F1 of **0.8368**
59
 
60
  ---
61
 
 
71
  > subtler or out-of-distribution toxicity may be harder to
72
  > catch. Zero LLM cost makes this the cheapest strategy.
73
 
74
+ **2. LLM-Only (Qwen2.5-14B-Instruct)**
75
+ > F1 = 0.7898 Β· Precision = 0.77 Β· Recall = 0.82
76
  >
77
  > The instruction-tuned LLM applies broad world knowledge and
78
  > pragmatic understanding to every comment, producing a
 
81
  > cost: every comment requires a full autoregressive generation
82
  > pass, making this the most expensive strategy.
83
 
84
+ **3. Hybrid (ML β†’ Qwen2.5-14B-Instruct)** ⭐
85
+ > F1 = 0.8368 Β· Precision = 0.97 Β· Recall = 0.70
86
  >
87
  > The hybrid routes only grey-zone comments β€” those where the
88
  > ML classifier's confidence falls in [0.15, 0.85] β€” to
requirements.txt CHANGED
@@ -1,6 +1,7 @@
1
  torch
2
  transformers>=4.52
3
  accelerate
 
4
  gradio>=5.0
5
  spaces
6
  numpy
 
1
  torch
2
  transformers>=4.52
3
  accelerate
4
+ bitsandbytes
5
  gradio>=5.0
6
  spaces
7
  numpy