Spaces:
Running on Zero
Running on Zero
Karl Qiu commited on
Commit Β·
7912365
1
Parent(s): 4cf849c
Switch to Qwen2.5-14B-Instruct from Qwen2.5-3B-Instruct
Browse files- README.md +3 -2
- app.py +78 -53
- assets/stage_distribution.png +2 -2
- assets/strategy_comparison.png +2 -2
- config.py +6 -1
- llm_reviewer.py +89 -34
- precomputed.py +17 -17
- requirements.txt +1 -0
README.md
CHANGED
|
@@ -23,7 +23,7 @@ tags:
|
|
| 23 |
A two-stage content moderation pipeline that routes comments intelligently:
|
| 24 |
|
| 25 |
1. **Stage 1 β ML Classifier** (DistilBERT, CPU, ~7 ms): handles confident predictions instantly.
|
| 26 |
-
2. **Stage 2 β LLM Reviewer** (Qwen2.5-
|
| 27 |
|
| 28 |
## Tabs
|
| 29 |
|
|
@@ -37,10 +37,11 @@ A two-stage content moderation pipeline that routes comments intelligently:
|
|
| 37 |
|
| 38 |
- **ML-only** works for everyone (CPU, no sign-in needed).
|
| 39 |
- **Hybrid** and **LLM-only** require a free HuggingFace account for ZeroGPU access.
|
|
|
|
| 40 |
|
| 41 |
## Reference
|
| 42 |
|
| 43 |
- Textbook: *Understanding Deep Learning* by Simon J.D. Prince
|
| 44 |
- Dataset: [google/civil_comments](https://huggingface.co/datasets/google/civil_comments)
|
| 45 |
- ML model: [martin-ha/toxic-comment-model](https://huggingface.co/martin-ha/toxic-comment-model)
|
| 46 |
-
- LLM: [Qwen/Qwen2.5-
|
|
|
|
| 23 |
A two-stage content moderation pipeline that routes comments intelligently:
|
| 24 |
|
| 25 |
1. **Stage 1 β ML Classifier** (DistilBERT, CPU, ~7 ms): handles confident predictions instantly.
|
| 26 |
+
2. **Stage 2 β LLM Reviewer** (Qwen2.5-14B-Instruct, NF4, GPU, ~1β2 s): reviews only ambiguous comments.
|
| 27 |
|
| 28 |
## Tabs
|
| 29 |
|
|
|
|
| 37 |
|
| 38 |
- **ML-only** works for everyone (CPU, no sign-in needed).
|
| 39 |
- **Hybrid** and **LLM-only** require a free HuggingFace account for ZeroGPU access.
|
| 40 |
+
- First LLM call loads the 14B model (~30β40 s); subsequent calls take ~1β2 s.
|
| 41 |
|
| 42 |
## Reference
|
| 43 |
|
| 44 |
- Textbook: *Understanding Deep Learning* by Simon J.D. Prince
|
| 45 |
- Dataset: [google/civil_comments](https://huggingface.co/datasets/google/civil_comments)
|
| 46 |
- ML model: [martin-ha/toxic-comment-model](https://huggingface.co/martin-ha/toxic-comment-model)
|
| 47 |
+
- LLM: [Qwen/Qwen2.5-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct)
|
app.py
CHANGED
|
@@ -1,13 +1,24 @@
|
|
| 1 |
"""
|
| 2 |
app.py β HuggingFace Spaces entry point for Hybrid Comment Moderation.
|
| 3 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
Three tabs:
|
| 5 |
-
π Live Demo
|
| 6 |
-
π Evaluation
|
| 7 |
-
ποΈ Architecture
|
| 8 |
"""
|
| 9 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
import time
|
|
|
|
| 11 |
import gradio as gr
|
| 12 |
from config import (
|
| 13 |
CONFIDENCE_LOW,
|
|
@@ -93,16 +104,28 @@ def classify_comment(comment: str, strategy: str):
|
|
| 93 |
return verdict, details
|
| 94 |
|
| 95 |
except Exception as e:
|
|
|
|
| 96 |
err = str(e).lower()
|
| 97 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
return (
|
| 99 |
"## β οΈ GPU Unavailable",
|
| 100 |
-
"
|
| 101 |
-
"
|
| 102 |
-
"
|
| 103 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 104 |
)
|
| 105 |
-
|
|
|
|
|
|
|
| 106 |
|
| 107 |
|
| 108 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
|
@@ -129,7 +152,9 @@ with gr.Blocks(
|
|
| 129 |
gr.Markdown(
|
| 130 |
"> π‘ **ML-only** works for everyone (CPU). \n"
|
| 131 |
"> **Hybrid** and **LLM-only** need a free HuggingFace "
|
| 132 |
-
"account for GPU access.\n"
|
|
|
|
|
|
|
| 133 |
)
|
| 134 |
|
| 135 |
with gr.Row():
|
|
@@ -257,33 +282,33 @@ with gr.Blocks(
|
|
| 257 |
# TAB 3: ARCHITECTURE
|
| 258 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 259 |
with gr.Tab("ποΈ Architecture"):
|
|
|
|
| 260 |
gr.Markdown(
|
| 261 |
"### Two-Stage Pipeline\n"
|
| 262 |
"\n"
|
| 263 |
"```\n"
|
| 264 |
"User Comment\n"
|
| 265 |
-
"
|
| 266 |
-
"
|
| 267 |
-
"
|
| 268 |
-
"
|
| 269 |
-
"
|
| 270 |
-
"
|
| 271 |
-
"
|
| 272 |
-
"
|
| 273 |
-
"
|
| 274 |
-
"
|
| 275 |
-
"
|
| 276 |
-
"
|
| 277 |
-
"
|
| 278 |
-
"
|
| 279 |
-
"
|
| 280 |
-
f"
|
| 281 |
-
|
| 282 |
-
"
|
| 283 |
-
"
|
| 284 |
-
"
|
| 285 |
-
"
|
| 286 |
-
" DONE\n"
|
| 287 |
"```\n"
|
| 288 |
)
|
| 289 |
|
|
@@ -291,21 +316,21 @@ with gr.Blocks(
|
|
| 291 |
"---\n"
|
| 292 |
"### Confidence Routing Logic\n"
|
| 293 |
"\n"
|
| 294 |
-
"The ML classifier outputs a toxic probability
|
| 295 |
-
"[0, 1]. Two thresholds define three zones:\n"
|
| 296 |
"\n"
|
| 297 |
"| Toxic Probability | Zone | Action |\n"
|
| 298 |
"|:------------------|:-----|:-------|\n"
|
| 299 |
-
f"| **
|
| 300 |
-
"|
|
| 301 |
-
f"| **
|
| 302 |
-
"|
|
| 303 |
-
f"| **{CONFIDENCE_LOW}
|
| 304 |
-
"Grey zone
|
| 305 |
"\n"
|
| 306 |
"Only grey-zone comments cost GPU time. On the "
|
| 307 |
-
"evaluation set,
|
| 308 |
-
"
|
| 309 |
)
|
| 310 |
|
| 311 |
gr.Markdown(
|
|
@@ -314,10 +339,10 @@ with gr.Blocks(
|
|
| 314 |
"\n"
|
| 315 |
"| Property | ML-Only | LLM-Only | Hybrid |\n"
|
| 316 |
"|:---------|:--------|:---------|:-------|\n"
|
| 317 |
-
"| **Speed** | ~7 ms | ~1
|
| 318 |
"| **Cost** | $0 (CPU) | High (every comment) "
|
| 319 |
-
"| Low (~
|
| 320 |
-
"| **Nuance** |
|
| 321 |
"| Broad world knowledge | Best of both |\n"
|
| 322 |
"| **Scalability** | Excellent | Poor | Good |\n"
|
| 323 |
"\n"
|
|
@@ -330,12 +355,12 @@ with gr.Blocks(
|
|
| 330 |
"---\n"
|
| 331 |
"### Models\n"
|
| 332 |
"\n"
|
| 333 |
-
"| Component | Model | Parameters | Serving |\n"
|
| 334 |
-
"|:----------|:------|:-----------|:--------|\n"
|
| 335 |
f"| ML Classifier | `{CLASSIFIER_MODEL_NAME}` "
|
| 336 |
-
"| ~67 M (
|
| 337 |
f"| LLM Reviewer | `{LLM_MODEL_NAME}` "
|
| 338 |
-
"| ~
|
| 339 |
)
|
| 340 |
|
| 341 |
gr.Markdown(
|
|
@@ -344,12 +369,12 @@ with gr.Blocks(
|
|
| 344 |
"\n"
|
| 345 |
"- **Course:** CS541 Artificial Intelligence, "
|
| 346 |
"Stevens Institute of Technology\n"
|
| 347 |
-
"- **Textbook:** *Understanding Deep Learning*
|
| 348 |
-
"Simon J.D. Prince\n"
|
| 349 |
"- **Dataset:** "
|
| 350 |
"[google/civil_comments]"
|
| 351 |
-
"(https://huggingface.co/datasets/
|
| 352 |
-
"civil_comments)\n"
|
| 353 |
)
|
| 354 |
|
| 355 |
# ββ Footer ββββββββββββββββββββββββββββββββββββββββββββββββ
|
|
|
|
| 1 |
"""
|
| 2 |
app.py β HuggingFace Spaces entry point for Hybrid Comment Moderation.
|
| 3 |
|
| 4 |
+
`import spaces` MUST be the first import so that ZeroGPU can patch
|
| 5 |
+
torch before any other module loads CUDA. All subsequent imports
|
| 6 |
+
(config, ml_classifier, llm_reviewer) inherit the patched torch.
|
| 7 |
+
|
| 8 |
Three tabs:
|
| 9 |
+
π Live Demo β interactive classification (ML / LLM / Hybrid)
|
| 10 |
+
π Evaluation β pre-computed metrics and charts from Colab
|
| 11 |
+
ποΈ Architecture β pipeline diagram and design rationale
|
| 12 |
"""
|
| 13 |
|
| 14 |
+
# ββ ZeroGPU patching (must precede every other import) ββββββββ
|
| 15 |
+
try:
|
| 16 |
+
import spaces # noqa: F401
|
| 17 |
+
except ImportError:
|
| 18 |
+
pass
|
| 19 |
+
|
| 20 |
import time
|
| 21 |
+
import traceback
|
| 22 |
import gradio as gr
|
| 23 |
from config import (
|
| 24 |
CONFIDENCE_LOW,
|
|
|
|
| 104 |
return verdict, details
|
| 105 |
|
| 106 |
except Exception as e:
|
| 107 |
+
tb = traceback.format_exc()
|
| 108 |
err = str(e).lower()
|
| 109 |
+
|
| 110 |
+
# ββ GPU / quota / auth errors β helpful message βββββββ
|
| 111 |
+
if any(k in err for k in [
|
| 112 |
+
"gpu", "quota", "zero", "login", "sign", "unauthorized",
|
| 113 |
+
]) or (isinstance(e, RuntimeError) and "gpu" not in err):
|
| 114 |
return (
|
| 115 |
"## β οΈ GPU Unavailable",
|
| 116 |
+
"This can happen if:\n"
|
| 117 |
+
"- You are not signed in to HuggingFace "
|
| 118 |
+
"([sign in here](https://huggingface.co/login) β free)\n"
|
| 119 |
+
"- Your daily ZeroGPU quota is exhausted\n"
|
| 120 |
+
"- The GPU is temporarily busy\n\n"
|
| 121 |
+
"**ML-only works without GPU** β select "
|
| 122 |
+
"**ML-only (CPU, no login needed)** above.\n\n"
|
| 123 |
+
f"<details><summary>Technical details</summary>\n\n"
|
| 124 |
+
f"```\n{tb}\n```\n</details>",
|
| 125 |
)
|
| 126 |
+
|
| 127 |
+
# ββ All other errors β show full traceback ββββββββββββ
|
| 128 |
+
return "## β Error", f"```\n{tb}\n```"
|
| 129 |
|
| 130 |
|
| 131 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
|
|
|
| 152 |
gr.Markdown(
|
| 153 |
"> π‘ **ML-only** works for everyone (CPU). \n"
|
| 154 |
"> **Hybrid** and **LLM-only** need a free HuggingFace "
|
| 155 |
+
"account for GPU access. \n"
|
| 156 |
+
f"> First LLM call loads the 14B model (~30β40 s); "
|
| 157 |
+
"subsequent calls take ~1β2 s.\n"
|
| 158 |
)
|
| 159 |
|
| 160 |
with gr.Row():
|
|
|
|
| 282 |
# TAB 3: ARCHITECTURE
|
| 283 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 284 |
with gr.Tab("ποΈ Architecture"):
|
| 285 |
+
|
| 286 |
gr.Markdown(
|
| 287 |
"### Two-Stage Pipeline\n"
|
| 288 |
"\n"
|
| 289 |
"```\n"
|
| 290 |
"User Comment\n"
|
| 291 |
+
" |\n"
|
| 292 |
+
" v\n"
|
| 293 |
+
"+----------------------------------------+\n"
|
| 294 |
+
"| Stage 1: ML Classifier |\n"
|
| 295 |
+
"| DistilBERT (CPU, ~7 ms per comment) |\n"
|
| 296 |
+
"| Fine-tuned on Jigsaw toxic comments |\n"
|
| 297 |
+
"+-------------------+--------------------+\n"
|
| 298 |
+
" |\n"
|
| 299 |
+
" Confident?\n"
|
| 300 |
+
" / \\\n"
|
| 301 |
+
" Yes No (grey zone)\n"
|
| 302 |
+
" | |\n"
|
| 303 |
+
" v v\n"
|
| 304 |
+
" DONE +----------------------------------+\n"
|
| 305 |
+
f" | Stage 2: LLM Reviewer |\n"
|
| 306 |
+
f" | {_LLM_SHORT} (GPU, ~1-2 s) |\n"
|
| 307 |
+
" | 14B params, NF4 quantized |\n"
|
| 308 |
+
" +-----------------+----------------+\n"
|
| 309 |
+
" |\n"
|
| 310 |
+
" v\n"
|
| 311 |
+
" DONE\n"
|
|
|
|
| 312 |
"```\n"
|
| 313 |
)
|
| 314 |
|
|
|
|
| 316 |
"---\n"
|
| 317 |
"### Confidence Routing Logic\n"
|
| 318 |
"\n"
|
| 319 |
+
"The ML classifier outputs a toxic probability "
|
| 320 |
+
"in [0, 1]. Two thresholds define three zones:\n"
|
| 321 |
"\n"
|
| 322 |
"| Toxic Probability | Zone | Action |\n"
|
| 323 |
"|:------------------|:-----|:-------|\n"
|
| 324 |
+
f"| **>= {CONFIDENCE_HIGH}** | High confidence "
|
| 325 |
+
"toxic | BLOCK (ML decides) |\n"
|
| 326 |
+
f"| **<= {CONFIDENCE_LOW}** | High confidence "
|
| 327 |
+
"safe | ALLOW (ML decides) |\n"
|
| 328 |
+
f"| **{CONFIDENCE_LOW} - {CONFIDENCE_HIGH}** | "
|
| 329 |
+
"Grey zone | Escalate to LLM |\n"
|
| 330 |
"\n"
|
| 331 |
"Only grey-zone comments cost GPU time. On the "
|
| 332 |
+
"evaluation set, **88%** of comments were handled "
|
| 333 |
+
"by the ML classifier alone.\n"
|
| 334 |
)
|
| 335 |
|
| 336 |
gr.Markdown(
|
|
|
|
| 339 |
"\n"
|
| 340 |
"| Property | ML-Only | LLM-Only | Hybrid |\n"
|
| 341 |
"|:---------|:--------|:---------|:-------|\n"
|
| 342 |
+
"| **Speed** | ~7 ms | ~1-2 s | ~7 ms (most) |\n"
|
| 343 |
"| **Cost** | $0 (CPU) | High (every comment) "
|
| 344 |
+
"| Low (~12% reach LLM) |\n"
|
| 345 |
+
"| **Nuance** | Training dist. only "
|
| 346 |
"| Broad world knowledge | Best of both |\n"
|
| 347 |
"| **Scalability** | Excellent | Poor | Good |\n"
|
| 348 |
"\n"
|
|
|
|
| 355 |
"---\n"
|
| 356 |
"### Models\n"
|
| 357 |
"\n"
|
| 358 |
+
"| Component | Model | Parameters | Quantization | Serving |\n"
|
| 359 |
+
"|:----------|:------|:-----------|:-------------|:--------|\n"
|
| 360 |
f"| ML Classifier | `{CLASSIFIER_MODEL_NAME}` "
|
| 361 |
+
"| ~67 M | None (full precision) | CPU |\n"
|
| 362 |
f"| LLM Reviewer | `{LLM_MODEL_NAME}` "
|
| 363 |
+
"| ~14 B | NF4 (~8 GB) | ZeroGPU A100 |\n"
|
| 364 |
)
|
| 365 |
|
| 366 |
gr.Markdown(
|
|
|
|
| 369 |
"\n"
|
| 370 |
"- **Course:** CS541 Artificial Intelligence, "
|
| 371 |
"Stevens Institute of Technology\n"
|
| 372 |
+
"- **Textbook:** *Understanding Deep Learning* "
|
| 373 |
+
"by Simon J.D. Prince\n"
|
| 374 |
"- **Dataset:** "
|
| 375 |
"[google/civil_comments]"
|
| 376 |
+
"(https://huggingface.co/datasets/"
|
| 377 |
+
"google/civil_comments)\n"
|
| 378 |
)
|
| 379 |
|
| 380 |
# ββ Footer ββββββββββββββββββββββββββββββββββββββββββββββββ
|
assets/stage_distribution.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/strategy_comparison.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
config.py
CHANGED
|
@@ -8,7 +8,12 @@ DECISION_ALLOW = "ALLOW"
|
|
| 8 |
|
| 9 |
# ββ Model identifiers ββββββββββββββββββββββββββββββββββββββββ
|
| 10 |
CLASSIFIER_MODEL_NAME = "martin-ha/toxic-comment-model" # DistilBERT, ~270 MB
|
| 11 |
-
LLM_MODEL_NAME = "Qwen/Qwen2.5-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
|
| 13 |
# ββ Generation parameters βββββββββββββββββββββββββββββββββββββ
|
| 14 |
LLM_MAX_NEW_TOKENS = 5
|
|
|
|
| 8 |
|
| 9 |
# ββ Model identifiers ββββββββββββββββββββββββββββββββββββββββ
|
| 10 |
CLASSIFIER_MODEL_NAME = "martin-ha/toxic-comment-model" # DistilBERT, ~270 MB
|
| 11 |
+
LLM_MODEL_NAME = "Qwen/Qwen2.5-14B-Instruct" # NF4, ~8 GB
|
| 12 |
+
|
| 13 |
+
# ββ Quantization ββββββββββββββββββββββββββββββββββββββββββββββ
|
| 14 |
+
# NF4 via bitsandbytes compresses 14B params from ~28 GB (bf16)
|
| 15 |
+
# to ~8 GB, fitting comfortably on a ZeroGPU A100 slice.
|
| 16 |
+
LLM_QUANTIZATION = "4bit"
|
| 17 |
|
| 18 |
# ββ Generation parameters βββββββββββββββββββββββββββββββββββββ
|
| 19 |
LLM_MAX_NEW_TOKENS = 5
|
llm_reviewer.py
CHANGED
|
@@ -1,24 +1,18 @@
|
|
| 1 |
"""
|
| 2 |
llm_reviewer.py β LLM reviewer for ambiguous comments (Stage 2).
|
| 3 |
|
| 4 |
-
Qwen2.5-
|
| 5 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
|
| 7 |
If the `spaces` package is unavailable (local testing), the GPU
|
| 8 |
decorator becomes a no-op so the code still works on any machine.
|
| 9 |
"""
|
| 10 |
|
| 11 |
-
|
| 12 |
-
from typing import Tuple
|
| 13 |
-
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 14 |
-
from config import (
|
| 15 |
-
LLM_MODEL_NAME,
|
| 16 |
-
LLM_MAX_NEW_TOKENS,
|
| 17 |
-
DECISION_BLOCK,
|
| 18 |
-
DECISION_ALLOW,
|
| 19 |
-
)
|
| 20 |
-
|
| 21 |
-
# ββ ZeroGPU decorator (graceful fallback for local runs) ββββββ
|
| 22 |
try:
|
| 23 |
import spaces
|
| 24 |
ZERO_GPU = True
|
|
@@ -35,6 +29,16 @@ except ImportError:
|
|
| 35 |
|
| 36 |
spaces = _SpacesFallback()
|
| 37 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
|
| 39 |
# ββ Prompt ββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 40 |
_SYSTEM_PROMPT = (
|
|
@@ -46,29 +50,71 @@ _SYSTEM_PROMPT = (
|
|
| 46 |
"Do not explain."
|
| 47 |
)
|
| 48 |
|
| 49 |
-
# ββ
|
| 50 |
-
#
|
| 51 |
-
#
|
| 52 |
-
#
|
| 53 |
-
|
|
|
|
|
|
|
| 54 |
|
| 55 |
-
_tokenizer = AutoTokenizer.from_pretrained(
|
| 56 |
-
LLM_MODEL_NAME, trust_remote_code=True
|
| 57 |
-
)
|
| 58 |
-
if _tokenizer.pad_token is None:
|
| 59 |
-
_tokenizer.pad_token = _tokenizer.eos_token
|
| 60 |
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
|
| 69 |
|
| 70 |
# ββ Answer parser βββββββββββββββββββββββββββββββββββββββββββββ
|
| 71 |
def _parse_answer(answer: str) -> Tuple[str, float]:
|
|
|
|
| 72 |
a = answer.lower().strip().rstrip(".")
|
| 73 |
if "non-toxic" in a or "non toxic" in a or "not toxic" in a:
|
| 74 |
return (DECISION_ALLOW, 0.85)
|
|
@@ -79,14 +125,22 @@ def _parse_answer(answer: str) -> Tuple[str, float]:
|
|
| 79 |
|
| 80 |
|
| 81 |
# ββ GPU-decorated classification function βββββββββββββββββββββ
|
| 82 |
-
@spaces.GPU(duration=
|
| 83 |
def gpu_classify(text: str) -> Tuple[str, float, str]:
|
| 84 |
"""
|
| 85 |
Classify one comment with the LLM.
|
| 86 |
|
| 87 |
On HuggingFace Spaces this runs on a ZeroGPU A100 slice.
|
| 88 |
-
The
|
|
|
|
| 89 |
"""
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
messages = [
|
| 91 |
{"role": "system", "content": _SYSTEM_PROMPT},
|
| 92 |
{"role": "user", "content": f'Classify this comment: "{text[:1024]}"'},
|
|
@@ -104,8 +158,9 @@ def gpu_classify(text: str) -> Tuple[str, float, str]:
|
|
| 104 |
)
|
| 105 |
|
| 106 |
new_tokens = output_ids[0][input_ids.shape[-1]:]
|
| 107 |
-
answer = _tokenizer.decode(
|
| 108 |
-
|
|
|
|
| 109 |
|
| 110 |
decision, confidence = _parse_answer(answer)
|
| 111 |
return decision, confidence, f"LLM said: '{answer}'"
|
|
|
|
| 1 |
"""
|
| 2 |
llm_reviewer.py β LLM reviewer for ambiguous comments (Stage 2).
|
| 3 |
|
| 4 |
+
Qwen2.5-14B-Instruct is loaded lazily in NF4 quantization via
|
| 5 |
+
bitsandbytes (~8 GB VRAM) on the first ZeroGPU call. The A100
|
| 6 |
+
handles this comfortably alongside DistilBERT on CPU.
|
| 7 |
+
|
| 8 |
+
The model stays in memory after first load, so subsequent calls
|
| 9 |
+
skip the loading step and run inference directly (~1β2 s).
|
| 10 |
|
| 11 |
If the `spaces` package is unavailable (local testing), the GPU
|
| 12 |
decorator becomes a no-op so the code still works on any machine.
|
| 13 |
"""
|
| 14 |
|
| 15 |
+
# ββ ZeroGPU must be imported before torch βββββββββββββββββββββ
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
try:
|
| 17 |
import spaces
|
| 18 |
ZERO_GPU = True
|
|
|
|
| 29 |
|
| 30 |
spaces = _SpacesFallback()
|
| 31 |
|
| 32 |
+
import torch
|
| 33 |
+
from typing import Tuple
|
| 34 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
|
| 35 |
+
from config import (
|
| 36 |
+
LLM_MODEL_NAME,
|
| 37 |
+
LLM_QUANTIZATION,
|
| 38 |
+
LLM_MAX_NEW_TOKENS,
|
| 39 |
+
DECISION_BLOCK,
|
| 40 |
+
DECISION_ALLOW,
|
| 41 |
+
)
|
| 42 |
|
| 43 |
# ββ Prompt ββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 44 |
_SYSTEM_PROMPT = (
|
|
|
|
| 50 |
"Do not explain."
|
| 51 |
)
|
| 52 |
|
| 53 |
+
# ββ Lazy-loaded model state βββββββββββββββββββββββββββββββββββ
|
| 54 |
+
# The model is loaded on the first @spaces.GPU call, not at
|
| 55 |
+
# import time. This avoids startup failures when GPU is not
|
| 56 |
+
# yet allocated and keeps Space boot time fast.
|
| 57 |
+
_model = None
|
| 58 |
+
_tokenizer = None
|
| 59 |
+
_load_error = None
|
| 60 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 61 |
|
| 62 |
+
def _ensure_model_loaded():
|
| 63 |
+
"""Load model and tokenizer once, on first use inside a GPU context."""
|
| 64 |
+
global _model, _tokenizer, _load_error
|
| 65 |
+
|
| 66 |
+
if _model is not None:
|
| 67 |
+
return True
|
| 68 |
+
if _load_error is not None:
|
| 69 |
+
return False
|
| 70 |
+
|
| 71 |
+
try:
|
| 72 |
+
print(f"β³ Loading {LLM_MODEL_NAME} ({LLM_QUANTIZATION}) β¦")
|
| 73 |
+
|
| 74 |
+
_tokenizer = AutoTokenizer.from_pretrained(
|
| 75 |
+
LLM_MODEL_NAME, trust_remote_code=True
|
| 76 |
+
)
|
| 77 |
+
if _tokenizer.pad_token is None:
|
| 78 |
+
_tokenizer.pad_token = _tokenizer.eos_token
|
| 79 |
+
|
| 80 |
+
# ββ Build quantization config βββββββββββββββββββββββββ
|
| 81 |
+
load_kwargs = {
|
| 82 |
+
"trust_remote_code": True,
|
| 83 |
+
"device_map": "auto",
|
| 84 |
+
}
|
| 85 |
+
|
| 86 |
+
if LLM_QUANTIZATION == "4bit" and torch.cuda.is_available():
|
| 87 |
+
load_kwargs["quantization_config"] = BitsAndBytesConfig(
|
| 88 |
+
load_in_4bit=True,
|
| 89 |
+
bnb_4bit_quant_type="nf4",
|
| 90 |
+
bnb_4bit_compute_dtype=torch.float16,
|
| 91 |
+
bnb_4bit_use_double_quant=True,
|
| 92 |
+
)
|
| 93 |
+
print(" NF4 quantization via bitsandbytes")
|
| 94 |
+
else:
|
| 95 |
+
load_kwargs["torch_dtype"] = torch.float16
|
| 96 |
+
print(" float16 (no quantization)")
|
| 97 |
+
|
| 98 |
+
_model = AutoModelForCausalLM.from_pretrained(
|
| 99 |
+
LLM_MODEL_NAME, **load_kwargs
|
| 100 |
+
)
|
| 101 |
+
|
| 102 |
+
if torch.cuda.is_available():
|
| 103 |
+
allocated = torch.cuda.memory_allocated() / 1e9
|
| 104 |
+
print(f" GPU memory allocated: {allocated:.1f} GB")
|
| 105 |
+
|
| 106 |
+
print(f"β
{LLM_MODEL_NAME} loaded successfully")
|
| 107 |
+
return True
|
| 108 |
+
|
| 109 |
+
except Exception as e:
|
| 110 |
+
_load_error = str(e)
|
| 111 |
+
print(f"β Failed to load {LLM_MODEL_NAME}: {_load_error}")
|
| 112 |
+
return False
|
| 113 |
|
| 114 |
|
| 115 |
# ββ Answer parser βββββββββββββββββββββββββββββββββββββββββββββ
|
| 116 |
def _parse_answer(answer: str) -> Tuple[str, float]:
|
| 117 |
+
"""Map free-text LLM output to a moderation decision."""
|
| 118 |
a = answer.lower().strip().rstrip(".")
|
| 119 |
if "non-toxic" in a or "non toxic" in a or "not toxic" in a:
|
| 120 |
return (DECISION_ALLOW, 0.85)
|
|
|
|
| 125 |
|
| 126 |
|
| 127 |
# ββ GPU-decorated classification function βββββββββββββββββββββ
|
| 128 |
+
@spaces.GPU(duration=120)
|
| 129 |
def gpu_classify(text: str) -> Tuple[str, float, str]:
|
| 130 |
"""
|
| 131 |
Classify one comment with the LLM.
|
| 132 |
|
| 133 |
On HuggingFace Spaces this runs on a ZeroGPU A100 slice.
|
| 134 |
+
The first call includes model loading (~30β40 s for 14B NF4);
|
| 135 |
+
subsequent calls run inference only (~1β2 s).
|
| 136 |
"""
|
| 137 |
+
if not _ensure_model_loaded():
|
| 138 |
+
return (
|
| 139 |
+
DECISION_ALLOW,
|
| 140 |
+
0.5,
|
| 141 |
+
f"LLM unavailable: {_load_error}",
|
| 142 |
+
)
|
| 143 |
+
|
| 144 |
messages = [
|
| 145 |
{"role": "system", "content": _SYSTEM_PROMPT},
|
| 146 |
{"role": "user", "content": f'Classify this comment: "{text[:1024]}"'},
|
|
|
|
| 158 |
)
|
| 159 |
|
| 160 |
new_tokens = output_ids[0][input_ids.shape[-1]:]
|
| 161 |
+
answer = _tokenizer.decode(
|
| 162 |
+
new_tokens, skip_special_tokens=True
|
| 163 |
+
).strip().lower()
|
| 164 |
|
| 165 |
decision, confidence = _parse_answer(answer)
|
| 166 |
return decision, confidence, f"LLM said: '{answer}'"
|
precomputed.py
CHANGED
|
@@ -5,7 +5,7 @@ All three strategies were evaluated on the same 100 balanced samples
|
|
| 5 |
(50 toxic + 50 non-toxic) from google/civil_comments test split.
|
| 6 |
|
| 7 |
ML Classifier : martin-ha/toxic-comment-model (DistilBERT)
|
| 8 |
-
LLM : Qwen/Qwen2.5-
|
| 9 |
Grey Zone : [0.15, 0.85]
|
| 10 |
"""
|
| 11 |
|
|
@@ -20,7 +20,7 @@ SYSTEM_CONFIG_MD = """\
|
|
| 20 |
| **Dataset** | google/civil_comments (test split) |
|
| 21 |
| **Eval Samples** | 100 (50 toxic + 50 non-toxic) |
|
| 22 |
| **ML Classifier** | martin-ha/toxic-comment-model (DistilBERT) |
|
| 23 |
-
| **LLM** | Qwen/Qwen2.5-
|
| 24 |
| **Grey Zone** | [0.15, 0.85] β ML uncertain β escalate to LLM |
|
| 25 |
| **Runtime** | Google Colab, NVIDIA T4 (15 GB VRAM) |
|
| 26 |
"""
|
|
@@ -29,16 +29,16 @@ SYSTEM_CONFIG_MD = """\
|
|
| 29 |
# Strategy comparison table
|
| 30 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 31 |
RESULTS_TABLE_MD = """\
|
| 32 |
-
| Metric | ML-Only (DistilBERT) | LLM-Only (Qwen2.5-
|
| 33 |
-
|:-------|---------------------:|-----------------------:|------------------------:|
|
| 34 |
-
| **F1 (macro)** | 0.7803 | 0.
|
| 35 |
-
| **F1 (toxic)** | 0.7342 | 0.
|
| 36 |
-
| **Precision** | **1.0000** | 0.
|
| 37 |
-
| **Recall** | 0.5800 | 0.
|
| 38 |
-
| **True Positives** | 29 |
|
| 39 |
-
| **True Negatives** | **50** |
|
| 40 |
-
| **False Positives** | **0** |
|
| 41 |
-
| **False Negatives** | 21 |
|
| 42 |
| **LLM Calls** | 0 | 100 | 12 |
|
| 43 |
| **LLM Call %** | 0% | 100% | 12% |
|
| 44 |
| **Est. Cost (USD)** | $0.0000 | $0.0366 | $0.0050 |
|
|
@@ -55,7 +55,7 @@ KEY_FINDINGS_MD = """\
|
|
| 55 |
> Hybrid: **$0.0050** vs LLM-Only: **$0.0366** β **7.3Γ cheaper**
|
| 56 |
|
| 57 |
#### π― Best F1 (macro)
|
| 58 |
-
> **Hybrid (MLβQwen2.5-
|
| 59 |
|
| 60 |
---
|
| 61 |
|
|
@@ -71,8 +71,8 @@ KEY_FINDINGS_MD = """\
|
|
| 71 |
> subtler or out-of-distribution toxicity may be harder to
|
| 72 |
> catch. Zero LLM cost makes this the cheapest strategy.
|
| 73 |
|
| 74 |
-
**2. LLM-Only (Qwen2.5-
|
| 75 |
-
> F1 = 0.
|
| 76 |
>
|
| 77 |
> The instruction-tuned LLM applies broad world knowledge and
|
| 78 |
> pragmatic understanding to every comment, producing a
|
|
@@ -81,8 +81,8 @@ KEY_FINDINGS_MD = """\
|
|
| 81 |
> cost: every comment requires a full autoregressive generation
|
| 82 |
> pass, making this the most expensive strategy.
|
| 83 |
|
| 84 |
-
**3. Hybrid (ML β Qwen2.5-
|
| 85 |
-
> F1 = 0.
|
| 86 |
>
|
| 87 |
> The hybrid routes only grey-zone comments β those where the
|
| 88 |
> ML classifier's confidence falls in [0.15, 0.85] β to
|
|
|
|
| 5 |
(50 toxic + 50 non-toxic) from google/civil_comments test split.
|
| 6 |
|
| 7 |
ML Classifier : martin-ha/toxic-comment-model (DistilBERT)
|
| 8 |
+
LLM : Qwen/Qwen2.5-14B-Instruct (NF4 quantization)
|
| 9 |
Grey Zone : [0.15, 0.85]
|
| 10 |
"""
|
| 11 |
|
|
|
|
| 20 |
| **Dataset** | google/civil_comments (test split) |
|
| 21 |
| **Eval Samples** | 100 (50 toxic + 50 non-toxic) |
|
| 22 |
| **ML Classifier** | martin-ha/toxic-comment-model (DistilBERT) |
|
| 23 |
+
| **LLM** | Qwen/Qwen2.5-14B-Instruct (NF4, ~8 GB VRAM) |
|
| 24 |
| **Grey Zone** | [0.15, 0.85] β ML uncertain β escalate to LLM |
|
| 25 |
| **Runtime** | Google Colab, NVIDIA T4 (15 GB VRAM) |
|
| 26 |
"""
|
|
|
|
| 29 |
# Strategy comparison table
|
| 30 |
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 31 |
RESULTS_TABLE_MD = """\
|
| 32 |
+
| Metric | ML-Only (DistilBERT) | LLM-Only (Qwen2.5-14B) | Hybrid (MLβQwen2.5-14B) |
|
| 33 |
+
|:-------|---------------------:|------------------------:|-------------------------:|
|
| 34 |
+
| **F1 (macro)** | 0.7803 | 0.7898 | **0.8368** |
|
| 35 |
+
| **F1 (toxic)** | 0.7342 | 0.7961 | **0.8140** |
|
| 36 |
+
| **Precision** | **1.0000** | 0.7736 | 0.9722 |
|
| 37 |
+
| **Recall** | 0.5800 | **0.8200** | 0.7000 |
|
| 38 |
+
| **True Positives** | 29 | **41** | 35 |
|
| 39 |
+
| **True Negatives** | **50** | 38 | 49 |
|
| 40 |
+
| **False Positives** | **0** | 12 | 1 |
|
| 41 |
+
| **False Negatives** | 21 | **9** | 15 |
|
| 42 |
| **LLM Calls** | 0 | 100 | 12 |
|
| 43 |
| **LLM Call %** | 0% | 100% | 12% |
|
| 44 |
| **Est. Cost (USD)** | $0.0000 | $0.0366 | $0.0050 |
|
|
|
|
| 55 |
> Hybrid: **$0.0050** vs LLM-Only: **$0.0366** β **7.3Γ cheaper**
|
| 56 |
|
| 57 |
#### π― Best F1 (macro)
|
| 58 |
+
> **Hybrid (MLβQwen2.5-14B-Instruct)** achieved the highest F1 of **0.8368**
|
| 59 |
|
| 60 |
---
|
| 61 |
|
|
|
|
| 71 |
> subtler or out-of-distribution toxicity may be harder to
|
| 72 |
> catch. Zero LLM cost makes this the cheapest strategy.
|
| 73 |
|
| 74 |
+
**2. LLM-Only (Qwen2.5-14B-Instruct)**
|
| 75 |
+
> F1 = 0.7898 Β· Precision = 0.77 Β· Recall = 0.82
|
| 76 |
>
|
| 77 |
> The instruction-tuned LLM applies broad world knowledge and
|
| 78 |
> pragmatic understanding to every comment, producing a
|
|
|
|
| 81 |
> cost: every comment requires a full autoregressive generation
|
| 82 |
> pass, making this the most expensive strategy.
|
| 83 |
|
| 84 |
+
**3. Hybrid (ML β Qwen2.5-14B-Instruct)** β
|
| 85 |
+
> F1 = 0.8368 Β· Precision = 0.97 Β· Recall = 0.70
|
| 86 |
>
|
| 87 |
> The hybrid routes only grey-zone comments β those where the
|
| 88 |
> ML classifier's confidence falls in [0.15, 0.85] β to
|
requirements.txt
CHANGED
|
@@ -1,6 +1,7 @@
|
|
| 1 |
torch
|
| 2 |
transformers>=4.52
|
| 3 |
accelerate
|
|
|
|
| 4 |
gradio>=5.0
|
| 5 |
spaces
|
| 6 |
numpy
|
|
|
|
| 1 |
torch
|
| 2 |
transformers>=4.52
|
| 3 |
accelerate
|
| 4 |
+
bitsandbytes
|
| 5 |
gradio>=5.0
|
| 6 |
spaces
|
| 7 |
numpy
|