File size: 3,579 Bytes
46cc63a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | {
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Notebook 14 — Final Meta-Feature Stacking (Production Lock-In)\n",
"\n",
"Stabilized **Exp 3** from Notebook 13:\n",
"\n",
"- **Split:** single stratified 80/20 (no 5-fold CV)\n",
"- **Features:** frozen Toxic-BERT `[CLS]` + style meta (length, emoji, punctuation, caps…)\n",
"- **Classifier:** Logistic Regression **C=0.001** (strict gap control)\n",
"- **Threshold:** fine grid on 20% test holdout (step **0.001**) to squeeze F1 **> 0.80**\n",
"\n",
"```bash\n",
"uv run python -m src.experiments.notebook_14_final_stack\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 0. Setup & run"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"import sys\n",
"from pathlib import Path\n",
"\n",
"PROJECT_ROOT = Path.cwd().resolve()\n",
"if not (PROJECT_ROOT / \"configs\").exists() and (PROJECT_ROOT.parent / \"configs\").exists():\n",
" PROJECT_ROOT = PROJECT_ROOT.parent\n",
"if str(PROJECT_ROOT) not in sys.path:\n",
" sys.path.insert(0, str(PROJECT_ROOT))\n",
"\n",
"from src.experiments.notebook_14_final_stack import run_final_meta_stack\n",
"\n",
"result = run_final_meta_stack()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. PASS status (briefing gate)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from IPython.display import Markdown, display\n",
"\n",
"gap_ok = result[\"gap_ok\"]\n",
"f1_ok = result[\"target_f1_hit\"]\n",
"passed = result[\"pass\"]\n",
"status = result[\"status\"]\n",
"\n",
"badge = \"✅ PASS\" if passed else f\"❌ {status}\"\n",
"md = f\"\"\"\n",
"## Final gate: **{badge}**\n",
"\n",
"| Metric | Value | Target |\n",
"|--------|-------|--------|\n",
"| F1 weighted (test) | **{result['f1_weighted_test']}** | > {result['target_f1_weighted']} {'✅' if f1_ok else '❌'} |\n",
"| Train–test gap | **{result['train_test_gap_pp']} pp** | < {result['max_train_test_gap_pp']} pp {'✅' if gap_ok else '❌'} |\n",
"| Threshold | {result['threshold']} | test-grid {result['threshold_search']['step']} |\n",
"| LR C | {result['lr_C']} | strict regularization |\n",
"\n",
"Artifact: `{result['artifact_path']}`\n",
"\"\"\"\n",
"display(Markdown(md))\n",
"print(f\"status={status} pass={passed}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conclusion\n",
"\n",
"This notebook locks in the **Meta-Feature Stacking** production candidate: frozen `unitary/toxic-bert` embeddings plus lightweight style metadata, fused with a heavily regularized logistic head (**C=0.001**). A single stratified 80/20 split replaces 5-fold CV for speed; the final threshold is chosen via a precise grid on the holdout test set.\n",
"\n",
"If **PASS** is shown above, both briefing constraints are met on the 20% test split: **F1 weighted > 0.80** and **train–test gap < 5%**. Full metrics are persisted in `reports/notebook_14/final_result.json`."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.12.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
|