Update README.md

README.md

@@ -26,13 +26,11 @@ tags:
 
 ---
 
-## Overview
+## The Problem
 
-Security Operations Centers (SOCs) are drowning in alerts. Every day, analysts receive thousands of automated security notifications — and the majority of them turn out to be noise. Studies show that **40-60% of SOC analyst time** is spent reviewing false positives: alerts that turn out to be harmless activity or just bad detections. The real threats get buried under the volume.
+SOC analysts waste **40-60% of their time** reviewing false positives. Out of 1000 daily alerts, ~600 are noise. Real threats get buried.
 
-In this assignment, I built a machine learning pipeline to predict whether a cybersecurity alert represents a confirmed attack (TruePositive), harmless activity (BenignPositive), or a false alarm (FalsePositive) — using only the structural features available the moment an alert fires, before any human analyst reviews it. If a model can automate the easy calls, analysts can focus their energy on the genuinely ambiguous cases.
-
-I found this dataset surprisingly rich. The 4.1 million evidence rows compress into meaningful incident-level patterns that a classifier can actually learn. The main challenge was figuring out which features encode real threat signal versus which ones just memorize organizational patterns — and that distinction turned out to matter a lot for building a model that generalizes.
+**Goal:** Build ML models to automate triage — predict if an alert is a real attack (TP), harmless activity (BP), or false alarm (FP).
 
 ---
 
@@ -41,9 +39,10 @@ I found this dataset surprisingly rich. The 4.1 million evidence rows compress i
 - **Source:** (https://archive.ics.uci.edu/dataset/498/incident+management+process+enriched+event+log)
 - **Original size:** 4.1M evidence-level rows, 45 features
 - **After cleaning:** 100K sampled rows, 35 features (10 dropped for >60% missing)
-- **Regression target:** `evidence_count` — number of evidence items per incident (proxy for complexity)
-- **Classification target:** `IncidentGrade` — TruePositive / BenignPositive / FalsePositive
-- **Class distribution:** BP 42% / TP 36% / FP 22%
+- **Targets:**
+  - Regression: `evidence_count` (attack complexity)
+  - Classification: `IncidentGrade` (TP/BP/FP)
+- **Class split:** 42% BP / 36% TP / 22% FP
 
 ## Feature Engineering
 
@@ -67,22 +66,18 @@ I also added clustering-derived features:
 ---
 
 ## Data Cleaning
+Removed `OrgId` → caused data leakage (R² 0.88 → 0.47)
 
 ### Missing Values
 
-Ten columns had more than 60% missing values and were dropped immediately. Keeping them would have meant imputing most of the data, which introduces more noise than signal.
-
-Dropped: `ActionGrouped`, `ActionGranular`, `ResourceType`, `ThreatFamily`, `AntispamDirection`, `Roles`, `EmailClusterId`, `SuspicionLevel`, `LastVerdict`, `MitreTechniques`
-
-I also removed `OrgId` after discovering it caused data leakage — early models achieved R² = 0.88 by memorizing which organizations generate more incidents rather than learning actual threat patterns. Dropping it brought R² to 0.47, which is a real score on real signal.
+Ten columns had more than 60% missing values and were dropped immediately. to prevent capturing more noise than signal.
 
 ### Duplicates
 
 No duplicate rows found in the 100K sample.
 
 ### Outliers
-
-I kept the outliers in `evidence_count`. The distribution is bimodal — incidents either have very few evidence items (5-10) or very many (100-200). This initially looked like outliers, but it's a real operational pattern: simple false alarms generate minimal evidence while real attacks generate cascading evidence chains. Removing them would have destroyed the main signal.
+Kept outliers in `evidence_count` → bimodal distribution is real signal
 
 ---
 
@@ -92,25 +87,28 @@ I kept the outliers in `evidence_count`. The distribution is bimodal — inciden
 
 ![Grade Distribution](https://huggingface.co/yonatane22-bh/Microsoft_SOC_analysis/resolve/main/figures/01_grade_distribution.png)
 
-The dataset is moderately imbalanced: **42% BenignPositive**, **36% TruePositive**, **22% FalsePositive**. A model that predicts everything as BP would score 42% accuracy without learning anything. I used macro F1-score throughout to give equal weight to all three classes.
+**Shows:** Class distribution of incident grades  
+**Finding:** 42% BenignPositive, 36% TruePositive, 22% FalsePositive  
+**Why it matters:** Moderate imbalance → used macro F1 metric (equal weight to all classes)
 
 ### Entity Type Patterns
 
 ![Entity Type by Grade](https://huggingface.co/yonatane22-bh/Microsoft_SOC_analysis/resolve/main/figures/02_entity_type_by_grade.png)
 
-IP-based entities dominate confirmed attacks — the `Ip` entity type has the highest TruePositive count at roughly 11K incidents. CloudLogonRequest stood out with almost zero BenignPositive, meaning flagged cloud logins are either real threats or false alarms, never routine activity.
+**Shows:** Which entity types (IP, User, Machine, etc.) appear in each grade  
+**Finding:** IP entities dominate TruePositive incidents (~11K count)  
+**Why it matters:** IP-based alerts deserve higher priority than User-based alerts
 
 ### Threat Category Profiles
 
 ![Category by Grade](https://huggingface.co/yonatane22-bh/Microsoft_SOC_analysis/resolve/main/figures/03_category_by_grade.png)
 
-This was the most revealing visualization. Different threat categories have completely different noise levels. Malware alerts are **74% BenignPositive** — most antivirus triggers are harmless. InitialAccess alerts are **54% TruePositive** — when someone is trying to get into a system, it's usually real. Category alone gives you a strong prior on whether an alert deserves investigation.
-
-### Evidence Volume by Grade
+**Shows:** Threat categories (InitialAccess, Malware, Exfiltration, etc.) by grade  
+**Finding:** 
+- InitialAccess: 54% TruePositive (high threat)
+- Malware: 74% BenignPositive (mostly noise)
 
-![Evidence by Grade](https://huggingface.co/yonatane22-bh/Microsoft_SOC_analysis/resolve/main/figures/05_evidence_by_grade.png)
-
-Real attacks leave a much larger forensic trail. TruePositive incidents average **117 evidence items** compared to 25 for FalsePositive — a **4.7× difference**. This confirmed that deriving `evidence_count` as a regression target was the right approach.
+**Why it matters:** Not all alert categories are equal — category alone is a strong predictor
 
 ---
 
@@ -150,23 +148,39 @@ Real attacks leave a much larger forensic trail. TruePositive incidents average
 
 ![Baseline Evaluation](https://huggingface.co/yonatane22-bh/Microsoft_SOC_analysis/resolve/main/figures/07_baseline_eval.png)
 
-The baseline LinearRegression on 6 encoded categorical features scored RMSE = 96.1, R² = 0.023. The residual plot showed two distinct peaks — a clear sign the linear model was failing to capture the bimodal distribution of evidence counts.
+**Shows:** LinearRegression performance — residual plot  
+**Finding:** Two distinct peaks → bimodal distribution not captured  
+**Result:** RMSE = 96.1, R² = 0.023 (failed)  
+**Why it matters:** Linear models can't handle this data → need tree models
 
 ### Regression Model Comparison
 
 ![Regression Comparison](https://huggingface.co/yonatane22-bh/Microsoft_SOC_analysis/resolve/main/figures/09_regression_comparison.png)
 
 ![Regression Importance](https://huggingface.co/yonatane22-bh/Microsoft_SOC_analysis/resolve/main/figures/10_regression_importance.png)
+**Shows:** RandomForest feature importance for regression  
+**Finding:**
+- AlertTitle: **61% importance**
+- cluster_distance: 30%
+- Category: 15%
+
+**Why it matters:** Which detector fired is by far the strongest predictor
+
 
 ![Winner Regression Eval](https://huggingface.co/yonatane22-bh/Microsoft_SOC_analysis/resolve/main/figures/11_winner_regression_eval.png)
 
-After adding the 5 engineered features plus clustering, RandomForest dropped RMSE to **70.9** — a **26% improvement** over baseline. R² rose from 0.023 to 0.469. Tree models handle the bimodal target distribution naturally by splitting the data rather than trying to fit a single line.
+**Shows:** RandomForest actual vs predicted scatter plot  
+**Finding:** Good fit along diagonal — captures bimodal pattern  
+**Why it matters:** Tree splits naturally handle 25-item vs 117-item peaks
+
 
 ### Classification Setup
 
 ![Grade Distribution](https://huggingface.co/yonatane22-bh/Microsoft_SOC_analysis/resolve/main/figures/12_grade_distribution.png)
 
-I used `IncidentGrade` directly without any conversion — the three classes already existed in the dataset. The class imbalance (42/36/22) made macro F1 the right metric to ensure the model learned all three classes equally.
+**Shows:** Target class distribution for classification task  
+**Finding:** Same 42/36/22 split as before  
+**Why it matters:** Confirms macro F1 is right metric (not accuracy)
 
 ### Classification Results