docs: relative link to README.tr.md

README.md

@@ -130,6 +130,44 @@ Bundle keys: `representation_order`, `representation_columns`, `numeric_feature_
 
 Training details and ablations: **`svc_sban.ipynb`**.
 
+### Feature pruning (TF-IDF columns)
+
+Implemented in **`svc_sban.ipynb`** (cells after the first per-representation `LinearSVC` bases):
+
+1. **Importance** — For each representation, mean `|coef_|` over classes from `final_base_models` (TF-IDF tokens + six numeric stats).
+2. **Sort ascending** — Lowest-importance names are dropped first.
+3. **Ratio sweep** — Validation macro-F1 was plotted for many removal ratios (roughly **5–60%** and **65–80%** in the analysis figures); the exported model uses a single setting.
+4. **Production choice** — **`feature_pruning_ratio = 0.65`**: remove the lowest **65%** of the ranked feature list for TF-IDF vocabulary entries. The six numeric columns (`char_count`, `line_count`, `token_count`, `avg_line_length`, `unique_token_ratio`, `char_entropy`) are **always kept** and re-appended via fixed column indices after TF-IDF subsetting.
+
+Validation macro-F1 at **65%** feature removal (same notebook run):
+
+| Representation | Macro F1 (val) | Columns after prune |
+|----------------|----------------|---------------------|
+| asm | 0.7009 | 26,259 |
+| binary | 0.6609 | 26,259 |
+| nld | 0.4870 | 26,259 |
+| source | 0.8802 | 26,257 |
+
+These pruned column sets are stored in the joblib bundle as `representations[r]["selected_indices"]` (feature step only; ID pruning below may reuse the same index vector).
+
+### ID pruning (training samples)
+
+Overlapping **bodmas** vs **sorel20m** IDs motivate dropping ambiguous training rows before refitting bases:
+
+1. Fix **feature pruning at 65%** and fit a temporary `LinearSVC` on pruned features.
+2. **Score** each training row in **`bodmas`** and **`sorel20m`** only: sparse TF-IDF presence (binary) dotted with pruned-model TF-IDF coefficient magnitudes → `importance_score`.
+3. **Grid** — For each representation, remove the lowest-scoring **`id_prune_ratios`** fraction **per class** (5%, 10%, …, 70%), refit on remaining train rows, measure validation macro-F1 → `id_pruning_summary` in the notebook.
+4. **Production choice** — `selected_id_prune_ratios`:
+
+| Representation | ID remove ratio | Val macro F1 | Train rows kept | Removed bodmas / sorel20m |
+|----------------|-----------------|--------------|-----------------|---------------------------|
+| asm | **10%** | 0.6978 | 102,538 | 5,689 / 4,941 |
+| binary | **5%** | 0.6625 | 107,854 | 2,844 / 2,470 |
+| source | **40%** | 0.8712 | 70,646 | 22,756 / 19,766 |
+| nld | **40%** | 0.4804 | 70,646 | 22,756 / 19,766 |
+
+Final stacking retrains ID-pruned bases (5-fold OOF decision scores), then class-weight search and meta learner on top of that pipeline. **`dike`** and **`malwarebazaar`** rows are never removed by this step.
+
 ### Split sizes used in training notebook
 
 | Split | Rows | bodmas | dike | malwarebazaar | sorel20m |