Sync expanded model card from GitHub

README.md

@@ -13,182 +13,233 @@ pipeline_tag: text-classification
 
 **[Türkçe dokümantasyon](README.tr.md)**
 
-Multiclass machine learning pipeline for **malware dataset origin classification** on the [SBAN](https://github.com/ma-soreto/SBAN) benchmark: given four synchronized representations of the same sample, predict which source corpus it belongs to among `bodmas`, `dike`, `malwarebazaar`, and `sorel20m`.
+Multiclass pipeline for **malware dataset origin classification** on [SBAN](https://github.com/ma-soreto/SBAN): four synchronized text views per sample → predict which sub-corpus it belongs to (`bodmas`, `dike`, `malwarebazaar`, `sorel20m`).
 
-This repository ships **code, notebooks, and the trained model weights**. It does **not** redistribute SBAN data or train/validation/test parquet files. You must acquire SBAN under its own terms and run the pipeline locally.
+This repository contains **code, notebooks, and `sban_weighted_stacking_model.joblib`**. **No SBAN parquet or raw JSON** is distributed; obtain SBAN separately.
 
 | Resource | Location |
 |----------|----------|
-| Source code | GitHub: [MalwareDatasetClassification](https://github.com/berkecanalan/MalwareDatasetClassification) |
-| Model weights | Same repo root + Hugging Face: [MalwareDatasetClassification](https://huggingface.co/canalan/MalwareDatasetClassification) |
+| Source | [github.com/berkecanalan/MalwareDatasetClassification](https://github.com/berkecanalan/MalwareDatasetClassification) |
+| Weights | Repo root + [huggingface.co/canalan/MalwareDatasetClassification](https://huggingface.co/canalan/MalwareDatasetClassification) |
 | License | Apache-2.0 ([LICENSE](LICENSE)) |
 
 ---
 
-## Problem formulation
+## Task and labels
 
-- **Input:** One malware sample described by four text fields (assembly listing, binary hex dump, source code, natural-language description NLD).
-- **Output:** One of four dataset labels (which SBAN sub-corpus the sample comes from).
-- **Note:** This is **dataset / provenance classification**, not a generic “malware vs benign” detector.
+| | |
+|---|---|
+| **Input** | `assembly_code`, `binary_code`, `source_code`, `NLD` for one sample |
+| **Output** | `dataset_name` ∈ {`bodmas`, `dike`, `malwarebazaar`, `sorel20m`} |
+| **Scope** | Dataset **provenance** classification, not generic malware detection |
 
-Reported metrics on held-out splits (from `inference.ipynb` / Colab run with exported bundle):
+---
+
+## Data preparation pipeline (scripts `01`–`11`)
+
+End-to-end flow on local SBAN exports:
+
+1. **`01_make_a_dataframe.py`** — Merge JSON shards under `data/M1/SBAN-MA-JUN25` into `SBAN.parquet` (four representations aligned by `ID`).
+2. **`02_validate_data.py`** — Schema, missing values, duplicates, cross-dataset `ID` overlap, content fingerprints (see [Data quality](#data-quality-findings)).
+3. **`03_make_clean_dataframe.py`** — Cleaning rules → `SBAN_clean.parquet`.
+4. **`04_analyze_prompt_residue.py`** — Count LLM/prompt boilerplate phrases per representation ([Prompt residue](#prompt-residue-analysis)).
+5. **`05_split_dataframe.py`** — Stratified train / validation / test parquet files.
+6. **`06_make_features.py`** — Optional TF-IDF `.npz` features for alternate experiments.
+7. **`07`–`11`** — Per-representation audits and source cleaning (`08_clean_source_code.py` uses `07_audit_source_code.py`).
 
-| Split | Accuracy | Macro F1 |
-|-------|----------|----------|
-| Validation | 0.9413 | 0.9097 |
-| Test | (run locally with your SBAN test parquet) | |
+Notebooks:
 
-Per-class validation F1 (macro average 0.9097): strong on `bodmas` and `sorel20m`; smaller classes `dike` and `malwarebazaar` are harder (typical class imbalance).
+- **`baseline.ipynb`** — Early fusion / baseline stacking comparisons.
+- **`svc_sban.ipynb`** — **Production model**: per-representation TF-IDF + numeric features, ID-based feature pruning, class-weight search, weighted `LinearSVC` bases, `HistGradientBoostingClassifier` meta learner, **joblib export**.
+- **`inference.ipynb`** — Load exported bundle; validation/test metrics; synthetic **demo** row.
+
+Canonical runtime entrypoint: **`inference.py`** (CLI + `StackingPredictor`).
 
 ---
 
-## Model architecture (production bundle)
+## Data quality findings
 
-The released artifact `sban_weighted_stacking_model.joblib` is produced in `svc_sban-temiz.ipynb`. High-level flow:
+Summaries below come from running the numbered scripts on the full merged SBAN table (before train/val/test split). Reproduce with your own copy of the data.
 
-```text
-For each representation r in {asm, binary, source, nld}:
-  raw text → TF-IDF (+ binary byte-tokenization for hex)
-           → numeric stats (char/line/token counts, entropy, …)
-           → StandardScaler on numeric block
-           → sparse hstack → feature selection (selected_indices)
-           → LinearSVC base model → decision_function scores
-
-Meta layer:
-  concat(all base decision scores + all scaled numeric blocks)
-  → HistGradientBoostingClassifier (weighted / tuned per representation)
-  → argmax → label_encoder inverse → prediction + predict_proba
-```
+### Cross-dataset `ID` overlap (content match rate)
 
-The joblib dict (`bundle_version: 1`) includes:
+Shared `ID`s across corpus pairs; percentages = share of common IDs where that column’s text is **byte-identical** (`02_validate_data.py`, section 9).
 
-- `representation_order`, `representation_columns`, `numeric_feature_names`
-- `label_encoder`, `meta_model`
-- per-representation: `vectorizer`, `numeric_scaler`, `selected_indices`, `base_model`
-- metadata: `sklearn_version`, `random_state`, `selected_class_weight_configs`
+| Pair | Common IDs | assembly | binary | source |
+|------|------------|----------|--------|--------|
+| bodmas × sorel20m | 806 | 72% | 74% | 91% |
+| bodmas × malwarebazaar | 520 | 27% | 25% | 0% |
+| bodmas × dike | 201 | 30% | 24% | 0% |
+| dike × malwarebazaar | 82 | 23% | 22% | 4% |
+| malwarebazaar × sorel20m | 99 | 41% | 47% | 0% |
+| dike × sorel20m | 46 | 44% | 48% | 0% |
 
-**Binary representation:** hex lines are parsed into space-separated byte tokens with `instsep` line separators before TF-IDF (see `inference.py`).
+High overlap for **bodmas × sorel20m** (especially source) motivates careful splitting and explains why the classifier must use subtle cues, not only exact string identity across corpora.
 
----
+### Rows with four aligned representations
 
-## Data schema (inference)
+After merge / alignment (`01_make_a_dataframe.py`):
 
-Parquet or DataFrame rows must include:
+| Dataset | Rows | Matched (4 repr.) |
+|---------|------|-------------------|
+| bodmas | 82,032 | 757 |
+| dike | 5,342 | 669 |
+| malwarebazaar | 6,048 | 905 |
+| sorel20m | 71,319 | 726 |
 
-| Column | Role |
-|--------|------|
-| `ID` | Sample identifier (optional for inference, kept in output) |
-| `dataset_name` | Ground-truth label (only for evaluation) |
-| `assembly_code` | Assembly text |
-| `binary_code` | Hex dump text |
-| `source_code` | Source text |
-| `NLD` | Natural language description |
+“Matched” = samples where all four representation fields are present for labeling and training.
 
----
+### Prompt residue analysis
+
+`04_analyze_prompt_residue.py` scans fixed English phrases (e.g. “your code”, “here”, “additional”) across columns. Illustrative totals on cleaned data:
+
+| Phrase | assembly | binary | source | NLD | Total |
+|--------|----------|--------|--------|-----|-------|
+| your code | 1 | 0 | 572 | 1 | 573 |
+| add main function | 0 | 0 | 69 | 0 | 69 |
+| code goes | 0 | 0 | 140 | 0 | 140 |
+| implementation goes | 0 | 0 | 49 | 0 | 49 |
+| corrected | 0 | 0 | 168 | 16 | 169 |
+| here | 128 | 0 | 1,493 | 323 | 1,798 |
+| no comments | 0 | 0 | 53 | 0 | 53 |
+| additional | 37 | 0 | 76 | 1,103 | 1,185 |
 
-## Repository layout
-
-| Path | Purpose |
-|------|---------|
-| `01_make_a_dataframe.py` | Build unified SBAN parquet from raw JSON layout under `data/` |
-| `02_validate_data.py` | Validation checks |
-| `03_make_clean_dataframe.py` | Cleaning |
-| `04_analyze_prompt_residue.py` | Prompt residue analysis |
-| `05_split_dataframe.py` | Train/val/test split |
-| `06_make_features.py` | TF-IDF feature export to `features/` (sparse npz) |
-| `07_audit_source_code.py` | Source audit |
-| `08_clean_source_code.py` | Source cleaning |
-| `09_audit_assembly_code.py` | Assembly audit |
-| `10_audit_binary_code.py` | Binary audit |
-| `11_audit_nld.py` | NLD audit |
-| `baseline.ipynb` | Baseline stacking experiments |
-| `svc_sban-temiz.ipynb` | Final weighted stacking training + **model export** |
-| `inference.ipynb` | Colab/local inference, metrics, **demo row** |
-| `inference.py` | Canonical Python inference + CLI |
-| `sban_weighted_stacking_model.joblib` | Exported model (~6.4 MB) — add to clone if not using Git LFS |
-
-Generated paths (`data/`, `*.parquet`, `features/`, `reports/`) are gitignored.
+Most residue sits in **source** and **NLD**; source cleaning scripts (`07`/`08`) target audit failures before modeling.
 
 ---
 
-## Installation
+## Model architecture
 
-**Inference only** (recommended for downstream use):
+Artifact: **`sban_weighted_stacking_model.joblib`** (`bundle_version: 1`, trained with **scikit-learn 1.6.1**).
 
-```bash
-python -m venv .venv
-source .venv/bin/activate
-pip install -r requirements-inference.txt
+```text
+For each r ∈ {asm, binary, source, nld}:
+  text → TF-IDF (binary: hex → byte tokens + instsep)
+       + 6 numeric stats (length, tokens, entropy, …)
+       → StandardScaler
+       → sparse hstack → column subset (selected_indices from ID pruning)
+       → LinearSVC (tuned class weights) → decision_function (4 scores)
+
+Meta:
+  hstack(all base decision scores + all scaled numeric blocks)
+  → HistGradientBoostingClassifier
+  → class probabilities
 ```
 
-**Full pipeline + notebooks:**
+Bundle keys: `representation_order`, `representation_columns`, `numeric_feature_names`, `label_encoder`, `meta_model`, `representations` (vectorizer, scaler, indices, base model), `selected_class_weight_configs`, metadata.
 
-```bash
-pip install -r requirements.txt
-```
+Training details and ablations: **`svc_sban.ipynb`**.
 
-Use **scikit-learn 1.6.1** (pinned) when loading the bundle to match training.
+### Split sizes used in training notebook
 
----
+| Split | Rows | bodmas | dike | malwarebazaar | sorel20m |
+|-------|------|--------|------|---------------|----------|
+| Train | 113,168 | 56,892 | 3,267 | 3,594 | 49,415 |
+| Validation | 16,167 | 8,128 | 467 | 513 | 7,059 |
+| Test | 32,334 | 16,255 | 933 | 1,027 | 14,119 |
 
-## Inference
+(Test counts from `inference.ipynb` evaluation on exported bundle.)
 
-### Command line
+### Base models on validation (`svc_sban.ipynb`)
 
-```bash
-python inference.py \
-  --model-path sban_weighted_stacking_model.joblib \
-  --input /path/to/SBAN_val.parquet \
-  --output predictions.parquet \
-  --evaluate
-```
+Single-representation `LinearSVC` decision scores, validation set:
 
-Output columns: all input columns plus `prediction` and `prob_<class>` for each label.
+| Representation | Accuracy | Macro F1 | Weighted F1 |
+|----------------|----------|----------|---------------|
+| asm | 0.8983 | 0.7099 | 0.8902 |
+| binary | 0.8426 | 0.6607 | 0.8345 |
+| source | 0.9253 | 0.8811 | 0.9251 |
+| nld | 0.6621 | 0.4997 | 0.6539 |
 
-### Python API
+**Source** is the strongest single view; **nld** alone is weakest but adds complementary signal in the stack.
 
-```python
-import pandas as pd
-from inference import load_predictor
+### Final exported model — validation & test
 
-predictor = load_predictor("sban_weighted_stacking_model.joblib")
-df = pd.read_parquet("/path/to/samples.parquet")
-result = predictor.predict(df)
-```
+Metrics from **`inference.ipynb`** with `sban_weighted_stacking_model.joblib` (matches weighted meta validation in `svc_sban.ipynb` before export).
+
+**Validation (n = 16,167)**
+
+| | Accuracy | Macro F1 | Weighted F1 |
+|---|----------|----------|-------------|
+| Overall | **0.9413** | **0.9097** | **0.9412** |
+
+| Class | Precision | Recall | F1 | Support |
+|-------|-----------|--------|-----|---------|
+| bodmas | 0.9551 | 0.9398 | 0.9474 | 8,128 |
+| dike | 0.9125 | 0.8266 | 0.8674 | 467 |
+| malwarebazaar | 0.8986 | 0.8635 | 0.8807 | 513 |
+| sorel20m | 0.9306 | 0.9562 | 0.9433 | 7,059 |
 
-### Notebook
+**Test (n = 32,334)**
 
-Open `inference.ipynb`: run **Colab** or **local** setup, then the demo cell (single synthetic row). Point `VAL_PATH` / `TEST_PATH` to your SBAN splits for full evaluation.
+| | Accuracy | Macro F1 | Weighted F1 |
+|---|----------|----------|-------------|
+| Overall | **0.9379** | **0.9012** | **0.9378** |
+
+| Class | Precision | Recall | F1 | Support |
+|-------|-----------|--------|-----|---------|
+| bodmas | 0.9532 | 0.9364 | 0.9447 | 16,255 |
+| dike | 0.8909 | 0.8489 | 0.8694 | 933 |
+| malwarebazaar | 0.8885 | 0.8150 | 0.8502 | 1,027 |
+| sorel20m | 0.9272 | 0.9545 | 0.9407 | 14,119 |
+
+Minority classes (`dike`, `malwarebazaar`) remain the hardest; weighted class tuning in `svc_sban.ipynb` targets that imbalance.
 
 ---
 
-## Reproducing training
+## Inference schema
+
+| Column | Required for predict | Notes |
+|--------|-------------------|--------|
+| `assembly_code`, `binary_code`, `source_code`, `NLD` | Yes | |
+| `ID` | No | Preserved in output |
+| `dataset_name` | No | For `--evaluate` / notebook metrics |
+
+---
+
+## Installation
+
+```bash
+pip install -r requirements-inference.txt   # predict only
+pip install -r requirements.txt             # full pipeline + notebooks
+```
 
-1. Place SBAN raw files under `data/M1/SBAN-MA-JUN25` (see `01_make_a_dataframe.py`).
-2. Run numbered scripts through `06_make_features.py` as needed for your experiment.
-3. Train and export with `svc_sban-temiz.ipynb` (Colab-oriented; expects parquet splits and writes `sban_weighted_stacking_model.joblib`).
+Use **scikit-learn 1.6.1** when loading the joblib bundle.
 
 ---
 
-## Hugging Face
+## Running inference
 
-Upload the model repository with:
+```bash
+python inference.py \
+  --model-path sban_weighted_stacking_model.joblib \
+  --input /path/to/SBAN_test.parquet \
+  --output predictions.parquet \
+  --evaluate
+```
 
-- `sban_weighted_stacking_model.joblib`
-- `README.md` (this file — YAML front matter is model-card compatible)
-- `inference.py`, `requirements-inference.txt`, `LICENSE`
+```python
+from inference import load_predictor
+import pandas as pd
+
+predictor = load_predictor("sban_weighted_stacking_model.joblib")
+out = predictor.predict(pd.read_parquet("/path/to/samples.parquet"))
+```
 
-Select **Apache-2.0** as the Hub license to match this repo.
+**Notebook:** `inference.ipynb` — Colab or local setup → demo row → validation/test cells (update parquet paths).
 
 ---
 
-## Citation and data
+## Reproducing the production model
 
-- **SBAN dataset:** cite and obtain data from the official SBAN project; this repo does not host samples.
-- **This model/code:** if you use the weights or pipeline, cite this repository and respect Apache-2.0.
+1. Obtain SBAN and build parquets via `01`–`05` (and cleaning/audit scripts as needed).
+2. Open **`svc_sban.ipynb`** (Colab or local), point to `SBAN_train/val/test.parquet`.
+3. Run training cells; export **`sban_weighted_stacking_model.joblib`** to the repo root.
+4. Verify with **`inference.py`** or **`inference.ipynb`**.
 
 ---
 
-## Security note
+## Citation and security
 
-`joblib.load` executes pickled objects. Only load `sban_weighted_stacking_model.joblib` from this project or your own trusted export.
+- Cite the **SBAN** dataset authors; this repo does not redistribute their files.
+- **`joblib.load` uses pickle** — only load bundles from this project or your own exports.