Expand model card

README.md

@@ -1,56 +1,199 @@
 ---
 base_model: microsoft/codebert-base
 library_name: pytorch
+pipeline_tag: text-classification
+language:
+- en
 tags:
 - security
 - command-analysis
 - malware-detection
+- mitre-attack
 - tf-idf
+- codebert
 ---
 
-# Genos model weights
+# Genos: command-line threat triage models
 
-Runtime model artifacts for the Genos command-analysis API (`genos_api`).
+Genos classifies a single shell or script command line (Linux, macOS, Windows,
+PowerShell) in two tiers. This repository holds the runtime artifacts used by
+the `genos_api` service.
+
+```
+command ─► (Base64/obfuscation decode) ─► Tier 1 gatekeeper ─► Benign / Context_Dependent / Malicious
+                                              │
+                                              └─► Tier 2: family specialist (11 tactic families)
+                                                          + behavior encoder (attack stage + action tags)
+```
+
+Benign commands skip Tier 2 by default. Every output is a model estimate, not a
+verdict to act on automatically.
 
 ## Files
 
-| File | Format | Purpose |
-|---|---|---|
-| `gatekeeper.pt` | PyTorch (CodeBERT fine-tune) | Tier 1 three-class gatekeeper: Benign / Context_Dependent / Malicious |
-| `behavior_encoder.pt`, `behavior_encoder.json` | PyTorch (CodeBERT fine-tune) + config | Behavior encoder: 13 attack stages and 9 action tags, `max_length` 256 |
-| `family_specialist_tfidf.joblib` | joblib (scikit-learn) | Tier 2 specialist: calibrated multi-label 11-family TF-IDF linear classifier |
-| `family_specialist_tfidf.json` | JSON | Training configuration and feature/calibration metadata for the Tier 2 specialist |
+| File | Format | Size | Purpose |
+|---|---|---|---|
+| `gatekeeper.pt` | PyTorch checkpoint | 502 MB | Tier 1 three-class gatekeeper |
+| `family_specialist_tfidf.joblib` | joblib (scikit-learn) | 33 MB | Tier 2 default specialist: calibrated multi-label 11-family classifier |
+| `family_specialist_tfidf.json` | JSON | 45 KB | Specialist training config, split audit, metrics, and checkpoint hash |
+| `behavior_encoder.pt` | PyTorch checkpoint | 499 MB | Attack-stage and action-tag encoder |
+| `behavior_encoder.json` | JSON | 1 KB | Behavior label maps and test metrics |
+
+The CodeBERT backbone (`microsoft/codebert-base`) is **not** included; download
+it separately (see Usage).
+
+## Models
+
+### Tier 1: gatekeeper (`gatekeeper.pt`)
+
+- **Architecture:** shared CodeBERT encoder with decision-decomposition heads
+  (`verdict_logits`, `non_benign_logit`, `malicious_given_non_benign_logit`,
+  `ordinal_risk_logit`).
+- **Classes:** `Benign` (0), `Malicious` (1), `Context_Dependent` (2).
+  `Context_Dependent` marks commands that cannot be judged benign or malicious
+  without environmental context.
+- **Training:** 5 epochs (best epoch 5), max length 256, learning rate 1e-5,
+  weight decay 0.01, label smoothing 0.05, effective batch size 256, balanced
+  sampling, seed 42, plus a 250-row benign operational patch.
+- **Reported metrics** (from `config/gatekeeper_meta.json` in the source repo):
+
+| Split | Accuracy | Macro F1 |
+|---|---:|---:|
+| Validation | 0.963 | 0.955 |
+| Test | 0.948 | 0.932 |
+
+Test per class: Benign P 0.968 / R 0.957 / F1 0.962; Malicious P 0.944 / R 0.970 /
+F1 0.957; Context_Dependent P 0.880 / R 0.871 / F1 0.876.
+
+> These numbers come from the original split used when this checkpoint was
+> trained. The project's later dataset audit found conflicting and
+> development-exposed rows in earlier splits, so these metrics should be treated
+> as optimistic and not as evidence of production accuracy.
+
+### Tier 2: family specialist (`family_specialist_tfidf.joblib`)
+
+- **Architecture:** TF-IDF features (character `char_wb` 2-5-grams, 250k features;
+  word 1-2-grams with a shell-aware token pattern, 100k features) feeding one
+  linear SVM per family, with per-family sigmoid calibration fit on train only
+  using template-group-isolated folds.
+- **Output:** multi-label scores over 11 families: Execution, Persistence,
+  Privilege Escalation, Defense Evasion, Credential Access, Discovery, Lateral
+  Movement, Command-and-Control / Payload Retrieval, Exfiltration, Impact,
+  Benign Admin. Decision threshold 0.5. No MITRE technique IDs are predicted.
+- **Data:** 37,846 train / 4,740 validation / 4,740 test commands, split 80/10/10
+  by parser residual-template group (seed 42) so normalized commands and
+  templates never cross splits. The split audit found zero command overlap
+  between splits.
+- **Speed:** about 2.9 ms median (3.1 ms p95) per command on CPU, excluding HTTP
+  and the other models.
+
+Test-set results (threshold 0.5):
+
+| Family | Precision | Recall | F1 | Support |
+|---|---:|---:|---:|---:|
+| Execution | 0.62 | 0.48 | 0.54 | 92 |
+| Persistence | 0.77 | 0.62 | 0.68 | 144 |
+| Privilege Escalation | 0.73 | 0.50 | 0.59 | 147 |
+| Defense Evasion | 0.77 | 0.56 | 0.65 | 280 |
+| Credential Access | 0.82 | 0.53 | 0.64 | 59 |
+| Discovery | 0.74 | 0.58 | 0.65 | 115 |
+| Lateral Movement | 1.00 | 0.14 | 0.25 | 7 |
+| Command-and-Control / Payload Retrieval | 0.81 | 0.54 | 0.65 | 39 |
+| Exfiltration | 1.00 | 0.11 | 0.20 | 9 |
+| Impact | 0.50 | 0.38 | 0.43 | 21 |
+| Benign Admin | 0.98 | 0.99 | 0.99 | 4,027 |
 
-The CodeBERT backbone used by the Tier 1 gatekeeper is not included. Fetch it
-separately from [`microsoft/codebert-base`](https://huggingface.co/microsoft/codebert-base).
+Macro F1 is 0.570 on test (0.604 on validation). The top-1 prediction matches at
+least one true family 93.7% of the time (top-2: 96.0%). The weakest families
+(Lateral Movement, Exfiltration, Impact) have very few test examples, so their
+scores are noisy. Precision is generally higher than recall: the model tends to
+miss attack families rather than invent them, and its most common error is
+predicting `Benign Admin` for a Defense Evasion or Discovery command.
+
+### Tier 2: behavior encoder (`behavior_encoder.pt`)
+
+- **Architecture:** CodeBERT encoder with a stage classifier and a multi-label
+  action head, max length 256.
+- **Stages (13):** C2 / Remote Access, Collection / Staging, Context Required,
+  Credential Access, Defense Evasion, Discovery / Recon, Execution, Exfiltration,
+  Impact, Lateral Movement, Payload Retrieval, Persistence, Privilege Escalation.
+- **Action tags (9):** archive_data, download_remote_resource,
+  execute_inline_code, execute_interpreter, extract_archive, remote_execution,
+  use_encoded_payload, use_obfuscation, use_signed_proxy_binary.
+- **Reported test metrics:** stage accuracy 0.565, stage macro F1 0.500, action
+  micro F1 0.936.
+
+> Action labels are derived from the same parser and rule features used for
+> input, so the high action F1 shows the model reproduces those rules. It is not
+> independent evidence of behavioral understanding. Stage accuracy of 0.565 over
+> 13 classes is modest.
 
 ## Usage
 
-Download into the repository's `models/` directory, which is where the
-application expects these files:
+Download the files into the application's `models/` directory:
 
 ```bash
 hf download genos-security/genos --local-dir models
 hf download microsoft/codebert-base --local-dir models/codebert-base
 ```
 
-## Security note
+Then point the service at them:
+
+```bash
+GENOS_CODEBERT_PATH=models/codebert-base
+GENOS_HF_LOCAL_ONLY=1
+GENOS_SPECIALIST_MODE=family
+GENOS_FAMILY_SPECIALIST_PATH=models/family_specialist_tfidf.joblib
+```
 
-The `.pt` and `.joblib` formats are Python pickles. Loading them runs code
-embedded in the file. Load these artifacts only from sources you trust, and
-verify checksums before deploying.
+The `.pt` files expect the `genos_api` code, label maps in `config/`, and the
+CodeBERT tokenizer. They are not drop-in `transformers` checkpoints.
 
 ## Intended use
 
-Defensive analysis and research on suspicious shell and script commands. These
-models are classifiers and should be used as one signal, not as a sole basis for
-blocking or attributing activity.
+- Triage and prioritization of suspicious command lines in defensive tooling,
+  research, and education.
+- Use as one signal alongside other telemetry, with a human in the loop.
+
+## Out-of-scope use
+
+- Automatically blocking, killing, or attributing activity from a score alone.
+- Judging full scripts, binaries, or multi-command sessions. Inputs are single
+  command lines truncated to 256 tokens.
+- Offensive use, such as tuning commands to evade the models.
+- Claims of production accuracy.
+
+## Limitations and known issues
+
+- **Weak labels.** Training labels come from source-derived tactic families and
+  parser rules, not human review. Independent annotation is not complete.
+- **Local evaluation only.** Results are from template-grouped splits of the
+  same data distribution. Generalization to new environments, attacker
+  tooling, or novel obfuscation has not been measured.
+- **Cross-component overlap.** Some test commands for one component appear in
+  another component's training data, so the tiers are not evaluated end to end
+  on untouched data.
+- **Uncalibrated gatekeeper.** Scores are estimates, not probabilities, unless a
+  validation-fitted calibration artifact is supplied.
+- **Known false positive.** The gatekeeper has scored the benign command `pwd` as
+  Malicious (about 52%, uncalibrated), and the behavior model labeled it
+  Persistence. Expect other false positives on very short commands.
+- **Low-support families.** Lateral Movement, Exfiltration, and Impact are
+  recalled poorly (recall 0.11-0.38).
+- **English, command-line text only.**
+- **Training data traces.** The TF-IDF vocabulary is built from training
+  commands and may contain fragments of them.
+
+## Security note
+
+`.pt` and `.joblib` files are Python pickles. Loading one runs code embedded in
+the file. Only load these artifacts from this repository's verified revisions,
+and check file hashes before deployment. The specialist records its own SHA-256
+in `family_specialist_tfidf.json` (`checkpoint_sha256`).
 
-## Limitations
+## Citation and license
 
-- Training labels are weak supervision. Independent human review of the labels is
-  not complete.
-- Scores are uncalibrated model estimates unless a validation-fitted calibration
-  artifact is supplied.
-- The TF-IDF artifact stores vocabulary built from training commands, so it may
-  contain fragments of that data.
+No license has been declared for these weights yet. Contact the maintainers
+before redistribution or commercial use. The base model,
+[`microsoft/codebert-base`](https://huggingface.co/microsoft/codebert-base), is
+released under the MIT license.