genos / README.md

Update README.md

36ae71c verified 12 days ago

8.81 kB

---
base_model: microsoft/codebert-base
library_name: pytorch
pipeline_tag: text-classification
language:
- en
tags:
- security
- command-analysis
- malware-detection
- mitre-attack
- tf-idf
- codebert
license: apache-2.0
---
# Genos: command-line threat triage models

Genos classifies a single shell or script command line (Linux, macOS, Windows,
PowerShell) in two tiers. This repository holds the runtime artifacts used by
the `genos_api` service.

```
command ─► (Base64/obfuscation decode) ─► Tier 1 gatekeeper ─► Benign / Context_Dependent / Malicious
                                              │
                                              └─► Tier 2: family specialist (11 tactic families)
                                                          + behavior encoder (attack stage + action tags)
```

Benign commands skip Tier 2 by default. Every output is a model estimate, not a
verdict to act on automatically.

## Files

| File | Format | Size | Purpose |
|---|---|---|---|
| `gatekeeper.pt` | PyTorch checkpoint | 502 MB | Tier 1 three-class gatekeeper |
| `family_specialist_tfidf.joblib` | joblib (scikit-learn) | 33 MB | Tier 2 default specialist: calibrated multi-label 11-family classifier |
| `family_specialist_tfidf.json` | JSON | 45 KB | Specialist training config, split audit, metrics, and checkpoint hash |
| `behavior_encoder.pt` | PyTorch checkpoint | 499 MB | Attack-stage and action-tag encoder |
| `behavior_encoder.json` | JSON | 1 KB | Behavior label maps and test metrics |

The CodeBERT backbone (`microsoft/codebert-base`) is **not** included; download
it separately (see Usage).

## Models

### Tier 1: gatekeeper (`gatekeeper.pt`)

- **Architecture:** shared CodeBERT encoder with decision-decomposition heads
  (`verdict_logits`, `non_benign_logit`, `malicious_given_non_benign_logit`,
  `ordinal_risk_logit`).
- **Classes:** `Benign` (0), `Malicious` (1), `Context_Dependent` (2).
  `Context_Dependent` marks commands that cannot be judged benign or malicious
  without environmental context.
- **Training:** 5 epochs (best epoch 5), max length 256, learning rate 1e-5,
  weight decay 0.01, label smoothing 0.05, effective batch size 256, balanced
  sampling, seed 42, plus a 250-row benign operational patch.
- **Reported metrics** (from `config/gatekeeper_meta.json` in the source repo):

| Split | Accuracy | Macro F1 |
|---|---:|---:|
| Validation | 0.963 | 0.955 |
| Test | 0.948 | 0.932 |

Test per class: Benign P 0.968 / R 0.957 / F1 0.962; Malicious P 0.944 / R 0.970 /
F1 0.957; Context_Dependent P 0.880 / R 0.871 / F1 0.876.
> These numbers come from the original split used when this checkpoint was
> trained. The project's later dataset audit found conflicting and
> development-exposed rows in earlier splits, so these metrics should be treated
> as optimistic and not as evidence of production accuracy.
### Tier 2: family specialist (`family_specialist_tfidf.joblib`)
- **Architecture:** TF-IDF features (character `char_wb` 2-5-grams, 250k features;
  word 1-2-grams with a shell-aware token pattern, 100k features) feeding one
  linear SVM per family, with per-family sigmoid calibration fit on train only
  using template-group-isolated folds.
- **Output:** multi-label scores over 11 families: Execution, Persistence,
  Privilege Escalation, Defense Evasion, Credential Access, Discovery, Lateral
  Movement, Command-and-Control / Payload Retrieval, Exfiltration, Impact,
  Benign Admin. Decision threshold 0.5. No MITRE technique IDs are predicted.
- **Data:** 37,846 train / 4,740 validation / 4,740 test commands, split 80/10/10
  by parser residual-template group (seed 42) so normalized commands and
  templates never cross splits. The split audit found zero command overlap
  between splits.
- **Speed:** about 2.9 ms median (3.1 ms p95) per command on CPU, excluding HTTP
  and the other models.

Test-set results (threshold 0.5):

| Family | Precision | Recall | F1 | Support |
|---|---:|---:|---:|---:|
| Execution | 0.62 | 0.48 | 0.54 | 92 |
| Persistence | 0.77 | 0.62 | 0.68 | 144 |
| Privilege Escalation | 0.73 | 0.50 | 0.59 | 147 |
| Defense Evasion | 0.77 | 0.56 | 0.65 | 280 |
| Credential Access | 0.82 | 0.53 | 0.64 | 59 |
| Discovery | 0.74 | 0.58 | 0.65 | 115 |
| Lateral Movement | 1.00 | 0.14 | 0.25 | 7 |
| Command-and-Control / Payload Retrieval | 0.81 | 0.54 | 0.65 | 39 |
| Exfiltration | 1.00 | 0.11 | 0.20 | 9 |
| Impact | 0.50 | 0.38 | 0.43 | 21 |
| Benign Admin | 0.98 | 0.99 | 0.99 | 4,027 |

Macro F1 is 0.570 on test (0.604 on validation). The top-1 prediction matches at
least one true family 93.7% of the time (top-2: 96.0%). The weakest families
(Lateral Movement, Exfiltration, Impact) have very few test examples, so their
scores are noisy. Precision is generally higher than recall: the model tends to
miss attack families rather than invent them, and its most common error is
predicting `Benign Admin` for a Defense Evasion or Discovery command.

### Tier 2: behavior encoder (`behavior_encoder.pt`)
- **Architecture:** CodeBERT encoder with a stage classifier and a multi-label
  action head, max length 256.
- **Stages (13):** C2 / Remote Access, Collection / Staging, Context Required,
  Credential Access, Defense Evasion, Discovery / Recon, Execution, Exfiltration,
  Impact, Lateral Movement, Payload Retrieval, Persistence, Privilege Escalation.
- **Action tags (9):** archive_data, download_remote_resource,
  execute_inline_code, execute_interpreter, extract_archive, remote_execution,
  use_encoded_payload, use_obfuscation, use_signed_proxy_binary.
- **Reported test metrics:** stage accuracy 0.565, stage macro F1 0.500, action
  micro F1 0.936.
> Action labels are derived from the same parser and rule features used for
> input, so the high action F1 shows the model reproduces those rules. It is not
> independent evidence of behavioral understanding. Stage accuracy of 0.565 over
> 13 classes is modest.
## Usage
Download the files into the application's `models/` directory:
```bash
hf download genos-security/genos --local-dir models
hf download microsoft/codebert-base --local-dir models/codebert-base
```
Then point the service at them:
```bash
GENOS_CODEBERT_PATH=models/codebert-base
GENOS_HF_LOCAL_ONLY=1
GENOS_SPECIALIST_MODE=family
GENOS_FAMILY_SPECIALIST_PATH=models/family_specialist_tfidf.joblib
```
The `.pt` files expect the `genos_api` code, label maps in `config/`, and the
CodeBERT tokenizer. They are not drop-in `transformers` checkpoints.

## Intended use

- Triage and prioritization of suspicious command lines in defensive tooling,
  research, and education.
- Use as one signal alongside other telemetry, with a human in the loop.

## Out-of-scope use

- Automatically blocking, killing, or attributing activity from a score alone.
- Judging full scripts, binaries, or multi-command sessions. Inputs are single
  command lines truncated to 256 tokens.
- Offensive use, such as tuning commands to evade the models.
- Claims of production accuracy.

## Limitations and known issues

- **Weak labels.** Training labels come from source-derived tactic families and
  parser rules, not human review. Independent annotation is not complete.
- **Local evaluation only.** Results are from template-grouped splits of the
  same data distribution. Generalization to new environments, attacker
  tooling, or novel obfuscation has not been measured.
- **Cross-component overlap.** Some test commands for one component appear in
  another component's training data, so the tiers are not evaluated end to end
  on untouched data.
- **Uncalibrated gatekeeper.** Scores are estimates, not probabilities, unless a
  validation-fitted calibration artifact is supplied.
- **Known false positive.** The gatekeeper has scored the benign command `pwd` as
  Malicious (about 52%, uncalibrated), and the behavior model labeled it
  Persistence. Expect other false positives on very short commands.
- **Low-support families.** Lateral Movement, Exfiltration, and Impact are
  recalled poorly (recall 0.11-0.38).
- **English, command-line text only.**
- **Training data traces.** The TF-IDF vocabulary is built from training
  commands and may contain fragments of them.

## Security note

`.pt` and `.joblib` files are Python pickles. Loading one runs code embedded in
the file. Only load these artifacts from this repository's verified revisions,
and check file hashes before deployment. The specialist records its own SHA-256
in `family_specialist_tfidf.json` (`checkpoint_sha256`).

## Citation and license

No license has been declared for these weights yet. Contact the maintainers
before redistribution or commercial use. The base model,
[`microsoft/codebert-base`](https://huggingface.co/microsoft/codebert-base), is
released under the MIT license.

	---
	base_model: microsoft/codebert-base
	library_name: pytorch
	pipeline_tag: text-classification
	language:
	- en
	tags:
	- security
	- command-analysis
	- malware-detection
	- mitre-attack
	- tf-idf
	- codebert
	license: apache-2.0
	---

	# Genos: command-line threat triage models

	Genos classifies a single shell or script command line (Linux, macOS, Windows,
	PowerShell) in two tiers. This repository holds the runtime artifacts used by
	the `genos_api` service.

	```
	command ─► (Base64/obfuscation decode) ─► Tier 1 gatekeeper ─► Benign / Context_Dependent / Malicious
	│
	└─► Tier 2: family specialist (11 tactic families)
	+ behavior encoder (attack stage + action tags)
	```

	Benign commands skip Tier 2 by default. Every output is a model estimate, not a
	verdict to act on automatically.

	## Files

	\| File \| Format \| Size \| Purpose \|
	\|---\|---\|---\|---\|
	\| `gatekeeper.pt` \| PyTorch checkpoint \| 502 MB \| Tier 1 three-class gatekeeper \|
	\| `family_specialist_tfidf.joblib` \| joblib (scikit-learn) \| 33 MB \| Tier 2 default specialist: calibrated multi-label 11-family classifier \|
	\| `family_specialist_tfidf.json` \| JSON \| 45 KB \| Specialist training config, split audit, metrics, and checkpoint hash \|
	\| `behavior_encoder.pt` \| PyTorch checkpoint \| 499 MB \| Attack-stage and action-tag encoder \|
	\| `behavior_encoder.json` \| JSON \| 1 KB \| Behavior label maps and test metrics \|

	The CodeBERT backbone (`microsoft/codebert-base`) is not included; download
	it separately (see Usage).

	## Models

	### Tier 1: gatekeeper (`gatekeeper.pt`)

	- Architecture: shared CodeBERT encoder with decision-decomposition heads
	(`verdict_logits`, `non_benign_logit`, `malicious_given_non_benign_logit`,
	`ordinal_risk_logit`).
	- Classes: `Benign` (0), `Malicious` (1), `Context_Dependent` (2).
	`Context_Dependent` marks commands that cannot be judged benign or malicious
	without environmental context.
	- Training: 5 epochs (best epoch 5), max length 256, learning rate 1e-5,
	weight decay 0.01, label smoothing 0.05, effective batch size 256, balanced
	sampling, seed 42, plus a 250-row benign operational patch.
	- Reported metrics (from `config/gatekeeper_meta.json` in the source repo):

	\| Split \| Accuracy \| Macro F1 \|
	\|---\|---:\|---:\|
	\| Validation \| 0.963 \| 0.955 \|
	\| Test \| 0.948 \| 0.932 \|

	Test per class: Benign P 0.968 / R 0.957 / F1 0.962; Malicious P 0.944 / R 0.970 /
	F1 0.957; Context_Dependent P 0.880 / R 0.871 / F1 0.876.

	> These numbers come from the original split used when this checkpoint was
	> trained. The project's later dataset audit found conflicting and
	> development-exposed rows in earlier splits, so these metrics should be treated
	> as optimistic and not as evidence of production accuracy.

	### Tier 2: family specialist (`family_specialist_tfidf.joblib`)

	- Architecture: TF-IDF features (character `char_wb` 2-5-grams, 250k features;
	word 1-2-grams with a shell-aware token pattern, 100k features) feeding one
	linear SVM per family, with per-family sigmoid calibration fit on train only
	using template-group-isolated folds.
	- Output: multi-label scores over 11 families: Execution, Persistence,
	Privilege Escalation, Defense Evasion, Credential Access, Discovery, Lateral
	Movement, Command-and-Control / Payload Retrieval, Exfiltration, Impact,
	Benign Admin. Decision threshold 0.5. No MITRE technique IDs are predicted.
	- Data: 37,846 train / 4,740 validation / 4,740 test commands, split 80/10/10
	by parser residual-template group (seed 42) so normalized commands and
	templates never cross splits. The split audit found zero command overlap
	between splits.
	- Speed: about 2.9 ms median (3.1 ms p95) per command on CPU, excluding HTTP
	and the other models.

	Test-set results (threshold 0.5):

	\| Family \| Precision \| Recall \| F1 \| Support \|
	\|---\|---:\|---:\|---:\|---:\|
	\| Execution \| 0.62 \| 0.48 \| 0.54 \| 92 \|
	\| Persistence \| 0.77 \| 0.62 \| 0.68 \| 144 \|
	\| Privilege Escalation \| 0.73 \| 0.50 \| 0.59 \| 147 \|
	\| Defense Evasion \| 0.77 \| 0.56 \| 0.65 \| 280 \|
	\| Credential Access \| 0.82 \| 0.53 \| 0.64 \| 59 \|
	\| Discovery \| 0.74 \| 0.58 \| 0.65 \| 115 \|
	\| Lateral Movement \| 1.00 \| 0.14 \| 0.25 \| 7 \|
	\| Command-and-Control / Payload Retrieval \| 0.81 \| 0.54 \| 0.65 \| 39 \|
	\| Exfiltration \| 1.00 \| 0.11 \| 0.20 \| 9 \|
	\| Impact \| 0.50 \| 0.38 \| 0.43 \| 21 \|
	\| Benign Admin \| 0.98 \| 0.99 \| 0.99 \| 4,027 \|

	Macro F1 is 0.570 on test (0.604 on validation). The top-1 prediction matches at
	least one true family 93.7% of the time (top-2: 96.0%). The weakest families
	(Lateral Movement, Exfiltration, Impact) have very few test examples, so their
	scores are noisy. Precision is generally higher than recall: the model tends to
	miss attack families rather than invent them, and its most common error is
	predicting `Benign Admin` for a Defense Evasion or Discovery command.

	### Tier 2: behavior encoder (`behavior_encoder.pt`)

	- Architecture: CodeBERT encoder with a stage classifier and a multi-label
	action head, max length 256.
	- Stages (13): C2 / Remote Access, Collection / Staging, Context Required,
	Credential Access, Defense Evasion, Discovery / Recon, Execution, Exfiltration,
	Impact, Lateral Movement, Payload Retrieval, Persistence, Privilege Escalation.
	- Action tags (9): archive_data, download_remote_resource,
	execute_inline_code, execute_interpreter, extract_archive, remote_execution,
	use_encoded_payload, use_obfuscation, use_signed_proxy_binary.
	- Reported test metrics: stage accuracy 0.565, stage macro F1 0.500, action
	micro F1 0.936.

	> Action labels are derived from the same parser and rule features used for
	> input, so the high action F1 shows the model reproduces those rules. It is not
	> independent evidence of behavioral understanding. Stage accuracy of 0.565 over
	> 13 classes is modest.

	## Usage

	Download the files into the application's `models/` directory:

	```bash
	hf download genos-security/genos --local-dir models
	hf download microsoft/codebert-base --local-dir models/codebert-base
	```

	Then point the service at them:

	```bash
	GENOS_CODEBERT_PATH=models/codebert-base
	GENOS_HF_LOCAL_ONLY=1
	GENOS_SPECIALIST_MODE=family
	GENOS_FAMILY_SPECIALIST_PATH=models/family_specialist_tfidf.joblib
	```

	The `.pt` files expect the `genos_api` code, label maps in `config/`, and the
	CodeBERT tokenizer. They are not drop-in `transformers` checkpoints.

	## Intended use

	- Triage and prioritization of suspicious command lines in defensive tooling,
	research, and education.
	- Use as one signal alongside other telemetry, with a human in the loop.

	## Out-of-scope use

	- Automatically blocking, killing, or attributing activity from a score alone.
	- Judging full scripts, binaries, or multi-command sessions. Inputs are single
	command lines truncated to 256 tokens.
	- Offensive use, such as tuning commands to evade the models.
	- Claims of production accuracy.

	## Limitations and known issues

	- Weak labels. Training labels come from source-derived tactic families and
	parser rules, not human review. Independent annotation is not complete.
	- Local evaluation only. Results are from template-grouped splits of the
	same data distribution. Generalization to new environments, attacker
	tooling, or novel obfuscation has not been measured.
	- Cross-component overlap. Some test commands for one component appear in
	another component's training data, so the tiers are not evaluated end to end
	on untouched data.
	- Uncalibrated gatekeeper. Scores are estimates, not probabilities, unless a
	validation-fitted calibration artifact is supplied.
	- Known false positive. The gatekeeper has scored the benign command `pwd` as
	Malicious (about 52%, uncalibrated), and the behavior model labeled it
	Persistence. Expect other false positives on very short commands.
	- Low-support families. Lateral Movement, Exfiltration, and Impact are
	recalled poorly (recall 0.11-0.38).
	- English, command-line text only.
	- Training data traces. The TF-IDF vocabulary is built from training
	commands and may contain fragments of them.

	## Security note

	`.pt` and `.joblib` files are Python pickles. Loading one runs code embedded in
	the file. Only load these artifacts from this repository's verified revisions,
	and check file hashes before deployment. The specialist records its own SHA-256
	in `family_specialist_tfidf.json` (`checkpoint_sha256`).

	## Citation and license

	No license has been declared for these weights yet. Contact the maintainers
	before redistribution or commercial use. The base model,
	[`microsoft/codebert-base`](https://huggingface.co/microsoft/codebert-base), is
	released under the MIT license.