Upload README.md with huggingface_hub

e51178b verified 3 months ago

7.53 kB

---
license: mit
library_name: pytorch
tags:
  - log-analysis
  - anomaly-detection
  - security
  - transformer
  - gpt
  - language-model
  - from-scratch
pipeline_tag: text-generation
---
# LogSentry-LM

A small, from-scratch **GPT-style language model** trained on real server logs to
(1) autoregressively complete log lines and (2) **detect anomalous log lines** by
how "surprised" the model is by them (perplexity-based scoring).

It is a compact, educational decoder-only Transformer (~2M parameters) built with
plain PyTorch — no `transformers` modeling class — intended for learning and for
lightweight log anomaly detection.

> 🚀 **Try the live demo:** https://log-sentry-lm.streamlit.app/ — paste log lines
> and see them scored in real time.

## Model details

- **Architecture:** decoder-only Transformer (GPT-style), causal self-attention,
  pre-norm residual blocks, learned token + positional embeddings.
- **Size:** `d_model=128`, `n_heads=4`, `n_layers=3`, `max_seq_len=128` (~2.03M params).
- **Tokenizer:** Byte-Pair Encoding (`tokenizers`), vocab ≈ 5.5k.
- **Training data:** combined **Loghub** samples — Linux + Apache + OpenSSH
  (`*_2k.log`, ~6k lines total).
- **Objective:** next-token prediction (cross-entropy).
- **Framework:** PyTorch (trained on Apple Silicon MPS).

## Files in this repo

| File | Purpose |
| --- | --- |
| `logsentry_lm.pt` | Trained weights + anomaly threshold + config |
| `log_tokenizer.json` | BPE tokenizer |
| `architecture.py` | Transformer building blocks |
| `dataset.py` | Tokenizer loader + dataset |
| `train_and_generate.py` | Model class, training, generation, anomaly scoring |
| `detect_anomalies.py` | Standalone anomaly predictor |

## Usage

> ⚠️ This is a **custom architecture**, so `AutoModel.from_pretrained` does **not**
> work. Download the files and load manually:
```python
from huggingface_hub import hf_hub_download
import torch
from train_and_generate import LogSentryLM, score_line
from dataset import build_or_load_tokenizer

repo = "sankath/LogSentry-LM"
ckpt_path = hf_hub_download(repo, "logsentry_lm.pt")
tok_path  = hf_hub_download(repo, "log_tokenizer.json")

ckpt = torch.load(ckpt_path, map_location="cpu")
model = LogSentryLM(vocab_size=ckpt["vocab_size"], max_seq_len=ckpt["max_seq_len"])
model.load_state_dict(ckpt["model_state"]); model.eval()
tokenizer = build_or_load_tokenizer(None, tok_path)  # loads the cached tokenizer
loss, ppl = score_line(model, tokenizer, "Failed password for root from 1.2.3.4 port 22 ssh2", "cpu")
print("anomaly" if loss > ckpt["threshold"] else "normal", f"(perplexity={ppl:.1f})")
```
## How anomaly detection works
The model learns what *normal* logs look like. A line it cannot predict scores a
high per-token loss (high perplexity) and is flagged as anomalous. The cutoff is
`threshold = mean + 2·std` of the training corpus line scores, stored in the
checkpoint. Reference behavior:
| Input | Perplexity | Verdict |
| --- | --- | --- |
| Normal Apache `[notice]` line | ~2 | normal |
| Real SSH login attempt | ~12–16 | borderline |
| Random gibberish / injected text | >100,000 | anomaly |
## Applications & use cases
Because the model learns the "grammar" of normal system logs, it can support a
range of log-analytics and security tasks:
- **Security monitoring / intrusion detection** — flag suspicious log lines
  (brute-force SSH attempts, injected commands, malformed requests) that don't
  match learned normal patterns. This is the core "sentry" use case.
- **Operational anomaly detection (AIOps)** — surface rare error conditions,
  misconfigurations, or failing components before they escalate, by ranking log
  lines by how "surprising" they are.
- **Log triage & noise reduction** — instead of reading thousands of lines,
  engineers review only the top-scoring (most unusual) ones.
- **Log autocompletion / templating** — suggest or complete log-line formats given
  a prefix (useful in tooling and for synthetic log generation for testing).
- **Data-quality / format validation** — detect corrupted, truncated, or
  non-conforming log entries in an ingestion pipeline.
- **Education** — a compact, readable reference for how a GPT-style Transformer,
  tokenizer, and perplexity-based anomaly scoring work end to end.
### How it can be used
- **Batch scan a log file** — score every line and export the anomalies:
  `python detect_anomalies.py --file /var/log/auth.log`
- **Score individual lines** programmatically via `score_line(...)` and compare
  against the saved `threshold` (see the usage snippet above).
- **Stream integration** — call the scorer inside a log shipper (Fluent Bit /
  Logstash / a Kafka consumer) to tag or route high-perplexity lines in near real
  time.
- **Alerting** — wire flagged lines into Slack/email/PagerDuty or a SIEM
  (Splunk, Elastic) as an enrichment signal.
- **Retrain on your own logs** — point `LOG_FILE` at your data and run
  `train_and_generate.py` to specialize the model to your environment.
## Future scope
Natural next steps to make this research prototype more capable and production-ready:
- **Per-source models & thresholds** — separate models for SSH, Apache, kernel,
  etc., removing the single-global-threshold weakness.
- **Session / sequence-level modeling** — detect anomalous *sequences* of events
  (e.g. login → privilege escalation → data access), not just single lines.
- **Larger context & scale** — bigger `d_model`, more layers, longer
  `max_seq_len`, and training on full Loghub datasets (millions of lines).
- **Template-aware tokenization** — parse logs into templates + variables (à la
  Drain) so IPs/IDs don't inflate the vocabulary or the anomaly score.
- **Calibrated scoring** — convert raw perplexity into probabilities/severity
  levels, with per-line-position attribution ("which token was surprising?").
- **Real-time serving** — expose the scorer as a REST/gRPC service or a Hugging
  Face **Space** (Gradio demo) for interactive use.
- **`transformers` integration** — wrap as a `PreTrainedModel` so it loads via
  `AutoModel.from_pretrained` and plugs into the standard ecosystem.
- **Evaluation & benchmarking** — measure precision/recall on labeled anomaly
  datasets (e.g. BGL, HDFS) against baselines like DeepLog and LogBERT.
- **Explainability & feedback loop** — let analysts confirm/dismiss flags to
  continuously refine the threshold and reduce false positives.
## Limitations
- Trained on a **small, mixed** corpus (three log sources). A single global
  threshold is imperfect across sources; per-source models/thresholds work better.
- Not production hardened — intended for education and experimentation.
- Small context window (128 tokens) and small vocab; it models line-level, not
  session-level, structure.
## Attribution & licensing
- Training data derived from the **[Loghub](https://github.com/logpai/loghub)**
  collection (Linux, Apache, OpenSSH samples). Please cite Loghub and consult its
  terms for any redistribution of the underlying logs.
- Model code released under the MIT license.
```bibtex
@inproceedings{zhu2023loghub,
  title     = {Loghub: A Large Collection of System Log
               Datasets for AI-driven Log Analytics},
  author    = {Zhu, Jieming and He, Shilin and He, Pinjia and
               Liu, Jinyang and Lyu, Michael R.},
  booktitle = {IEEE International Symposium on Software
               Reliability Engineering (ISSRE)},
  year      = {2023}
}
```

	---
	license: mit
	library_name: pytorch
	tags:
	- log-analysis
	- anomaly-detection
	- security
	- transformer
	- gpt
	- language-model
	- from-scratch
	pipeline_tag: text-generation
	---

	# LogSentry-LM

	A small, from-scratch GPT-style language model trained on real server logs to
	(1) autoregressively complete log lines and (2) detect anomalous log lines by
	how "surprised" the model is by them (perplexity-based scoring).

	It is a compact, educational decoder-only Transformer (~2M parameters) built with
	plain PyTorch — no `transformers` modeling class — intended for learning and for
	lightweight log anomaly detection.

	> 🚀 Try the live demo: https://log-sentry-lm.streamlit.app/ — paste log lines
	> and see them scored in real time.

	## Model details

	- Architecture: decoder-only Transformer (GPT-style), causal self-attention,
	pre-norm residual blocks, learned token + positional embeddings.
	- Size: `d_model=128`, `n_heads=4`, `n_layers=3`, `max_seq_len=128` (~2.03M params).
	- Tokenizer: Byte-Pair Encoding (`tokenizers`), vocab ≈ 5.5k.
	- Training data: combined Loghub samples — Linux + Apache + OpenSSH
	(`*_2k.log`, ~6k lines total).
	- Objective: next-token prediction (cross-entropy).
	- Framework: PyTorch (trained on Apple Silicon MPS).

	## Files in this repo

	\| File \| Purpose \|
	\| --- \| --- \|
	\| `logsentry_lm.pt` \| Trained weights + anomaly threshold + config \|
	\| `log_tokenizer.json` \| BPE tokenizer \|
	\| `architecture.py` \| Transformer building blocks \|
	\| `dataset.py` \| Tokenizer loader + dataset \|
	\| `train_and_generate.py` \| Model class, training, generation, anomaly scoring \|
	\| `detect_anomalies.py` \| Standalone anomaly predictor \|

	## Usage

	> ⚠️ This is a custom architecture, so `AutoModel.from_pretrained` does not
	> work. Download the files and load manually:

	```python
	from huggingface_hub import hf_hub_download
	import torch
	from train_and_generate import LogSentryLM, score_line
	from dataset import build_or_load_tokenizer

	repo = "sankath/LogSentry-LM"
	ckpt_path = hf_hub_download(repo, "logsentry_lm.pt")
	tok_path = hf_hub_download(repo, "log_tokenizer.json")

	ckpt = torch.load(ckpt_path, map_location="cpu")
	model = LogSentryLM(vocab_size=ckpt["vocab_size"], max_seq_len=ckpt["max_seq_len"])
	model.load_state_dict(ckpt["model_state"]); model.eval()

	tokenizer = build_or_load_tokenizer(None, tok_path) # loads the cached tokenizer

	loss, ppl = score_line(model, tokenizer, "Failed password for root from 1.2.3.4 port 22 ssh2", "cpu")
	print("anomaly" if loss > ckpt["threshold"] else "normal", f"(perplexity={ppl:.1f})")
	```

	## How anomaly detection works

	The model learns what normal logs look like. A line it cannot predict scores a
	high per-token loss (high perplexity) and is flagged as anomalous. The cutoff is
	`threshold = mean + 2·std` of the training corpus line scores, stored in the
	checkpoint. Reference behavior:

	\| Input \| Perplexity \| Verdict \|
	\| --- \| --- \| --- \|
	\| Normal Apache `[notice]` line \| ~2 \| normal \|
	\| Real SSH login attempt \| ~12–16 \| borderline \|
	\| Random gibberish / injected text \| >100,000 \| anomaly \|

	## Applications & use cases

	Because the model learns the "grammar" of normal system logs, it can support a
	range of log-analytics and security tasks:

	- Security monitoring / intrusion detection — flag suspicious log lines
	(brute-force SSH attempts, injected commands, malformed requests) that don't
	match learned normal patterns. This is the core "sentry" use case.
	- Operational anomaly detection (AIOps) — surface rare error conditions,
	misconfigurations, or failing components before they escalate, by ranking log
	lines by how "surprising" they are.
	- Log triage & noise reduction — instead of reading thousands of lines,
	engineers review only the top-scoring (most unusual) ones.
	- Log autocompletion / templating — suggest or complete log-line formats given
	a prefix (useful in tooling and for synthetic log generation for testing).
	- Data-quality / format validation — detect corrupted, truncated, or
	non-conforming log entries in an ingestion pipeline.
	- Education — a compact, readable reference for how a GPT-style Transformer,
	tokenizer, and perplexity-based anomaly scoring work end to end.

	### How it can be used

	- Batch scan a log file — score every line and export the anomalies:
	`python detect_anomalies.py --file /var/log/auth.log`
	- Score individual lines programmatically via `score_line(...)` and compare
	against the saved `threshold` (see the usage snippet above).
	- Stream integration — call the scorer inside a log shipper (Fluent Bit /
	Logstash / a Kafka consumer) to tag or route high-perplexity lines in near real
	time.
	- Alerting — wire flagged lines into Slack/email/PagerDuty or a SIEM
	(Splunk, Elastic) as an enrichment signal.
	- Retrain on your own logs — point `LOG_FILE` at your data and run
	`train_and_generate.py` to specialize the model to your environment.

	## Future scope

	Natural next steps to make this research prototype more capable and production-ready:

	- Per-source models & thresholds — separate models for SSH, Apache, kernel,
	etc., removing the single-global-threshold weakness.
	- Session / sequence-level modeling — detect anomalous sequences of events
	(e.g. login → privilege escalation → data access), not just single lines.
	- Larger context & scale — bigger `d_model`, more layers, longer
	`max_seq_len`, and training on full Loghub datasets (millions of lines).
	- Template-aware tokenization — parse logs into templates + variables (à la
	Drain) so IPs/IDs don't inflate the vocabulary or the anomaly score.
	- Calibrated scoring — convert raw perplexity into probabilities/severity
	levels, with per-line-position attribution ("which token was surprising?").
	- Real-time serving — expose the scorer as a REST/gRPC service or a Hugging
	Face Space (Gradio demo) for interactive use.
	- `transformers` integration — wrap as a `PreTrainedModel` so it loads via
	`AutoModel.from_pretrained` and plugs into the standard ecosystem.
	- Evaluation & benchmarking — measure precision/recall on labeled anomaly
	datasets (e.g. BGL, HDFS) against baselines like DeepLog and LogBERT.
	- Explainability & feedback loop — let analysts confirm/dismiss flags to
	continuously refine the threshold and reduce false positives.

	## Limitations

	- Trained on a small, mixed corpus (three log sources). A single global
	threshold is imperfect across sources; per-source models/thresholds work better.
	- Not production hardened — intended for education and experimentation.
	- Small context window (128 tokens) and small vocab; it models line-level, not
	session-level, structure.

	## Attribution & licensing

	- Training data derived from the [Loghub](https://github.com/logpai/loghub)
	collection (Linux, Apache, OpenSSH samples). Please cite Loghub and consult its
	terms for any redistribution of the underlying logs.
	- Model code released under the MIT license.

	```bibtex
	@inproceedings{zhu2023loghub,
	title = {Loghub: A Large Collection of System Log
	Datasets for AI-driven Log Analytics},
	author = {Zhu, Jieming and He, Shilin and He, Pinjia and
	Liu, Jinyang and Lyu, Michael R.},
	booktitle = {IEEE International Symposium on Software
	Reliability Engineering (ISSRE)},
	year = {2023}
	}
	```