Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,107 @@
+---
+license: mit
+library_name: pytorch
+tags:
+  - log-analysis
+  - anomaly-detection
+  - security
+  - transformer
+  - gpt
+  - language-model
+  - from-scratch
+pipeline_tag: text-generation
+---
+
+# LogSentry-LM
+
+A small, from-scratch **GPT-style language model** trained on real server logs to
+(1) autoregressively complete log lines and (2) **detect anomalous log lines** by
+how "surprised" the model is by them (perplexity-based scoring).
+
+It is a compact, educational decoder-only Transformer (~2M parameters) built with
+plain PyTorch — no `transformers` modeling class — intended for learning and for
+lightweight log anomaly detection.
+
+## Model details
+
+- **Architecture:** decoder-only Transformer (GPT-style), causal self-attention,
+  pre-norm residual blocks, learned token + positional embeddings.
+- **Size:** `d_model=128`, `n_heads=4`, `n_layers=3`, `max_seq_len=128` (~2.03M params).
+- **Tokenizer:** Byte-Pair Encoding (`tokenizers`), vocab ≈ 5.5k.
+- **Training data:** combined **Loghub** samples — Linux + Apache + OpenSSH
+  (`*_2k.log`, ~6k lines total).
+- **Objective:** next-token prediction (cross-entropy).
+- **Framework:** PyTorch (trained on Apple Silicon MPS).
+
+## Files in this repo
+
+| File | Purpose |
+| --- | --- |
+| `logsentry_lm.pt` | Trained weights + anomaly threshold + config |
+| `log_tokenizer.json` | BPE tokenizer |
+| `architecture.py` | Transformer building blocks |
+| `dataset.py` | Tokenizer loader + dataset |
+| `train_and_generate.py` | Model class, training, generation, anomaly scoring |
+| `detect_anomalies.py` | Standalone anomaly predictor |
+
+## Usage
+
+> ⚠️ This is a **custom architecture**, so `AutoModel.from_pretrained` does **not**
+> work. Download the files and load manually:
+
+```python
+from huggingface_hub import hf_hub_download
+import torch
+from train_and_generate import LogSentryLM, score_line
+from dataset import build_or_load_tokenizer
+
+repo = "YOUR_USERNAME/LogSentry-LM"
+ckpt_path = hf_hub_download(repo, "logsentry_lm.pt")
+tok_path  = hf_hub_download(repo, "log_tokenizer.json")
+
+ckpt = torch.load(ckpt_path, map_location="cpu")
+model = LogSentryLM(vocab_size=ckpt["vocab_size"], max_seq_len=ckpt["max_seq_len"])
+model.load_state_dict(ckpt["model_state"]); model.eval()
+
+tokenizer = build_or_load_tokenizer(None, tok_path)  # loads the cached tokenizer
+
+loss, ppl = score_line(model, tokenizer, "Failed password for root from 1.2.3.4 port 22 ssh2", "cpu")
+print("anomaly" if loss > ckpt["threshold"] else "normal", f"(perplexity={ppl:.1f})")
+```
+
+## How anomaly detection works
+
+The model learns what *normal* logs look like. A line it cannot predict scores a
+high per-token loss (high perplexity) and is flagged as anomalous. The cutoff is
+`threshold = mean + 2·std` of the training corpus line scores, stored in the
+checkpoint. Reference behavior:
+
+| Input | Perplexity | Verdict |
+| --- | --- | --- |
+| Normal Apache `[notice]` line | ~2 | normal |
+| Real SSH login attempt | ~12–16 | borderline |
+| Random gibberish / injected text | >100,000 | anomaly |
+
+## Limitations
+
+- Trained on a **small, mixed** corpus (three log sources). A single global
+  threshold is imperfect across sources; per-source models/thresholds work better.
+- Not production hardened — intended for education and experimentation.
+- Small context window (128 tokens) and small vocab; it models line-level, not
+  session-level, structure.
+
+## Attribution & licensing
+
+- Training data derived from the **[Loghub](https://github.com/logpai/loghub)**
+  collection (Linux, Apache, OpenSSH samples). Please cite Loghub and consult its
+  terms for any redistribution of the underlying logs.
+- Model code released under the MIT license.
+
+```bibtex
+@inproceedings{zhu2023loghub,
+  title={Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics},
+  author={Zhu, Jieming and He, Shilin and He, Pinjia and Liu, Jinyang and Lyu, Michael R.},
+  booktitle={IEEE International Symposium on Software Reliability Engineering (ISSRE)},
+  year={2023}
+}
+```