ghost-small-gen: 45M from-scratch generalist (GhostLM v0.12.0)

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+ghostlm_mark_512.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,103 @@
+---
+license: mit
+language:
+- en
+library_name: ghostlm
+pipeline_tag: text-generation
+tags:
+- ghostlm
+- from-scratch
+- generalist
+- cybersecurity
+- small-language-model
+---
+
+# ghost-small-gen
+
+A small (~45M parameter) decoder-only language model **trained entirely from
+scratch** in PyTorch, no pretrained weights and no fine-tuning. It is the first
+generalist checkpoint in the [GhostLM](https://github.com/joemunene-by/GhostLM)
+project: a model that broadened from a cybersecurity-only corpus into a small
+generalist while keeping cybersecurity as its deepest specialty.
+
+- Parameters: ~45M (6 layers, d_model 512, RoPE + SwiGLU + RMSNorm)
+- Tokenizer: GPT-2 BPE (50,257) + 7 special tokens
+- Training: 30,000 steps on a Mac M4 (MPS), final val_loss 3.76
+- Corpus: a decontaminated 258.9M-token multi-domain corpus that is only
+  **8.6% cybersecurity** (general web, broad Wikipedia, code, math, instruction,
+  and a cybersecurity layer), with 0.004% measured benchmark contamination
+- Recipe: intra-document attention masking + a multi-stage domain curriculum
+
+## Evaluation (honest, full benchmark sets)
+
+Debiased multi-permutation text-scoring. `+` means the 95% bootstrap CI lower
+bound is above the 25% random baseline (significantly above chance). Peer
+numbers are published zero-shot references for the small-model class; harnesses
+differ, so treat them as context, not an exact comparison.
+
+| Benchmark | n | ghost-small-gen (45M) | 95% CI | vs random | Peer reference |
+|---|---:|---:|---:|:--:|---|
+| ARC-Easy | 2365 | 27.2% | 25.4-28.9 | + | Pythia-160M 43.5, 111M 34.8, 256M 37.6 |
+| ARC-Challenge | 1165 | 24.3% | 22.1-26.6 | ~ | Pythia-160M 18.8, SmolLM2-360M 36.6 |
+| OpenBookQA | 500 | 27.4% | 23.7-31.1 | ~ | 111M 27.8, 256M 25.4, LaMini-35M 26.2 |
+| SecQA (cyber) | 210 | 34.3% | 28.5-40.6 | + | retained specialty |
+| CTF eval (cyber) | 30 | 63.3% | 46.7-80.0 | + | retained specialty |
+
+What this says, calibrated for a 45M model trained from scratch on free compute:
+
+- The generalist pivot worked. Three of five benchmarks are statistically above
+  the 25% random baseline, on a corpus only 8.6% cybersecurity.
+- Cybersecurity is fully retained and is the standout (SecQA 34.3%, CTF 63.3%).
+- It is competitive within its own size class: OpenBookQA beats the 256M and 35M
+  survey peers, and ARC-Challenge beats Pythia-160M, a roughly 3.5x larger model.
+- It does not clear the 35-45% competitive band on ARC-Easy (27.2%): above
+  chance, not state of the art there.
+
+This is a solid, defensible result for the size and the compute, not a
+"beats everything" claim.
+
+## Intended use and limitations
+
+Research and education: a transparent, hand-written small model for studying
+from-scratch training, generalist corpus design, and cybersecurity-aware
+language modeling. It is a base model (not instruction-tuned), small, and will
+hallucinate. Do not use it for safety-critical decisions. The cybersecurity
+content is for defensive and educational understanding.
+
+## How to load
+
+This is a custom architecture, not a `transformers` model. Load it with the
+GhostLM code:
+
+```python
+import torch
+from safetensors.torch import load_model
+from ghostlm.config import GhostLMConfig
+from ghostlm.model import GhostLM
+from huggingface_hub import hf_hub_download
+import json
+
+repo = "Ghostgim/ghost-small-gen"
+cfg = GhostLMConfig(**{k: v for k, v in json.load(open(hf_hub_download(repo, "config.json"))).items()
+                       if k in GhostLMConfig().__dataclass_fields__})
+model = GhostLM(cfg)
+load_model(model, hf_hub_download(repo, "model.safetensors"))
+model.eval()
+```
+
+See the [GhostLM repository](https://github.com/joemunene-by/GhostLM) for the
+model code, tokenizer, generation utilities, and the full scorecard.
+
+## Training details
+
+- Base architecture: ghost-small-v0.5 (RoPE, SwiGLU, RMSNorm)
+- Context length: 512, batch 16, grad-accum 4, lr 3e-4, 2000-step warmup
+- Intra-document attention masking so packed documents do not attend across
+  EOS boundaries
+- Multi-stage domain curriculum that shifts the data mixture across training
+  (broad web early, code/math/knowledge upweighted later)
+- Corpus decontaminated against every evaluation benchmark before training
+
+## License
+
+MIT. Built and trained by Joe Munene.

config.json

@@ -0,0 +1,44 @@
+{
+  "vocab_size": 50264,
+  "context_length": 512,
+  "d_model": 512,
+  "n_heads": 8,
+  "n_layers": 6,
+  "d_ff": 2048,
+  "dropout": 0.1,
+  "bias": true,
+  "use_rope": true,
+  "rope_base": 10000.0,
+  "use_swiglu": true,
+  "use_rmsnorm": true,
+  "use_flash_attention": false,
+  "n_kv_heads": null,
+  "use_qk_norm": false,
+  "intra_doc_mask": true,
+  "eos_token_id": 50258,
+  "use_moe": false,
+  "n_experts": 4,
+  "n_experts_active": 2,
+  "moe_aux_loss_coef": 0.01,
+  "batch_size": 16,
+  "learning_rate": 0.0003,
+  "weight_decay": 0.1,
+  "beta1": 0.9,
+  "beta2": 0.95,
+  "grad_clip": 1.0,
+  "grad_accum_steps": 4,
+  "warmup_steps": 2000,
+  "max_steps": 30000,
+  "eval_interval": 1500,
+  "save_interval": 3000,
+  "data_dir": "data/processed",
+  "checkpoint_dir": "checkpoints/ghost_small_gen",
+  "log_dir": "logs/ghost_small_gen",
+  "device": "mps",
+  "dtype": "float32",
+  "seed": 42,
+  "use_wandb": false,
+  "wandb_project": "ghostlm",
+  "use_compile": false,
+  "gradient_checkpointing": false
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4eb2841582dc5a9570412ca1387865d79174a9cae0cc8128f21ed5503c1a6637
+size 90048640

tokenizer_config.json

@@ -0,0 +1,5 @@
+{
+  "tokenizer_type": "tiktoken",
+  "vocab_size": 50264,
+  "model": "gpt2"
+}