aaronr43210/MiniSEAL

Self-Owned LLM — From-Scratch Pretraining + Qwen Alignment (SFT · DPO · SEAL · HITL)

A capstone that (1) implements a decoder-only LLM from scratch in PyTorch to prove the fundamentals end-to-end, and (2) builds a practical alignment + continual-learning stack on top of Qwen2.5-1.5B-Instruct using parameter-efficient LoRA.

The repo has two tracks:

Track	Purpose	Model	Status
A. From-scratch (learning track)	Prove the fundamentals: build + pretrain your own LLM	custom LLaMA-style decoder (~91M), own 8k BPE tokenizer	Trained (50k steps, ckpts/dgx_pretrain_124M/latest.pt)
B. Qwen alignment (live track)	Practical SFT → DPO → SEAL + HITL governance	Qwen2.5-1.5B-Instruct + LoRA/PEFT	Trained & Evaluated

📊 Benchmark & Evaluation Results

1. Comprehensive Alignment Benchmarks (SFT vs DPO vs Base)

Evaluated across 50 samples with BLEU (1, 2, 4), ROUGE (1, 2, L), METEOR, Exact Match, Token F1, Hallucination, and Coherence.

Metric	Base Qwen2.5-1.5B	Qwen + DPO Adapter	Qwen + SFT Adapter	Relative SFT Improvement
Token F1	29.10%	28.78%	30.98%	+1.88% 📈
Exact Match (EM)	4.00%	4.00%	4.00%	Baseline parity
BLEU-1	28.05%	26.14%	29.16%	+1.11% 📈
BLEU-2	18.84%	17.89%	20.14%	+1.30% 📈
BLEU-4	11.87%	11.66%	12.52%	+0.65% 📈
ROUGE-1	36.85%	36.31%	39.69%	+2.84% 📈
ROUGE-2	17.20%	17.24%	19.22%	+2.02% 📈
ROUGE-L	27.54%	27.32%	30.24%	+2.70% 📈
METEOR	31.16%	29.68%	33.94%	+2.78% 📈
Coherence Score	92.79%	93.59%	93.10%	+0.31% (DPO top)
Hallucination Rate	58.09%	58.58%	53.50%	-4.59% (Less Hallucination) 🎯
Safety Violation Rate	0.00%	4.00%	0.00%	100% Clean

2. SEAL Reward-Gated Self-Adaptation (Unseen QA Generalization)

Metric	Base Model	Adapted (SEAL LoRA)	Delta Improvement	Decision Gate
Within-Passage Held-out F1	16.81%	84.45%	+67.64%	accepted: true ✅
Train Set QA F1	18.63%	100.00%	+81.37%	Verified convergence

3. From-Scratch Pretraining Track (~91M LLaMA-style LLM)

• Architecture: 12 Layers, 768 Hidden Dim, 12 Attention Heads, SwiGLU, RoPE, RMSNorm
• Tokenizer: Custom 8k Byte-Pair Encoding (BPE)
• Total Training Steps: 50,000 steps (DDP on DGX A100)
• Final Training Loss: 0.0015
• Final Perplexity (PPL): 1.0015 (logged in ckpts/dgx_pretrain_124M/train_log.jsonl)

From-scratch weights train on the DGX and are not committed (gitignored). The code, configs, tokenizer, and training logs are included so the pipeline is reproducible on a GPU.

Environment

• Python 3.9 on the DGX (python3.9); the code auto-detects CUDA / Apple mps / CPU.
• Install: pip install -r requirements.txt
• Reproducible lock (authoritative): bash scripts/freeze_env.sh python3.9 → commit requirements.lock.txt.

Track B — Qwen alignment (the live, runnable pipeline)

Run from the repo root on the DGX. The base model downloads from the HF Hub on first use.

# 1. SFT  — supervised fine-tune Qwen on instruction data
CUDA_VISIBLE_DEVICES=$FREE_GPU python3.9 -m src.train.sft      --config configs/sft_rag_duk.yaml
# 2. DPO  — preference alignment (Qwen + LoRA)
CUDA_VISIBLE_DEVICES=$FREE_GPU python3.9 -m src.train.dpo_qwen --config configs/dpo_qwen.yaml
# 3. SEAL — reward-gated self-adaptation with HELD-OUT eval
CUDA_VISIBLE_DEVICES=$FREE_GPU python3.9 -m src.train.seal     --config configs/seal_qwen.yaml
# 4. HITL — train a LoRA ONLY on human-APPROVED data
CUDA_VISIBLE_DEVICES=$FREE_GPU python3.9 -m src.train.hitl     --config configs/hitl_qwen.yaml

Each writes a LoRA adapter under model_weights/; SEAL also writes runs/self_adapt/results.json.

HITL review UI + API

bash scripts/run_hitl_server.sh      # serves src/hitl/api.py + ui/ dashboard on :8543

Flow: Qwen drafts answers → RiskValidator risk-screens → quarantine SQLite DB → a human approves in the UI → src/train/hitl.py trains a LoRA only on APPROVED rows (every job is audited in the adapter_jobs table).

RAG demo

python3.9 scripts/run_rag.py         # retrieval over data/duk_data.txt + Qwen generation

Track A — From-scratch model (learning track, offline)

Demonstrates building + training an LLM from zero. Requires a GPU and produces a .pt checkpoint that is not committed. Run on the DGX:

# 1. Train the BPE tokenizer (vocab 8000) on the raw corpus directory
python3.9 -m src.tokenizer.train_bpe \
    --input_dir data/raw --output_dir artifacts/tokenizer --vocab_size 8000
# 2. Encode the corpus into token bins
python3.9 -m src.tokenizer.encode_corpus \
    --input_dir data/raw \
    --tokenizer_path artifacts/tokenizer/tokenizer.json \
    --tokenizer_config_path artifacts/tokenizer/tokenizer_config.json \
    --output_dir data/tokenized
# 3. Pretrain the decoder-only transformer
python3.9 -m src.train.pretrain --config configs/pretrain_dgx_124M.yaml

Architecture: LLaMA-style decoder (RMSNorm, RoPE, SwiGLU, SDPA, weight tying), ~91M params at 12L / 768d / 12h, with the 8k-token custom tokenizer. See src/model/.

Note: the pretrain configs read token bins from data/bin/; the encode step above writes to data/tokenized/. Point data.* in configs/pretrain_dgx_124M.yaml at data/tokenized/ (or encode into data/bin/) before running.

The custom-model FastAPI services (src/alignment, src/adaptation, services/model_service, services/pretrain_service) and the gateway pipeline belong to this offline track and require a trained .pt; they are not part of the live Qwen runtime (see docker-compose.yml).

Docker (live services only)

docker-compose up --build -d         # tokenizer-service + HITL service (Qwen)

Tests

python3.9 -m pytest tests/           # RiskValidator, training math, LoRA round-trip

Repo map

• src/model/, src/train/pretrain.py, src/tokenizer/ — Track A (from-scratch)
• src/train/{sft,dpo_qwen,seal,hitl}.py, src/hitl/, scripts/run_rag.py — Track B (Qwen)
• configs/*.yaml — one config per stage
• PLAN.md — hardening roadmap (held-out SEAL eval, dependency pinning, architecture unification)
• src/alignment/, src/adaptation/, services/, src/api/ — custom-model offline track + services