aaronr43210 commited on
Commit
45e264d
Β·
verified Β·
1 Parent(s): 8ab258c

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +94 -0
README.md ADDED
@@ -0,0 +1,94 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Self-Owned LLM β€” From-Scratch Pretraining + Qwen Alignment (SFT Β· DPO Β· SEAL Β· HITL)
2
+
3
+ A capstone that (1) implements a decoder-only LLM **from scratch** in PyTorch to prove the
4
+ fundamentals end-to-end, and (2) builds a practical alignment + continual-learning stack on
5
+ top of **Qwen2.5-1.5B-Instruct** using parameter-efficient LoRA.
6
+
7
+ The repo has **two tracks**:
8
+
9
+ | Track | Purpose | Model | Status |
10
+ |-------|---------|-------|--------|
11
+ | **A. From-scratch** (learning track) | Prove the fundamentals: build + pretrain your own LLM | custom LLaMA-style decoder (~91M), own 8k BPE tokenizer | Offline β€” trained on the DGX; the `.pt` is gitignored, not in the repo |
12
+ | **B. Qwen alignment** (live track) | Practical SFT β†’ DPO β†’ SEAL + HITL governance | Qwen2.5-1.5B-Instruct + LoRA/PEFT | Runnable |
13
+
14
+ > From-scratch weights train on the DGX and are **not** committed (gitignored). The code,
15
+ > configs, tokenizer, and training logs are included so the pipeline is reproducible on a GPU.
16
+
17
+ ## Environment
18
+ - **Python 3.9** on the DGX (`python3.9`); the code auto-detects CUDA / Apple `mps` / CPU.
19
+ - Install: `pip install -r requirements.txt`
20
+ - Reproducible lock (authoritative): `bash scripts/freeze_env.sh python3.9` β†’ commit `requirements.lock.txt`.
21
+
22
+ ## Track B β€” Qwen alignment (the live, runnable pipeline)
23
+ Run from the repo root on the DGX. The base model downloads from the HF Hub on first use.
24
+
25
+ ```bash
26
+ # 1. SFT β€” supervised fine-tune Qwen on instruction data
27
+ CUDA_VISIBLE_DEVICES=$FREE_GPU python3.9 -m src.train.sft --config configs/sft_rag_duk.yaml
28
+ # 2. DPO β€” preference alignment (Qwen + LoRA)
29
+ CUDA_VISIBLE_DEVICES=$FREE_GPU python3.9 -m src.train.dpo_qwen --config configs/dpo_qwen.yaml
30
+ # 3. SEAL β€” reward-gated self-adaptation with HELD-OUT eval
31
+ CUDA_VISIBLE_DEVICES=$FREE_GPU python3.9 -m src.train.seal --config configs/seal_qwen.yaml
32
+ # 4. HITL β€” train a LoRA ONLY on human-APPROVED data
33
+ CUDA_VISIBLE_DEVICES=$FREE_GPU python3.9 -m src.train.hitl --config configs/hitl_qwen.yaml
34
+ ```
35
+ Each writes a LoRA adapter under `model_weights/`; SEAL also writes `runs/self_adapt/results.json`.
36
+
37
+ ### HITL review UI + API
38
+ ```bash
39
+ bash scripts/run_hitl_server.sh # serves src/hitl/api.py + ui/ dashboard on :8543
40
+ ```
41
+ Flow: Qwen drafts answers β†’ `RiskValidator` risk-screens β†’ quarantine SQLite DB β†’ a human
42
+ approves in the UI β†’ `src/train/hitl.py` trains a LoRA only on APPROVED rows (every job is
43
+ audited in the `adapter_jobs` table).
44
+
45
+ ### RAG demo
46
+ ```bash
47
+ python3.9 scripts/run_rag.py # retrieval over data/duk_data.txt + Qwen generation
48
+ ```
49
+
50
+ ## Track A β€” From-scratch model (learning track, offline)
51
+ Demonstrates building + training an LLM from zero. Requires a GPU and produces a `.pt`
52
+ checkpoint that is **not** committed. Run on the DGX:
53
+
54
+ ```bash
55
+ # 1. Train the BPE tokenizer (vocab 8000) on the raw corpus directory
56
+ python3.9 -m src.tokenizer.train_bpe \
57
+ --input_dir data/raw --output_dir artifacts/tokenizer --vocab_size 8000
58
+ # 2. Encode the corpus into token bins
59
+ python3.9 -m src.tokenizer.encode_corpus \
60
+ --input_dir data/raw \
61
+ --tokenizer_path artifacts/tokenizer/tokenizer.json \
62
+ --tokenizer_config_path artifacts/tokenizer/tokenizer_config.json \
63
+ --output_dir data/tokenized
64
+ # 3. Pretrain the decoder-only transformer
65
+ python3.9 -m src.train.pretrain --config configs/pretrain_dgx_124M.yaml
66
+ ```
67
+ Architecture: LLaMA-style decoder (RMSNorm, RoPE, SwiGLU, SDPA, weight tying), ~91M params
68
+ at 12L / 768d / 12h, with the 8k-token custom tokenizer. See `src/model/`.
69
+
70
+ > **Note:** the pretrain configs read token bins from `data/bin/`; the encode step above
71
+ > writes to `data/tokenized/`. Point `data.*` in `configs/pretrain_dgx_124M.yaml` at
72
+ > `data/tokenized/` (or encode into `data/bin/`) before running.
73
+ >
74
+ > The custom-model FastAPI services (`src/alignment`, `src/adaptation`,
75
+ > `services/model_service`, `services/pretrain_service`) and the gateway pipeline belong to
76
+ > this offline track and require a trained `.pt`; they are **not** part of the live Qwen
77
+ > runtime (see `docker-compose.yml`).
78
+
79
+ ## Docker (live services only)
80
+ ```bash
81
+ docker-compose up --build -d # tokenizer-service + HITL service (Qwen)
82
+ ```
83
+
84
+ ## Tests
85
+ ```bash
86
+ python3.9 -m pytest tests/ # RiskValidator, training math, LoRA round-trip
87
+ ```
88
+
89
+ ## Repo map
90
+ - `src/model/`, `src/train/pretrain.py`, `src/tokenizer/` β€” **Track A** (from-scratch)
91
+ - `src/train/{sft,dpo_qwen,seal,hitl}.py`, `src/hitl/`, `scripts/run_rag.py` β€” **Track B** (Qwen)
92
+ - `configs/*.yaml` β€” one config per stage
93
+ - `PLAN.md` β€” hardening roadmap (held-out SEAL eval, dependency pinning, architecture unification)
94
+ - `src/alignment/`, `src/adaptation/`, `services/`, `src/api/` β€” custom-model offline track + services