Text Generation
PEFT
Safetensors
PyTorch
English
causal-lm
lora
qwen
sft
dpo
seal
self-adaptation
hitl
Instructions to use aaronr43210/MiniSEAL with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use aaronr43210/MiniSEAL with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Upload README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Self-Owned LLM β From-Scratch Pretraining + Qwen Alignment (SFT Β· DPO Β· SEAL Β· HITL)
|
| 2 |
+
|
| 3 |
+
A capstone that (1) implements a decoder-only LLM **from scratch** in PyTorch to prove the
|
| 4 |
+
fundamentals end-to-end, and (2) builds a practical alignment + continual-learning stack on
|
| 5 |
+
top of **Qwen2.5-1.5B-Instruct** using parameter-efficient LoRA.
|
| 6 |
+
|
| 7 |
+
The repo has **two tracks**:
|
| 8 |
+
|
| 9 |
+
| Track | Purpose | Model | Status |
|
| 10 |
+
|-------|---------|-------|--------|
|
| 11 |
+
| **A. From-scratch** (learning track) | Prove the fundamentals: build + pretrain your own LLM | custom LLaMA-style decoder (~91M), own 8k BPE tokenizer | Offline β trained on the DGX; the `.pt` is gitignored, not in the repo |
|
| 12 |
+
| **B. Qwen alignment** (live track) | Practical SFT β DPO β SEAL + HITL governance | Qwen2.5-1.5B-Instruct + LoRA/PEFT | Runnable |
|
| 13 |
+
|
| 14 |
+
> From-scratch weights train on the DGX and are **not** committed (gitignored). The code,
|
| 15 |
+
> configs, tokenizer, and training logs are included so the pipeline is reproducible on a GPU.
|
| 16 |
+
|
| 17 |
+
## Environment
|
| 18 |
+
- **Python 3.9** on the DGX (`python3.9`); the code auto-detects CUDA / Apple `mps` / CPU.
|
| 19 |
+
- Install: `pip install -r requirements.txt`
|
| 20 |
+
- Reproducible lock (authoritative): `bash scripts/freeze_env.sh python3.9` β commit `requirements.lock.txt`.
|
| 21 |
+
|
| 22 |
+
## Track B β Qwen alignment (the live, runnable pipeline)
|
| 23 |
+
Run from the repo root on the DGX. The base model downloads from the HF Hub on first use.
|
| 24 |
+
|
| 25 |
+
```bash
|
| 26 |
+
# 1. SFT β supervised fine-tune Qwen on instruction data
|
| 27 |
+
CUDA_VISIBLE_DEVICES=$FREE_GPU python3.9 -m src.train.sft --config configs/sft_rag_duk.yaml
|
| 28 |
+
# 2. DPO β preference alignment (Qwen + LoRA)
|
| 29 |
+
CUDA_VISIBLE_DEVICES=$FREE_GPU python3.9 -m src.train.dpo_qwen --config configs/dpo_qwen.yaml
|
| 30 |
+
# 3. SEAL β reward-gated self-adaptation with HELD-OUT eval
|
| 31 |
+
CUDA_VISIBLE_DEVICES=$FREE_GPU python3.9 -m src.train.seal --config configs/seal_qwen.yaml
|
| 32 |
+
# 4. HITL β train a LoRA ONLY on human-APPROVED data
|
| 33 |
+
CUDA_VISIBLE_DEVICES=$FREE_GPU python3.9 -m src.train.hitl --config configs/hitl_qwen.yaml
|
| 34 |
+
```
|
| 35 |
+
Each writes a LoRA adapter under `model_weights/`; SEAL also writes `runs/self_adapt/results.json`.
|
| 36 |
+
|
| 37 |
+
### HITL review UI + API
|
| 38 |
+
```bash
|
| 39 |
+
bash scripts/run_hitl_server.sh # serves src/hitl/api.py + ui/ dashboard on :8543
|
| 40 |
+
```
|
| 41 |
+
Flow: Qwen drafts answers β `RiskValidator` risk-screens β quarantine SQLite DB β a human
|
| 42 |
+
approves in the UI β `src/train/hitl.py` trains a LoRA only on APPROVED rows (every job is
|
| 43 |
+
audited in the `adapter_jobs` table).
|
| 44 |
+
|
| 45 |
+
### RAG demo
|
| 46 |
+
```bash
|
| 47 |
+
python3.9 scripts/run_rag.py # retrieval over data/duk_data.txt + Qwen generation
|
| 48 |
+
```
|
| 49 |
+
|
| 50 |
+
## Track A β From-scratch model (learning track, offline)
|
| 51 |
+
Demonstrates building + training an LLM from zero. Requires a GPU and produces a `.pt`
|
| 52 |
+
checkpoint that is **not** committed. Run on the DGX:
|
| 53 |
+
|
| 54 |
+
```bash
|
| 55 |
+
# 1. Train the BPE tokenizer (vocab 8000) on the raw corpus directory
|
| 56 |
+
python3.9 -m src.tokenizer.train_bpe \
|
| 57 |
+
--input_dir data/raw --output_dir artifacts/tokenizer --vocab_size 8000
|
| 58 |
+
# 2. Encode the corpus into token bins
|
| 59 |
+
python3.9 -m src.tokenizer.encode_corpus \
|
| 60 |
+
--input_dir data/raw \
|
| 61 |
+
--tokenizer_path artifacts/tokenizer/tokenizer.json \
|
| 62 |
+
--tokenizer_config_path artifacts/tokenizer/tokenizer_config.json \
|
| 63 |
+
--output_dir data/tokenized
|
| 64 |
+
# 3. Pretrain the decoder-only transformer
|
| 65 |
+
python3.9 -m src.train.pretrain --config configs/pretrain_dgx_124M.yaml
|
| 66 |
+
```
|
| 67 |
+
Architecture: LLaMA-style decoder (RMSNorm, RoPE, SwiGLU, SDPA, weight tying), ~91M params
|
| 68 |
+
at 12L / 768d / 12h, with the 8k-token custom tokenizer. See `src/model/`.
|
| 69 |
+
|
| 70 |
+
> **Note:** the pretrain configs read token bins from `data/bin/`; the encode step above
|
| 71 |
+
> writes to `data/tokenized/`. Point `data.*` in `configs/pretrain_dgx_124M.yaml` at
|
| 72 |
+
> `data/tokenized/` (or encode into `data/bin/`) before running.
|
| 73 |
+
>
|
| 74 |
+
> The custom-model FastAPI services (`src/alignment`, `src/adaptation`,
|
| 75 |
+
> `services/model_service`, `services/pretrain_service`) and the gateway pipeline belong to
|
| 76 |
+
> this offline track and require a trained `.pt`; they are **not** part of the live Qwen
|
| 77 |
+
> runtime (see `docker-compose.yml`).
|
| 78 |
+
|
| 79 |
+
## Docker (live services only)
|
| 80 |
+
```bash
|
| 81 |
+
docker-compose up --build -d # tokenizer-service + HITL service (Qwen)
|
| 82 |
+
```
|
| 83 |
+
|
| 84 |
+
## Tests
|
| 85 |
+
```bash
|
| 86 |
+
python3.9 -m pytest tests/ # RiskValidator, training math, LoRA round-trip
|
| 87 |
+
```
|
| 88 |
+
|
| 89 |
+
## Repo map
|
| 90 |
+
- `src/model/`, `src/train/pretrain.py`, `src/tokenizer/` β **Track A** (from-scratch)
|
| 91 |
+
- `src/train/{sft,dpo_qwen,seal,hitl}.py`, `src/hitl/`, `scripts/run_rag.py` β **Track B** (Qwen)
|
| 92 |
+
- `configs/*.yaml` β one config per stage
|
| 93 |
+
- `PLAN.md` β hardening roadmap (held-out SEAL eval, dependency pinning, architecture unification)
|
| 94 |
+
- `src/alignment/`, `src/adaptation/`, `services/`, `src/api/` β custom-model offline track + services
|