Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# RunPod setup for kyrael + sorelith Krea2 LoRA training
|
| 2 |
+
|
| 3 |
+
## What's already done
|
| 4 |
+
- Both datasets are uploaded as private HF dataset repos:
|
| 5 |
+
- `JBARU/kyrael-dataset`
|
| 6 |
+
- `JBARU/sorelith-dataset`
|
| 7 |
+
- No captions yet -- `setup_pod.sh` captions them on the pod (Qwen2.5-VL-7B, fast on a real GPU).
|
| 8 |
+
|
| 9 |
+
## Files in this folder
|
| 10 |
+
- `setup_pod.sh` -- full bootstrap: installs musubi-tuner, downloads models, downloads datasets, captions, caches, trains both LoRAs
|
| 11 |
+
- `caption_dataset.py` -- auto-captioning script (called by setup_pod.sh)
|
| 12 |
+
- `dataset_kyrael.toml` / `dataset_sorelith.toml` -- musubi-tuner dataset configs
|
| 13 |
+
|
| 14 |
+
## Steps you need to do yourself (I can't do these for you)
|
| 15 |
+
|
| 16 |
+
### 1. Accept gated model access
|
| 17 |
+
Before anything else, visit https://huggingface.co/krea/Krea-2-Raw while logged in
|
| 18 |
+
and accept/request access (it auto-approves). Skip this and the download in step 4
|
| 19 |
+
of setup_pod.sh will fail.
|
| 20 |
+
|
| 21 |
+
### 2. Create a RunPod account + pod
|
| 22 |
+
1. Sign up at https://runpod.io (this needs your own payment method -- I can't do this part)
|
| 23 |
+
2. Go to **Pods** -> **Deploy**
|
| 24 |
+
3. Pick a GPU: **RTX 4090 (24GB)** is the recommended sweet spot for this job (~$0.34-0.69/hr)
|
| 25 |
+
4. Pick a template with PyTorch + CUDA pre-installed (e.g. "RunPod PyTorch 2.x")
|
| 26 |
+
5. Deploy the pod, wait for it to start
|
| 27 |
+
6. Open its **web terminal** (or connect via SSH if you set up a key)
|
| 28 |
+
|
| 29 |
+
### 3. Get the scripts onto the pod
|
| 30 |
+
Once you have the pod's terminal open:
|
| 31 |
+
```bash
|
| 32 |
+
mkdir -p /workspace
|
| 33 |
+
cd /workspace
|
| 34 |
+
```
|
| 35 |
+
Then either:
|
| 36 |
+
- **Easiest**: use the RunPod web UI's file upload to drop `caption_dataset.py`,
|
| 37 |
+
`dataset_kyrael.toml`, and `dataset_sorelith.toml` into `/workspace/`
|
| 38 |
+
- Or paste their contents directly using `cat > filename.py << 'EOF' ... EOF` in the terminal
|
| 39 |
+
|
| 40 |
+
### 4. Run the bootstrap
|
| 41 |
+
Upload `setup_pod.sh` the same way, then:
|
| 42 |
+
```bash
|
| 43 |
+
chmod +x setup_pod.sh
|
| 44 |
+
./setup_pod.sh
|
| 45 |
+
```
|
| 46 |
+
It'll pause at `hf auth login` for you to paste your token interactively --
|
| 47 |
+
same rule as before, paste it only when prompted, never on the command line.
|
| 48 |
+
|
| 49 |
+
### 5. When it's done
|
| 50 |
+
Trained LoRAs land in `/workspace/output/kyrael/kyrael_lora.safetensors` and
|
| 51 |
+
`/workspace/output/sorelith/sorelith_lora.safetensors`. Push them back to HF
|
| 52 |
+
(so you can grab them locally) with:
|
| 53 |
+
```bash
|
| 54 |
+
hf upload <your-username>/kyrael-lora /workspace/output/kyrael --repo-type model
|
| 55 |
+
hf upload <your-username>/sorelith-lora /workspace/output/sorelith --repo-type model
|
| 56 |
+
```
|
| 57 |
+
Then locally: `hf download <your-username>/kyrael-lora --local-dir D:\ComfyModels\loras\kyrael`
|
| 58 |
+
|
| 59 |
+
### 6. Don't forget to stop the pod
|
| 60 |
+
RunPod bills by the hour while running -- stop/terminate it once training's
|
| 61 |
+
done so you're not paying for idle GPU time.
|
| 62 |
+
|
| 63 |
+
## Vaelith run: what changed after kyrael
|
| 64 |
+
|
| 65 |
+
- **Network volume: use at least 100GB**, not 50GB. Kyrael's pod hit disk-full
|
| 66 |
+
twice at 50GB -- base models alone (~33GB) plus the Qwen2.5-VL-7B captioning
|
| 67 |
+
model's cache (~16GB, easy to forget about) leave almost no margin at 50GB.
|
| 68 |
+
- **`num_repeats` dropped from 10 to 3** in `dataset_vaelith.toml`. Kyrael's
|
| 69 |
+
LoRA came out overtrained/rigid (locked pose, completely resistant to style
|
| 70 |
+
LoRA blending even at 5.0 weight) -- traced back to 34 images x 10 repeats x
|
| 71 |
+
16 epochs = 5,440 total training exposures on a small dataset. Lower repeats
|
| 72 |
+
should fix this without hurting identity retention.
|
| 73 |
+
- **`setup_pod_vaelith.sh` cleans up the captioning model cache immediately**
|
| 74 |
+
after captioning finishes (`rm -rf /workspace/.cache`), instead of leaving
|
| 75 |
+
it sitting there until disk fills up mid-training like last time.
|
| 76 |
+
|
| 77 |
+
## If something fails partway through
|
| 78 |
+
Each numbered section in `setup_pod.sh` is independent enough to re-run on its
|
| 79 |
+
own -- if training crashes on kyrael, you don't need to redo the downloads or
|
| 80 |
+
sorelith's caching, just re-run that one training command.
|