docs/local_dev_setup.md

# Local Dev Setup — Indic Heritage Studio v2 (NVIDIA 8×80GB Dev Box)

This guide walks you through setting up Indic Heritage Studio v2 on the 8 × NVIDIA 80GB dev box. The same code runs on AMD Radeon Cloud for the final demo — only the PyTorch wheel differs.

## Prerequisites

- **OS:** Ubuntu 22.04 LTS (or compatible)
- **Python:** 3.11
- **NVIDIA driver:** 535+ (CUDA 12.1 compatible)
- **Disk:** 50 GB free (35 GB for models, 5 GB for datasets, 10 GB for outputs)
- **RAM:** 32 GB minimum, 64 GB recommended (for LoRA training)
- **GPUs:** 8 × A100/H100 80GB (this is the dev target; the code also runs on 1+ GPUs)

## Step 1 — Clone

```bash
git clone https://github.com/<your-handle>/Radeon-hackathon-2026-07.git
cd Radeon-hackathon-2026-07/indic-heritage-studio-v2
```

## Step 2 — Python environment

```bash
# Install Python 3.11 if not already
sudo apt install python3.11 python3.11-venv python3.11-dev

# Create venv
python3.11 -m venv .venv
source .venv/bin/activate

pip install --upgrade pip wheel
```

## Step 3 — Install PyTorch (CUDA 12.1)

The dev box has CUDA 12.x drivers, so we use the cu121 wheel:

```bash
pip install torch==2.4.1 torchvision==0.19.1 --index-url https://download.pytorch.org/whl/cu121
```

Verify:

```bash
python -c "
import torch
print('PyTorch:', torch.__version__)
print('CUDA available:', torch.cuda.is_available())
print('GPU count:', torch.cuda.device_count())
for i in range(torch.cuda.device_count()):
    print(f'  GPU {i}: {torch.cuda.get_device_name(i)} '
          f'({torch.cuda.get_device_properties(i).total_memory / 1e9:.1f} GB)')
"
```

You should see 8 GPUs, each reporting ~80 GB.

## Step 4 — Install project deps

```bash
pip install -r requirements.txt
```

This installs: diffusers 0.30, transformers 4.44, accelerate, peft (LoRA), controlnet-aux, compel, gradio, openai, bitsandbytes (8-bit optimizer for LoRA training), datasets, wandb, and more.

## Step 5 — Configure environment

```bash
cp .env.example .env
```

Edit `.env`:
- `AMD_MODEL_API_KEY=<your key>` — get one free at https://developer.amd.com.cn/radeon/modelapis (used by the optional agent layer)
- Leave everything else at defaults

## Step 6 — Download models (~35 GB, ~10 min on fast connection)

```bash
python scripts/download_models.py
```

Downloads:
- SDXL 1.0 base + DreamShaper-XL turbo (~7 GB)
- SDXL refiner (~6 GB)
- SDXL inpainting checkpoint (~6 GB)
- IP-Adapter XL + image encoder (~3 GB)
- Stable Video Diffusion XT 1.1 (~10 GB)
- ControlNet Canny / Depth / OpenPose SDXL (~3 GB)
- controlnet-aux Annotators (~1 GB)

## Step 7 — Verify

```bash
python scripts/verify_rocm.py
```

Should report:
- 8 GPUs visible
- ~640 GB total VRAM
- All library versions installed
- "Multi-GPU mode: pipelines will be pinned to dedicated GPUs"

## Step 8 — Source heritage art datasets

For each of the 5 heritage styles, source 30-50 high-quality reference images:

```bash
mkdir -p assets/datasets/raw/{madhubani,warli,pattachitra,mughal,tanjore}
# Place JPG/PNG files in each subdirectory
```

See `assets/styles/README.md` for source recommendations (Wikipedia Commons, museum archives, etc.).

## Step 9 — Prepare datasets + train LoRAs

```bash
# Prepare (resize, caption, write metadata) — ~5 min
python training/prepare_dataset.py

# Train LoRA for each style — ~30 min per style on 1 GPU
python -m training.train_lora --style madhubani --steps 800
python -m training.train_lora --style warli --steps 800
python -m training.train_lora --style pattachitra --steps 800
python -m training.train_lora --style mughal --steps 800
python -m training.train_lora --style tanjore --steps 800
```

Or train all 5 in parallel across 5 GPUs:

```bash
for style in madhubani warli pattachitra mughal tanjore; do
  CUDA_VISIBLE_DEVICES=$((i++)) python -m training.train_lora --style $style --steps 800 &
done
wait
```

Each LoRA saves to `assets/loras/<style>.safetensors` (~150 MB).

## Step 10 — Place IP-Adapter reference images

For each style, place one canonical reference image in `assets/styles/`:

| File | Style |
|---|---|
| `madhubani_ref.png` | Madhubani |
| `warli_ref.png` | Warli |
| `pattachitra_ref.png` | Pattachitra |
| `mughal_ref.png` | Mughal |
| `tanjore_ref.png` | Tanjore |

These are used by the IP-Adapter XL style transfer pipeline as conditioning inputs. Recommended: 1024×1024 RGB PNG.

## Step 11 — Run smoke tests

```bash
pytest tests/test_agents.py -v
```

Should pass all 9 tests (config, styles, agents, gpu_utils, image_utils, batch).

## Step 12 — Launch the UI

```bash
python app.py
```

Open http://localhost:7860 in your browser. You should see the 6-tab UI with the GPU monitor sidebar showing 8 GPUs.

## Step 13 — Generate demo gallery (optional, for PDF/PPT)

```bash
python scripts/generate_demo_outputs.py
```

Produces ~62 demo outputs in `examples/` (~30 min on 8×80GB).

## Troubleshooting

| Issue | Fix |
|---|---|
| `torch.cuda.is_available() == False` | Reinstall PyTorch with `--index-url https://download.pytorch.org/whl/cu121` |
| `RuntimeError: CUDA out of memory` during LoRA training | Lower `--batch-size` or reduce `--rank` from 32 → 16 |
| `controlnet_aux` download fails | Manually run: `python -c "from controlnet_aux import CannyDetector; CannyDetector.from_pretrained('lllyasviel/Annotators')"` |
| SVD generates black frames | Increase `noise_aug_strength` from 0.02 → 0.05 |
| Multi-GPU batch workers fail to spawn | Set `MULTI_GPU=0` in `.env` and rerun in single-GPU mode |
| Gradio UI loads but pipelines fail | Check `outputs/*.log` — most likely a missing model download |