| # selfhostedmodels |
|
|
| # Evoneural MVP – Local Mesh & Skybox |
|
|
| Localhost MVP for **text → 3D mesh** and **text → 360° skybox** using local models (no hosted APIs). |
|
|
| - **Mesh**: Text → image (Stable Diffusion) → 3D mesh (TripoSR). Output: `.obj` or `.glb`. |
| - **Skybox**: Text → 2:1 equirectangular image (Stable Diffusion). Optional seamless edge check. |
|
|
| **Default model:** `runwayml/stable-diffusion-v1-5` (no Hugging Face login required; first run downloads ~4GB). |
|
|
| ## Prerequisites |
|
|
| - **Python 3.10** (recommended) — [python.org](https://www.python.org/downloads/) |
| - **NVIDIA GPU** with CUDA (recommended; CPU is slower) |
| - **Git** (for cloning TripoSR; mesh only) |
|
|
| **No Conda?** Use **venv** (built into Python) — steps below. |
|
|
| ## 1. Environment |
|
|
| ### Option A: venv + pip (no Conda) |
|
|
| From PowerShell (project folder is `evoneural`): |
|
|
| ```powershell |
| cd D:\project\evoneural |
| python -m venv .venv |
| .venv\Scripts\Activate.ps1 |
| pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118 |
| pip install -r requirements.txt |
| ``` |
|
|
| - **CPU only:** use `pip install torch torchvision` (no `--index-url`). |
| - **CUDA 12.x:** use `cu121` instead of `cu118`. |
|
|
| ### Option B: Conda |
|
|
| ```powershell |
| cd D:\project\evoneural |
| conda env create -f environment.yml |
| conda activate evoneural-mvp |
| ``` |
|
|
| If you use CPU-only or a different CUDA version, edit `environment.yml` (e.g. remove `pytorch-cuda=11.8` or set `pytorch-cuda=12.1`). |
|
|
| ## 2. TripoSR (for mesh) |
|
|
| Mesh generation needs the TripoSR repo and its dependencies. |
|
|
| ```powershell |
| cd D:\project\evoneural |
| git clone https://github.com/VAST-AI-Research/TripoSR.git TripoSR |
| pip install -r TripoSR/requirements.txt |
| ``` |
|
|
| On Windows, if `torchmcubes` fails, see [TripoSR README](https://github.com/VAST-AI-Research/TripoSR#troubleshooting) (CUDA version match, then reinstall torchmcubes). |
|
|
| ## 2b. Stable Diffusion model (Hugging Face) |
|
|
| If you see **"Cannot load model ... model is not cached locally and an error occurred while trying to fetch metadata"**, the app cannot reach Hugging Face. Use one of these: |
|
|
| **Option 1 – Log in (uses cached token)** |
| From a terminal with internet: |
|
|
| ```powershell |
| huggingface-cli login |
| ``` |
|
|
| Paste a token from [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) (read access is enough). Then run the app again. |
|
|
| **Option 2 – Set token in env** |
| Create a token at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens), then: |
|
|
| ```powershell |
| $env:HF_TOKEN = "hf_xxxxxxxx" |
| streamlit run app.py |
| ``` |
|
|
| **Option 3 – Download model once, then use offline** |
| On a machine that can reach Hugging Face: |
|
|
| ```powershell |
| cd D:\project\evoneural |
| .venv\Scripts\Activate.ps1 |
| python -m scripts.download_sd_model |
| ``` |
|
|
| Then set the path and run the app (no Hugging Face needed): |
|
|
| ```powershell |
| $env:SD_MODEL_PATH = "D:\project\evoneural\weights\stable-diffusion-2-1-base" |
| streamlit run app.py |
| ``` |
|
|
| ## How it works |
|
|
| 1. **Skybox tab:** You enter a text prompt → the app loads Stable Diffusion (from cache or Hugging Face) → generates a 2:1 image → saves to `outputs/` and shows a download button. Optional “seamless” check compares left/right edges. |
| 2. **Mesh tab:** You enter a prompt (or upload an image) → the app generates an image with SD (if needed) → runs TripoSR on that image → outputs a `.obj` or `.glb` to `outputs/` (requires TripoSR repo cloned in `./TripoSR`). |
| 3. **Model loading:** The app first tries a local folder (`SD_MODEL_PATH` or `weights/stable-diffusion-2-1-base` if complete). If none, it loads `runwayml/stable-diffusion-v1-5` from the Hub (first run downloads the model; later runs use the cache). No token needed unless your network restricts Hugging Face. |
|
|
| ## 3. Run the app |
|
|
| From the project root (with venv activated): |
|
|
| ```powershell |
| cd D:\project\evoneural |
| .venv\Scripts\Activate.ps1 |
| streamlit run app.py |
| ``` |
|
|
| Open **http://localhost:8501**. |
|
|
| - **Text → 3D Mesh**: Enter a prompt (or upload an image). First run downloads SD 2.1 and TripoSR weights. |
| - **Text → Skybox**: Enter a prompt; image is 2:1 (e.g. 1024×512). Use “Run seamless edge check” to compare left/right edges. |
|
|
| Outputs are under `outputs/`. Use the download buttons to save mesh (`.glb`/`.obj`) and skybox (`.png`). |
|
|
| ## 4. Performance |
|
|
| - **Skybox**: ~6–8 GB VRAM (SD 2.1, 1024×512, FP16). Use 2048×1024 only if you have enough VRAM. |
| - **Mesh**: ~6 GB for TripoSR + ~6 GB for SD (text-to-image). Total peak can be ~10–12 GB if both run in same process. |
|
|
| If you run out of VRAM: |
|
|
| - Use 1024×512 for skybox. |
| - Close other GPU apps. |
| - Consider quantization (e.g. 8-bit) or CPU offload in diffusers (see [Optimization](#optimization)). |
|
|
| ## 5. Optimization (if VRAM is exceeded) |
|
|
| - **Quantization**: Use `load_in_8bit=True` or `load_in_4bit=True` with `bitsandbytes` where supported in diffusers. |
| - **Model CPU offload**: In diffusers, `pipe.enable_sequential_cpu_offload()` or `pipe.enable_model_cpu_offload()` to move parts to CPU and reduce peak VRAM (slower). |
| - **Smaller resolution**: 512×512 for text-to-image; 1024×512 for skybox. |
|
|
| ## Project layout |
|
|
| ``` |
| evoneural/ |
| ├── README.md |
| ├── app.py # Streamlit UI |
| ├── requirements.txt |
| ├── environment.yml |
| ├── scripts/ |
| │ ├── skybox_generator.py |
| │ ├── mesh_generator.py |
| │ ├── text_to_image.py |
| │ └── check_seamless.py |
| ├── outputs/ # Generated meshes and skybox images |
| └── TripoSR/ # Clone here (see step 2) |
| ``` |
|
|
| ## License |
|
|
| See TripoSR and Stable Diffusion model licenses (MIT / Stability). This MVP is for local use and evaluation. |
|
|
