File size: 7,477 Bytes
5d64530 c280886 5d64530 c280886 5d64530 c280886 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 | ---
title: Drape
emoji: π
colorFrom: yellow
colorTo: red
sdk: gradio
sdk_version: 6.31.0
app_file: app.py
pinned: false
license: mit
short_description: Snap an outfit, shop every piece with fashion AI
models:
- Marqo/marqo-fashionSigLIP
- mattmdjaga/segformer_b2_clothes
datasets:
- Marqo/polyvore
tags:
- fashion
- visual-search
- retrieval
- multimodal
---
# Drape β shop the look
**Snap an outfit, shop every piece.** Drape finds each garment in a photo, matches it to real products in a catalog, lets you tweak a piece in plain words ("this jacket, but in black leather"), and suggests what would complete the look.
**Live demo:** https://huggingface.co/spaces/Imene-ee/Drape
<!-- After deploying, add a screenshot: docs/screenshot.png -->
| | |
|---|---|
| **Find the pieces** | Pixel-level garment parsing (top, dress, skirt, pants, shoes, bag, hat, scarfβ¦) |
| **Match to products** | Fashion-tuned image embeddings, searched only within the right category |
| **Refine with words** | Blend the photo with a text instruction; you choose the balance |
| **Complete the look** | Style-coherent suggestions for the categories the outfit is missing |
| **Search by description** | "chunky white sneakers" searches product images, not keywords |
| **Free to run** | No GPU and no paid API at query time: a few seconds per photo on CPU |
## How it works
```
photo ββΊ SegFormer clothes parsing ββΊ one mask per piece
β
βΌ
cut out onto white, square-padded (looks like a product shot)
β
βΌ
Marqo-FashionSigLIP embedding (each piece + the whole look)
β
ββββββββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββββ
βΌ βΌ βΌ
nearest products in the image + text scores, z-scored nearest items to the whole look
piece's category (MMR and blended ("β¦but in black") in the categories it lacks
removes near-duplicates) (shoes, bag, a layerβ¦)
```
Design choices worth knowing:
- **Cut-out onto white.** Catalog products are shot on white; street photos are not. Masking each garment onto a white square narrows that domain gap before embedding.
- **Category routing.** Each detected garment is searched only within its own shelves (a top within tops and outerwear), so a shoe never returns a handbag.
- **Score fusion that respects scale.** Imageβimage cosines sit around 0.7 and imageβtext cosines around 0.1. Adding them raw lets the image side drown out the text, so each is z-scored over the candidates before blending.
- **Diversity.** Results are re-ranked with maximal marginal relevance and exact near-duplicates are skipped, so you see ten different products rather than one product ten times.
- **Graceful fallbacks.** No person in the photo? Drape treats it as one product and routes it to a category with zero-shot text prompts. Catalog or model missing at startup? The app still loads and says what is wrong.
## Benchmark
<!-- Replace this section with the contents of benchmark/results.md after running scripts/benchmark.py -->
Known-item retrieval under photo conditions: catalog products are degraded the way a garment looks when cropped out of a real photo (off-centre crop, tilt, colour cast, blur, JPEG, and in the *wild* setting a non-white background plus a partial occlusion), then searched against a 5,000-product gallery. Same gallery and same queries for every model: the fashion-tuned model Drape ships, the same architecture with generic pre-training, and a classic CLIP.
Run it yourself: `python scripts/benchmark.py --index-dir index_out` (about 10 minutes on a free Colab T4).
## Project layout
```
app.py Gradio app (Hugging Face Space entry point)
drape/
taxonomy.py garment kinds, catalog groups, keyword + zero-shot routing
models.py open_clip embedder, SegFormer segmenter (torch imported lazily)
garments.py masks -> clean crops, overlay rendering
index.py catalog index: exact cosine search, filters, fusion, MMR
pipeline.py analyze / refine / complete the look / text search
augment.py photo-condition degradations for the benchmark
scripts/
build_index.py embed a Hugging Face image dataset into a catalog index
benchmark.py retrieval benchmark vs. generic baselines
notebooks/
build_index_colab.ipynb one-click index build + benchmark on a free GPU
tests/ pytest suite with fake models (no GPU, no downloads)
```
## Run locally
```bash
git clone https://github.com/<your-username>/drape && cd drape
pip install -r requirements.txt
python app.py # downloads the models and the catalog index on first run
pytest -q # tests use fake models, no downloads needed
```
## Rebuild the catalog index
The Space loads a prebuilt index (embeddings + thumbnails) from the Hub dataset `Imene-ee/drape-catalog`. To rebuild it, or to index a different catalog, open `notebooks/build_index_colab.ipynb` in Google Colab with a GPU runtime and run all cells, or:
```bash
pip install -r requirements-build.txt
huggingface-cli login
python scripts/build_index.py --dataset Marqo/polyvore --max-items 40000 --push-repo <you>/drape-catalog
```
Any Hugging Face dataset with an image column works; the builder picks up title and category columns when they exist and falls back to zero-shot categorisation when they do not.
## Configuration
| Variable | Default | Purpose |
|---|---|---|
| `DRAPE_INDEX_REPO` | `Imene-ee/drape-catalog` | Hub dataset holding the index |
| `DRAPE_INDEX_DIR` | β | Local index folder (overrides the Hub) |
| `DRAPE_EMBED_MODEL` | `hf-hub:Marqo/marqo-fashionSigLIP` | Must match the model the index was built with |
| `DRAPE_SEG_MODEL` | `mattmdjaga/segformer_b2_clothes` | Garment parser |
| `DRAPE_MAX_SIDE` | `1024` | Photos are downscaled to this before parsing |
## Limitations
- "Complete the look" is zero-shot style similarity, not a trained compatibility model.
- The benchmark uses synthetic photo degradations of catalog images, not real street-to-shop pairs, and the fashion model saw Polyvore images during its own training (see caveats in the results).
- The garment parser does not separate a jacket from the top under it; both are searched together.
- Catalog images come from the public Polyvore research dataset and are used for demonstration only.
## Roadmap
- Outfit-compatibility model trained on Polyvore Outfits, evaluated on the standard fill-in-the-blank task.
- Consumer-to-shop evaluation on real photos (DeepFashion2).
- Bring-your-own-catalog: an API that indexes a retailer's product feed.
- Multilingual search (French, Arabic) for North African e-commerce.
## Credits
Marqo-FashionSigLIP (Marqo, Apache-2.0) Β· SegFormer clothes parser by mattmdjaga (fine-tuned on ATR) Β· Polyvore product images via `Marqo/polyvore` Β· built with open_clip, Transformers and Gradio.
Built by **Imene Guizani** Β· [LinkedIn](https://linkedin.com/in/imene-guizani)
|