File size: 7,477 Bytes
5d64530
 
c280886
 
 
5d64530
 
 
 
 
 
c280886
 
 
 
 
 
 
 
 
 
5d64530
 
c280886
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
---
title: Drape
emoji: πŸ‘—
colorFrom: yellow
colorTo: red
sdk: gradio
sdk_version: 6.31.0
app_file: app.py
pinned: false
license: mit
short_description: Snap an outfit, shop every piece with fashion AI
models:
  - Marqo/marqo-fashionSigLIP
  - mattmdjaga/segformer_b2_clothes
datasets:
  - Marqo/polyvore
tags:
  - fashion
  - visual-search
  - retrieval
  - multimodal
---

# Drape β€” shop the look

**Snap an outfit, shop every piece.** Drape finds each garment in a photo, matches it to real products in a catalog, lets you tweak a piece in plain words ("this jacket, but in black leather"), and suggests what would complete the look.

**Live demo:** https://huggingface.co/spaces/Imene-ee/Drape  
<!-- After deploying, add a screenshot: docs/screenshot.png -->

| | |
|---|---|
| **Find the pieces** | Pixel-level garment parsing (top, dress, skirt, pants, shoes, bag, hat, scarf…) |
| **Match to products** | Fashion-tuned image embeddings, searched only within the right category |
| **Refine with words** | Blend the photo with a text instruction; you choose the balance |
| **Complete the look** | Style-coherent suggestions for the categories the outfit is missing |
| **Search by description** | "chunky white sneakers" searches product images, not keywords |
| **Free to run** | No GPU and no paid API at query time: a few seconds per photo on CPU |

## How it works

```
photo ─► SegFormer clothes parsing ─► one mask per piece
                                         β”‚
                                         β–Ό
                    cut out onto white, square-padded (looks like a product shot)
                                         β”‚
                                         β–Ό
          Marqo-FashionSigLIP embedding (each piece + the whole look)
                                         β”‚
          β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
          β–Ό                              β–Ό                               β–Ό
  nearest products in the        image + text scores, z-scored      nearest items to the whole look
  piece's category (MMR          and blended ("…but in black")      in the categories it lacks
  removes near-duplicates)                                          (shoes, bag, a layer…)
```

Design choices worth knowing:

- **Cut-out onto white.** Catalog products are shot on white; street photos are not. Masking each garment onto a white square narrows that domain gap before embedding.
- **Category routing.** Each detected garment is searched only within its own shelves (a top within tops and outerwear), so a shoe never returns a handbag.
- **Score fusion that respects scale.** Image–image cosines sit around 0.7 and image–text cosines around 0.1. Adding them raw lets the image side drown out the text, so each is z-scored over the candidates before blending.
- **Diversity.** Results are re-ranked with maximal marginal relevance and exact near-duplicates are skipped, so you see ten different products rather than one product ten times.
- **Graceful fallbacks.** No person in the photo? Drape treats it as one product and routes it to a category with zero-shot text prompts. Catalog or model missing at startup? The app still loads and says what is wrong.

## Benchmark

<!-- Replace this section with the contents of benchmark/results.md after running scripts/benchmark.py -->

Known-item retrieval under photo conditions: catalog products are degraded the way a garment looks when cropped out of a real photo (off-centre crop, tilt, colour cast, blur, JPEG, and in the *wild* setting a non-white background plus a partial occlusion), then searched against a 5,000-product gallery. Same gallery and same queries for every model: the fashion-tuned model Drape ships, the same architecture with generic pre-training, and a classic CLIP.

Run it yourself: `python scripts/benchmark.py --index-dir index_out` (about 10 minutes on a free Colab T4).

## Project layout

```
app.py                  Gradio app (Hugging Face Space entry point)
drape/
  taxonomy.py           garment kinds, catalog groups, keyword + zero-shot routing
  models.py             open_clip embedder, SegFormer segmenter (torch imported lazily)
  garments.py           masks -> clean crops, overlay rendering
  index.py              catalog index: exact cosine search, filters, fusion, MMR
  pipeline.py           analyze / refine / complete the look / text search
  augment.py            photo-condition degradations for the benchmark
scripts/
  build_index.py        embed a Hugging Face image dataset into a catalog index
  benchmark.py          retrieval benchmark vs. generic baselines
notebooks/
  build_index_colab.ipynb   one-click index build + benchmark on a free GPU
tests/                  pytest suite with fake models (no GPU, no downloads)
```

## Run locally

```bash
git clone https://github.com/<your-username>/drape && cd drape
pip install -r requirements.txt
python app.py            # downloads the models and the catalog index on first run
pytest -q                # tests use fake models, no downloads needed
```

## Rebuild the catalog index

The Space loads a prebuilt index (embeddings + thumbnails) from the Hub dataset `Imene-ee/drape-catalog`. To rebuild it, or to index a different catalog, open `notebooks/build_index_colab.ipynb` in Google Colab with a GPU runtime and run all cells, or:

```bash
pip install -r requirements-build.txt
huggingface-cli login
python scripts/build_index.py --dataset Marqo/polyvore --max-items 40000 --push-repo <you>/drape-catalog
```

Any Hugging Face dataset with an image column works; the builder picks up title and category columns when they exist and falls back to zero-shot categorisation when they do not.

## Configuration

| Variable | Default | Purpose |
|---|---|---|
| `DRAPE_INDEX_REPO` | `Imene-ee/drape-catalog` | Hub dataset holding the index |
| `DRAPE_INDEX_DIR` | β€” | Local index folder (overrides the Hub) |
| `DRAPE_EMBED_MODEL` | `hf-hub:Marqo/marqo-fashionSigLIP` | Must match the model the index was built with |
| `DRAPE_SEG_MODEL` | `mattmdjaga/segformer_b2_clothes` | Garment parser |
| `DRAPE_MAX_SIDE` | `1024` | Photos are downscaled to this before parsing |

## Limitations

- "Complete the look" is zero-shot style similarity, not a trained compatibility model.
- The benchmark uses synthetic photo degradations of catalog images, not real street-to-shop pairs, and the fashion model saw Polyvore images during its own training (see caveats in the results).
- The garment parser does not separate a jacket from the top under it; both are searched together.
- Catalog images come from the public Polyvore research dataset and are used for demonstration only.

## Roadmap

- Outfit-compatibility model trained on Polyvore Outfits, evaluated on the standard fill-in-the-blank task.
- Consumer-to-shop evaluation on real photos (DeepFashion2).
- Bring-your-own-catalog: an API that indexes a retailer's product feed.
- Multilingual search (French, Arabic) for North African e-commerce.

## Credits

Marqo-FashionSigLIP (Marqo, Apache-2.0) Β· SegFormer clothes parser by mattmdjaga (fine-tuned on ATR) Β· Polyvore product images via `Marqo/polyvore` Β· built with open_clip, Transformers and Gradio.

Built by **Imene Guizani** Β· [LinkedIn](https://linkedin.com/in/imene-guizani)