Drape / README.md
Imene-ee's picture
First version of Drape
c280886 verified
|
Raw History Blame Contribute Delete
7.48 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade
metadata
title: Drape
emoji: πŸ‘—
colorFrom: yellow
colorTo: red
sdk: gradio
sdk_version: 6.31.0
app_file: app.py
pinned: false
license: mit
short_description: Snap an outfit, shop every piece with fashion AI
models:
  - Marqo/marqo-fashionSigLIP
  - mattmdjaga/segformer_b2_clothes
datasets:
  - Marqo/polyvore
tags:
  - fashion
  - visual-search
  - retrieval
  - multimodal

Drape β€” shop the look

Snap an outfit, shop every piece. Drape finds each garment in a photo, matches it to real products in a catalog, lets you tweak a piece in plain words ("this jacket, but in black leather"), and suggests what would complete the look.

Live demo: https://huggingface.co/spaces/Imene-ee/Drape

Find the pieces Pixel-level garment parsing (top, dress, skirt, pants, shoes, bag, hat, scarf…)
Match to products Fashion-tuned image embeddings, searched only within the right category
Refine with words Blend the photo with a text instruction; you choose the balance
Complete the look Style-coherent suggestions for the categories the outfit is missing
Search by description "chunky white sneakers" searches product images, not keywords
Free to run No GPU and no paid API at query time: a few seconds per photo on CPU

How it works

photo ─► SegFormer clothes parsing ─► one mask per piece
                                         β”‚
                                         β–Ό
                    cut out onto white, square-padded (looks like a product shot)
                                         β”‚
                                         β–Ό
          Marqo-FashionSigLIP embedding (each piece + the whole look)
                                         β”‚
          β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
          β–Ό                              β–Ό                               β–Ό
  nearest products in the        image + text scores, z-scored      nearest items to the whole look
  piece's category (MMR          and blended ("…but in black")      in the categories it lacks
  removes near-duplicates)                                          (shoes, bag, a layer…)

Design choices worth knowing:

  • Cut-out onto white. Catalog products are shot on white; street photos are not. Masking each garment onto a white square narrows that domain gap before embedding.
  • Category routing. Each detected garment is searched only within its own shelves (a top within tops and outerwear), so a shoe never returns a handbag.
  • Score fusion that respects scale. Image–image cosines sit around 0.7 and image–text cosines around 0.1. Adding them raw lets the image side drown out the text, so each is z-scored over the candidates before blending.
  • Diversity. Results are re-ranked with maximal marginal relevance and exact near-duplicates are skipped, so you see ten different products rather than one product ten times.
  • Graceful fallbacks. No person in the photo? Drape treats it as one product and routes it to a category with zero-shot text prompts. Catalog or model missing at startup? The app still loads and says what is wrong.

Benchmark

Known-item retrieval under photo conditions: catalog products are degraded the way a garment looks when cropped out of a real photo (off-centre crop, tilt, colour cast, blur, JPEG, and in the wild setting a non-white background plus a partial occlusion), then searched against a 5,000-product gallery. Same gallery and same queries for every model: the fashion-tuned model Drape ships, the same architecture with generic pre-training, and a classic CLIP.

Run it yourself: python scripts/benchmark.py --index-dir index_out (about 10 minutes on a free Colab T4).

Project layout

app.py                  Gradio app (Hugging Face Space entry point)
drape/
  taxonomy.py           garment kinds, catalog groups, keyword + zero-shot routing
  models.py             open_clip embedder, SegFormer segmenter (torch imported lazily)
  garments.py           masks -> clean crops, overlay rendering
  index.py              catalog index: exact cosine search, filters, fusion, MMR
  pipeline.py           analyze / refine / complete the look / text search
  augment.py            photo-condition degradations for the benchmark
scripts/
  build_index.py        embed a Hugging Face image dataset into a catalog index
  benchmark.py          retrieval benchmark vs. generic baselines
notebooks/
  build_index_colab.ipynb   one-click index build + benchmark on a free GPU
tests/                  pytest suite with fake models (no GPU, no downloads)

Run locally

git clone https://github.com/<your-username>/drape && cd drape
pip install -r requirements.txt
python app.py            # downloads the models and the catalog index on first run
pytest -q                # tests use fake models, no downloads needed

Rebuild the catalog index

The Space loads a prebuilt index (embeddings + thumbnails) from the Hub dataset Imene-ee/drape-catalog. To rebuild it, or to index a different catalog, open notebooks/build_index_colab.ipynb in Google Colab with a GPU runtime and run all cells, or:

pip install -r requirements-build.txt
huggingface-cli login
python scripts/build_index.py --dataset Marqo/polyvore --max-items 40000 --push-repo <you>/drape-catalog

Any Hugging Face dataset with an image column works; the builder picks up title and category columns when they exist and falls back to zero-shot categorisation when they do not.

Configuration

Variable Default Purpose
DRAPE_INDEX_REPO Imene-ee/drape-catalog Hub dataset holding the index
DRAPE_INDEX_DIR β€” Local index folder (overrides the Hub)
DRAPE_EMBED_MODEL hf-hub:Marqo/marqo-fashionSigLIP Must match the model the index was built with
DRAPE_SEG_MODEL mattmdjaga/segformer_b2_clothes Garment parser
DRAPE_MAX_SIDE 1024 Photos are downscaled to this before parsing

Limitations

  • "Complete the look" is zero-shot style similarity, not a trained compatibility model.
  • The benchmark uses synthetic photo degradations of catalog images, not real street-to-shop pairs, and the fashion model saw Polyvore images during its own training (see caveats in the results).
  • The garment parser does not separate a jacket from the top under it; both are searched together.
  • Catalog images come from the public Polyvore research dataset and are used for demonstration only.

Roadmap

  • Outfit-compatibility model trained on Polyvore Outfits, evaluated on the standard fill-in-the-blank task.
  • Consumer-to-shop evaluation on real photos (DeepFashion2).
  • Bring-your-own-catalog: an API that indexes a retailer's product feed.
  • Multilingual search (French, Arabic) for North African e-commerce.

Credits

Marqo-FashionSigLIP (Marqo, Apache-2.0) Β· SegFormer clothes parser by mattmdjaga (fine-tuned on ATR) Β· Polyvore product images via Marqo/polyvore Β· built with open_clip, Transformers and Gradio.

Built by Imene Guizani Β· LinkedIn