Drape / docs /how_it_works.md
Imene-ee's picture
First version of Drape
c280886 verified
|
Raw History Blame Contribute Delete
2.64 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

How it works

photo ──► SegFormer clothes parsing ──► one mask per piece (top, pants, shoes, bag…)
                                           β”‚
                                           β–Ό
                       cut out onto white, square-padded (looks like a product shot)
                                           β”‚
                                           β–Ό
             Marqo-FashionSigLIP image embedding  (+ one for the whole look)
                                           β”‚
               β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
               β–Ό                           β–Ό                             β–Ό
     nearest products inside       "this, but in black leather":   complete the look: nearest
     the piece's own category      image + text scores, z-scored    items to the whole look in the
     (MMR to avoid duplicates)     and blended                     categories the outfit lacks
  1. Find the pieces. A SegFormer-B2 model fine-tuned for clothes parsing labels every pixel (upper clothes, skirt, pants, dress, shoes, bag, hat, scarf, sunglasses, belt). Speckles are removed and each piece becomes a mask.
  2. Make it look like a catalog photo. Each piece is cut out onto a white, square canvas with a feathered edge, because catalog products are shot on white. This closes much of the gap between a street photo and a product image.
  3. Embed. Marqo-FashionSigLIP maps images and text into one space where similar products sit close together.
  4. Search inside the right shelf. A shoe is only compared with shoes. Search is exact cosine similarity over the whole catalog, then re-ranked with maximal marginal relevance so the results are not ten copies of the same item.
  5. Refine with words. Image and text similarities live on different scales (β‰ˆ0.7 vs β‰ˆ0.1), so each is z-scored over the candidates before blending with the weight you choose.
  6. Complete the look. The whole outfit's embedding captures its palette, materials and formality. Within each category the outfit is missing (e.g. shoes, bag, a layer), Drape suggests the items closest to that style. This is zero-shot: no outfit-compatibility training yet.
  7. No person in the photo? Drape treats the image as a single product and routes it to a category by comparing it with text prompts ("a pair of boots", "a handbag", …).

Everything runs on a free CPU: no GPU and no paid API at query time.