|
Download docs/how_it_works.md from Imene-ee/Drape: direct link, hf CLI and curl.
- Browser
- Download file 2.64 kB
-
https://huggingface.co/spaces/Imene-ee/Drape/resolve/main/docs/how_it_works.md
- Command line
-
hf download hf://spaces/Imene-ee/Drape/docs/how_it_works.md
-
curl -L -o how_it_works.md https://huggingface.co/spaces/Imene-ee/Drape/resolve/main/docs/how_it_works.md
2.64 kB
A newer version of the Gradio SDK is available: 6.32.0
How it works
photo βββΊ SegFormer clothes parsing βββΊ one mask per piece (top, pants, shoes, bagβ¦)
β
βΌ
cut out onto white, square-padded (looks like a product shot)
β
βΌ
Marqo-FashionSigLIP image embedding (+ one for the whole look)
β
βββββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββ
βΌ βΌ βΌ
nearest products inside "this, but in black leather": complete the look: nearest
the piece's own category image + text scores, z-scored items to the whole look in the
(MMR to avoid duplicates) and blended categories the outfit lacks
- Find the pieces. A SegFormer-B2 model fine-tuned for clothes parsing labels every pixel (upper clothes, skirt, pants, dress, shoes, bag, hat, scarf, sunglasses, belt). Speckles are removed and each piece becomes a mask.
- Make it look like a catalog photo. Each piece is cut out onto a white, square canvas with a feathered edge, because catalog products are shot on white. This closes much of the gap between a street photo and a product image.
- Embed. Marqo-FashionSigLIP maps images and text into one space where similar products sit close together.
- Search inside the right shelf. A shoe is only compared with shoes. Search is exact cosine similarity over the whole catalog, then re-ranked with maximal marginal relevance so the results are not ten copies of the same item.
- Refine with words. Image and text similarities live on different scales (β0.7 vs β0.1), so each is z-scored over the candidates before blending with the weight you choose.
- Complete the look. The whole outfit's embedding captures its palette, materials and formality. Within each category the outfit is missing (e.g. shoes, bag, a layer), Drape suggests the items closest to that style. This is zero-shot: no outfit-compatibility training yet.
- No person in the photo? Drape treats the image as a single product and routes it to a category by comparing it with text prompts ("a pair of boots", "a handbag", β¦).
Everything runs on a free CPU: no GPU and no paid API at query time.