Home-decor SigLIP 2 (v3-clean-nostyle)

A fine-tune of google/siglip2-base-patch16-224 for furniture and home-decor search: real room photo β†’ product, text β†’ product, and text β†’ room.

Code (data cleaning, training, evaluation): github.com/jainalphin/home-decor-siglip. Every evaluation table, including runs that did not help: RESULTS.md.

Results

All numbers are on held-out data that was never used for training or for choosing the checkpoint.

Task Base SigLIP 2 This model
Real room photo β†’ product (Shop the Look, 2,250 queries, 19k-product gallery), R@1 / R@10 0.076 / 0.364 0.164 / 0.508
Real furniture photo β†’ other photos of the same model (Pix3D, 6,795 queries), R@1 0.778 0.827
Search query β†’ room (DeepFurniture test rooms with no near-copy in training), R@10 0.281 0.391
Search query β†’ furniture crop (DeepFurniture test), R@10 0.361 0.440
Search query β†’ product photo (ABO test), R@1 0.620 0.680

95% bootstrap intervals for the "this model βˆ’ base" differences exclude zero for the first four rows. For ABO only R@10 was bootstrapped (+0.009, interval [0.000, 0.018]); both models are near the ceiling there.

Usage

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

model_id = "AlphJain/home-decor-siglip2"
model = AutoModel.from_pretrained(model_id).eval()
processor = AutoProcessor.from_pretrained(model_id)

image = Image.open("room.jpg").convert("RGB")
texts = ["green velvet sofa", "scandinavian bedroom", "brass floor lamp"]   # lowercase, as in training
with torch.no_grad():
    img = model.get_image_features(**processor(images=image, return_tensors="pt"))
    txt = model.get_text_features(**processor(text=texts, padding="max_length", max_length=64,
                                              truncation=True, return_tensors="pt"))
img = torch.nn.functional.normalize(img, dim=-1)
txt = torch.nn.functional.normalize(txt, dim=-1)
scores = (img @ txt.T)[0]
for text, score in sorted(zip(texts, scores.tolist()), key=lambda x: -x[1]):
    print(f"{score:.3f}  {text}")
Downloads last month
41
Safetensors
Model size
0.4B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for AlphJain/home-decor-siglip2

Finetuned
(154)
this model