match

Next Match

Next Match finds a catalog item from a reference image plus a short edit in Turkish or English — "bunun kırmızı olanı", "çizgili olanı", "same but a hat". The test catalog is built so that the image alone (it finds look-alikes in the wrong colour) and the text alone (it finds red things of any kind) both fail; the fused query succeeds. The catalog is synthetic: procedurally drawn product icons (8 categories × 8 colours × 3 patterns).

Space: https://huggingface.co/spaces/thelamapi/Next-Match — pick a product (or upload a picture), type a change, and compare the three rankings live.

Results (1600 test queries, 192-item catalog, unseen renderings)

Recall@1 / Recall@5:

Query group n Image + text (Next Match) Image only Text only
all 1600 0.932 / 0.966 0.000 / 0.316 0.054 / 0.263
edit=color 648 0.994 / 1.000 0.000 / 0.358 0.051 / 0.218
edit=pattern 311 0.836 / 0.897 0.000 / 0.878 0.016 / 0.074
edit=category 322 1.000 / 1.000 0.000 / 0.000 0.031 / 0.183
edit=color+pattern 319 0.834 / 0.931 0.000 / 0.000 0.122 / 0.621
lang=tr 747 0.995 / 1.000 0.000 / 0.317 0.051 / 0.258
lang=en 853 0.878 / 0.937 0.000 / 0.314 0.057 / 0.267
template=seen 1104 1.000 / 1.000 0.000 / 0.323 0.057 / 0.251
template=held-out 496 0.782 / 0.891 0.000 / 0.298 0.048 / 0.290
combo=seen 1192 0.933 / 0.966 0.000 / 0.348 0.057 / 0.296
combo=held-out 408 0.931 / 0.966 0.000 / 0.221 0.047 / 0.167

template=held-out: phrasings never used in training (e.g. "show me the red one", "kırmızı olanını göster"). combo=held-out: the reference or target is one of 24 attribute combinations never rendered during training.

Test queries and Recall@K

Method

  • Image encoder: 4 conv blocks (16/32/64/64 channels) + linear → 64-d. Catalog keys are L2-normalised image features.
  • Text: Turkish-aware lower-casing and longest-prefix stemming over a 61-word vocabulary (so "kırmızısı" or "olanını" still match), mean word embedding → MLP.
  • Fusion: q = normalise(f_img + MLP([f_img, f_text])) — the text predicts a residual edit of the image feature.
  • Baselines: image-only (image encoder + MLP head) and text-only (text encoder vs its own image keys), same architecture budget, 6000 steps, batch 128, InfoNCE with τ = 0.1.
  • The Space runs all three encoders with nlrt.js on exported weights; Node parity max abs difference 1.6653345369377348e-16.

Usage

from inference import load, search
spec = load()
ids = search(spec, ref_image, "bunun kırmızı olanı")   # ref_image: (48, 48, 3) uint8; ids index catalog.json

Limitations

  • Toy domain: flat icons on plain backgrounds. Real product photos need a pretrained vision encoder and real catalog data.
  • Vocabulary is closed: words outside it are ignored (the Space shows which words were understood).
  • Uploaded photos are far outside the training distribution and are a curiosity, not a supported use.
  • Trained on NVIDIA L4; profile full. Failure examples are in evaluation.json.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Space using thelamapi/Next-Match 1

Collection including thelamapi/Next-Match