File size: 3,393 Bytes
bda739d
 
8b916da
 
 
bda739d
8b916da
bda739d
8b916da
 
bda739d
 
8b916da
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
---
title: Next Match
emoji: 🧩
colorFrom: green
colorTo: gray
sdk: static
app_file: index.html
pinned: false
license: apache-2.0
short_description: Search with an image plus a sentence (TR/EN), live
---

# Next Match

Next Match finds a catalog item from a **reference image plus a short edit in Turkish or English** —
*"bunun kırmızı olanı"*, *"çizgili olanı"*, *"same but a hat"*. The test catalog is built so that the image alone (it
finds look-alikes in the wrong colour) and the text alone (it finds red things of any kind) both fail; the fused query
succeeds. The catalog is **synthetic**: procedurally drawn product icons (8 categories × 8 colours × 3 patterns).

**Model:** https://huggingface.co/thelamapi/Next-Match — pick a product (or upload a picture), type a change, and compare the three rankings live.

## Results (1600 test queries, 192-item catalog, unseen renderings)

Recall@1 / Recall@5:

| Query group | n | Image + text (Next Match) | Image only | Text only |
|---|---:|---:|---:|---:|
| all | 1600 | 0.932 / 0.966 | 0.000 / 0.316 | 0.054 / 0.263 |
| edit=color | 648 | 0.994 / 1.000 | 0.000 / 0.358 | 0.051 / 0.218 |
| edit=pattern | 311 | 0.836 / 0.897 | 0.000 / 0.878 | 0.016 / 0.074 |
| edit=category | 322 | 1.000 / 1.000 | 0.000 / 0.000 | 0.031 / 0.183 |
| edit=color+pattern | 319 | 0.834 / 0.931 | 0.000 / 0.000 | 0.122 / 0.621 |
| lang=tr | 747 | 0.995 / 1.000 | 0.000 / 0.317 | 0.051 / 0.258 |
| lang=en | 853 | 0.878 / 0.937 | 0.000 / 0.314 | 0.057 / 0.267 |
| template=seen | 1104 | 1.000 / 1.000 | 0.000 / 0.323 | 0.057 / 0.251 |
| template=held-out | 496 | 0.782 / 0.891 | 0.000 / 0.298 | 0.048 / 0.290 |
| combo=seen | 1192 | 0.933 / 0.966 | 0.000 / 0.348 | 0.057 / 0.296 |
| combo=held-out | 408 | 0.931 / 0.966 | 0.000 / 0.221 | 0.047 / 0.167 |

*template=held-out*: phrasings never used in training (e.g. "show me the red one", "kırmızı olanını göster").
*combo=held-out*: the reference or target is one of 24 attribute combinations never rendered during training.

![Test queries and Recall@K](preview.png)

## Method
* Image encoder: 4 conv blocks (16/32/64/64 channels) + linear → 64-d. Catalog keys are L2-normalised image features.
* Text: Turkish-aware lower-casing and longest-prefix stemming over a 61-word vocabulary (so "kırmızısı" or "olanını" still match), mean word embedding → MLP.
* Fusion: q = normalise(f_img + MLP([f_img, f_text])) — the text predicts a residual edit of the image feature.
* Baselines: image-only (image encoder + MLP head) and text-only (text encoder vs its own image keys), same architecture budget, 6000 steps, batch 128, InfoNCE with τ = 0.1.
* The Space runs all three encoders with `nlrt.js` on exported weights; Node parity max abs difference 1.6653345369377348e-16.

## Usage
```python
from inference import load, search
spec = load()
ids = search(spec, ref_image, "bunun kırmızı olanı")   # ref_image: (48, 48, 3) uint8; ids index catalog.json
```

## Limitations
* Toy domain: flat icons on plain backgrounds. Real product photos need a pretrained vision encoder and real catalog data.
* Vocabulary is closed: words outside it are ignored (the Space shows which words were understood).
* Uploaded photos are far outside the training distribution and are a curiosity, not a supported use.
* Trained on NVIDIA L4; profile full. Failure examples are in `evaluation.json`.