Add full cross-modal metrics and qualitative retrieval examples
Browse files
README.md
CHANGED
|
@@ -76,6 +76,36 @@ its audio–text alignment is emergent. This model trains on audio–text only;
|
|
| 76 |
audio–image alignment is emergent. Both evaluated with identical clips, frames, and
|
| 77 |
scoring; ImageBind numbers computed with the released imagebind_huge checkpoint.*
|
| 78 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 79 |
Text, image, and video benchmarks are the base model's published MMEB-V2 results, which
|
| 80 |
are unaffected by this extension.
|
| 81 |
|
|
|
|
| 76 |
audio–image alignment is emergent. Both evaluated with identical clips, frames, and
|
| 77 |
scoring; ImageBind numbers computed with the released imagebind_huge checkpoint.*
|
| 78 |
|
| 79 |
+
Full audio→image metrics for this model (per-modality mean-centered gallery — the readout
|
| 80 |
+
implemented by `FusionEmbedder.center`; chance R@10 = 0.014):
|
| 81 |
+
|
| 82 |
+
| R@1 | R@5 | R@10 | mAP@10 |
|
| 83 |
+
|---|---|---|---|
|
| 84 |
+
| 0.085 | 0.260 | 0.368 | 0.155 |
|
| 85 |
+
|
| 86 |
+
**What audio→image retrieval looks like.** The 0.368 above is not only an aggregate — the
|
| 87 |
+
retrievals are organized by sound. Real examples from this checkpoint on VGGSound-696:
|
| 88 |
+
|
| 89 |
+
*Direct hits* — the clip's own frame is returned in the top 5, among the same kind of scene:
|
| 90 |
+
|
| 91 |
+
| Sound | Top-5 retrieval | Exact frame |
|
| 92 |
+
|---|---|---|
|
| 93 |
+
| A siren | its own ambulance, then more emergency vehicles | rank 2 |
|
| 94 |
+
| *"Switch on the good piece"* (speech) | the appliance being switched on | rank 1 |
|
| 95 |
+
| A whirring kitchen motor | its own blender, among mixers | rank 3 |
|
| 96 |
+
| An emergency-vehicle siren | fire engines and ambulances | rank 4 |
|
| 97 |
+
| Metallic clanking and banging | the kitchen it came from | rank 5 |
|
| 98 |
+
|
| 99 |
+
*Right neighbourhood* — the exact frame ranks lower (often a poor still), but every top
|
| 100 |
+
result is the correct sound category:
|
| 101 |
+
|
| 102 |
+
| Sound | Top-5 retrieval | Exact frame |
|
| 103 |
+
|---|---|---|
|
| 104 |
+
| A distinct click | people typing at keyboards | rank 109 |
|
| 105 |
+
| A power motor | an angle grinder, a belt sander, a table saw | rank 68 |
|
| 106 |
+
| A single cowbell note | cattle herds, one wearing a bell | rank 6 |
|
| 107 |
+
| High squeaks and chirps | chickens, crows, a quail, a parrot | rank 104 |
|
| 108 |
+
|
| 109 |
Text, image, and video benchmarks are the base model's published MMEB-V2 results, which
|
| 110 |
are unaffected by this extension.
|
| 111 |
|