abtonmoy commited on
Commit
5874cec
·
verified ·
1 Parent(s): 6d3bd56

Add full cross-modal metrics and qualitative retrieval examples

Browse files
Files changed (1) hide show
  1. README.md +30 -0
README.md CHANGED
@@ -76,6 +76,36 @@ its audio–text alignment is emergent. This model trains on audio–text only;
76
  audio–image alignment is emergent. Both evaluated with identical clips, frames, and
77
  scoring; ImageBind numbers computed with the released imagebind_huge checkpoint.*
78
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
79
  Text, image, and video benchmarks are the base model's published MMEB-V2 results, which
80
  are unaffected by this extension.
81
 
 
76
  audio–image alignment is emergent. Both evaluated with identical clips, frames, and
77
  scoring; ImageBind numbers computed with the released imagebind_huge checkpoint.*
78
 
79
+ Full audio→image metrics for this model (per-modality mean-centered gallery — the readout
80
+ implemented by `FusionEmbedder.center`; chance R@10 = 0.014):
81
+
82
+ | R@1 | R@5 | R@10 | mAP@10 |
83
+ |---|---|---|---|
84
+ | 0.085 | 0.260 | 0.368 | 0.155 |
85
+
86
+ **What audio→image retrieval looks like.** The 0.368 above is not only an aggregate — the
87
+ retrievals are organized by sound. Real examples from this checkpoint on VGGSound-696:
88
+
89
+ *Direct hits* — the clip's own frame is returned in the top 5, among the same kind of scene:
90
+
91
+ | Sound | Top-5 retrieval | Exact frame |
92
+ |---|---|---|
93
+ | A siren | its own ambulance, then more emergency vehicles | rank 2 |
94
+ | *"Switch on the good piece"* (speech) | the appliance being switched on | rank 1 |
95
+ | A whirring kitchen motor | its own blender, among mixers | rank 3 |
96
+ | An emergency-vehicle siren | fire engines and ambulances | rank 4 |
97
+ | Metallic clanking and banging | the kitchen it came from | rank 5 |
98
+
99
+ *Right neighbourhood* — the exact frame ranks lower (often a poor still), but every top
100
+ result is the correct sound category:
101
+
102
+ | Sound | Top-5 retrieval | Exact frame |
103
+ |---|---|---|
104
+ | A distinct click | people typing at keyboards | rank 109 |
105
+ | A power motor | an angle grinder, a belt sander, a table saw | rank 68 |
106
+ | A single cowbell note | cattle herds, one wearing a bell | rank 6 |
107
+ | High squeaks and chirps | chickens, crows, a quail, a parrot | rank 104 |
108
+
109
  Text, image, and video benchmarks are the base model's published MMEB-V2 results, which
110
  are unaffected by this extension.
111