is_longer at batch 1: which branch is the intended one?
Your write-up of the is_longer lines is the clearest description of this I have found,
and the measurement on short system sounds settles the behaviour: at batch 1 it is always
True, so the fusion branch is what the stock processor takes.
One thing you may want to test, because it points the other way. The is_longer=False
build is not necessarily the wrong branch; it may be the branch the checkpoint was meant
to take. Two observations from a retrieval benchmark I ran on the same checkpoint:
laion_clap's inference API goes throughget_audio_features, which setslonger=False
for audio at or under the window and randomiseslongeronly in a training dataset. Its
inference path is the False branch.- Running 957 AudioCaps clips through the HF path with
is_longerforced False gives T2A
R@1 0.349, against 0.342 for thelaion_clappackage on the identical fixture and trim.
Through the fusion-forced path it is 0.232. The False build lands on the reference
pipeline's numbers; the True build sits 11.0 points below them.
The test that would settle it on your own data: compare both of your builds against
embeddings from laion_clap itself on the same clips, which shows directly which branch
matches the reference. One caveat for that comparison: laion_clap sets longer=False
regardless of length, but at clip lengths as short as your 0.76 to 1.65 s the padding mode
also matters, so it is worth matching the reference padding there too.
I read your cosine 0.78 to 0.87 result as the two branches measured against each other,
which differ by construction. The question underneath is which branch the checkpoint
expects for short audio, and on retrieval quality the answer looks like False.
Separately, the RNG is unseeded, so batch sizes above 1 also shift which clips take which
branch between runs: two seeded batch-8 runs reproduce exactly, the unseeded pair differs
in 478 of 957 ranks. That may matter for anyone comparing your export against the stock
model on a batch larger than one.
I have filed this upstream as https://github.com/huggingface/transformers/issues/50097 so the discussion lives in one place.
Details: https://huggingface.co/datasets/b4ph/clap-retrieval-benchmark
Thank you for this, and for the benchmark behind it. You're right, and the card's verdict on the is_longer=False build ("That build is wrong") does not hold.
I read laion_clap's inference path to check. For the fused model, get_audio_embedding_from_data calls get_audio_features with data_truncating='fusion' and data_filling='repeatpad'. A clip of up to 480,000 samples (10 s at 48 kHz) has its mel stacked four times and longer set to False, after being repeat-padded to that length if it is shorter; only longer clips take the fusion branch. So for the short clips an app sends, the reference takes the False branch. My 0.78 to 0.87 cosine compared that build with the stock HF pipeline at batch 1, which is the behaviour your transformers issue reports. As you say, it shows that the two branches differ, and nothing about which one the checkpoint expects.
Next, on my side:
- run your test: both builds against embeddings from laion_clap itself on the same clips, with laion_clap's repeat-padding;
- add the
is_longer=Falsebuild to this repo next to the fusion build, with those numbers; - rewrite the
is_longersection to say which branch the reference takes, with links to your benchmark and to transformers#50097.
On the RNG: agreed. This graph does not read is_longer and has a fixed batch of one, so the file always takes the same path, but anyone comparing it with the stock model at a larger batch should set is_longer or seed the RNG. I'll add a line on that too.
I'll post the numbers here when they're in.
I ran it, on the five clips from the card (0.76 to 1.65 s), against laion_clap's own pipeline: its repeat-padding, its mel, and its model in eval mode. The script is in the repo, scripts/compare_laion_clap.py.
As your checkpoint-identity file found, laion_clap's default fusion checkpoint is not the one in laion/clap-htsat-fused, and neither is 630k-fusion-best.pt. So the reference loads the HF weights into laion_clap's own model, through transformers' conversion map in reverse; text embeddings then match at cosine 1.000000.
Cosine against that reference, fp32 files:
is_longer=Falsebuild: 0.938 to 0.975- fusion build: 0.724 to 0.824
Given laion_clap's mel instead of the HF extractor's, the False branch matches at 1.000000 on all five. The rest of its gap is laion_clap truncating the waveform to int16: with the same step on the HF side the mels agree to within 0.01 dB, and without it they differ by up to 59 dB in near-silent frames. So for short clips the reference takes the False branch, as you said.
In the repo now:
- The False build sits next to the fusion one:
clap_audio_global_xnnpack_fp32.pteand_int8.pte, and a Core ML build. It takes only the first mel chunk,input_features[:, 0:1], since for a clip of 10 s or less all four are the same. The fusion files stay for clips over 10 s, a case I have not measured. - The
is_longerindex was also what kept Core ML off the global graph. Without it, Core ML runs that graph in 8.6 ms on an M4 Max, against 66 ms on XNNPACK. - The
is_longersection is rewritten: which branch the reference takes, these numbers, links to your benchmark and transformers#50097, and the RNG note, which now also covers the random crops of clips over 10 s.
Thank you for the benchmark and for pointing at the right test. It corrected the card.