Post
3172
I audited one of my own evaluations. The ranking did not hold up the way I expected.
Eight open models, one task: infer the structure of a prompt. Then ask again with the identical call. Caching off.
- Agreement between repeated identical calls (mean Jaccard) ranged from 0.39 to 0.96 across models.
- Only 35 of 127 prompt-model cells were perfectly reproducible on every run.
- I bootstrapped the reproducibility ranking over prompts. The two least reproducible models kept their rank in 99% and 86% of resamples. The middle four kept theirs in 27% to 48%.
So the table reliably finds the worst model. It does not reliably find the best.
Reproducible is also not the same as correct. F1 against gold annotations ran from 0.56 to 0.99.
By the audit date, 4 of the 8 model variants had been retired (HTTP 410). The study as specified can no longer be re-run. The saved outputs are what survives, so I published all of them: every inferred structure, every run, the prompts and the annotations.
Paper: How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure (2609.30074)
Dataset: dipankarsarkar/llm-evaluation-self-audit
Code: https://github.com/sarkar-dipankar/llm-evaluation-self-audit
How many of the leaderboard rankings you rely on would survive re-running the same calls?
Eight open models, one task: infer the structure of a prompt. Then ask again with the identical call. Caching off.
- Agreement between repeated identical calls (mean Jaccard) ranged from 0.39 to 0.96 across models.
- Only 35 of 127 prompt-model cells were perfectly reproducible on every run.
- I bootstrapped the reproducibility ranking over prompts. The two least reproducible models kept their rank in 99% and 86% of resamples. The middle four kept theirs in 27% to 48%.
So the table reliably finds the worst model. It does not reliably find the best.
Reproducible is also not the same as correct. F1 against gold annotations ran from 0.56 to 0.99.
By the audit date, 4 of the 8 model variants had been retired (HTTP 410). The study as specified can no longer be re-run. The saved outputs are what survives, so I published all of them: every inferred structure, every run, the prompts and the annotations.
Paper: How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure (2609.30074)
Dataset: dipankarsarkar/llm-evaluation-self-audit
Code: https://github.com/sarkar-dipankar/llm-evaluation-self-audit
How many of the leaderboard rankings you rely on would survive re-running the same calls?
