This is a great resource for the SLM community! Cross-encoders are often overlooked in small-model discussions because everyone focuses on causal LMs, but a well-tuned 20-50M cross-encoder can punch well above its weight on NLI and retrieval tasks.

One thing I've noticed working with sub-100M models: positional encoding choices matter enormously at that scale. If you're accepting LLM contributions, even a tiny 5M-param encoder with RoPE vs learned absolute positions can shift NLI accuracy by several points on short premises/hypotheses.

Would love to see a row for a distilled MiniLM-style model or a from-scratch 10M cross-encoder. Happy to help benchmark if you want a second pair of eyes on task formatting. 🤗

The parenthesized replacements make declared test-set training overlap visible. For comparing a few models, a companion view using only their shared benchmarks without that overlap would be useful alongside the imputed scores. It would separate directly measured comparisons from the kNN and z-score estimates, while making the reduced dataset coverage explicit.