File size: 903 Bytes
398cfcf | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | # Evaluation summary
`summary.json` records the completed Sol Nano checkpoint's full zero-shot evaluation on HellaSwag, ARC-Easy, ARC-Challenge, PIQA, and ArithMark-3.0.
The Axiomic Labs Open SLM Intelligence Index is calculated with:
`N(s,c) = 100 * (s-c) / (100-c)`
`ARC = (ARC-Easy + ARC-Challenge) / 2`
`Index = [N(HellaSwag,25) + N(ARC,25) + N(PIQA,50) + 0.65*N(ArithMark-3,25)] / 3.65`
Result: **6.068393579554266 / 100**.
This is a local result, not an independently verified leaderboard submission. The original checkpoint path in the summary identifies the Molab training environment; model weights in this repository are exported from that checkpoint.
References: [Axiomic index methodology](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard/blob/main/index.html), [official ArithMark-3.0 dataset and evaluator](https://huggingface.co/datasets/AxiomicLabs/Arithmark-3.0).
|