PolyPythias - a EleutherAI Collection

EleutherAI 's Collections

Pythia Scaling Suite

Quirky Models and Datasets

Quirky Math (archive)

Pile-T5 T5x Checkpoints

Sparse Autoencoders

PolyPythias

updated Apr 9, 2025

EleutherAI/polypythias-evals

Preview • Updated Sep 11, 2024 • 1.85k
EleutherAI/pile-preshuffled-seeds

Updated Feb 12 • 272 • 1

Note Training data information for each seed.
EleutherAI/pythia-14m-deduped

Text Generation • 39.2M • Updated Feb 12 • 9.67k • 30
EleutherAI/pythia-14m-seed1

Text Generation • Updated Jan 11 • 2.41k
EleutherAI/pythia-14m-seed2

Text Generation • Updated Jan 11 • 1.75k
EleutherAI/pythia-14m-seed3

Text Generation • Updated Jan 11 • 1.78k
EleutherAI/pythia-14m-seed4

Text Generation • Updated Jan 11 • 1.49k
EleutherAI/pythia-14m-seed5

Text Generation • Updated Jan 11 • 1.82k
EleutherAI/pythia-14m-seed6

Text Generation • Updated Jan 11 • 1.61k
EleutherAI/pythia-14m-seed7

Text Generation • Updated Jan 11 • 1.82k
EleutherAI/pythia-14m-seed8

Text Generation • Updated Jan 11 • 1.59k
EleutherAI/pythia-14m-seed9

Text Generation • Updated Jan 11 • 1.8k
EleutherAI/pythia-31m-deduped

Text Generation • 55.7M • Updated Feb 12 • 2.11k • 5
EleutherAI/pythia-31m-seed1

Text Generation • Updated Jan 11 • 1.54k
EleutherAI/pythia-31m-seed2

Text Generation • Updated Jan 11 • 1.44k
EleutherAI/pythia-31m-seed3

Text Generation • Updated Jan 11 • 1.38k
EleutherAI/pythia-31m-seed4

Text Generation • Updated Jan 11 • 1.39k
EleutherAI/pythia-31m-seed5

Text Generation • Updated Jan 11 • 1.39k
EleutherAI/pythia-31m-seed6

Text Generation • Updated Jan 11 • 1.38k
EleutherAI/pythia-31m-seed7

Text Generation • Updated Jan 11 • 1.41k
EleutherAI/pythia-31m-seed8

Text Generation • Updated Jan 11 • 1.38k
EleutherAI/pythia-31m-seed9

Text Generation • Updated Jan 11 • 1.54k
EleutherAI/pythia-70m

95.6M • Updated Nov 21, 2023 • 526k • 85
EleutherAI/pythia-70m-seed1

Text Generation • Updated Jan 11 • 8.8k
EleutherAI/pythia-70m-seed2

Text Generation • Updated Jan 11 • 8.38k
EleutherAI/pythia-70m-seed3

Text Generation • Updated Jan 11 • 1.61k
EleutherAI/pythia-70m-seed4

Text Generation • Updated Jan 11 • 1.54k
EleutherAI/pythia-70m-seed5

Text Generation • Updated Jan 11 • 1.56k
EleutherAI/pythia-70m-seed6

Text Generation • Updated Jan 11 • 1.77k
EleutherAI/pythia-70m-seed7

Text Generation • Updated Jan 11 • 1.72k
EleutherAI/pythia-70m-seed8

Text Generation • Updated Jan 11 • 1.72k
EleutherAI/pythia-70m-seed9

Text Generation • Updated Jan 11 • 1.59k
EleutherAI/pythia-160m

Text Generation • 0.2B • Updated Jul 9, 2023 • 2.29M • 44
EleutherAI/pythia-160m-seed1

Text Generation • 0.2B • Updated Jan 11 • 21.8k
EleutherAI/pythia-160m-seed2

Text Generation • 0.2B • Updated Jan 11 • 21.4k
EleutherAI/pythia-160m-seed3

Text Generation • 0.2B • Updated Jan 11 • 3.53k
EleutherAI/pythia-160m-seed4

Text Generation • Updated Jan 11 • 1.99k • 1
EleutherAI/pythia-160m-seed5

Text Generation • Updated Jan 11 • 2.12k
EleutherAI/pythia-160m-seed6

Text Generation • Updated Jan 11 • 1.55k
EleutherAI/pythia-160m-seed7

Text Generation • Updated Jan 11 • 1.68k
EleutherAI/pythia-160m-seed8

Text Generation • Updated Jan 11 • 1.56k
EleutherAI/pythia-160m-seed9

Text Generation • Updated Jan 11 • 1.67k
EleutherAI/pythia-410m

Text Generation • 0.5B • Updated Jul 9, 2023 • 309k • 39
EleutherAI/pythia-410m-seed1

Text Generation • Updated Jan 11 • 6.42k
EleutherAI/pythia-410m-seed2

Text Generation • Updated Jan 11 • 6.39k
EleutherAI/pythia-410m-seed3

Text Generation • Updated Jan 11 • 2.17k
EleutherAI/pythia-410m-seed4

Text Generation • Updated Jan 11 • 1.8k
EleutherAI/pythia-410m-seed5

Text Generation • Updated Jan 11 • 1.87k
EleutherAI/pythia-410m-seed6

Text Generation • Updated Jan 11 • 1.71k • 1
EleutherAI/pythia-410m-seed7

Text Generation • Updated Jan 11 • 1.65k
EleutherAI/pythia-410m-seed8

Text Generation • Updated Jan 11 • 1.56k
EleutherAI/pythia-410m-seed9

Text Generation • Updated Jan 11 • 1.62k
EleutherAI/pythia-160m-data-seed1

Text Generation • Updated Jan 11 • 128k

Note Version where the data order and weight initialization seeds are decoupled. Here, only the data seed is different from pythia-160m ("seed 0").
EleutherAI/pythia-160m-data-seed2

Text Generation • Updated Jan 11 • 54.9k

Note Version where the data order and weight initialization seeds are decoupled. Here, only the data seed is different from pythia-160m ("seed 0").
EleutherAI/pythia-160m-data-seed3

Text Generation • Updated Jan 11 • 54.6k

Note Version where the data order and weight initialization seeds are decoupled. Here, only the data seed is different from pythia-160m ("seed 0").
EleutherAI/pythia-160m-weight-seed1

Text Generation • Updated Jan 11 • 65k

Note Version where the data order and weight initialization seeds are decoupled. Here, only the weight initialization seed is different from pythia-160m ("seed 0").
EleutherAI/pythia-160m-weight-seed2

Text Generation • Updated Jan 11 • 60.5k

Note Version where the data order and weight initialization seeds are decoupled. Here, only the weight initialization seed is different from pythia-160m ("seed 0").
EleutherAI/pythia-160m-weight-seed3

Text Generation • Updated Jan 11 • 54.1k

Note Version where the data order and weight initialization seeds are decoupled. Here, only the weight initialization seed is different from pythia-160m ("seed 0").
PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs

Paper • 2503.09543 • Published Mar 12, 2025