IDiom-300M

IDiom is an autoregressive protein language model trained on IDiom-DB, a dataset of 54 million intrinsically disordered protein regions (IDRs) curated from the AlphaFold Database. IDiom generates IDRs with or without flanking protein context using a fill-in-the-middle (FIM) format.

Preprint   |   IDiom GitHub   |   IDiom Models   |   IDiom-DB dataset

Model family

Model	Parameters	Layers	Width	Heads
IDiom-300M	302M	24	1,024	16
IDiom-85M	85M	12	768	12
IDiom-20M	18.9M	6	512	8

All three models use RMSNorm, SwiGLU, QK normalization, RoPE, and tied input/output embeddings. They share a context length of 1,024 input tokens, including special tokens, and a vocabulary of 20 amino acids, three FIM markers, and four control tokens.

Training data

The models were trained on IDiom-DB-v1, a dataset of 54M IDRs curated from the AlphaFold Database.

Usage

Please see the IDiom Github repository for usage information.

Sparse autoencoder

Only IDiom-300M is accompanied by IDiomSAE, a Top-K sparse autoencoder trained on the residual stream of layer-18.

License

Model weights and code: MIT. The IDiom-DB corpus is CC BY 4.0.