README.md

---
language:
- en
license: mit
library_name: transformers
tags:
- materials-science
- crystallography
- generative-ai
- inverse-design
- chemistry
- unconditional
datasets:
- c-bone/lematerial_clean
pipeline_tag: text-generation
---

# Model Card for CrystaLLM-pi_base

## Model Details

### Model Description

**CrystaLLM-pi_base** is an unconditional generative model designed for the generation of valid inorganic crystal structures. It serves as the foundational pre-trained model for the `CrystaLLM-pi` framework. Based on a GPT-2 decoder-only architecture, it is trained on a large corpus of Crystallographic Information Files (CIFs) to learn the syntax, symmetry, and chemical rules governing crystalline matter.

This model does not accept property conditioning vectors. It generates structures based on text prompts (e.g., chemical composition or space group) or unconditionally (ab-initio generation).

- **Developed by:** Bone et al. (University College London)
- **Model type:** Autoregressive Transformer (GPT-2)
- **Language(s):** CIF (Crystallographic Information File) syntax
- **License:** MIT

### Model Sources

- **Repository:** [GitHub: CrystaLLM-pi](https://github.com/C-Bone-UCL/CrystaLLM-pi)
- **Paper:** [Discovery and recovery of crystalline materials with property-conditioned transformers (arXiv:2511.21299)](https://arxiv.org/abs/2511.21299)
- **Dataset:** [HuggingFace: c-bone/lematerial_clean](https://huggingface.co/datasets/c-bone/lematerial_clean)

## Uses

### Direct Use

The model is intended for:
1. **Unconditional Generation:** Exploring the general chemical space of stable crystals.
2. **Composition/Space Group Completion:** Generating valid structures given a partial prompt (e.g., a chemical formula).
3. **Fine-tuning base:** Serving as the pre-trained initialization for property-conditional models (like `CrystaLLM-pi_bandgap` or `CrystaLLM-pi_density`).

### Out-of-Scope Use

- **Property Conditioning:** This model cannot be steered by properties like band gap or density. Use the specific fine-tuned variants for those tasks.
- **Large Unit Cells:** Context window limit of 1024 tokens (~20 atoms/cell).

## Bias, Risks, and Limitations

- **Training Distribution:** The model reflects the biases present in the LeMaterial dataset. It is most effective at generating structures similar to known stable inorganic compounds.
- **Validity:** While it learns CIF syntax robustly, it may still generate physically invalid structures (e.g., overlapping atoms) or chemically unstable compositions.

## How to Get Started with the Model

For instructions on how to load and run generation with this model, please refer to the `_load_and_generate.py` script in the [CrystaLLM-pi GitHub Repository](https://github.com/C-Bone-UCL/CrystaLLM-pi).

## Training Details

### Training Data

The model was pre-trained on the **LeMaterial** dataset (specifically `c-bone/lematerial_clean`), a large-scale collection of ~4.35 million augmented CIFs derived from major materials databases.

- **Source:** LeMaterial (via `c-bone/lematerial_clean`)
- **Preprocessing:** CIFs are deduplicated, augmented (with symmetry operations), and tokenized.

### Training Procedure

- **Architecture:** GPT-2 Small (~25.9M parameters).
- **Objective:** Causal Language Modeling (Next-token prediction).
- **Loss Function:** Cross-entropy with specific weighting for fixed syntax tokens to accelerate learning of the CIF format.

## Evaluation

### Metrics

The model is evaluated based on:
1. **Validity:** The rate at which generated sequences can be parsed as valid CIF files.
2. **Structural Consistency:** Adherence to space group symmetry and reasonable bond lengths.

### Results

The base model achieves high validity rates and effectively learns to generate chemically plausible structures, serving as a robust foundation for downstream property-conditioning tasks.

## Citation

```bibtex
@misc{bone2025discoveryrecoverycrystallinematerials,
      title={Discovery and recovery of crystalline materials with property-conditioned transformers}, 
      author={Cyprien Bone and Matthew Walker and Kuangdai Leng and Luis M. Antunes and Ricardo Grau-Crespo and Amil Aligayev and Javier Dominguez and Keith T. Butler},
      year={2025},
      eprint={2511.21299},
      archivePrefix={arXiv},
      primaryClass={cond-mat.mtrl-sci},
      url={[https://arxiv.org/abs/2511.21299](https://arxiv.org/abs/2511.21299)}, 
}