| # AstroMLab |
|
|
| AstroMLab is a diverse group of researchers dedicated to advancing the application of Large Language Models (LLMs) in astronomy. Our team includes: |
| - Leading astronomers, astrophysicists, and cosmologists. |
| - Natural language processing experts. |
| - Frontier arXivists from the NASA Astrophysics Data System |
|
|
| ## Objectives |
| - Develop specialized LLMs for astronomy |
| - Create open-source models for advanced research |
| - Facilitate LLM-driven end-to-end agentic research in astronomy |
|
|
| ## Current Work |
|
|
| Our ongoing projects include: |
|
|
| - Curation of an astronomy-based benchmarking dataset |
| - Development of specialized astronomy LLMs |
| - Performance evaluation of models on astronomical tasks |
|
|
| ## Models and Performance |
|
|
| We have developed several models, including AstroSage-LLaMA-3.1-70B ([de Haan et al. 2025b](https://arxiv.org/abs/2505.17592)) AstroSage-LLaMA-3.1-8B ([de Haan et al. 2025a](https://arxiv.org/abs/2411.09012)), AstroLLaMA-2-70B ([Pan et al. 2024](https://arxiv.org/abs/2409.19750)), and AstroLLaMA-3-8B ([Pan et al. 2024](https://arxiv.org/abs/2409.19750)). Our AstroSage-LLaMA-3.1-8B model has demonstrated strong performance in astronomy Q&A tasks ([Ting et al. 2024](https://arxiv.org/abs/2407.11194)): |
|
|
| | Model | Score (%) | |
| |-------|-----------| |
| | **AstroSage-LLaMA-3.1-70B (AstroMLab)** | **86.2** | |
| | Claude-4-Opus | **86.3** | |
| | o3 | 85.4 | |
| | Claude-4-Sonnet | 85.0 | |
| | GPT-4.1 | 84.7 | |
| | o4-Mini | 84.7 | |
| | Gemini-2.5-Pro | 84.8 | |
| | Deepseek-R1 | 84.4 | |
| | Qwen-3-235B | 84.0 | |
| | LLaMA-4-Maverick | 83.4 | |
| | Deepseek-v3-2503 | 82.9 | |
| | Gemini-2.5-Flash-0520 | 82.3 | |
| | LLaMA-4-Scout | 82.2 | |
| | Grok-3 | 81.7 | |
| | Mistral-Medium-v3 | 81.8 | |
| | **AstroSage-LLaMA-3.1-8B (AstroMLab)** | **80.9** | |
| | Mistral-Large-v2 | 80.8 | |
| | Qwen-3-32B | 79.7 | |
| | Mistral-Small-v3.1 | 78.6 | |
| | GPT-4.1-Nano | 78.0 | |
| | Gemini-2-Flash-Lite | 78.4 | |
| | Gemma-3-27B | 76.9 | |
| | Qwen-3-14B | 76.4 | |
| | AstroLLaMA-2-7B | 44.3 | |
|
|
| As of this writing in May 2025, AstroSage-LLaMA-3.1-70B ([de Haan et al. 2025b](https://arxiv.org/abs/2505.17592)) achieves among the highest scores on AstroBench ([Ting et al. 2024](https://arxiv.org/abs/2407.11194)), tying with Claude-4-Opus and outperforming other leading models including GPT-4.1, o3, Gemini-2.5-Pro, and Claude-4-Sonnet. |
|
|
|  |
|
|
|
|
| ## Support and Resources |
|
|
| Our research benefits from: |
| - Access to the Frontier nodes at Oak Ridge Leadership Computing Facility |
| - Support from Microsoft's Accelerating Foundation Models Research (AFMR) program |
|
|
| ## Contact |
|
|
| For inquiries or collaboration opportunities, please contact: astromachinelearninglab@gmail.com |
