--- license: apache-2.0 library_name: transformers pipeline_tag: image-text-to-text base_model: Qwen/Qwen2.5-VL-3B-Instruct tags: - chemistry - multimodal - reasoning - vision-language - grpo - qwen2_5_vl - smiles - iupac - spectroscopy language: - en --- # CheMM-R1 CheMM-R1 is a chemistry-specific multimodal large language model (MLLM) for molecular structure recognition and spectral elucidation. It is built on top of **Qwen2.5-VL-3B-Instruct** and trained with **CheMMGRPO** — a domain-adapted Group Relative Policy Optimisation pipeline that combines a chemistry cold-start SFT stage with reinforcement learning driven by chemistry-specific reward functions. The model is introduced in the paper *"CheMM-R1: Enhancing Chemical Structure Recognition and Elucidation with Reasoning Multimodal Large Language Models"*. - Code & benchmark: https://github.com/liting980713/CheMM-R1 ## Capabilities CheMM-R1 is designed for the following tasks: - **SmilesQA** — predict the SMILES string of a molecule from its 2D structure image. - **IupacQA** — predict the IUPAC name of a molecule from its 2D structure image. - **MwQA** — predict the molecular weight of a molecule from its 2D structure image. - **SpectraQA** — predict the SMILES of a molecule from its spectral images (IR, ¹H-NMR, ¹³C-NMR, positive-ion MS, negative-ion MS). The model produces explicit step-by-step reasoning in `` tags, intermediate SMILES in `` tags, and the final answer in `` tags. ## Method ### Base model - Architecture: `Qwen2_5_VLForConditionalGeneration` (Qwen2.5-VL-3B-Instruct) - Precision: bfloat16 - Context length: up to 128K tokens, trained with max sequence length 20,480 (cold-start) and output length 5,120 (RL) ### Training pipeline (CheMMGRPO) 1. **Cold start (SFT)** — the base model is fine-tuned on 40,000 chemistry QA instances from CheMM-Bench with multimodal Chain-of-Thought reasoning distilled from Gemini-2.5-Pro, to inject organic chemistry knowledge and reasoning patterns. 2. **Reinforcement learning (GRPO)** — the cold-started model is further optimised with GRPO using four chemistry-specific reward functions: - `smiles_acc` — chemical validity of generated SMILES - `atom_acc`, `func_group_acc` — structural accuracy - `output_format` — format compliance with `//` tags - `answer_acc` — factual correctness via Levenshtein-based fuzzy match for textual answers (SMILES, IUPAC) and ±0.05 g/mol tolerance for molecular weights ### Key hyperparameters - Framework: MS-SWIFT - Cold start: lr 1e-4, Adam (β1=0.9, β2=0.95, ε=1e-8), batch size 64, max model length 20,480, max image pixels 262,144 - RL (GRPO): 4 rollouts per question, sampling temperature 1.0, lr 1e-4, KL coefficient β=0.001, batch size 48, max output length 5,120 - Hardware: 8× H200 (141 GB) GPUs, ~12 hours total training ## Dataset: CheMM-Bench CheMM-R1 is trained and evaluated on **CheMM-Bench**, a multimodal chemistry reasoning benchmark with 48,500 long Chain-of-Thought reasoning steps across four tasks: - 26,500 molecules for structure recognition (SmilesQA / IupacQA / MwQA) - 22,000 molecules for structure elucidation (SpectraQA) with five spectral image types (IR, ¹H-NMR, ¹³C-NMR, +ion MS, −ion MS) Molecules are derived from the Alberts spectroscopic dataset (USPTO reaction database). SMILES → 2D structure conversion is performed with RDKit; IUPAC names are sourced from PubChem. ## Results Averaged accuracy / F1 on CheMM-Bench (CheMM-R1 vs. strongest baselines): | Model | SR Avg ACC | SR Avg F1 | SpectraQA ACC | SpectraQA F1 | Overall ACC | Overall F1 | |--------------------|-----------:|----------:|--------------:|-------------:|------------:|-----------:| | GPT-o3 | 5.78 | 10.94 | 1.50 | 2.96 | 3.34 | 6.46 | | Gemini-2.5-Pro | 16.13 | 27.78 | 1.80 | 3.54 | 7.95 | 14.72 | | Claude-Sonnet-4 | 1.99 | 3.91 | 1.60 | 3.15 | 1.77 | 3.48 | | Grok-4 | 2.79 | 5.43 | 4.05 | 7.78 | 3.51 | 6.78 | | Gemini-2.5-Flash | 8.58 | 15.80 | 1.10 | 2.18 | 4.31 | 8.26 | | **CheMM-R1 (3B)** | **23.73** | **38.35** | **36.32** | **53.28** | **30.92** | **47.23** | Tanimoto@1.0 structural match on SmilesQA / SpectraQA: | Model | SmilesQA | SpectraQA | |--------------------|---------:|----------:| | Gemini-2.5-Pro | 44.52 | 11.46 | | Grok-4 | 2.18 | 18.28 | | ChemVLM-8B | 31.99 | — | | **CheMM-R1 (3B)** | **60.00**| **56.57** | See the paper for full tables including BLEU-1 and Levenshtein-distance similarity and ablations on cold-start vs. GRPO vs. CheMMGRPO. ## Usage ```python from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration from PIL import Image import torch model_id = "zzha6204/CheMM-R1" processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True) model = Qwen2_5_VLForConditionalGeneration.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True, ) image = Image.open("molecule.png") messages = [ { "role": "user", "content": [ {"type": "image", "image": image}, {"type": "text", "text": "What is the SMILES representation of this molecule?"}, ], } ] text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device) output_ids = model.generate(**inputs, max_new_tokens=2048) response = processor.batch_decode( output_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=True )[0] print(response) ``` For SpectraQA, pass all available spectral images (IR, ¹H-NMR, ¹³C-NMR, +ion MS, −ion MS) as a multi-image message. ## Output format CheMM-R1 is trained to produce: ``` step-by-step chemical reasoning intermediate SMILES final answer ``` Downstream parsers should extract the content of `` as the final prediction. ## Intended use and limitations - Intended for **research on multimodal chemistry reasoning**: molecular structure recognition and spectral elucidation of small organic molecules. - Molecules are drawn from the USPTO-derived Alberts dataset; out-of-distribution performance on larger natural products, organometallics, or experimentally noisy real-world spectra is not guaranteed. - Outputs — including SMILES, IUPAC names, molecular weights, and reasoning traces — may be incorrect and **must not be used for safety-critical decisions in chemistry or medicinal research** without expert verification. - The model is derived from Qwen2.5-VL-3B-Instruct and inherits its license and any biases of the base model and distilled reasoning data (Gemini-2.5-Pro). ## Citation If you use CheMM-R1 or CheMM-Bench in your research, please cite: ```bibtex @inproceedings{huang2025chemmr1, title = {CheMM-R1: Enhancing Chemical Structure Recognition and Elucidation with Reasoning Multimodal Large Language Models}, author = {Huang, Liting and Zhang, Zhihao and Wang, Shoujin}, year = {2025} } ``` ## Acknowledgements - Base model: [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) - Training framework: [MS-SWIFT](https://github.com/modelscope/ms-swift) - Spectroscopic data: Alberts et al. (USPTO-derived), PubChem, RDKit