Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,107 @@
+---
+license: apache-2.0
+language:
+  - en
+tags:
+  - chemistry
+  - markush
+  - cxsmiles
+  - molecular-structure
+  - ocr
+  - document-understanding
+  - image-to-text
+  - vision-language-model
+  - patent-analysis
+library_name: transformers
+pipeline_tag: image-to-text
+datasets:
+  - docling-project/MarkushGrapher-2-Datasets
+---
+
+<p align="center">
+  <img src="markushgrapher_2_repo_banner.png" alt="MarkushGrapher-2 Banner" width="100%">
+</p>
+
+# MarkushGrapher-2
+
+**MarkushGrapher-2** is an end-to-end multimodal model for recognizing chemical structures from patent document images. It jointly encodes vision, text, and layout information to convert Markush structure images into machine-readable CXSMILES representations.
+
+## Overview
+
+MarkushGrapher-2 is a transformer-based model that integrates two complementary encoders:
+- A **Vision Encoder** (Swin-B ViT), pretrained for Optical Chemical Structure Recognition (OCSR)
+- A **Vision-Text-Layout (VTL) Encoder** (T5-base), trained for Markush feature extraction
+
+The model also includes **ChemicalOCR**, a dedicated OCR module fine-tuned for chemical images, enabling fully end-to-end processing without external OCR dependencies.
+
+### Architecture
+
+The input image is processed through two parallel pipelines:
+1. **Vision pipeline**: The image is encoded by the OCSR vision encoder and projected via an MLP.
+2. **VTL pipeline**: The image is passed through ChemicalOCR to extract text and bounding boxes, which are then fused with image patches in the VTL encoder.
+
+The outputs of both pipelines are concatenated and fed to a text decoder that autoregressively generates a CXSMILES sequence describing the Markush backbone and a substituent table.
+
+### Two-Stage Training Strategy
+
+- **Phase 1 (Adaptation)**: The Vision Encoder is frozen while the projector and text decoder are trained on 243K real-world image-SMILES pairs for standard molecular structure recognition.
+- **Phase 2 (Fusion)**: The VTL encoder is introduced and trained jointly with the text decoder on 235K synthetic and 145K real-world Markush structure samples for CXSMILES and substituent table prediction.
+
+## Performance
+
+### Markush Structure Recognition (CXSMILES Accuracy)
+
+| Benchmark | MarkushGrapher-2 | MolParser-Base | MolScribe | MarkushGrapher-1 | DeepSeek-OCR | GPT-5 |
+|---|---|---|---|---|---|---|
+| **M2S** (103) | **56** | 39 | 21 | 38 | 0 | 3 |
+| **USPTO-M** (74) | **55** | 30 | 7 | 32 | 0 | - |
+| **WildMol-M** (10K) | **48.0** | 38.1 | 28.1 | - | 1.9 | - |
+| **IP5-M** (1K) | **53.7** | 47.7 | 22.3 | - | 0.0 | - |
+
+### Molecular Structure Recognition (SMILES Accuracy)
+
+| Benchmark | MarkushGrapher-2 | MolParser-Base | MolScribe | MolGrapher |
+|---|---|---|---|---|
+| **WildMol** (10K) | 68.4 | **76.9** | 66.4 | 45.5 |
+| **JPO** (450) | 71.0 | **78.9** | 76.2 | 67.5 |
+| **UOB** (5.7K) | **96.6** | 91.8 | 87.4 | 94.9 |
+| **USPTO** (5.7K) | 89.8 | 93.0 | **93.1** | 91.5 |
+
+## Model Details
+
+- **Parameters**: 831M total (744M trainable)
+- **Vision Encoder**: Swin-B ViT (from MolScribe, frozen during Phase 2)
+- **VTL Encoder/Decoder**: T5-base architecture with UDOP fusion
+- **OCR Module**: ChemicalOCR (256M parameters, based on SmolDocling)
+- **Input**: Patent document image crop (1024x1024)
+- **Output**: CXSMILES sequence + substituent table
+
+## Usage
+
+```python
+# Load the model checkpoint
+from transformers import AutoModel
+
+model = AutoModel.from_pretrained("docling-project/markushgrapher-2-checkpoint")
+```
+
+For full inference pipeline usage, see the [MarkushGrapher-IBM repository](https://github.ibm.com/DeepSearch/MarkushGrapher-IBM).
+
+## Datasets
+
+Training and evaluation datasets are available at [docling-project/MarkushGrapher-2-Datasets](https://huggingface.co/datasets/docling-project/MarkushGrapher-2-Datasets).
+
+## Citation
+
+```bibtex
+@inproceedings{strohmeyer2025markushgrapher2,
+  title={MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures},
+  author={Strohmeyer, Tim},
+  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
+  year={2026}
+}
+```
+
+## License
+
+This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).