| --- |
| license: apache-2.0 |
| language: |
| - en |
| base_model: |
| - openmmlab/mask-rcnn |
| - microsoft/swin-base-patch4-window7-224-in22k |
| pipeline_tag: image-segmentation |
| --- |
| |
| # Model Card for ChartPointNet-InstanceSeg |
|
|
| ChartPointNet-InstanceSeg is a high-precision data point instance segmentation model for scientific charts. It uses Mask R-CNN with a Swin Transformer backbone to detect and segment individual data points, especially in dense and small-object scenarios common in scientific figures. |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| ChartPointNet-InstanceSeg is designed for pixel-precise instance segmentation of data points in scientific charts (e.g., scatter plots). It leverages Mask R-CNN with a Swin Transformer backbone, trained on enhanced COCO-style datasets with instance masks for data points. The model is ideal for extracting quantitative data from scientific figures and for downstream chart analysis. |
|
|
| - **Developed by:** Hansheng Zhu |
| - **Model type:** Instance Segmentation |
| - **License:** Apache-2.0 |
| - **Finetuned from model:** openmmlab/mask-rcnn |
|
|
| ### Model Sources |
|
|
| - **Repository:** [https://github.com/hanszhu/ChartSense](https://github.com/hanszhu/ChartSense) |
| - **Paper:** https://arxiv.org/abs/2106.01841 |
|
|
| ## Uses |
|
|
| ### Direct Use |
|
|
| - Instance segmentation of data points in scientific charts |
| - Automated extraction of quantitative data from figures |
| - Preprocessing for downstream chart understanding and data mining |
|
|
| ### Downstream Use |
|
|
| - As a preprocessing step for chart structure parsing or data extraction |
| - Integration into document parsing, digital library, or accessibility systems |
|
|
| ### Out-of-Scope Use |
|
|
| - Segmentation of non-data-point elements |
| - Use on figures outside the supported chart types |
| - Medical or legal decision making |
|
|
| ## Bias, Risks, and Limitations |
|
|
| - The model is limited to data point segmentation in scientific charts. |
| - May not generalize to figures with highly unusual styles or poor image quality. |
| - Potential dataset bias: Training data is sourced from scientific literature. |
|
|
| ### Recommendations |
|
|
| Users should verify predictions on out-of-domain data and be aware of the model’s limitations regarding chart style and domain. |
|
|
| ## How to Get Started with the Model |
|
|
| ```python |
| import torch |
| from mmdet.apis import inference_detector, init_detector |
| |
| config_file = 'legend_match_swin/mask_rcnn_swin_datapoint.py' |
| checkpoint_file = 'chart_datapoint.pth' |
| model = init_detector(config_file, checkpoint_file, device='cuda:0') |
| |
| result = inference_detector(model, 'example_chart.png') |
| # result: list of detected masks and class labels |
| ``` |
|
|
| ## Training Details |
|
|
| ### Training Data |
|
|
| - **Dataset:** Enhanced COCO-style scientific chart dataset with instance masks |
| - Data point class with pixel-precise segmentation masks |
| - Images and annotations filtered and preprocessed for optimal Swin Transformer performance |
|
|
| ### Training Procedure |
|
|
| - Images resized to 1120x672 |
| - Mask R-CNN with Swin Transformer backbone |
| - **Training regime:** fp32 |
| - **Optimizer:** AdamW |
| - **Batch size:** 8 |
| - **Epochs:** 36 |
| - **Learning rate:** 1e-4 |
|
|
| ## Evaluation |
|
|
| ### Testing Data, Factors & Metrics |
|
|
| - **Testing Data:** Held-out split from enhanced COCO-style dataset |
| - **Factors:** Data point density, image quality |
| - **Metrics:** mAP (mean Average Precision), AP50, AP75, per-class AP |
|
|
| ### Results |
|
|
| | Category | mAP | mAP_50 | mAP_75 | mAP_s | mAP_m | mAP_l | |
| |-----------------|-------|--------|--------|-------|-------|-------| |
| | data-point | 0.485 | 0.687 | 0.581 | 0.487 | 0.05 | nan | |
| |
| #### Summary |
| |
| The model achieves strong mAP for data point segmentation, excelling in dense and small-object scenarios. It is highly effective for scientific figures requiring pixel-level accuracy. |
| |
| ## Environmental Impact |
| |
| - **Hardware Type:** NVIDIA V100 GPU |
| - **Hours used:** 10 |
| - **Cloud Provider:** Google Cloud |
| - **Compute Region:** us-central1 |
| - **Carbon Emitted:** ~15 kg CO2eq (estimated) |
| |
| ## Technical Specifications |
| |
| ### Model Architecture and Objective |
| |
| - Mask R-CNN with Swin Transformer backbone |
| - Instance segmentation head for data point class |
| |
| ### Compute Infrastructure |
| |
| - **Hardware:** NVIDIA V100 GPU |
| - **Software:** PyTorch 1.13, MMDetection 2.x, Python 3.9 |
| |
| ## Citation |
| |
| **BibTeX:** |
| |
| ```bibtex |
| @article{DocFigure2021, |
| title={DocFigure: A Dataset for Scientific Figure Classification}, |
| author={S. Afzal, et al.}, |
| journal={arXiv preprint arXiv:2106.01841}, |
| year={2021} |
| } |
| ``` |
| |
| **APA:** |
| |
| Afzal, S., et al. (2021). DocFigure: A Dataset for Scientific Figure Classification. arXiv preprint arXiv:2106.01841. |
| |
| ## Glossary |
| |
| - **Data Point:** An individual visual marker representing a value in a scientific chart (e.g., a dot in a scatter plot) |
| |
| ## More Information |
| |
| - [DocFigure Paper](https://arxiv.org/abs/2106.01841) |
| |
| ## Model Card Authors |
| |
| Hansheng Zhu |
| |
| ## Model Card Contact |
| |
| hanszhu05@gmail.com |
