| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen2.5-VL-7B-Instruct |
| pipeline_tag: image-text-to-text |
| library_name: transformers |
| language: |
| - en |
| datasets: |
| - Pengyu965/StructHub |
| tags: |
| - svg |
| - xml |
| - image-to-svg |
| - structured-images |
| - diagrams |
| - vector-graphics |
| - qwen2.5-vl |
| - grpo |
| - reinforcement-learning |
| - multimodal |
| --- |
| |
| # Back2Struct: Making Structured Images Editable Again |
|
|
| *Recovering editable, object-level SVG / XML code from images of structured graphics.* |
|
|
| - π Paper: [arXiv:2609.37016](https://arxiv.org/abs/2609.37016) β *Back2Struct: Making Structured Images Editable Again* |
| - π Project page: https://pengyu965.github.io/Back2Struct.github.io/ |
| - π€ Model: [`Pengyu965/Back2Struct-Image2SVG-7B`](https://huggingface.co/Pengyu965/Back2Struct-Image2SVG-7B) |
| - π Dataset: [`Pengyu965/StructHub`](https://huggingface.co/datasets/Pengyu965/StructHub) |
|
|
| Back2Struct "makes structured images editable again" by directly recovering vector |
| graphics code (SVG / XML) from image representations. Given an image of a structured |
| graphic β a diagram, chart, flowchart, architecture figure, UML diagram, or schema β |
| it predicts semantically **object-level** SVG/XML that explicitly encodes text, shapes, |
| topology, and layout (not low-level pixel tracing), so the code imports straight into |
| tools like PowerPoint to edit, restyle, and reuse while preserving structure. Training |
| is supervised fine-tuning from `Qwen2.5-VL-7B-Instruct`, followed by GRPO with a |
| composite reward (syntactic validity, length fidelity, perceptual quality). |
|
|
| This project ships two artifacts β a **model** and a **dataset** β documented together |
| in this shared card. |
|
|
| ## π€ Model β `Back2Struct-Image2SVG-7B` |
|
|
| | | | |
| |---|---| |
| | **Base** | `Qwen/Qwen2.5-VL-7B-Instruct` (7B) | |
| | **Stage 1 β SFT** | Supervised fine-tuning on StructHub imageβSVG pairs | |
| | **Stage 2 β GRPO** | Group-Relative Policy Optimization (TRL) | |
| | **Reward** | Composite, render-gated: syntactic validity (must render) Γ perceptual fidelity (baseline-subtracted DINOv2 similarity between the rendered prediction and the target), encouraging compilability, length consistency, and structural/semantic faithfulness | |
| | **Max output** | 8,192 tokens Β· LoRA (r=64, Ξ±=128), merged into the released weights | |
|
|
| ## π Dataset β `StructHub` |
|
|
| Structured images paired with their editable SVG/XML source code. |
|
|
| | Split | Description | # examples | |
| |-------|-------------|-----------:| |
| | `train` | Full training set (all sources, all token lengths), benchmark held out | 86,102 | |
| | `benchmark` | Held-out evaluation benchmark (starvector 569 + crawled 334 + nn_diagram 97) | 1,000 | |
| |
| Fields: `id` (string), `source` (string), `token_count` (int32), `image` (embedded PNG), |
| `svg` (ground-truth SVG/XML). The benchmark is stratified by SVG length into easy |
| (β€2,048), medium (2,048β4,096), and hard (>4,096) tiers (333/333/334). |
|
|
| ## Results |
|
|
| **StructHub benchmark, render-gated** (a non-rendering prediction scores 0 on DINO/SSIM/GPT, 1 on LPIPS). SR = render success (%); GPT = 0β100 GPTScore. |
|
|
| | Model | SR β | DINO β | LPIPS β | SSIM β | GPT β | |
| |-------|-----:|-------:|--------:|-------:|------:| |
| | ***Overall*** | | | | | | |
| | Qwen3-VL-30B-A3B | 18.8 | 0.1566 | 0.9067 | 0.1201 | 13.29 | |
| | StarVector-8B | 31.4 | 0.2694 | 0.7936 | 0.2170 | 21.05 | |
| | Qwen2.5-VL-7B (base) | 52.3 | 0.3527 | 0.7834 | 0.3276 | 27.56 | |
| | Qwen2.5-VL-7B (SFT) | 42.8 | 0.3715 | 0.7403 | 0.2891 | 33.49 | |
| | **Back2Struct (RL)** | **71.5** | **0.5622** | **0.6473** | **0.4966** | **51.85** | |
| | ***Easy*** | | | | | | |
| | Qwen3-VL-30B-A3B | 26.5 | 0.2231 | 0.8659 | 0.1702 | 19.51 | |
| | StarVector-8B | 46.1 | 0.3923 | 0.7023 | 0.3123 | 31.90 | |
| | Qwen2.5-VL-7B (base) | 69.3 | 0.4776 | 0.7123 | 0.4196 | 39.66 | |
| | Qwen2.5-VL-7B (SFT) | 56.0 | 0.4866 | 0.6630 | 0.3789 | 44.58 | |
| | **Back2Struct (RL)** | **84.0** | **0.6384** | **0.5956** | **0.5796** | **62.63** | |
| | ***Medium*** | | | | | | |
| | Qwen3-VL-30B-A3B | 19.6 | 0.1631 | 0.9004 | 0.1288 | 13.38 | |
| | StarVector-8B | 37.0 | 0.3232 | 0.7533 | 0.2583 | 25.27 | |
| | Qwen2.5-VL-7B (base) | 51.8 | 0.3546 | 0.7816 | 0.3306 | 26.88 | |
| | Qwen2.5-VL-7B (SFT) | 45.5 | 0.3985 | 0.7161 | 0.3091 | 36.50 | |
| | **Back2Struct (RL)** | **75.6** | **0.6088** | **0.6182** | **0.5275** | **55.14** | |
| | ***Hard*** | | | | | | |
| | Qwen3-VL-30B-A3B | 10.2 | 0.0836 | 0.9536 | 0.0614 | 7.01 | |
| | StarVector-8B | 11.1 | 0.0932 | 0.9249 | 0.0808 | 6.02 | |
| | Qwen2.5-VL-7B (base) | 35.7 | 0.2263 | 0.8561 | 0.2331 | 16.17 | |
| | Qwen2.5-VL-7B (SFT) | 27.0 | 0.2297 | 0.8413 | 0.1795 | 19.43 | |
| | **Back2Struct (RL)** | **55.0** | **0.4399** | **0.7278** | **0.3830** | **37.83** | |
|
|
| **On successfully compiled outputs**, the 7B model matches or beats frontier closed models: |
|
|
| | Model | DINO β | LPIPS β | SSIM β | GPT β | |
| |-------|-------:|--------:|-------:|------:| |
| | Gemini-2.5-Pro | **0.8855** | 0.4648 | 0.6532 | 77.95 | |
| | GPT-5 | 0.8852 | 0.4350 | 0.6573 | 77.21 | |
| | **Back2Struct (7B)** | 0.8697 | **0.3896** | **0.6738** | **78.63** | |
|
|
| ## Usage |
|
|
| ### Run the model |
| ```python |
| import torch |
| from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor |
| from qwen_vl_utils import process_vision_info |
| |
| model_id = "Pengyu965/Back2Struct-Image2SVG-7B" |
| model = Qwen2_5_VLForConditionalGeneration.from_pretrained( |
| model_id, torch_dtype=torch.bfloat16, device_map="auto") |
| processor = AutoProcessor.from_pretrained(model_id) |
| |
| messages = [{"role": "user", "content": [ |
| {"type": "image", "image": "diagram.png"}, |
| {"type": "text", "text": "Convert this structured image into editable SVG/XML code."}, |
| ]}] |
| text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| image_inputs, video_inputs = process_vision_info(messages) |
| inputs = processor(text=[text], images=image_inputs, videos=video_inputs, |
| padding=True, return_tensors="pt").to(model.device) |
| generated = model.generate(**inputs, max_new_tokens=8192, do_sample=False) |
| print(processor.batch_decode(generated[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]) |
| ``` |
|
|
| ### Load the dataset |
| ```python |
| from datasets import load_dataset |
| ds = load_dataset("Pengyu965/StructHub") |
| ex = ds["benchmark"][0] |
| ex["image"].save("input.png"); print(ex["svg"][:500]) |
| ``` |
|
|
| ## Intended use & limitations |
|
|
| - **Intended use:** recovering editable vector code from images of *structured* graphics. |
| - **Not for:** photographic / natural-image vectorization. |
| - **Limitations:** built on a general (non-code-specialized) VLM; outputs capped at 8,192 |
| tokens; very long/dense diagrams (hard tier) remain the main failure mode (render reliability). |
|
|
| ## Licensing |
|
|
| Model weights are released under **Apache-2.0** (inheriting the Qwen2.5-VL-7B-Instruct base). |
| The **StructHub** dataset aggregates multiple sources, including crawled open-domain graphics; |
| it is released for **research use** β review and respect the original content licenses. |
| *(Confirm before wide distribution.)* |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{yan2026back2structmakingstructuredimages, |
| title={Back2Struct: Making Structured Images Editable Again}, |
| author={Pengyu Yan and Yixin Wu and Yunjie Tian and David Doermann}, |
| year={2026}, |
| eprint={2609.37016}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CV}, |
| url={https://arxiv.org/abs/2609.37016}, |
| } |
| ``` |
|
|
