Add model card

README.md

@@ -0,0 +1,118 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen2.5-VL-7B-Instruct
+pipeline_tag: image-text-to-text
+library_name: transformers
+language:
+- en
+datasets:
+- Pengyu965/StructHub
+tags:
+- svg
+- xml
+- image-to-svg
+- structured-images
+- diagrams
+- vector-graphics
+- qwen2.5-vl
+- grpo
+- reinforcement-learning
+- multimodal
+---
+
+# Back2Struct-Image2SVG-7B
+
+**Back2Struct** recovers **editable, object-level SVG / XML code** from images of structured
+graphics (diagrams, charts, flowcharts, architecture figures, UML, schemas). Instead of
+low-level pixel vectorization, it predicts vector primitives that explicitly encode text,
+shapes, topology, and layout — so the output can be imported into tools like PowerPoint and
+edited, restyled, and reused while preserving structural fidelity.
+
+This checkpoint is the reinforcement-learning model (**GRPO checkpoint-2200**), fine-tuned
+from `Qwen/Qwen2.5-VL-7B-Instruct`.
+
+- 📄 Paper: *Back2Struct: Making Structured Images Editable Again*
+- 🔗 Project page: https://pengyu965.github.io/Back2Struct.github.io/
+- 📚 Dataset: [`Pengyu965/StructHub`](https://huggingface.co/datasets/Pengyu965/StructHub)
+
+## Training
+
+| | |
+|---|---|
+| **Base model** | Qwen/Qwen2.5-VL-7B-Instruct (7B) |
+| **Stage 1 — SFT** | Supervised fine-tuning on StructHub image→SVG pairs (≤ 8,192 tokens) |
+| **Stage 2 — GRPO** | Group-Relative Policy Optimization (TRL), checkpoint **2200** |
+| **Reward** | Composite, render-gated: syntactic validity (must render) × perceptual fidelity — a baseline-subtracted DINOv2 similarity between the rendered prediction and the target image; encourages compilability, length consistency, and structural/semantic faithfulness |
+| **Max output length** | 8,192 tokens |
+| **Adapter** | LoRA (r=64, α=128), merged into the released weights |
+
+## Results
+
+**StructHub benchmark (1,000 samples), render-gated evaluation** — a non-rendering prediction
+scores 0 on DINO/SSIM/GPT and 1 on LPIPS. SR = render success (%); GPT = 0–100 GPTScore.
+
+| Model | SR ↑ | DINO ↑ | LPIPS ↓ | SSIM ↑ | GPT ↑ |
+|-------|-----:|-------:|--------:|-------:|------:|
+| Qwen3-VL-30B-A3B | 18.8 | 0.1566 | 0.9067 | 0.1201 | 13.29 |
+| StarVector-8B | 31.4 | 0.2694 | 0.7936 | 0.2170 | 21.05 |
+| Qwen2.5-VL-7B (base) | 52.3 | 0.3527 | 0.7834 | 0.3276 | 27.56 |
+| Qwen2.5-VL-7B (SFT) | 42.8 | 0.3715 | 0.7403 | 0.2891 | 33.49 |
+| **Back2Struct (RL, this model)** | **71.5** | **0.5622** | **0.6473** | **0.4966** | **51.85** |
+
+**On successfully compiled outputs**, the 7B model matches or beats frontier closed models:
+
+| Model | DINO ↑ | LPIPS ↓ | SSIM ↑ | GPT ↑ |
+|-------|-------:|--------:|-------:|------:|
+| Gemini-2.5-Pro | **0.8855** | 0.4648 | 0.6532 | 77.95 |
+| GPT-5 | 0.8852 | 0.4350 | 0.6573 | 77.21 |
+| **Back2Struct (7B)** | 0.8697 | **0.3896** | **0.6738** | **78.63** |
+
+## Usage
+
+```python
+import torch
+from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
+from qwen_vl_utils import process_vision_info
+
+model_id = "Pengyu965/Back2Struct-Image2SVG-7B"
+model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
+    model_id, torch_dtype=torch.bfloat16, device_map="auto")
+processor = AutoProcessor.from_pretrained(model_id)
+
+messages = [{
+    "role": "user",
+    "content": [
+        {"type": "image", "image": "diagram.png"},
+        {"type": "text",  "text": "Convert this structured image into editable SVG/XML code."},
+    ],
+}]
+
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+image_inputs, video_inputs = process_vision_info(messages)
+inputs = processor(text=[text], images=image_inputs, videos=video_inputs,
+                   padding=True, return_tensors="pt").to(model.device)
+
+generated = model.generate(**inputs, max_new_tokens=8192, do_sample=False)
+svg = processor.batch_decode(
+    generated[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]
+print(svg)  # render with cairosvg or import into PowerPoint
+```
+
+## Intended use & limitations
+
+- **Intended use:** recovering editable vector code from images of *structured* graphics.
+- **Not for:** photographic / natural-image vectorization.
+- **Limitations:** built on a general (non-code-specialized) VLM backbone; outputs are capped
+  at 8,192 tokens, which bounds the complexity of very large diagrams; very long / dense
+  graphics (the *hard* tier) remain the main failure mode (render reliability).
+
+## Citation
+
+```bibtex
+@article{yan2026back2struct,
+  title   = {Back2Struct: Making Structured Images Editable Again},
+  author  = {Yan, Pengyu and Wu, Yixin and Tian, Yunjie and Doermann, David},
+  year    = {2026},
+  note    = {Preprint}
+}
+```