Image-Text-to-Text
Transformers
Safetensors
qwen2_5_vl
qwen3_vl
vision-language
visual-token-pruning
visionzip
scopd
Instructions to use enmingzhangzz/SCOPD with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use enmingzhangzz/SCOPD with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="enmingzhangzz/SCOPD")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("enmingzhangzz/SCOPD", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use enmingzhangzz/SCOPD with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "enmingzhangzz/SCOPD" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "enmingzhangzz/SCOPD", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/enmingzhangzz/SCOPD
- SGLang
How to use enmingzhangzz/SCOPD with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "enmingzhangzz/SCOPD" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "enmingzhangzz/SCOPD", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "enmingzhangzz/SCOPD" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "enmingzhangzz/SCOPD", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use enmingzhangzz/SCOPD with Docker Model Runner:
docker model run hf.co/enmingzhangzz/SCOPD
|
Download README.md from enmingzhangzz/SCOPD: direct link, hf CLI and curl.
- Browser
- Download file 12 kB
-
https://huggingface.co/enmingzhangzz/SCOPD/resolve/main/README.md
- Command line
-
hf download hf://enmingzhangzz/SCOPD/README.md
-
curl -L -o README.md https://huggingface.co/enmingzhangzz/SCOPD/resolve/main/README.md
12 kB
| license: apache-2.0 | |
| base_model: | |
| - Qwen/Qwen2.5-VL-7B-Instruct | |
| - Qwen/Qwen3-VL-4B-Instruct | |
| library_name: transformers | |
| pipeline_tag: image-text-to-text | |
| tags: | |
| - qwen2_5_vl | |
| - qwen3_vl | |
| - vision-language | |
| - visual-token-pruning | |
| - visionzip | |
| - scopd | |
| # SCOPD / SCOPD+ Checkpoints | |
| LoRA adapters for Qwen2.5-VL-7B-Instruct and Qwen3-VL-4B-Instruct with visual-token-pruned inference. | |
| This repository provides **adapter weights only** and evaluation instructions. | |
| The pinned base model is required separately. No full/merged model weights, | |
| teacher weights, optimizer state, or training data are included. | |
| ## Model Variants | |
| | Model | Objective | Response-token selection | Weights | | |
| |---|---|---|---| | |
| | SCOPD | forward KL: KL(teacher \|\| student) | all response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd) | | |
| | SCOPD+ | forward KL: KL(teacher \|\| student) | top 10% budget-JSD response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd-plus) | | |
| | SCOPD+ Top20 | forward KL: KL(teacher \|\| student) | top 20% budget-JSD response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd-plus-top20) | | |
| | SCOPD+ Top40 | forward KL: KL(teacher \|\| student) | top 40% budget-JSD response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd-plus-top40) | | |
| | SCOPD+ Top60 | forward KL: KL(teacher \|\| student) | top 60% budget-JSD response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd-plus-top60) | | |
| | SFT | supervised fine-tuning | supervised response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/sft) | | |
| | EPIC | official EPIC configuration | EPIC response supervision | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/epic) | | |
| | GRPO | group-relative policy optimization | reward-based policy objective | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/grpo) | | |
| | SCOPD Off-Policy | forward KL on teacher-generated responses | all response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd-offpolicy) | | |
| | SCOPD+ Reverse KL | reverse KL: KL(student \|\| teacher) | top 10% budget-JSD response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd-plus-reverse-kl) | | |
| | SCOPD+ JSD | symmetric JSD(teacher, student) | top 10% budget-JSD response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd-plus-jsd) | | |
| | Qwen3-VL-4B SCOPD | forward KL: KL(teacher \|\| student) | all response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/qwen3-vl-4b-scopd) | | |
| | Qwen3-VL-4B SCOPD+ | forward KL: KL(teacher \|\| student) | top 10% budget-JSD response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/qwen3-vl-4b-scopd-plus) | | |
| Each subfolder contains `adapter_model.safetensors`, `adapter_config.json`, | |
| and provenance. Qwen2.5 adapters are approximately **154 MiB** each; Qwen3 | |
| adapters are approximately **126 MiB** each. All 13 adapters total approximately | |
| **1.90 GiB**. Their tensors retain the saved FP32 precision. | |
| Inference uses a BF16 base model. All adapters use rank 16, alpha 32, | |
| zero dropout, and language-decoder attention/MLP projections only. | |
| **Top10/20/40/60 refer to selected response tokens, not visual retention.** | |
| The SCOPD+ variants use a native VisionZip intervention from 10% to 11% | |
| visual retention to select tokens. The released checkpoints support evaluation | |
| at 10%, 20%, 30%, and no pruning using the same runner. | |
| The Qwen2.5 base revision is `cc594898137f460bfe9f0759e9844b3ce807cfb5`. | |
| The Qwen3 base revision is `ebb281ec70b05090aa6165b016eac8ec08e71b17`. | |
| Qwen3 adapters must not be loaded onto Qwen2.5 models or vice versa. | |
| The JSD and reverse-KL variants here are **LoRA adapters**, not full-parameter | |
| fine-tuning variants. The GRPO checkpoint is the final policy at 2560 optimizer | |
| updates after 10240 processed questions; these units are not interchangeable. | |
| See [model identifiers](models.json) and each model's `provenance.json`. | |
| ## Download | |
| The collection does not have a single root model configuration. | |
| Download the adapter variant and evaluation code: | |
| ```bash | |
| python -m pip install 'huggingface-hub==0.34.3' | |
| export RELEASE_DIR="$PWD/scopd-checkpoints" | |
| export VARIANT=scopd-plus | |
| export RELEASE_REVISION=main | |
| python - <<'PY' | |
| import os | |
| from huggingface_hub import HfApi, snapshot_download | |
| repo = "enmingzhangzz/SCOPD" | |
| revision = HfApi().model_info(repo, revision=os.environ["RELEASE_REVISION"]).sha | |
| snapshot_download(repo, revision=revision, local_dir=os.environ["RELEASE_DIR"], | |
| allow_patterns=[f'models/{os.environ["VARIANT"]}/*', 'evaluation/**', | |
| 'models.json', 'README.md', 'LICENSE', 'NOTICE', 'checksums.sha256']) | |
| print("Pinned release revision:", revision) | |
| PY | |
| ``` | |
| Record the printed commit hash and use it as `MODEL_REVISION` below. | |
| The download can be resumed with the same revision. | |
| ## Evaluation Environment | |
| Use Python 3.11 and a CUDA-compatible environment. The evaluation code and | |
| patched Transformers/VisionZip sources are included under `evaluation/`; | |
| a separate code repository is not needed. | |
| ```bash | |
| cd "$RELEASE_DIR/evaluation" | |
| python -m pip install torch==2.9.1 torchvision==0.24.1 \ | |
| --index-url https://download.pytorch.org/whl/cu126 | |
| python -m pip install -r requirements.txt | |
| python -m pip install -r third_party/VLMEvalKit/requirements.txt | |
| python -m pip install -r requirements.txt | |
| python -m pip install 'flash-attn==2.8.3' --no-build-isolation | |
| export RUNTIME=eval # Use qwen3 for qwen3-vl-4b-scopd or qwen3-vl-4b-scopd-plus | |
| python runtime/prepare.py --mode "$RUNTIME" | |
| source runtime/activate.sh "$RUNTIME" | |
| ``` | |
| For Qwen2.5, installing an unmodified `transformers==4.57.0` alone is not sufficient | |
| for VisionZip: the preparer verifies and applies the evaluation overlays. | |
| For Qwen3, it builds an **isolated stock** Transformers 4.57.0 tree; the included | |
| Qwen3 VisionZip wrapper handles token selection, contextual merging, DeepStack | |
| features and position IDs without Qwen2.5 overlays. Do not mix these runtimes | |
| within one Python process. A different | |
| Pillow implementation can change image preprocessing: consult | |
| [environment details](evaluation/docs/ENVIRONMENT.md) before a numerical | |
| reproduction attempt. The listed environment is a source-version specification, | |
| not a guarantee of wheel availability on every system. | |
| ## Run a Benchmark | |
| First download the pinned base model and merge the adapter **locally for | |
| evaluation**. This preparation step is not training, and does not upload a | |
| merged model. The default CPU merge needs enough host RAM for the base model | |
| and merge buffers; use `--device cuda` only on an available GPU with sufficient | |
| memory. Allow approximately 16 GiB of disk for Qwen2.5 or 9 GiB for Qwen3. | |
| ```bash | |
| export MERGED_MODEL="$RELEASE_DIR/local-merged/$VARIANT" | |
| python scripts/merge_adapter.py \ | |
| --adapter "$RELEASE_DIR/models/$VARIANT" \ | |
| --output "$MERGED_MODEL" | |
| ``` | |
| The helper reads the correct base model and revision from the adapter's | |
| provenance automatically. Alternatively, `--base-model` can point to an already | |
| downloaded Hub snapshot directory named by that base model's pinned revision. | |
| For direct PEFT inference, the same adapter can be loaded onto that base using | |
| `PeftModel.from_pretrained`; the benchmark runner below uses a merged model. | |
| ```bash | |
| export MODEL_REVISION=THE_PRINTED_COMMIT_HASH | |
| export EVAL_ROOT="$PWD/results/$VARIANT" | |
| python -m scopd_eval.infer \ | |
| --model "$MERGED_MODEL" \ | |
| --model-revision "$MODEL_REVISION" \ | |
| --dataset mmstar --retention 0.10 \ | |
| --output-dir "$EVAL_ROOT/mmstar-r010" | |
| ``` | |
| Use `--retention 0.20` or `0.30` for milder pruning, and `--retention 1` | |
| for no pruning. The argument is **retention**, not the backend's | |
| `visionzip_ratio` parameter. The runner detects the model family from | |
| `config.json`. Greedy inference uses `<think>...</think><answer>...</answer>` | |
| for Qwen2.5 and **`<analysis>...</analysis><answer>...</answer>` for Qwen3**, | |
| at most 2048 generated tokens, and image bounds of 1280 to 4096 nominal visual | |
| tokens. A nominal token corresponds to 28x28 input pixels for Qwen2.5 and | |
| 32x32 for Qwen3; these budgets are not interchangeable pixel counts. | |
| For Qwen3, choose `VARIANT=qwen3-vl-4b-scopd` or | |
| `VARIANT=qwen3-vl-4b-scopd-plus` in the download command and `RUNTIME=qwen3` | |
| in the environment command. All other merge, inference and judging commands | |
| are unchanged. The Qwen3 checkpoints are the final student adapters after | |
| 10240 processed questions (320 optimizer updates), not teacher/EMA weights. | |
| Their original rollout cap was 512; the evaluation cap above is 2048. | |
| The included Qwen3 runner supports images, including multi-image questions, | |
| but does not support video pruning. | |
| Supported image tasks include MME, MMStar, MathVista-MINI, | |
| MathVerse-MINI-Vision-Only, MMMU-Pro-4c, HallusionBench, CVBench, | |
| LogicVista, BLINK, VisOnlyQA, HRBench4K, MMVP, MME-RealWorld-Lite, | |
| RealWorldQA, POPE, and MathVision-MINI. Use `--help` for the dataset keys. | |
| Dataset access and licenses are separate; benchmark data is not included. | |
| For multiple GPUs, run independent `--shard N --shards K` processes with one | |
| GPU each, then merge **all** shards before scoring: | |
| ```bash | |
| python -m scopd_eval.merge_shards --input-dir "$EVAL_ROOT/mmstar-r010" \ | |
| --shards 4 --output "$EVAL_ROOT/mmstar-r010/all_predictions.jsonl" | |
| ``` | |
| The unsharded command above writes `shard_000/predictions.jsonl` and does not | |
| need merging. Resume skips completed examples only when the input/model/config | |
| contract matches. Missing or conflicting shards are errors, not partial scores. | |
| ## Unified Answer Judging | |
| Use protocol `qwen27-visual-gt-mismatch-v1` for **every compared model and task**, | |
| including MMStar. Deterministic matches are accepted directly. All nonmatches | |
| are sent to **Qwen/Qwen3.6-27B-FP8** with the original question, options, | |
| reference answer, candidate final answer, and image(s). The judge must evaluate | |
| the candidate, not solve the question to repair it. | |
| Start the judge in a **separate environment and GPU process**, without the | |
| evaluation Transformers override: | |
| ```bash | |
| python -m pip install -r requirements-judge.txt | |
| export JUDGE_MODEL_PATH=/path/to/pinned/Qwen3.6-27B-FP8 | |
| export JUDGE_MODEL_REVISION=PINNED_JUDGE_COMMIT | |
| bash scripts/serve_judge.sh | |
| ``` | |
| From the evaluation environment: | |
| ```bash | |
| python -m scopd_eval.postprocess \ | |
| --input "$EVAL_ROOT/mmstar-r010/shard_000/predictions.jsonl" \ | |
| --output-dir "$EVAL_ROOT/mmstar-r010/judged" \ | |
| --judge-revision "$JUDGE_MODEL_REVISION" | |
| ``` | |
| Use the merged input for a sharded run. Preserve raw predictions, judge output, | |
| model revisions, and protocol metadata. See the | |
| [full scoring policy and prompt location](evaluation/docs/EVALUATION.md). | |
| This is a versioned evaluation protocol, not a claim of identity with every | |
| benchmark's official evaluator. Scores from other judge protocols must be | |
| reprocessed before comparison. No benchmark scores are asserted in this release. | |
| ## Validation and Limitations | |
| Adapter identities, safetensor structure, trainable module keys, | |
| base-model revision, and publication contents are checked before release. Runtime and judge-routing | |
| tests are included. The real 27B image-aware judge and a clean-machine | |
| installation still need end-to-end validation; mock tests do not establish GPU | |
| memory or throughput guarantees. Models may hallucinate and are intended for | |
| research evaluation, not high-stakes decisions. | |
| ## License | |
| The weights are derivatives of | |
| [Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) and | |
| [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct), | |
| whose pinned model cards specify Apache-2.0. See [LICENSE](LICENSE) and | |
| [NOTICE](NOTICE). Third-party evaluation and pruning implementations retain | |
| their individual licenses and source notices. | |