--- license: apache-2.0 base_model: - Qwen/Qwen2.5-VL-7B-Instruct - Qwen/Qwen3-VL-4B-Instruct library_name: transformers pipeline_tag: image-text-to-text tags: - qwen2_5_vl - qwen3_vl - vision-language - visual-token-pruning - visionzip - scopd --- # SCOPD / SCOPD+ Checkpoints LoRA adapters for Qwen2.5-VL-7B-Instruct and Qwen3-VL-4B-Instruct with visual-token-pruned inference. This repository provides **adapter weights only** and evaluation instructions. The pinned base model is required separately. No full/merged model weights, teacher weights, optimizer state, or training data are included. ## Model Variants | Model | Objective | Response-token selection | Weights | |---|---|---|---| | SCOPD | forward KL: KL(teacher \|\| student) | all response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd) | | SCOPD+ | forward KL: KL(teacher \|\| student) | top 10% budget-JSD response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd-plus) | | SCOPD+ Top20 | forward KL: KL(teacher \|\| student) | top 20% budget-JSD response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd-plus-top20) | | SCOPD+ Top40 | forward KL: KL(teacher \|\| student) | top 40% budget-JSD response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd-plus-top40) | | SCOPD+ Top60 | forward KL: KL(teacher \|\| student) | top 60% budget-JSD response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd-plus-top60) | | SFT | supervised fine-tuning | supervised response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/sft) | | EPIC | official EPIC configuration | EPIC response supervision | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/epic) | | GRPO | group-relative policy optimization | reward-based policy objective | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/grpo) | | SCOPD Off-Policy | forward KL on teacher-generated responses | all response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd-offpolicy) | | SCOPD+ Reverse KL | reverse KL: KL(student \|\| teacher) | top 10% budget-JSD response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd-plus-reverse-kl) | | SCOPD+ JSD | symmetric JSD(teacher, student) | top 10% budget-JSD response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/scopd-plus-jsd) | | Qwen3-VL-4B SCOPD | forward KL: KL(teacher \|\| student) | all response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/qwen3-vl-4b-scopd) | | Qwen3-VL-4B SCOPD+ | forward KL: KL(teacher \|\| student) | top 10% budget-JSD response tokens | [Download](https://huggingface.co/enmingzhangzz/SCOPD/tree/main/models/qwen3-vl-4b-scopd-plus) | Each subfolder contains `adapter_model.safetensors`, `adapter_config.json`, and provenance. Qwen2.5 adapters are approximately **154 MiB** each; Qwen3 adapters are approximately **126 MiB** each. All 13 adapters total approximately **1.90 GiB**. Their tensors retain the saved FP32 precision. Inference uses a BF16 base model. All adapters use rank 16, alpha 32, zero dropout, and language-decoder attention/MLP projections only. **Top10/20/40/60 refer to selected response tokens, not visual retention.** The SCOPD+ variants use a native VisionZip intervention from 10% to 11% visual retention to select tokens. The released checkpoints support evaluation at 10%, 20%, 30%, and no pruning using the same runner. The Qwen2.5 base revision is `cc594898137f460bfe9f0759e9844b3ce807cfb5`. The Qwen3 base revision is `ebb281ec70b05090aa6165b016eac8ec08e71b17`. Qwen3 adapters must not be loaded onto Qwen2.5 models or vice versa. The JSD and reverse-KL variants here are **LoRA adapters**, not full-parameter fine-tuning variants. The GRPO checkpoint is the final policy at 2560 optimizer updates after 10240 processed questions; these units are not interchangeable. See [model identifiers](models.json) and each model's `provenance.json`. ## Download The collection does not have a single root model configuration. Download the adapter variant and evaluation code: ```bash python -m pip install 'huggingface-hub==0.34.3' export RELEASE_DIR="$PWD/scopd-checkpoints" export VARIANT=scopd-plus export RELEASE_REVISION=main python - <<'PY' import os from huggingface_hub import HfApi, snapshot_download repo = "enmingzhangzz/SCOPD" revision = HfApi().model_info(repo, revision=os.environ["RELEASE_REVISION"]).sha snapshot_download(repo, revision=revision, local_dir=os.environ["RELEASE_DIR"], allow_patterns=[f'models/{os.environ["VARIANT"]}/*', 'evaluation/**', 'models.json', 'README.md', 'LICENSE', 'NOTICE', 'checksums.sha256']) print("Pinned release revision:", revision) PY ``` Record the printed commit hash and use it as `MODEL_REVISION` below. The download can be resumed with the same revision. ## Evaluation Environment Use Python 3.11 and a CUDA-compatible environment. The evaluation code and patched Transformers/VisionZip sources are included under `evaluation/`; a separate code repository is not needed. ```bash cd "$RELEASE_DIR/evaluation" python -m pip install torch==2.9.1 torchvision==0.24.1 \ --index-url https://download.pytorch.org/whl/cu126 python -m pip install -r requirements.txt python -m pip install -r third_party/VLMEvalKit/requirements.txt python -m pip install -r requirements.txt python -m pip install 'flash-attn==2.8.3' --no-build-isolation export RUNTIME=eval # Use qwen3 for qwen3-vl-4b-scopd or qwen3-vl-4b-scopd-plus python runtime/prepare.py --mode "$RUNTIME" source runtime/activate.sh "$RUNTIME" ``` For Qwen2.5, installing an unmodified `transformers==4.57.0` alone is not sufficient for VisionZip: the preparer verifies and applies the evaluation overlays. For Qwen3, it builds an **isolated stock** Transformers 4.57.0 tree; the included Qwen3 VisionZip wrapper handles token selection, contextual merging, DeepStack features and position IDs without Qwen2.5 overlays. Do not mix these runtimes within one Python process. A different Pillow implementation can change image preprocessing: consult [environment details](evaluation/docs/ENVIRONMENT.md) before a numerical reproduction attempt. The listed environment is a source-version specification, not a guarantee of wheel availability on every system. ## Run a Benchmark First download the pinned base model and merge the adapter **locally for evaluation**. This preparation step is not training, and does not upload a merged model. The default CPU merge needs enough host RAM for the base model and merge buffers; use `--device cuda` only on an available GPU with sufficient memory. Allow approximately 16 GiB of disk for Qwen2.5 or 9 GiB for Qwen3. ```bash export MERGED_MODEL="$RELEASE_DIR/local-merged/$VARIANT" python scripts/merge_adapter.py \ --adapter "$RELEASE_DIR/models/$VARIANT" \ --output "$MERGED_MODEL" ``` The helper reads the correct base model and revision from the adapter's provenance automatically. Alternatively, `--base-model` can point to an already downloaded Hub snapshot directory named by that base model's pinned revision. For direct PEFT inference, the same adapter can be loaded onto that base using `PeftModel.from_pretrained`; the benchmark runner below uses a merged model. ```bash export MODEL_REVISION=THE_PRINTED_COMMIT_HASH export EVAL_ROOT="$PWD/results/$VARIANT" python -m scopd_eval.infer \ --model "$MERGED_MODEL" \ --model-revision "$MODEL_REVISION" \ --dataset mmstar --retention 0.10 \ --output-dir "$EVAL_ROOT/mmstar-r010" ``` Use `--retention 0.20` or `0.30` for milder pruning, and `--retention 1` for no pruning. The argument is **retention**, not the backend's `visionzip_ratio` parameter. The runner detects the model family from `config.json`. Greedy inference uses `......` for Qwen2.5 and **`......` for Qwen3**, at most 2048 generated tokens, and image bounds of 1280 to 4096 nominal visual tokens. A nominal token corresponds to 28x28 input pixels for Qwen2.5 and 32x32 for Qwen3; these budgets are not interchangeable pixel counts. For Qwen3, choose `VARIANT=qwen3-vl-4b-scopd` or `VARIANT=qwen3-vl-4b-scopd-plus` in the download command and `RUNTIME=qwen3` in the environment command. All other merge, inference and judging commands are unchanged. The Qwen3 checkpoints are the final student adapters after 10240 processed questions (320 optimizer updates), not teacher/EMA weights. Their original rollout cap was 512; the evaluation cap above is 2048. The included Qwen3 runner supports images, including multi-image questions, but does not support video pruning. Supported image tasks include MME, MMStar, MathVista-MINI, MathVerse-MINI-Vision-Only, MMMU-Pro-4c, HallusionBench, CVBench, LogicVista, BLINK, VisOnlyQA, HRBench4K, MMVP, MME-RealWorld-Lite, RealWorldQA, POPE, and MathVision-MINI. Use `--help` for the dataset keys. Dataset access and licenses are separate; benchmark data is not included. For multiple GPUs, run independent `--shard N --shards K` processes with one GPU each, then merge **all** shards before scoring: ```bash python -m scopd_eval.merge_shards --input-dir "$EVAL_ROOT/mmstar-r010" \ --shards 4 --output "$EVAL_ROOT/mmstar-r010/all_predictions.jsonl" ``` The unsharded command above writes `shard_000/predictions.jsonl` and does not need merging. Resume skips completed examples only when the input/model/config contract matches. Missing or conflicting shards are errors, not partial scores. ## Unified Answer Judging Use protocol `qwen27-visual-gt-mismatch-v1` for **every compared model and task**, including MMStar. Deterministic matches are accepted directly. All nonmatches are sent to **Qwen/Qwen3.6-27B-FP8** with the original question, options, reference answer, candidate final answer, and image(s). The judge must evaluate the candidate, not solve the question to repair it. Start the judge in a **separate environment and GPU process**, without the evaluation Transformers override: ```bash python -m pip install -r requirements-judge.txt export JUDGE_MODEL_PATH=/path/to/pinned/Qwen3.6-27B-FP8 export JUDGE_MODEL_REVISION=PINNED_JUDGE_COMMIT bash scripts/serve_judge.sh ``` From the evaluation environment: ```bash python -m scopd_eval.postprocess \ --input "$EVAL_ROOT/mmstar-r010/shard_000/predictions.jsonl" \ --output-dir "$EVAL_ROOT/mmstar-r010/judged" \ --judge-revision "$JUDGE_MODEL_REVISION" ``` Use the merged input for a sharded run. Preserve raw predictions, judge output, model revisions, and protocol metadata. See the [full scoring policy and prompt location](evaluation/docs/EVALUATION.md). This is a versioned evaluation protocol, not a claim of identity with every benchmark's official evaluator. Scores from other judge protocols must be reprocessed before comparison. No benchmark scores are asserted in this release. ## Validation and Limitations Adapter identities, safetensor structure, trainable module keys, base-model revision, and publication contents are checked before release. Runtime and judge-routing tests are included. The real 27B image-aware judge and a clean-machine installation still need end-to-end validation; mock tests do not establish GPU memory or throughput guarantees. Models may hallucinate and are intended for research evaluation, not high-stakes decisions. ## License The weights are derivatives of [Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) and [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct), whose pinned model cards specify Apache-2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE). Third-party evaluation and pruning implementations retain their individual licenses and source notices.