Reproduce the local MMEB evaluation
This is a snapshot of the local Nexus MMEB evaluator on 2026-10-15, packaged with tencent/WeMM-Embedding-9B weights. These are the upstream released models, not local training checkpoints.
Setup and execution
Use Python 3.10 and a CUDA-capable PyTorch environment. Install dependencies:
python -m pip install -r evaluation/requirements.txt
DATA_BASEDIR=/path/to/vlm2vec_eval NPROC=8 bash evaluation/run_eval.sh image_retrieval
DATA_BASEDIR=/path/to/vlm2vec_eval NPROC=8 bash evaluation/run_eval.sh video_ret_5
DATA_BASEDIR=/path/to/vlm2vec_eval NPROC=8 bash evaluation/run_eval.sh moment_retrieval
DATA_BASEDIR=/path/to/vlm2vec_eval NPROC=8 bash evaluation/run_eval.sh visdoc_full
Each process loads a full model on its GPU; NPROC is data parallelism, not model sharding. Reduce BATCH_SIZE if memory is insufficient. One GPU is the default. MODEL_PATH, OUTPUT_DIR, GENERATED_MEDIA_BASEDIR, MASTER_PORT, BATCH_SIZE and FORCE_RECOMPUTE may be overridden.
The launcher sets NEXUS_WEMM_OFFICIAL=1 before importing Nexus. WeMM uses its released custom model, user-only prompt, processor settings and embedding() path. Qwen uses the Nexus Qwen-style conversation and last-token pooling. Both normalize embeddings. Task instructions, candidate selection, local/global evaluation, video sampling and metrics are determined by the included dataset parsers and YAML files; preserve them when comparing results. The video configuration uses 64 frames. The evaluator writes *_score.json files into OUTPUT_DIR.
Data requirements
Benchmark media are not included in this model package. DATA_BASEDIR must contain image-tasks/, video-tasks/ and visdoc-tasks/ as required by the YAMLs. The dataset parsers download annotations/corpora from their configured Hugging Face sources; see nexus/Nexus/evaluation/mmeb_v2/constant.py and data/datasets/. Visual-document page images can be generated under GENERATED_MEDIA_BASEDIR, which must be writable. Internet access or an existing dataset cache is required.
Runtime snapshot
The original runtime uses transformers 5.6.0.dev0, installed from a development source archive without a recorded Git commit. To avoid depending on a moving main branch, vendor/ contains the installed Transformers source and distribution metadata (excluding bytecode and install records, with an upstream test token redacted in testing_utils.py). The launcher prepends this runtime snapshot to PYTHONPATH. Other installed dependency versions are recorded in requirements.txt. The upstream model's recommended environment may differ; see UPSTREAM_README.md. Vendored source retains its upstream licensing.
Scope of verification
Packaging checks include weight shard references, JSON/YAML parsing, source compilation, evaluator CLI imports, and file hashes. No new GPU benchmark run is performed during packaging. This package does not claim to reproduce any particular score without the original benchmark data and runtime hardware.