UMM Stage-2 portable training bundle

This public repository is a self-contained handoff for retraining UMM Stage 2: Qwen3-VL CoVT curriculum with SAM, DINOv2, VGGT, PiDiNet and SigLIP frozen-teacher supervision.

The deployed visual-token counts are [8,4,4,4,4]; VGGT/depth uses four tokens, aligned with CoVT.

Current status (2026-08-11)

Stage-2 continuation training is complete at step 9,000. Portable dense bfloat16 models for steps 7,000, 8,000 and 9,000 are included at:

• models/ckpts/stage2_covt_from3k_bcd_2k/merged/checkpoint-7000-merged;
• models/ckpts/stage2_covt_from3k_bcd_2k/merged/checkpoint-8000-merged;
• models/ckpts/stage2_covt_from3k_bcd_2k/merged/checkpoint-9000-merged.

All three models passed the 767-tensor dense validation, exact 54-expert comparison, exact LoRA merge probe and finite-value scan. Step 7,000 was evaluated on full CV-Bench test, labelled BLINK val and V* core test. Steps 8,000 and 9,000 are merged and validated but not yet benchmarked.

Read reports/PROGRESS_2026-08-11.md before continuing work. It contains the exact curriculum, benchmark results, V* resolution correction, published artifact paths and remaining tasks.

Download and deploy

unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY
hf download Orangerl/umm --local-dir umm
cd umm
bash scripts/bootstrap.sh
DRY_RUN=1 bash code/umm/stage2_covt/run_train.sh

bootstrap.sh extracts the relocatable unvideo environment, reinstalls the exact bundled editable sources, materializes 756,894 images from tar shards, renders machine-local paths and runs the Stage-2 validation suite.

Reserve at least 650 GB of free disk space: the downloaded image shards and their extracted images coexist after deployment. The target should be Linux x86_64 with an NVIDIA driver compatible with CUDA 12.8.

For Code Agent handoff, use this prompt from the downloaded repository:

Read skills/deploy-umm-stage2/SKILL.md completely and use it to deploy and
validate this repository. Do not start formal training without my explicit
instruction.

Provide SWANLAB_API_KEY or secrets/swanlab_key.txt before real training. Formal training is never started automatically. The launcher defaults to GPUs 0,1,2,3; override CUDA_VISIBLE_DEVICES for a different topology.

After a completed run, merge the Stage-2 LoRA/non-LoRA output with:

MODEL_PATH="$PWD/outputs/stage2_covt_sam_vggt_covt4_v2" \
  bash code/umm/stage2_covt/run_merge.sh

For an exact saved checkpoint, always set both source and destination:

MODEL_PATH="$PWD/models/ckpts/stage2_covt_from3k_bcd_2k/checkpoint-9000" \
SAVE_MODEL_PATH="$PWD/models/ckpts/stage2_covt_from3k_bcd_2k/merged/checkpoint-9000-merged" \
DTYPE=bfloat16 DEVICE_MAP=cpu \
  bash code/umm/stage2_covt/run_merge.sh

Included artifacts

• portable UMM code and DeepSpeed config;
• exact customized Transformers and Diffusers sources;
• Qwen3-VL-4B-Instruct base model;
• Stage-1 checkpoint-32000.bin connector/embedding initialization;
• SAM ViT-H, VGGT-1B, DINOv2 ViT-L/14, PiDiNet and SigLIP2-Large teacher assets;
• exact local CoVT dataset: 889,489 conversations and 756,894 images;
• relocatable packed unvideo environment;
• $deploy-umm-stage2 Code Agent skill;
• validated merged Stage-2 checkpoints at steps 7,000, 8,000 and 9,000;
• full evaluation metrics, audits, selected logs and progress reports under reports/;
• SHA256 integrity manifest and provenance metadata.

FLUX weights and the 560 GB Stage-1 DeepSpeed optimizer history are not included because the active Stage-2 code does not load them. Stage 2 consumes only the included 1.06 GB Stage-1 connector file.

Licensing

No unified relicensing is asserted. Each bundled model, dataset subset and source tree remains subject to its upstream license. In particular, the original facebook/VGGT-1B checkpoint has non-commercial restrictions; replace it with the commercial checkpoint before commercial use.