---
license: other
tags:
- multimodal
- qwen3-vl
- covt
- segmentation
- depth
- training-bundle
---

# UMM Stage-2 portable training bundle

This public repository is a self-contained handoff for retraining UMM Stage 2: Qwen3-VL CoVT curriculum with SAM, DINOv2, VGGT, PiDiNet and SigLIP frozen-teacher supervision.

The deployed visual-token counts are `[8,4,4,4,4]`; VGGT/depth uses four tokens, aligned with CoVT.

## Current status (2026-08-11)

Stage-2 continuation training is complete at step 9,000. Portable dense bfloat16
models for steps 7,000 and 9,000 are included at:

- `models/ckpts/stage2_covt_from3k_bcd_2k/merged/checkpoint-7000-merged`;
- `models/ckpts/stage2_covt_from3k_bcd_2k/merged/checkpoint-9000-merged`.

Both models passed the 767-tensor dense validation, exact 54-expert comparison,
exact LoRA merge probe and finite-value scan. Step 7,000 was evaluated on full
CV-Bench test, labelled BLINK val and V* core test. Step 9,000 is merged and
validated but not yet benchmarked.

Read [`reports/PROGRESS_2026-08-11.md`](reports/PROGRESS_2026-08-11.md) before
continuing work. It contains the exact curriculum, benchmark results, V*
resolution correction, published artifact paths and remaining tasks.

## Download and deploy

```bash
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY
hf download Orangerl/umm --local-dir umm
cd umm
bash scripts/bootstrap.sh
DRY_RUN=1 bash code/umm/stage2_covt/run_train.sh
```

`bootstrap.sh` extracts the relocatable `unvideo` environment, reinstalls the exact bundled editable sources, materializes 756,894 images from tar shards, renders machine-local paths and runs the Stage-2 validation suite.

Reserve at least 650 GB of free disk space: the downloaded image shards and
their extracted images coexist after deployment. The target should be Linux
x86_64 with an NVIDIA driver compatible with CUDA 12.8.

For Code Agent handoff, use this prompt from the downloaded repository:

```text
Read skills/deploy-umm-stage2/SKILL.md completely and use it to deploy and
validate this repository. Do not start formal training without my explicit
instruction.
```

Provide `SWANLAB_API_KEY` or `secrets/swanlab_key.txt` before real training. Formal training is never started automatically.
The launcher defaults to GPUs `0,1,2,3`; override `CUDA_VISIBLE_DEVICES` for a
different topology.

After a completed run, merge the Stage-2 LoRA/non-LoRA output with:

```bash
MODEL_PATH="$PWD/outputs/stage2_covt_sam_vggt_covt4_v2" \
  bash code/umm/stage2_covt/run_merge.sh
```

For an exact saved checkpoint, always set both source and destination:

```bash
MODEL_PATH="$PWD/models/ckpts/stage2_covt_from3k_bcd_2k/checkpoint-9000" \
SAVE_MODEL_PATH="$PWD/models/ckpts/stage2_covt_from3k_bcd_2k/merged/checkpoint-9000-merged" \
DTYPE=bfloat16 DEVICE_MAP=cpu \
  bash code/umm/stage2_covt/run_merge.sh
```

## Included artifacts

- portable UMM code and DeepSpeed config;
- exact customized Transformers and Diffusers sources;
- Qwen3-VL-4B-Instruct base model;
- Stage-1 `checkpoint-32000.bin` connector/embedding initialization;
- SAM ViT-H, VGGT-1B, DINOv2 ViT-L/14, PiDiNet and SigLIP2-Large teacher assets;
- exact local CoVT dataset: 889,489 conversations and 756,894 images;
- relocatable packed `unvideo` environment;
- `$deploy-umm-stage2` Code Agent skill;
- validated merged Stage-2 checkpoints at steps 7,000 and 9,000;
- full evaluation metrics, audits, selected logs and progress reports under `reports/`;
- SHA256 integrity manifest and provenance metadata.

FLUX weights and the 560 GB Stage-1 DeepSpeed optimizer history are not included because the active Stage-2 code does not load them. Stage 2 consumes only the included 1.06 GB Stage-1 connector file.

## Licensing

No unified relicensing is asserted. Each bundled model, dataset subset and source tree remains subject to its upstream license. In particular, the original `facebook/VGGT-1B` checkpoint has non-commercial restrictions; replace it with the commercial checkpoint before commercial use.