VAR 18ch Multi-view Checkpoints

This repository stores VAR checkpoints for an 18-channel text-conditioned multi-view hand-object generation experiment. Each generated sample is arranged as six RGB views stacked along the channel dimension.

Primary Checkpoint

checkpoints/var_d12_18ch_scale_avg_final.pth

Architecture: text-conditioned VAR
Depth: 12
Embedding dimension: 768
Attention heads: 12
Output channels: 18
VQ-VAE quantization: Euclidean / L2 (using_znorm=False)
Training strategy: no progressive training, scale_avg loss weighting
Regularization: dropout 0.2, attention dropout 0.1, drop path 0.1, label smoothing 0.15, weight decay 0.25

Optional Experimental Checkpoint

checkpoints/experimental/var_mv_v2_final.pth

This is an experimental multi-view attention v2 checkpoint. It is archived for research comparison and should be treated separately from the primary depth-12 inference path.

Archive Checkpoint

checkpoints/archive/var_d12_mv_init.pth

This checkpoint stores the MV-layer initialization generated from the primary depth-12 VAR checkpoint. It is not the recommended inference checkpoint, but it is preserved for reproducibility of the MV-attention experiments.

Strict Reproducibility Artifacts

checkpoints/pretrain/var_d12_init.pth

This is the depth-12 VAR initialization checkpoint used by the final 18-channel training branch. It is preserved for retraining or ablation runs, not as the recommended inference checkpoint.

checkpoints/resume/train_d12_scale_avg_resume_20250115/last.ckpt

This is the final PyTorch Lightning checkpoint for the recommended base run. It contains training state for resume/debugging workflows and is larger than the plain inference checkpoint.

Training logs for the same run are stored under:

training_logs/train_d12_scale_avg_resume_20250115/
  csv_logs/version_0/hparams.yaml
  csv_logs/version_0/metrics.csv
  tb_logs/version_0/hparams.yaml
  tb_logs/version_0/events.out.tfevents.1768496525.tueilmt-simgpu.137100.0

Dependencies

This VAR checkpoint requires the matching 18-channel VQ-VAE checkpoint and the 18-channel stacked NPY data format used during training. The model uses openai/clip-vit-large-patch14 as the text encoder.

Inference Skeleton

python infer_var.py \
  --var_ckpt checkpoints/var_d12_18ch_scale_avg_final.pth \
  --vae_ckpt /path/to/vqvae_npy_final.pth \
  --depth 12 \
  --vae_in_channels 18 \
  --using_znorm 0 \
  --text "A hand is holding a cylindrical cup with a handle" \
  --preset sharp \
  --num_samples 2 \
  --output_dir outputs/demo

Limitations

The training corpus is relatively small for a text-conditioned generative model.
The primary depth-12 branch plateaued at about 55% validation accuracy.
The multi-view attention branch is experimental and not the default release path.

Attribution

The codebase is derived from the MIT-licensed VAR project by FoundationVision. This repository contains experiment checkpoints and metadata only.

Downloads last month: -; Downloads are not tracked for this model. How to track