--- library_name: pytorch tags: - procedure-planning - diffusion - vision-language - pytorch --- # Compact CLAD I reproduced CLAD and used it as the starting point for a lighter formulation. The question I tested was simple: can the diffusion state be made smaller? Compact CLAD keeps the VAE and diffusion structure from CLAD, but the noisy state contains only the action plan. Task identity, endpoint observations, and the VAE latents are given to the denoiser as context. This makes the diffusion state smaller while keeping the original planning objective. This repository contains the original CLAD reproduction, my Compact CLAD implementation, and a Transformer baseline called Simple CLAD. [Model and experiment artifacts](https://huggingface.co/grKnight/compact-clad) · [Original CLAD paper](https://arxiv.org/abs/2503.06637) · [PDPP codebase](https://github.com/MCG-NJU/PDPP) ## Why I worked on this CLAD performs diffusion over a tensor that combines the task class, action class, and visual observation. The task and observation information is repeated at each point in the plan. This produces a wide diffusion state, especially when the visual feature has 1,536 dimensions. In this formulation, I kept latent endpoint conditioning and iterative action generation while supplying the remaining information through conditioning. Compact CLAD therefore performs diffusion on the one hot action grid alone. The reduction is substantial. For NIV, the state width falls from 1,589 values per step to 48. For CrossTask, it falls from 1,659 to 105. For COIN, it falls from 2,494 to 778. ## What I changed 1. I wrote an endpoint VAE trainer for signed observation and text features. It standardizes the inputs using statistics from the training split and uses MSE reconstruction. 2. I built a small conditional temporal U Net for action diffusion. It receives the start observation, goal observation, task class, and the two endpoint latents through a separate condition encoder. 3. I kept the clean sample prediction objective used by CLAD. The loss is summed over the full action grid, with ten times more weight on the first and last planning positions. 4. I added checkpoint metadata and strict compatibility checks. A checkpoint cannot be loaded silently with a different VAE format, conditioning mode, or planner shape. 5. I added CSV, JSONL, and TensorBoard logs, validation based checkpoint selection, early stopping for the VAE, and evaluation over five random seeds. 6. I kept two conditioning settings. `protocol` follows the benchmark setup and uses endpoint action labels. `deployable` removes that label access and requires an image only VAE. ## Results I completed horizon 3 runs on NIV and CrossTask. Planner values below are means over seeds 0 through 4.
Dataset Parameters Selected checkpoint Acc@1 Acc@5 Success mIoU1 mIoU2
NIV 1,079,536 Epoch 125 53.70 81.16 33.63 79.45 62.66
CrossTask 1,087,721 Epoch 80 63.11 84.97 30.44 51.05 60.60
The best validation ELBO was 597.9396 for the NIV VAE and 617.4352 for the CrossTask VAE. On CrossTask, endpoint accuracy was 73.68 for `a0` and 73.29 for `aT`. `Acc@1` and `Acc@5` measure action prediction. `Success` requires the whole trajectory to be correct. `mIoU1` and `mIoU2` are the instruction and video overlap measures used by the CLAD and PDPP evaluation code. ## A note about the NIV result The official NIV test manifest enriched with event class labels was also used for validation and model selection. For that reason, I treat the NIV numbers as a record of the completed experiment, not as an untouched test estimate. The CrossTask numbers come from the saved five seed test run. I am not making a state of the art claim from these experiments. The main result is that the smaller action focused formulation trains, produces usable plans, and can be reproduced from the saved code and artifacts. ## Saved artifacts The [Compact CLAD model repository](https://huggingface.co/grKnight/compact-clad) contains the selected checkpoints, periodic checkpoints, training logs, validation logs, TensorBoard files, run configurations, and five seed inference outputs for NIV and CrossTask. It also contains `artifact_manifest.json`, which records the file path, byte size, and SHA 256 checksum for each uploaded artifact. ## Main files `train_endpoint_vae.py` trains the normalized endpoint VAE. `main_compact_clad.py` trains the compact diffusion planner. `inference_compact_clad.py` runs repeated inference and reports the planning metrics. `model/compact_temporal.py` defines the conditional temporal U Net. `model/compact_diffusion.py` implements the training objective and the DDPM and DDIM samplers. `utils/compact_pipeline.py` prepares endpoint inputs, creates the VAE, and manages checkpoint metadata. `utils/metric_logging.py` writes the experiment records. `tests/` contains the unit and synthetic integration tests. ## Reproducing the run Create the environment first. ```bash conda env create -f environment.yaml conda activate thvae ``` The split files, action metadata, and language embeddings are included in the repository. The S3D visual features must be placed in the matching folder under `dataset`. Train the endpoint VAE. These are the commands for the two completed settings. ```bash DATASET=NIV HORIZON=3 bash train_vae_thvae_vanilla_kepp.sh DATASET=crosstask HORIZON=3 bash train_vae_thvae_vanilla_kepp.sh ``` Set `data` and `horizon` near the top of `train_compact.sh`, check `vae_path`, and train the planner. ```bash bash train_compact.sh ``` Run evaluation with a saved checkpoint. ```bash bash inference_compact.sh path/to/compact_checkpoint.pth.tar ``` Run the test suite with the following command. ```bash python -m unittest discover -s tests -v ``` More implementation notes and the original ablation plan are in [`LIGHTWEIGHT_VAE_DIFFUSION_PLAN.md`](LIGHTWEIGHT_VAE_DIFFUSION_PLAN.md). ## Attribution This is my independent reproduction and extension. It is not an official implementation from the CLAD authors. The repository builds on the public PDPP planning framework and on the CLAD work by Lei Shi and Andreas Bulling. If you use the original CLAD method, please cite its authors. ```bibtex @article{shi2025clad, title={CLAD: Constrained Latent Action Diffusion for Vision Language Procedure Planning}, author={Shi, Lei and Bulling, Andreas}, journal={arXiv preprint arXiv:2503.06637}, year={2025} } ```