grKnight/compact-clad

Compact CLAD

I reproduced CLAD and used it as the starting point for a lighter formulation. The question I tested was simple: can the diffusion state be made smaller?

Compact CLAD keeps the VAE and diffusion structure from CLAD, but the noisy state contains only the action plan. Task identity, endpoint observations, and the VAE latents are given to the denoiser as context. This makes the diffusion state smaller while keeping the original planning objective.

This repository contains the original CLAD reproduction, my Compact CLAD implementation, and a Transformer baseline called Simple CLAD.

Model and experiment artifacts · Original CLAD paper · PDPP codebase

Why I worked on this

CLAD performs diffusion over a tensor that combines the task class, action class, and visual observation. The task and observation information is repeated at each point in the plan. This produces a wide diffusion state, especially when the visual feature has 1,536 dimensions.

In this formulation, I kept latent endpoint conditioning and iterative action generation while supplying the remaining information through conditioning. Compact CLAD therefore performs diffusion on the one hot action grid alone.

The reduction is substantial. For NIV, the state width falls from 1,589 values per step to 48. For CrossTask, it falls from 1,659 to 105. For COIN, it falls from 2,494 to 778.

What I changed

1. I wrote an endpoint VAE trainer for signed observation and text features. It standardizes the inputs using statistics from the training split and uses MSE reconstruction.

2. I built a small conditional temporal U Net for action diffusion. It receives the start observation, goal observation, task class, and the two endpoint latents through a separate condition encoder.

3. I kept the clean sample prediction objective used by CLAD. The loss is summed over the full action grid, with ten times more weight on the first and last planning positions.

4. I added checkpoint metadata and strict compatibility checks. A checkpoint cannot be loaded silently with a different VAE format, conditioning mode, or planner shape.

5. I added CSV, JSONL, and TensorBoard logs, validation based checkpoint selection, early stopping for the VAE, and evaluation over five random seeds.

6. I kept two conditioning settings. protocol follows the benchmark setup and uses endpoint action labels. deployable removes that label access and requires an image only VAE.

Results

I completed horizon 3 runs on NIV and CrossTask. Planner values below are means over seeds 0 through 4.

Dataset	Parameters	Selected checkpoint	Acc@1	Acc@5	Success	mIoU1	mIoU2
NIV	1,079,536	Epoch 125	53.70	81.16	33.63	79.45	62.66
CrossTask	1,087,721	Epoch 80	63.11	84.97	30.44	51.05	60.60

The best validation ELBO was 597.9396 for the NIV VAE and 617.4352 for the CrossTask VAE. On CrossTask, endpoint accuracy was 73.68 for a0 and 73.29 for aT.

Acc@1 and Acc@5 measure action prediction. Success requires the whole trajectory to be correct. mIoU1 and mIoU2 are the instruction and video overlap measures used by the CLAD and PDPP evaluation code.

A note about the NIV result

The official NIV test manifest enriched with event class labels was also used for validation and model selection. For that reason, I treat the NIV numbers as a record of the completed experiment, not as an untouched test estimate. The CrossTask numbers come from the saved five seed test run.

I am not making a state of the art claim from these experiments. The main result is that the smaller action focused formulation trains, produces usable plans, and can be reproduced from the saved code and artifacts.

Saved artifacts

The Compact CLAD model repository contains the selected checkpoints, periodic checkpoints, training logs, validation logs, TensorBoard files, run configurations, and five seed inference outputs for NIV and CrossTask.

It also contains artifact_manifest.json, which records the file path, byte size, and SHA 256 checksum for each uploaded artifact.

Main files

train_endpoint_vae.py trains the normalized endpoint VAE.

main_compact_clad.py trains the compact diffusion planner.

inference_compact_clad.py runs repeated inference and reports the planning metrics.

model/compact_temporal.py defines the conditional temporal U Net.

model/compact_diffusion.py implements the training objective and the DDPM and DDIM samplers.

utils/compact_pipeline.py prepares endpoint inputs, creates the VAE, and manages checkpoint metadata.

utils/metric_logging.py writes the experiment records.

tests/ contains the unit and synthetic integration tests.

Reproducing the run

Create the environment first.

conda env create -f environment.yaml
conda activate thvae

The split files, action metadata, and language embeddings are included in the repository. The S3D visual features must be placed in the matching folder under dataset.

Train the endpoint VAE. These are the commands for the two completed settings.

DATASET=NIV HORIZON=3 bash train_vae_thvae_vanilla_kepp.sh
DATASET=crosstask HORIZON=3 bash train_vae_thvae_vanilla_kepp.sh

Set data and horizon near the top of train_compact.sh, check vae_path, and train the planner.

bash train_compact.sh

Run evaluation with a saved checkpoint.

bash inference_compact.sh path/to/compact_checkpoint.pth.tar

Run the test suite with the following command.

python -m unittest discover -s tests -v

More implementation notes and the original ablation plan are in LIGHTWEIGHT_VAE_DIFFUSION_PLAN.md.

Attribution

This is my independent reproduction and extension. It is not an official implementation from the CLAD authors. The repository builds on the public PDPP planning framework and on the CLAD work by Lei Shi and Andreas Bulling.

If you use the original CLAD method, please cite its authors.

@article{shi2025clad,
  title={CLAD: Constrained Latent Action Diffusion for Vision Language Procedure Planning},
  author={Shi, Lei and Bulling, Andreas},
  journal={arXiv preprint arXiv:2503.06637},
  year={2025}
}