README.md

---
license: apache-2.0
tags:
- computer-vision
- robotics
- articulated-objects
language:
- en
---

# L-CAMP: Language-Conditioned Axis and Motion Prediction for Articulated Object Manipulation

[Ikechukwu Daniel Adebi](https://danieladebi.github.io), [Peter Stone](https://www.cs.utexas.edu/~pstone/), [Mitchell Pryor](https://www.me.utexas.edu/people/faculty-directory/pryor)

The University of Texas at Austin

Code: [UTNuclearRobotics/l_camp](https://github.com/UTNuclearRobotics/l_camp)

## About this checkpoint

This repository hosts a pretrained L-CAMP checkpoint, `lcamp_model.pt`, available from the Files tab above. It predicts the screw axis and motion of an articulated object part from an RGB(-D) image conditioned on a language instruction.

To use it, download `lcamp_model.pt` from the Files tab and place it in a `checkpoints/` directory inside a clone of the [GitHub repo](https://github.com/UTNuclearRobotics/l_camp), then follow the setup and evaluation steps below.

## Setup

1. Clone the [GitHub repo](https://github.com/UTNuclearRobotics/l_camp) and create the conda environment from `environment_lcamp.yml`:
   ```bash
   conda create -f environment_lcamp.yml 
   conda activate lcamp
   ```
2. Build the LCAMP dataset (only needed once per dataset version), see
   `process_lcamp_dataset.py`. Whatever path you pass as its `--data_dir` is
   the dataset directory (containing `train_dataset.json`, `test_dataset.json`,
   and `intrinsics.npy`) referenced below as `<DATASET_DIR>`
   (`--data_root`/`--data_dir` in `train.py`/`eval.py`).
3. Get the background images: `LCAMPDataset` (in `lcamp_dataset.py`)
   unconditionally loads a background image list on init (used for the
   `--randomize_background`/`--random_backgrounds` augmentation, but read
   regardless), sourced from the
   [MIT Indoor 67](http://web.mit.edu/torralba/www/indoor.html) scene dataset.
   From that page, download the images archive and the `TrainImages.txt` /
   `TestImages.txt` split files, then lay them out as:
   ```
   backgrounds/mit_indoor_67/TrainImages.txt
   backgrounds/mit_indoor_67/TestImages.txt
   backgrounds/mit_indoor_67/raw/Images/<category>/<image>.jpg
   ```
   relative to wherever you run `train.py`/`eval.py` from (or under
   `$SCRATCH/backgrounds/mit_indoor_67/...` if using `--cluster_loc
   stampede3` — see the `--cluster_loc` note below).

All commands below (`train.py`, `eval.py`) assume the `lcamp` environment is
active.

## Training

If you'd rather train your own checkpoint instead of using this one, training is done with `train.py` via `torchrun` (DDP). The dataset must already be built (see `process_lcamp_dataset.py`) as a directory containing `train_dataset.json`, `test_dataset.json`, and `intrinsics.npy`.

Set `--nproc_per_node` to the number of GPUs on your machine (use `1` for a
single-GPU/local run):
```bash
torchrun --nproc_per_node=<NUM_GPUS> train.py \
    --save_path checkpoints/my_lcamp_model \
    --data_root <DATASET_DIR> \
    --model_type resnet \
    --epochs 50 \
    --batch_size 64 \
    --lr 1e-3 \
    --min_lr 1e-5 \
    --use_camera_frame \
    --use_depth \
    --use_text_instructions \
    --emphasize_part_mask \
    --bbox_loc \
    --cluster_loc <CLUSTER_LOC> \
    --lambda_axis 8 \
    --lambda_anchor 8 \
    --lambda_joint 1
```

Notable flags (`train.py --help` for the complete list):
- `--model_type`: `resnet` (default LCAMP backbone) or `dino`.
- `--use_text_instructions`: enables the language-conditioned model
  (`LCAMPModelResnetLanguage`); required if you plan to run the `lcamp`
  backend with instructions in `lcamp_service_node.py`.
- `--use_depth`: adds depth as an input channel.
- `--bbox_loc`: predict axis location as `(u,v,z)` decoded via the part
  bounding box instead of raw XYZ, must match `--bbox_loc` in `eval.py` and
  `use_bbox_loc` in `lcamp_service_node.py`/`lcamp.launch.py` for the same
  checkpoint. Whenever this is active, pass `--bbox_loc` explicitly so the
  flag stays consistent across `train.py`/`eval.py`/deployment.
- `--resume` / `--resume_from <path>`: resume from `checkpoint_best.pt` in
  `--save_path`, or from an explicit checkpoint path.
- `--save_freq`, `--eval_freq`: checkpoint/eval cadence in epochs.
- `--cluster_loc`: the dataset JSON (from `process_lcamp_dataset.py`) stores
  absolute file paths for the rgb/depth/mask files, so if you're loading the
  dataset on a different machine/path layout than where it was built, those
  paths won't resolve. `nrg` (default) uses the JSON paths as-is, `stampede3`
  is a legacy alias for `$SCRATCH`. For anything else, just pass your own
  dataset root directory as `--cluster_loc /path/to/your/root` — `lcamp_
  dataset.py` will swap it in for the legacy `/storage/danieladebi` prefix in
  the rgb/depth/mask paths, and root the background images (see above) under
  it too. No code changes needed.

Checkpoints are written to `--save_path` as `checkpoint_best.pt`,
`checkpoint_last.pt`, and periodic `checkpoint_{epoch}.pt` files.

## Evaluation

Evaluation is done with `eval.py` (single process, no DDP). To evaluate this checkpoint:
```bash
python eval.py \
    --model_path checkpoints/lcamp_model.pt \
    --data_dir <DATASET_DIR> \
    --split test \
    --model_type resnet \
    --use_camera_frame \
    --use_depth \
    --use_text_instructions \
    --bbox_loc \
    --cluster_loc <CLUSTER_LOC> \
    --visualize
```

This must use the **same architecture flags used at training time**
(`--model_type`, `--use_depth`, `--use_text_instructions`,
`--exclude_object_mask`, `--emphasize_part_mask`/`--emphasize_object_mask`,
`--bbox_loc`, etc.), mismatches will load state dict weights incorrectly or
error, and `--use_camera_frame`/`--cluster_loc` must match how the dataset
was built.

Notable flags:
- `--split`: `train` or `test`.
- `--visualize`: saves per-sample prediction images.
- `--random_preds`: baseline using random axis/location predictions.
- `--free_motion_query_prob`: fraction of eval samples using a free-motion
  (zero-vector) query instead of the GT axis, to measure reliance on visual
  cues vs. the GT axis itself.

Results (per-object/per-category success rates, axis/location error, joint
type accuracy) are written to `results/<model_name>/`, where `<model_name>`
is derived from `--model_path`.

Once you're happy with a checkpoint (this one or your own), point the
`model_name` launch argument in `launch/lcamp.launch.py` at it (see the
launch file for details on `use_bbox_loc` and other runtime flags).

## Deploying on a Robot

Running predictions on a real Boston Dynamics Spot (via the ROS2 service node
in `scripts/lcamp_service_node.py`) is outside the scope of this model card,
see [ros2_setup/ROS2_SETUP.md](https://github.com/UTNuclearRobotics/l_camp/blob/main/ros2_setup/ROS2_SETUP.md) in the GitHub repo for that setup.