CLIP β†’ language-conditioned Panda control

Frozen pretrained representations. A locally trained policy. Closed-loop MuJoCo rollouts.

Source & setup Β· Dataset Β· Six demos Β· Full results

Selected learned-policy success

A selected success, not an aggregate performance claim. Watch the complete success and a same-task timeout.

What is here

  • 18 unique training checkpoint files, representing 21 original snapshot paths, including diagnostic/smoke runs, single-task training, and multi-task training.
  • 1,529 unique experiment files from Stages 2–7: raw rollout videos, action/latency traces, reports, configurations, and logs, totaling 867,299,299 bytes before cards/media.
  • Forty duplicate paths are represented by aliases.json; the source downloader restores their original experiment paths from the single retained payload.
  • A SHA-256 inventory in artifact_manifest.json, plus six curated MP4 demos and an animated preview.

The selected policy is outputs/stage7/frozen_epoch06.ckpt (42,775,514 bytes), SHA-256:

3731f123462be5bee742cfcb45a595386b68a5963150e1a7079328315eca36cf

Other checkpoints are archival and have not all been evaluated under the final protocol.

Architecture

Component Status
CLIP ViT-B/16 image/text encoders PRETRAINED + FROZEN
Visual/text projectors, state MLP, modality embeddings, action query TRAINED BY US
Four-layer, width-256, eight-head transformer and seven-value action head TRAINED BY US
Preprocessing, OSC_POSE, MuJoCo, LIBERO success predicates NON-LEARNED INFRASTRUCTURE

There are 3,557,639 trainable parameters and no fine-tuned pretrained components. The frozen CLIP backbone is downloaded separately from openai/clip-vit-base-patch16, revision 57c216476eefef5ab752ec549e440a49ae4ae5f3.

Input: one external RGB image, instruction text, and seven joint plus two finger positions. Output: one normalized [dx, dy, dz, dRx, dRy, dRz, gripper] command. Training uses behavior cloning, not reinforcement learning.

Recorded results

Per-task success and 95% Wilson intervals

Goal Successes / rollouts
put the bowl on the plate 8 / 20
put the bowl on top of the cabinet 14 / 20
open the middle drawer of the cabinet 0 / 20
open the top drawer and put the bowl inside 0 / 20
push the plate to the front of the stove 11 / 20
put the bowl on the stove 5 / 20
put the cream cheese in the bowl 0 / 20
put the wine bottle on the rack 1 / 20
put the wine bottle on top of the cabinet 14 / 20
turn on the stove 2 / 20

55 / 200 successes (27.5%), one training seed. Protocol: fixed reset indices 20–39 per goal, ten open-gripper settling steps, 600-action horizon, and 20 Hz simulated control. All 145 unsuccessful outcomes reached the horizon; no execution faults occurred.

The eight-task extension was selected after the first two task outcomes were visible. Reset IDs were withheld from development selection, but simulator starting states are not claimed disjoint from demonstrations. Separate offline action-error tests use disjoint whole demonstration episodes.

Evaluation remains partial: final image/proprioception ablations, novel placement/visual shifts, contact taxonomy, and multiple training seeds are incomplete. This is a local supervised experiment, not an official lifelong-learning benchmark result.

Training evidence

Training and validation behavior-cloning loss

Epoch index 6 was selected using validation/development evidence. Lower offline loss is not itself a closed-loop success claim.

Reproduce

Clone the source repository and follow docs/SETUP.md for the pinned Python 3.11 / native Windows / CUDA environment. From the project root:

python scripts\download_project_artifacts.py --dataset --checkpoint
python scripts\download_clip_stage3.py
python scripts\rollout_libero_policy.py --checkpoint outputs\stage7\frozen_epoch06.ckpt --manifest data\manifests\libero_goal_stage6.json --task put_the_bowl_on_top_of_the_cabinet --reset-indices 20 --settling-steps 10 --settling-gripper -1 --horizon 600 --output outputs\reproduction\cabinet_reset20

Use --all-artifacts to restore all saved experiment files. The source downloader pins Hub revisions, verifies SHA-256, and refuses to overwrite conflicting local files. This is a custom Lightning policy checkpoint, not a Transformers AutoModel export. PyTorch serialization should only be loaded from trusted sources.

Intended use, limitations, and licensing

Intended for learning and reproducing behavior cloning in simulation. The policy has one camera, no temporal memory or action chunking, and uneven goal performance. It is not a general-purpose robot foundation model and has not been deployed on physical hardware. Runs used an RTX 4050 Laptop GPU with 6 GB VRAM. Simulated 20 Hz does not establish hard real-time operation.

Original code and trained checkpoints: MIT, copyright 2026 Samyak Jain. The dataset retains Apache-2.0 terms. LIBERO assets, pretrained CLIP, and dependencies retain their own licenses; see upstream notices. Videos show LIBERO/MuJoCo simulation assets. No third-party data or pretrained backbone is relicensed here.

Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
loading

Dataset used to train Samisaliveagain/libero-clip-vla

Collection including Samisaliveagain/libero-clip-vla