bosungkim/sensorcade-v0.1

Sensorcade-BC v0.1

Behavior-cloning baselines for the Sensorcade benchmark, where robot policies must read many kinds of sensors, live, while acting. Every model here was trained only on the Sensorcade v0.1 dataset, so anyone can retrain them from the download.

There are 24 models: 6 robot setups × 2 input sets × 2 training seeds.

Folder prefix	Robot	Families	Sensor encoders (full only)
bcD_yam_	I2RT YAM arm	Delicate Grasp, Precision Pour, Kitchen Prep	fingertip tactile, wrist force/torque, microphone, joint torque
bcD_g1_	Unitree G1	Load Carry	wrist force/torque, foot pressure, IMU, joint torque
bcD_g1_pour_	Unitree G1	Carry and Pour	wrist force/torque, foot pressure, IMU, joint torque
bcD_x2_	Skydio X2	Disturbed Hover, Gate Racing, Person Following	IMU, motor current, lidar, thermal camera
bcD_go1_	Unitree Go1	Person Following	IMU, thermal camera
bcD_rov_	8-thruster ROV	Subsea Inspection	hydrophone, imaging and scanning sonar, DVL, INS, USBL, IMU, magnetometer, depth, altimeter, CTD, water speed, tether tension, motor current and RPM

• full: cameras, language, robot state and the robot's sensors.
• nosense: the same model with its sensor encoders removed: cameras, language and robot state only, the default VLA input set.
• s0 / s1: training seed.

Each folder holds:

• last.pt: the final checkpoint (235–263 MB);
• args.json: the training arguments;
• inputs.json: the cameras, streams and task the model reads;
• train.jsonl and val.jsonl: the training and validation logs.

Model

Camera frames go through a small convolutional image encoder trained from scratch. The instruction becomes one token: a frozen Qwen3.5-4B sentence embedding, mean-pooled. Each sensor type gets its own encoder, a causal convolution tower over the raw window at the sensor's own rate. Tokens from all encoders go into a small transformer with a flow-matching action head. The sensor, image and text tokens are computed at a slow rate (every 10 control steps, ready 3 steps later), while the action head acts every control step on the latest robot state.

Training: 30,000 steps (40,000 for g1 and g1_pour), batch 32, learning rate 3e-4, on the dataset's training split (seed 3000); validation uses seed 3001.

Results

Success in % on the benchmark's scored test cases (main and hard), mean of 2 training seeds × 2 eval seeds (1234, 777), 1,024 episodes each:

Family	Expert (ceiling)	BC: cameras + language + state	BC: + sensors
Delicate Grasp	53.5	0.0	0.0
Precision Pour	73.5	0.8	10.0
Load Carry	91.2	0.0	0.0
Carry and Pour	84.1	0.0	0.0
Disturbed Hover	99.5	0.0	0.0
Gate Racing	72.9	0.0	0.0
Person Following (Skydio X2)	83.3	0.0	0.0
Person Following (Unitree Go1)	97.2	0.5	0.3
Subsea Inspection	87.2	0.0	29.2
Kitchen Prep	99.0	0.0	0.0
Overall	83.5	0.1	4.4

These are a starting point, not a solution. The gap between the two BC columns shows what the sensors are worth, and the gap to the expert is the room left. While the expert drives, the models' predicted actions track it closely, but when a model drives it drifts off the expert's states, and plain behavior cloning has nothing to correct that. The benchmark's README compares them with interactively trained policies.

Use

git clone https://github.com/bosung/sensorcade.git && cd sensorcade
uv sync --extra baselines
hf download bosungkim/sensorcade-v0.1 --include "bcD_rov_full_s0/*" --local-dir checkpoints
uv run --extra baselines python scripts/eval_set.py --model bcD_rov_full_s0 \
  --checkpoint checkpoints/bcD_rov_full_s0/last.pt --realism checkpoint --kinds main hard --prompt-batch

To retrain one, see scripts/train_bc.py and the Baselines section of the code repository.

License

Apache-2.0, like the benchmark code. The training data is the CC-BY-4.0 Sensorcade v0.1 dataset.

Citation

TBA.