Instructions to use nhatcm/hitbench-smolvla-30k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use nhatcm/hitbench-smolvla-30k with LeRobot:
# See https://github.com/huggingface/lerobot?tab=readme-ov-file#installation for more details git clone https://github.com/huggingface/lerobot.git cd lerobot pip install -e .[smolvla]
# Launch finetuning on your dataset python lerobot/scripts/train.py \ --policy.path=nhatcm/hitbench-smolvla-30k \ --dataset.repo_id=lerobot/svla_so101_pickplace \ --batch_size=64 \ --steps=20000 \ --output_dir=outputs/train/my_smolvla \ --job_name=my_smolvla_training \ --policy.device=cuda \ --wandb.enable=true
# Run the policy using the record function python -m lerobot.record \ --robot.type=so101_follower \ --robot.port=/dev/ttyACM0 \ # <- Use your port --robot.id=my_blue_follower_arm \ # <- Use your robot id --robot.cameras="{ front: {type: opencv, index_or_path: 8, width: 640, height: 480, fps: 30}}" \ # <- Use your cameras --dataset.single_task="Grasp a lego block and put it in the bin." \ # <- Use the same task description you used in your dataset recording --dataset.repo_id=HF_USER/dataset_name \ # <- This will be the dataset name on HF Hub --dataset.episode_time_s=50 \ --dataset.num_episodes=10 \ --policy.path=nhatcm/hitbench-smolvla-30k - Notebooks
- Google Colab
- Kaggle
SO-101 SmolVLA: balloon to tray, front RGB + EEF delta
This private evaluation checkpoint is the completed 30,000-step SmolVLA run.
It was trained on 1,565 causal segments (349,611 frames) exported from 477
front-view Task 0-9 trajectories. The base instruction is
put balloon onto tray; h2r-air and r2h append their exact event phrase only
after the recorded authority edge.
Exact interface
Input to one policy call:
observation.images.rgb: rawuint8RGB, HWC; resize to 256x256, convert to float32 CHW, add a batch dimension, and divide by 255.observation.state: raw measured SO-101 joint positions, float32[6], in this order:shoulder_pan.pos, shoulder_lift.pos, elbow_flex.pos, wrist_flex.pos, wrist_roll.pos, gripper.pos.task:['put balloon onto tray'].
The state MEAN_STD statistics are embedded in model.safetensors; do not
normalize the state twice. RGB and action normalization are IDENTITY.
Output from predict_action_chunk is [batch, 50, 7]; one action is ordered
[dx, dy, dz, dRx, dRy, dRz, gripper] and clipped to [-1, 1].
The representation is normalized-eef-delta-v2.
Decode each executed action against the latest measured EEF pose:
a = np.clip(action, -1.0, 1.0)
target_position_m = latest_position_m + a[:3] * 0.004
delta_rotvec_rad = a[3:6] * 0.5
target_rotation = Rotation.from_rotvec(delta_rotvec_rad).as_matrix() @ latest_rotation
fraction_closed = (a[6] + 1.0) / 2.0
gripper_units = 100.0 - fraction_closed * (100.0 - 0.0)
Thus gripper -1 is open and +1 is closed. Send decoded EEF targets through
the calibrated SO-101 IK and joint-limit controller.
Debug fixture
debug_sample/observations.npz contains 50 consecutive,
unaugmented training frames from episode 0, beginning
at frame 0. Arrays:
rgb: uint8[50, 256, 256, 3]in RGB HWC orderobservation_state: float32[50, 6]ground_truth_action: float32[50, 7]timestamp,frame_index,episode_index,task_indextask: a scalar Unicode string
Use debug_sample/preview.gif for a visual sanity check and
debug_sample/manifest.json for provenance and checksums. This fixture is for
input-pipeline debugging, not an unbiased evaluation split.
Files
model.safetensors,config.json: policy weights and architecturetrain_config.json: full training and augmentation configurationaction_contract.json: authoritative EEF output decodereval_contract.json: evaluator-facing input/output schema and embedded state statisticsdebug_sample/*: aligned RGB/state/action fixture
Repository: nhatcm/hitbench-so101-smolvla-front-eef-30k.
Training recipe
- 30,000 optimizer updates; effective batch 28 on four GPUs
- 80% globally frame-uniform anchors and 20% balanced Task-3/Task-5 authority-boundary anchors
- semantic wall replacement applied only to labeled wall pixels
- measured sim-real RGB gain/bias bridge on every example
- camera color, illumination, blur, sensor noise, and JPEG pools ramped linearly with optimizer progress
- no crops, flips, warps, dropout masks, or changes to state, language, padding, or actions
The final observed training loss was approximately 0.005. This is a training
objective, not a physical task-success measurement. See
sim_real_augmentation_recipe.json, augmentation_curriculum.json,
sampling_contract.json, and wall_augmentation_contract.json for the exact
reproducibility contracts.
- Downloads last month
- 14