SO-101 SmolVLA: balloon to tray, front RGB + EEF delta

This private evaluation checkpoint is the completed 30,000-step SmolVLA run. It was trained on 1,565 causal segments (349,611 frames) exported from 477 front-view Task 0-9 trajectories. The base instruction is put balloon onto tray; h2r-air and r2h append their exact event phrase only after the recorded authority edge.

Exact interface

Input to one policy call:

  • observation.images.rgb: raw uint8 RGB, HWC; resize to 256x256, convert to float32 CHW, add a batch dimension, and divide by 255.
  • observation.state: raw measured SO-101 joint positions, float32 [6], in this order: shoulder_pan.pos, shoulder_lift.pos, elbow_flex.pos, wrist_flex.pos, wrist_roll.pos, gripper.pos.
  • task: ['put balloon onto tray'].

The state MEAN_STD statistics are embedded in model.safetensors; do not normalize the state twice. RGB and action normalization are IDENTITY.

Output from predict_action_chunk is [batch, 50, 7]; one action is ordered [dx, dy, dz, dRx, dRy, dRz, gripper] and clipped to [-1, 1]. The representation is normalized-eef-delta-v2.

Decode each executed action against the latest measured EEF pose:

a = np.clip(action, -1.0, 1.0)
target_position_m = latest_position_m + a[:3] * 0.004
delta_rotvec_rad = a[3:6] * 0.5
target_rotation = Rotation.from_rotvec(delta_rotvec_rad).as_matrix() @ latest_rotation
fraction_closed = (a[6] + 1.0) / 2.0
gripper_units = 100.0 - fraction_closed * (100.0 - 0.0)

Thus gripper -1 is open and +1 is closed. Send decoded EEF targets through the calibrated SO-101 IK and joint-limit controller.

Debug fixture

debug_sample/observations.npz contains 50 consecutive, unaugmented training frames from episode 0, beginning at frame 0. Arrays:

  • rgb: uint8 [50, 256, 256, 3] in RGB HWC order
  • observation_state: float32 [50, 6]
  • ground_truth_action: float32 [50, 7]
  • timestamp, frame_index, episode_index, task_index
  • task: a scalar Unicode string

Use debug_sample/preview.gif for a visual sanity check and debug_sample/manifest.json for provenance and checksums. This fixture is for input-pipeline debugging, not an unbiased evaluation split.

Files

  • model.safetensors, config.json: policy weights and architecture
  • train_config.json: full training and augmentation configuration
  • action_contract.json: authoritative EEF output decoder
  • eval_contract.json: evaluator-facing input/output schema and embedded state statistics
  • debug_sample/*: aligned RGB/state/action fixture

Repository: nhatcm/hitbench-so101-smolvla-front-eef-30k.

Training recipe

  • 30,000 optimizer updates; effective batch 28 on four GPUs
  • 80% globally frame-uniform anchors and 20% balanced Task-3/Task-5 authority-boundary anchors
  • semantic wall replacement applied only to labeled wall pixels
  • measured sim-real RGB gain/bias bridge on every example
  • camera color, illumination, blur, sensor noise, and JPEG pools ramped linearly with optimizer progress
  • no crops, flips, warps, dropout masks, or changes to state, language, padding, or actions

The final observed training loss was approximately 0.005. This is a training objective, not a physical task-success measurement. See sim_real_augmentation_recipe.json, augmentation_curriculum.json, sampling_contract.json, and wall_augmentation_contract.json for the exact reproducibility contracts.

Downloads last month
14
Safetensors
Model size
0.5B params
Tensor type
F32
·
BF16
·
Video Preview
loading