{ "_readme_scheme_switch": "Top-level \"scheme\": xvla_pybullet_drone_demo.py uses the matching block directly. xvla_widowx_drone_demo.py merges widowx_ee6d workspace/task with numeric scheme 1/2/3 gui, sim_steps, xvla_infer_gap, etc.; scheme widowx_ee6d uses only that block. Legacy key widowx_scheme can still select the workspace block. CLI: --scheme overrides top-level scheme. Numeric scheme blocks may set gate_pose_estimator (opencv|xvla) to override widowx_ee6d gate pose estimation.", "scheme": 2, "common": { "_readme_gate_pose_estimator": "xvla_widowx_drone_demo only: rectangular portal opening orientation (when fpv_use_sim_truth_pose=false). gate_pose_estimator must be opencv or xvla; common may provide default, schemes.widowx_ee6d or numeric 1/2/3 merge may override. Top-level same-name key overrides last (see merge logic).", "gate_pose_estimator": "opencv", "server_url": "http://127.0.0.1:8000/act", "qs_policy_path": "QS.json", "vvla_url": "", "vvla_chat_completions_url": "", "vvla_chat_model": "gpt-4o-mini", "vvla_chat_api_key": "", "xvla_act_probe_request_timeout_s": 300, "xvla_act_request_timeout_s": 300, "xvla_act_probe_max_wait_s": 900, "xvla_act_probe_retry_s": 3, "xvla_act_probe_image_height": 256, "xvla_act_probe_image_width": 256, "auto_start_xvla_server": true, "xvla_server_access_log": false, "xvla_server_log_level": "warning", "instruction": "Fly to the target while avoiding obstacles", "record_visualization": true, "recording_path": "recordings", "recording_fps": 18, "record_every": 1, "recording_append_timestamp": true, "recording_width": 640, "recording_height": 480 }, "schemes": { "1": { "label": "Visual debug: default GUI + faster simulation", "recording_format": "mp4", "sim_steps": 120, "xvla_steps": 6, "gui": true, "speed": 3.0, "xvla_infer_gap": 1, "dt": 0.06, "realtime": false, "log_every": 1 }, "2": { "label": "Fast batch run: no GUI + large infer steps + less logging", "recording_format": "gif", "sim_steps": 100, "xvla_steps": 2, "gui": false, "speed": 4.0, "xvla_infer_gap": 20, "dt": 0.08, "realtime": false, "log_every": 20, "record_every": 4 }, "3": { "label": "Demo playback: window + near-real-time replay", "recording_format": "mp4", "sim_steps": 180, "xvla_steps": 6, "gui": true, "speed": 2.5, "xvla_infer_gap": 1, "dt": 0.05, "realtime": true, "log_every": 5 }, "widowx_ee6d": { "label": "Drone=WidowX EE: EE6D semantics flight in arm workspace (xvla_widowx_drone_demo.py)", "recording_path": "recordings", "recording_format": "mp4", "recording_folder_prefix": "", "recording_scene_fov": 48.0, "recording_scene_margin": 1.75, "recording_camera_distance_scale": 0.4, "recording_stereo45_distance_scale": 1.5, "instruction": "Sequential flight through rectangular portals in task order; each subtask names billboard_id matching the top marker (1–20).", "sim_steps": 7200, "xvla_steps": 6, "gui": true, "speed": 1.0, "xvla_infer_gap": 1, "dt": 0.06, "realtime": false, "log_every": 10, "with_objects": true, "treat_pos_as": "absolute", "delta_pos_scale": 0.04, "pos_lerp_alpha": 0.25, "rot_lerp_alpha": 0.20, "_auto_motion": "When true, auto-adjust pos_lerp_alpha and delta_pos_scale by workspace size, infer gap, and language-only task (--cmd or task_sequence: []). auto_motion_ref_diag_m is calibration reference diagonal (m). CLI: --auto-motion / --no-auto-motion.", "auto_motion_scales": true, "auto_motion_ref_diag_m": 0.55, "_language_only_motion": "Only without task_sequence / --cmd: amplify wrist-local target displacement from model. language_only_motion_amplify=multiplier along drone→target displacement; language_only_min_step_local_m=local min target distance after workspace clip (m). CLI: --cmd-motion-amplify / --cmd-min-step.", "language_only_motion_amplify": 3.0, "language_only_min_step_local_m": 0.04, "_cmd_color_disambiguation": "With --cmd: if sentence contains a color word present in scene cubes (e.g. yellow), append English sentence to X-VLA language_instruction emphasizing that color portal only; independent of billboard append. billboard_id numbering usually skips color append (may trigger id-specific append). false disables color append.", "cmd_color_disambiguation": true, "_infer_displacement_scale": "On each X-VLA /act target refresh, multiply decoded target displacement in wrist-local frame (after target_pull and language-only boost), then clip to workspace; 1.0 unchanged. Independent of sim-step pos_lerp. CLI: --infer-displacement-scale.", "infer_displacement_scale": 1.0, "_xvla_scene_semantic_context": "When true (default) with with_objects and scene objects: prepend full-object English catalog aligned to simulation and workspace lo/hi to each /act language_instruction, plus path/safety suffix for spatial semantics from top/slant images. --no-xvla-scene-catalog disables.", "xvla_scene_semantic_context": true, "_xvla_path_planning_instruction_suffix": "Non-empty string overrides default English path-planning suffix; empty string uses built-in default.", "xvla_path_planning_instruction_suffix": "", "_local_avoidance": "Near-field artificial-potential repulsion on inflated obstacle AABBs: deflect world-frame target_pos (phase1 lerp); phase2 P-control error vector plus repulsion. Objects near goal/subgoal (center distance ≤ local_avoidance_exclude_goal_tol_m) excluded. --no-local-avoidance disables.", "local_avoidance_enabled": true, "local_avoidance_robot_radius": 0.048, "local_avoidance_influence_m": 0.16, "local_avoidance_gain": 0.065, "local_avoidance_target_max_shift_m": 0.09, "local_avoidance_phase2_gain": 0.85, "local_avoidance_exclude_goal_tol_m": 0.11, "_cmd_global_camera": "Only --cmd without task_sequence: color segmentation at workspace X-VLA camera resolution/FOV, ray intersect portal opening plane for world target, blend with decoded target via cmd_global_cam_pull_alpha (0=off). Opening normal from sim tilt_deg/yaw_deg (real robot needs extrinsics/depth). Below cmd_global_cam_min_area_ratio mask area falls back to portal catalog center. CLI: --cmd-global-cam / --no-cmd-global-cam / --cmd-global-cam-pull P.", "cmd_use_global_camera_target": true, "cmd_global_cam_pull_alpha": 0.5, "workspace_camera_width": 256, "workspace_camera_height": 256, "workspace_camera_fov": 55.0, "cmd_global_cam_min_area_ratio": 0.012, "cmd_global_cam_weak_mask_scale": 0.35, "cmd_global_cam_weak_mask_min_pull": 0.12, "cmd_global_cam_portal_min_pull": 0.18, "cmd_coarse_plan_once": true, "cmd_coarse_plan_steps": 56, "cmd_coarse_plan_smooth_window": 5, "_navigation_phase2": "Phase 2: top-down corridor grid A* (inflate Phase1 obstacle XY) for coarse path and blue polyline; portal frames use four-bar AABBs (hole passable). navigation_phase1_corridor_margin_m expands green corridor margin. Closed-loop /act still uses slant workspace camera; top-down X-VLA only when navigation_phase2_optional_topdown_xvla=true.", "navigation_phase2_xvla_steps": null, "navigation_phase2_sync_root_config": false, "navigation_phase2_sync_qs": false, "navigation_phase2_extra_instruction": "", "navigation_phase2_geom_astar": true, "navigation_phase2_astar_cell_m": 0.04, "navigation_phase1_corridor_margin_m": 0.18, "navigation_phase1_corridor_bandwidth_m": 0.12, "navigation_collision_pad_m": null, "navigation_use_phase3_refined_path_in_sim": true, "navigation_sim_recording_right_distance_scale": 0.8, "navigation_sim_recording_top_distance_scale": 0.8, "navigation_sim_recording_stereo45_distance_scale": 0.7, "_navigation_sim_recording_view_scales": "sim/ recording camera distance scale per view (× base scale; >1 farther, <1 closer). right=1.3, top=1.1, 45deg=0.7; front unchanged.", "_navigation_use_phase3_path": "true: after Phase3 do not exit; replay refined trajectory as polyline interpolation into main sim (no X-VLA /act during sim), record world_*.mp4|gif in recordings//sim/. May combine with cmd_coarse_plan_once:true (Phase1+2+3 then sim). false with cmd_coarse_plan_once:true exits after Phase3. navigation_collision_pad_m null aligns with astar_obstacle_pad.", "navigation_phase2_optional_topdown_xvla": false, "navigation_phase3_xvla_action_classify": true, "navigation_phase3_xvla_steps": 1, "navigation_phase3_feedback_alpha": 0.10, "navigation_recording_use_opengl": true, "_navigation_phase3": "Phase 3: feedback sphere alpha; with GUI and navigation_recording_use_opengl=true use ER_BULLET_HARDWARE_OPENGL for true transparency (DIRECT/TinyRenderer uses OpenCV 2D overlay).", "cmd_goal_coupled_virtual_base": true, "vb_smooth_alpha": 0.42, "vb_max_speed_m_s": 4.0, "vb_jump_warn_m": 0.35, "cmd_precision_zone_enable": true, "cmd_precision_zone_scale": 1.2, "cmd_precision_zone_inflate_m": 0.08, "cmd_precision_zone_min_r": 0.10, "cmd_precision_infer_every": 1, "_language_air_path": "Language-only --cmd with figure-eight / in the air / over the workspace / patrol etc.: clamp decoded local Z to mid cruise band in workspace and cap global camera pull to language_only_air_path_max_cam_pull to avoid ground portal targets pulling low; segments still from X-VLA. language_only_cruise_z_clamp=false disables Z clamp.", "language_only_cruise_z_clamp": true, "language_only_cruise_z_margin_lo_frac": 0.1, "language_only_cruise_z_margin_hi_frac": 0.08, "language_only_air_path_max_cam_pull": 0.22, "_language_only_demo_trajectory": "No task_sequence and scenic command (pattern above workspace, figure-eight, circuit, etc.) not through-portal: each /act raises local Z to workspace ground (virtual_base) + language_only_demo_height_factor×max object height above ground (default 1.44≈20% above prior 1.2); blend decoded XY with parametric overlay via language_only_demo_plan_blend_beta; fill/edge_margin/period below. false disables.", "language_only_demo_trajectory_fill": true, "language_only_demo_height_factor": 1.44, "language_only_demo_plan_blend_beta": 0.38, "language_only_demo_plan_fill_frac": 0.40, "language_only_demo_plan_edge_margin_frac": 0.08, "language_only_demo_traj_period_infers": 32, "_language_only_demo_xvla_trajectory_once": "Scenic non-portal commands: first /act with steps=max(--xvla-steps, language_only_demo_traj_plan_steps) for full action sequence, decode waypoints and lift to z_min; sim steps interpolate by arc length only, no further X-VLA. false reverts to per-step inference + original figure-eight blend.", "language_only_demo_xvla_trajectory_once": true, "language_only_demo_traj_plan_steps": 64, "_language_only_demo_plan_topdown": "Scenic one-shot planning: true uses get_world_recording_rgb(view_kind=top) same as recording top for X-VLA input; false uses slant workspace camera.", "language_only_demo_plan_use_topdown_camera": true, "_language_only_demo_fill_xy": "After planning, affine-map decoded waypoint XY bbox to scene object top-down AABB (shrunk margin) ∩ current workspace top rectangle so trajectory fills object occupancy in top view; large margin may degenerate.", "language_only_demo_fill_xy_margin_m": 0.025, "_language_only_demo_scenic_xy_force": "One-shot scenic: after prior top rectangle, replace with workspace XY center square, side = language_only_demo_scenic_xy_force_long_scale × workspace long side, shared by figure-eight, ellipse, and rectangle-remap stunts.", "language_only_demo_scenic_xy_force_long_scale": 0.85, "workspace_lo": [0.15, -0.25, 0.02], "workspace_hi": [0.55, 0.25, 0.40], "camera_eye": [-0.55, 0.05, 0.40], "camera_look_at": [0.35, 0.0, 0.18], "recording_width": 640, "recording_height": 480, "record_every": 1, "recording_final_hold_frames": 18, "trail_enabled": true, "trail_rgba": [1.0, 0.68, 0.32, 0.75], "trail_radius": 0.006, "trail_min_distance": 0.01, "virtual_workspace_enabled": true, "virtual_workspace_margin": 0.02, "_fpv_slot_align": "Near target within fpv_slot_align_dist: gate-pose camera render + rect frame pose override target_R. gate_pose_use_dedicated_camera uses dedicated pose. fpv_use_sim_truth_pose=true uses sim quaternion; false gate_pose_estimator=opencv (color seg + long-edge tilt) or xvla (gate RGB + X-VLA 6D rotation, slower). gate_pose_cv_fallback_map_pose / gate_pose_xvla_fallback_opencv control fallback.", "fpv_slot_align": true, "fpv_slot_align_dist": 0.45, "fpv_use_sim_truth_pose": true, "portal_match_tol": 0.15, "fpv_cam_offset_body": [0.06, 0.0, 0.02], "fpv_cam_look_body": [0.85, 0.0, 0.0], "fpv_cam_width": 256, "fpv_cam_height": 256, "fpv_cam_fov": 72.0, "gate_pose_use_dedicated_camera": true, "gate_pose_cam_offset_body": [0.06, 0.0, 0.032], "gate_pose_cam_look_body": [0.95, 0.0, -0.08], "gate_pose_cam_width": 320, "gate_pose_cam_height": 320, "gate_pose_cam_fov": 82.0, "gate_pose_cv_rgb_pad": 45, "gate_pose_cv_min_area_ratio": 0.012, "gate_pose_cv_fallback_map_pose": true, "_gate_pose_estimator": "Matches gate_pose_estimator in this block: only when fpv_use_sim_truth_pose=false; opencv=color seg+long-edge tilt; xvla=gate image + X-VLA /act for 6D rotation (high latency). Allowed: opencv | xvla only.", "gate_pose_estimator": "xvla", "_xvla_gate_instruction": "English sent to dedicated gate-pose camera when gate_pose_estimator=xvla; placeholders {billboard_id}, {color}.", "xvla_gate_instruction_template": "Orient end-effector with long opening of rectangular portal billboard_id={billboard_id} ({color}); pass through.", "xvla_gate_steps": 4, "xvla_gate_infer_width": 256, "xvla_gate_infer_height": 256, "gate_pose_xvla_fallback_opencv": true, "task_default_phrase": "Fly through rectangular portal billboard_id={billboard_id} ({color} visual); align with the long narrow opening.", "task_default_arrive_radius": 0.06, "task_default_dwell_steps": 3, "stop_when_all_visited": true, "_precision_notes": "Two-phase algorithm: Phase1=X-VLA approach, switch to Phase2=P-control precision on stall or phase1_max_steps. target_pull_alpha: pull VLA output toward task_xyz (0=pure VLA, 1=pure geometry). target_snap_radius: snap to target center within this distance. movement_deadband: no move below this displacement, suppress jitter. phase1_stall_window: history window steps for stall detection. phase1_stall_threshold: min approach in window to count as stall. phase1_max_steps: max Phase1 steps per target for Phase2 time. phase2_p_gain: P-controller gain. phase2_max_speed: Phase2 max speed (m/s).", "target_pull_alpha": 0.7, "target_snap_radius": 0.02, "movement_deadband": 0.005, "phase1_stall_window": 20, "phase1_stall_threshold": 0.005, "phase1_max_steps": 45, "phase2_p_gain": 4.0, "phase2_max_speed": 0.24, "cubes": [ {"color": "red", "shape": "rect_frame", "pos": [2.60, 0.000, 0.670], "rgba": [0.90, 0.20, 0.20, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "orange", "shape": "rect_frame", "pos": [3.218, 0.441, 0.659], "rgba": [1.00, 0.48, 0.05, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "yellow", "shape": "rect_frame", "pos": [3.776, 0.713, 0.628], "rgba": [0.95, 0.85, 0.20, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "green", "shape": "rect_frame", "pos": [4.218, 0.713, 0.579], "rgba": [0.20, 0.85, 0.30, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "cyan", "shape": "rect_frame", "pos": [4.502, 0.441, 0.518], "rgba": [0.10, 0.85, 0.90, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "blue", "shape": "rect_frame", "pos": [4.60, 0.000, 0.450], "rgba": [0.20, 0.45, 0.95, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "purple", "shape": "rect_frame", "pos": [4.502,-0.441, 0.382], "rgba": [0.55, 0.25, 0.90, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "light-red", "shape": "rect_frame", "pos": [4.218,-0.713, 0.321], "rgba": [1.00, 0.55, 0.55, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "light-green", "shape": "rect_frame", "pos": [3.776,-0.713, 0.272], "rgba": [0.60, 1.00, 0.60, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "light-purple","shape": "rect_frame", "pos": [3.218,-0.441, 0.241], "rgba": [0.78, 0.62, 1.00, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "red", "shape": "rect_frame", "pos": [2.60, 0.000, 0.230], "rgba": [0.90, 0.20, 0.20, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "orange", "shape": "rect_frame", "pos": [1.982, 0.441, 0.241], "rgba": [1.00, 0.48, 0.05, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "yellow", "shape": "rect_frame", "pos": [1.424, 0.713, 0.272], "rgba": [0.95, 0.85, 0.20, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "green", "shape": "rect_frame", "pos": [0.982, 0.713, 0.321], "rgba": [0.20, 0.85, 0.30, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "cyan", "shape": "rect_frame", "pos": [0.698, 0.441, 0.382], "rgba": [0.10, 0.85, 0.90, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "blue", "shape": "rect_frame", "pos": [0.60, 0.000, 0.450], "rgba": [0.20, 0.45, 0.95, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "purple", "shape": "rect_frame", "pos": [0.698,-0.441, 0.518], "rgba": [0.55, 0.25, 0.90, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "light-red", "shape": "rect_frame", "pos": [0.982,-0.713, 0.579], "rgba": [1.00, 0.55, 0.55, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "light-green", "shape": "rect_frame", "pos": [1.424,-0.713, 0.628], "rgba": [0.60, 1.00, 0.60, 1.0], "side": 0.392, "thickness": 0.026}, {"color": "light-purple","shape": "rect_frame", "pos": [1.982,-0.441, 0.659], "rgba": [0.78, 0.62, 1.00, 1.0], "side": 0.392, "thickness": 0.026} ], "_readme_task_sequence_billboard": "Each subtask instruction uses billboard_id=N, matching rect_frame order in cubes front-to-back and portal top markers 1–20; target_xyz remains corresponding frame center.", "task_sequence": [ {"label": "through-billboard-01", "instruction": "Fly through rectangular portal billboard_id=1 only; align with the long narrow opening.", "target_xyz": [2.60, 0.000, 0.670], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-02", "instruction": "Fly through rectangular portal billboard_id=2 only; align with the long narrow opening.", "target_xyz": [3.218, 0.441, 0.659], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-03", "instruction": "Fly through rectangular portal billboard_id=3 only; align with the long narrow opening.", "target_xyz": [3.776, 0.713, 0.628], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-04", "instruction": "Fly through rectangular portal billboard_id=4 only; align with the long narrow opening.", "target_xyz": [4.218, 0.713, 0.579], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-05", "instruction": "Fly through rectangular portal billboard_id=5 only; align with the long narrow opening.", "target_xyz": [4.502, 0.441, 0.518], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-06", "instruction": "Fly through rectangular portal billboard_id=6 only; align with the long narrow opening.", "target_xyz": [4.60, 0.000, 0.450], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-07", "instruction": "Fly through rectangular portal billboard_id=7 only; align with the long narrow opening.", "target_xyz": [4.502,-0.441, 0.382], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-08", "instruction": "Fly through rectangular portal billboard_id=8 only; align with the long narrow opening.", "target_xyz": [4.218,-0.713, 0.321], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-09", "instruction": "Fly through rectangular portal billboard_id=9 only; align with the long narrow opening.", "target_xyz": [3.776,-0.713, 0.272], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-10", "instruction": "Fly through rectangular portal billboard_id=10 only; align with the long narrow opening.", "target_xyz": [3.218,-0.441, 0.241], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-11", "instruction": "Fly through rectangular portal billboard_id=11 only; align with the long narrow opening.", "target_xyz": [2.60, 0.000, 0.230], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-12", "instruction": "Fly through rectangular portal billboard_id=12 only; align with the long narrow opening.", "target_xyz": [1.982, 0.441, 0.241], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-13", "instruction": "Fly through rectangular portal billboard_id=13 only; align with the long narrow opening.", "target_xyz": [1.424, 0.713, 0.272], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-14", "instruction": "Fly through rectangular portal billboard_id=14 only; align with the long narrow opening.", "target_xyz": [0.982, 0.713, 0.321], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-15", "instruction": "Fly through rectangular portal billboard_id=15 only; align with the long narrow opening.", "target_xyz": [0.698, 0.441, 0.382], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-16", "instruction": "Fly through rectangular portal billboard_id=16 only; align with the long narrow opening.", "target_xyz": [0.60, 0.000, 0.450], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-17", "instruction": "Fly through rectangular portal billboard_id=17 only; align with the long narrow opening.", "target_xyz": [0.698,-0.441, 0.518], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-18", "instruction": "Fly through rectangular portal billboard_id=18 only; align with the long narrow opening.", "target_xyz": [0.982,-0.713, 0.579], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-19", "instruction": "Fly through rectangular portal billboard_id=19 only; align with the long narrow opening.", "target_xyz": [1.424,-0.713, 0.628], "arrive_radius": 0.06, "dwell_steps": 2}, {"label": "through-billboard-20", "instruction": "Fly through rectangular portal billboard_id=20 only; align with the long narrow opening.", "target_xyz": [1.982,-0.441, 0.659], "arrive_radius": 0.06, "dwell_steps": 2} ] } } }