Add final FastWAM zipper_pulling checkpoint (16380 steps)
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- baseline/fastwam_zipper/README.md +44 -0
- baseline/fastwam_zipper/THIRD_PARTY.md +3 -0
- baseline/fastwam_zipper/context.pt +3 -0
- baseline/fastwam_zipper/inference.py +107 -0
- baseline/fastwam_zipper/inference_config.json +78 -0
- baseline/fastwam_zipper/inference_manifest.json +539 -0
- baseline/fastwam_zipper/models/LICENSE-WAN-APACHE-2.0.txt +202 -0
- baseline/fastwam_zipper/models/Wan2.2-TI2V-5B/Wan2.2_VAE.pth +3 -0
- baseline/fastwam_zipper/norm_stats.json +136 -0
- baseline/fastwam_zipper/provenance.json +20 -0
- baseline/fastwam_zipper/requirements-inference.txt +12 -0
- baseline/fastwam_zipper/source/LICENSE +24 -0
- baseline/fastwam_zipper/source/README.md +437 -0
- baseline/fastwam_zipper/source/configs/data/galaxea_r1_lite_3cam.yaml +87 -0
- baseline/fastwam_zipper/source/configs/data/libero_2cam.yaml +68 -0
- baseline/fastwam_zipper/source/configs/data/libero_2cam_lerobot_v30.yaml +59 -0
- baseline/fastwam_zipper/source/configs/data/robotwin.yaml +125 -0
- baseline/fastwam_zipper/source/configs/model/fastwam.yaml +59 -0
- baseline/fastwam_zipper/source/configs/model/fastwam_idm.yaml +60 -0
- baseline/fastwam_zipper/source/configs/model/fastwam_joint.yaml +58 -0
- baseline/fastwam_zipper/source/configs/model/fastwam_optional_idm.yaml +62 -0
- baseline/fastwam_zipper/source/configs/sim_libero.yaml +59 -0
- baseline/fastwam_zipper/source/configs/sim_robotwin.yaml +51 -0
- baseline/fastwam_zipper/source/configs/task/galaxea_r1_lite_3cam_1e-4.yaml +46 -0
- baseline/fastwam_zipper/source/configs/task/libero_idm_2cam224_1e-4.yaml +27 -0
- baseline/fastwam_zipper/source/configs/task/libero_joint_2cam224_1e-4.yaml +27 -0
- baseline/fastwam_zipper/source/configs/task/libero_optional_idm_2cam224_1e-4.yaml +24 -0
- baseline/fastwam_zipper/source/configs/task/libero_uncond_2cam224_1e-4.yaml +27 -0
- baseline/fastwam_zipper/source/configs/task/robotwin_idm_3cam_384_1e-4.yaml +27 -0
- baseline/fastwam_zipper/source/configs/task/robotwin_joint_3cam_384_1e-4.yaml +27 -0
- baseline/fastwam_zipper/source/configs/task/robotwin_uncond_3cam_384_1e-4.yaml +27 -0
- baseline/fastwam_zipper/source/configs/train.yaml +48 -0
- baseline/fastwam_zipper/source/pyproject.toml +48 -0
- baseline/fastwam_zipper/source/scripts/__init__.py +1 -0
- baseline/fastwam_zipper/source/scripts/accelerate_configs/accelerate_zero0.yaml +22 -0
- baseline/fastwam_zipper/source/scripts/accelerate_configs/accelerate_zero1_ds.yaml +18 -0
- baseline/fastwam_zipper/source/scripts/accelerate_configs/accelerate_zero2_ds.yaml +18 -0
- baseline/fastwam_zipper/source/scripts/check_galaxea_r1_lite_contract.py +91 -0
- baseline/fastwam_zipper/source/scripts/check_galaxea_text_cache.py +47 -0
- baseline/fastwam_zipper/source/scripts/dryrun_fastwam.py +227 -0
- baseline/fastwam_zipper/source/scripts/ds_configs/ds_zero1_config.json +18 -0
- baseline/fastwam_zipper/source/scripts/ds_configs/ds_zero2_config.json +19 -0
- baseline/fastwam_zipper/source/scripts/precompute_text_embeds.py +389 -0
- baseline/fastwam_zipper/source/scripts/prepare_galaxea_r1_lite_overlay.py +141 -0
- baseline/fastwam_zipper/source/scripts/preprocess_action_dit_backbone.py +240 -0
- baseline/fastwam_zipper/source/scripts/train.py +16 -0
- baseline/fastwam_zipper/source/scripts/train_galaxea_r1_lite.sh +125 -0
- baseline/fastwam_zipper/source/scripts/train_zero1.sh +116 -0
- baseline/fastwam_zipper/source/scripts/train_zero2.sh +116 -0
- baseline/fastwam_zipper/source/src/fastwam.egg-info/PKG-INFO +37 -0
baseline/fastwam_zipper/README.md
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# FastWAM — zipper_pulling
|
| 2 |
+
|
| 3 |
+
Final checkpoint from `bifrost-2026102616142400-shouqx`: **30 epochs**, **16,380 optimizer steps**. The checkpoint is uploaded unchanged; its SHA-256 is recorded in `provenance.json`.
|
| 4 |
+
|
| 5 |
+
## Included inference assets
|
| 6 |
+
|
| 7 |
+
- `step_016380.pt`: final MoT (video/action) and proprioception encoder weights.
|
| 8 |
+
- `models/Wan2.2-TI2V-5B/Wan2.2_VAE.pth`: the frozen VAE actually used in training.
|
| 9 |
+
- `context.pt`: exact cached native UMT5 embedding for **Pull the zipper.**.
|
| 10 |
+
- `norm_stats.json`: this task's original state/action q01/q99 statistics, byte-identical to training.
|
| 11 |
+
- `inference_config.json`, `inference.py`, `requirements-inference.txt`, and exact training source in `source/`.
|
| 12 |
+
|
| 13 |
+
The fixed-task path works offline after downloading this entire directory and installing Python dependencies. No external model checkpoint, tokenizer, or text encoder is required: it uses the included text cache. Changing the instruction requires a newly computed matching native UMT5 cache; editing the prompt string alone does not change conditioning.
|
| 14 |
+
|
| 15 |
+
## Download
|
| 16 |
+
|
| 17 |
+
Download the complete task directory (including the VAE and source):
|
| 18 |
+
|
| 19 |
+
```python
|
| 20 |
+
from huggingface_hub import snapshot_download
|
| 21 |
+
snapshot_download("qshou-coder/SpaceDreamer_Galexea", allow_patterns=["baseline/fastwam_zipper/**"], local_dir="fastwam_export")
|
| 22 |
+
```
|
| 23 |
+
|
| 24 |
+
Then enter `fastwam_export/baseline/fastwam_zipper/`.
|
| 25 |
+
|
| 26 |
+
## Setup and inference
|
| 27 |
+
|
| 28 |
+
Use Python 3.10+ and a CUDA GPU with BF16 support and sufficient free VRAM (training used PyTorch 2.7.1 / CUDA 12.8). From this directory:
|
| 29 |
+
|
| 30 |
+
```bash
|
| 31 |
+
python -m pip install torch==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu128
|
| 32 |
+
python -m pip install -r requirements-inference.txt
|
| 33 |
+
python inference.py --input observation.npz --output prediction.npz --device cuda:0 --steps 20 --seed 0
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
`observation.npz` must contain `head_left`, `wrist_left`, `wrist_right` (HxWx3 **RGB uint8**, no BGR conversion), and `state` (float32, shape `[14]`, original physical state units). The order is **left arm 6, left gripper, right arm 6, right gripper**.
|
| 37 |
+
|
| 38 |
+
The head image is resized to 256x320 and the two wrists to 128x160, then packed as head on top, left/right wrists below, giving 384x320. Inference applies the training q01/q99 transform and no random crop/color augmentation. `prediction.npz` contains `actions` `[40,14]` in original physical target units and `normalized_actions`. Actions are **absolute original `motion_target.position` labels**, not deltas, Cartesian commands, or velocity. Keep the native gripper encoding and coordinate conventions. The data rate was 20 Hz; execution/replanning is controlled by the downstream robot integration, not this file-based runner.
|
| 39 |
+
|
| 40 |
+
Python API: `from inference import Policy; policy = Policy(); result = policy.predict(images, state)`. `predict_video=True` additionally uses the native joint sampler to return nine video frames; default inference uses the native cached action-only path. Default 20 denoising steps is the native inference default, not a claimed evaluation optimum.
|
| 41 |
+
|
| 42 |
+
## Validation scope
|
| 43 |
+
|
| 44 |
+
See `validation.json` for actual checks. Packaging checks cover source identity, checkpoint step and architecture, cached text, training preprocessing and normalization parity, inverse normalization, and relative asset resolution. They do not constitute a new real-robot rollout or GPU performance measurement. Training completion receipts confirm final save and strict reload; the original training smoke used an earlier smoke checkpoint.
|
baseline/fastwam_zipper/THIRD_PARTY.md
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Third-party components
|
| 2 |
+
|
| 3 |
+
The exact FastWAM training source is included under `source/`, with its original MIT license and source headers. The frozen `Wan2.2_VAE.pth` is the original Wan-AI/Wan2.2-TI2V-5B component, SHA-256 `20eb789667fa5e60e7516bf509512f6cb61f01b0aa0695eadaea930c13892b36`. Wan2.2 is distributed under Apache-2.0; see https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B and https://github.com/Wan-Video/Wan2.2 . The native task text cache was produced with the frozen Wan UMT5 encoder.
|
baseline/fastwam_zipper/context.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:24c9c95ba634779ba7aeb33743ee38b3cf688805471889400a2e841fd4ea4ad1
|
| 3 |
+
size 1050977
|
baseline/fastwam_zipper/inference.py
ADDED
|
@@ -0,0 +1,107 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Portable, fixed-task FastWAM inference using the exact training source."""
|
| 2 |
+
import argparse
|
| 3 |
+
import json
|
| 4 |
+
import os
|
| 5 |
+
from pathlib import Path
|
| 6 |
+
import sys
|
| 7 |
+
import numpy as np
|
| 8 |
+
import torch
|
| 9 |
+
from torchvision.transforms import functional as F, InterpolationMode
|
| 10 |
+
|
| 11 |
+
CAMERAS=('head_left','wrist_left','wrist_right')
|
| 12 |
+
|
| 13 |
+
def pack_images(images):
|
| 14 |
+
views=[]
|
| 15 |
+
for name in CAMERAS:
|
| 16 |
+
image=np.asarray(images[name])
|
| 17 |
+
if image.ndim!=3 or image.shape[-1]!=3 or image.dtype!=np.uint8:
|
| 18 |
+
raise ValueError(f'{name} must be HxWx3 uint8 RGB (not BGR).')
|
| 19 |
+
v=torch.from_numpy(image.copy()).permute(2,0,1).float()/255.
|
| 20 |
+
size=[256,320] if name=='head_left' else [128,160]
|
| 21 |
+
views.append(F.resize(v,size,InterpolationMode.BILINEAR,antialias=True))
|
| 22 |
+
return torch.cat([views[0],torch.cat(views[1:],dim=-1)],dim=-2).contiguous()*2-1
|
| 23 |
+
|
| 24 |
+
def normalize(x,stats):
|
| 25 |
+
x=np.asarray(x,dtype=np.float32)
|
| 26 |
+
lo,hi=np.array(stats['q01']),np.array(stats['q99'])
|
| 27 |
+
if x.shape[-1]!=14 or lo.shape!=(14,) or hi.shape!=(14,) or not np.isfinite(x).all():
|
| 28 |
+
raise ValueError('Expected finite 14D state/actions and 14D quantile bounds.')
|
| 29 |
+
result=np.zeros_like(x,dtype=np.float32);mask=~np.isclose(hi,lo)
|
| 30 |
+
result[...,mask]=(x[...,mask]-lo[mask])/(hi[mask]-lo[mask])
|
| 31 |
+
result[...,mask]=2*result[...,mask]-1
|
| 32 |
+
return np.clip(result,-1,1)
|
| 33 |
+
|
| 34 |
+
def denormalize(x,stats):
|
| 35 |
+
x=np.asarray(x,dtype=np.float32)
|
| 36 |
+
lo,hi=np.array(stats['q01']),np.array(stats['q99'])
|
| 37 |
+
if x.shape[-1]!=14 or not np.isfinite(x).all():raise ValueError('Invalid model actions.')
|
| 38 |
+
result=(x+1)/2*(hi-lo)+lo
|
| 39 |
+
result[...,np.isclose(hi,lo)]=lo[np.isclose(hi,lo)]
|
| 40 |
+
return result.astype(np.float32)
|
| 41 |
+
|
| 42 |
+
class Policy:
|
| 43 |
+
def __init__(self,bundle=None,device='cuda:0'):
|
| 44 |
+
self.root=Path(bundle or Path(__file__).parent).resolve()
|
| 45 |
+
cfg=json.loads((self.root/'inference_config.json').read_text())
|
| 46 |
+
self.config=cfg
|
| 47 |
+
self.stats=json.loads((self.root/'norm_stats.json').read_text())['norm_stats']
|
| 48 |
+
context=torch.load(self.root/'context.pt',map_location='cpu',weights_only=True)
|
| 49 |
+
if context['prompt']!=cfg['encoded_prompt']:raise ValueError('Task/context mismatch.')
|
| 50 |
+
if context['context'].shape!=(128,4096) or context['mask'].shape!=(128,):raise ValueError('Invalid text cache.')
|
| 51 |
+
self.context=context
|
| 52 |
+
sys.path.insert(0,str(self.root/'source/src'))
|
| 53 |
+
os.environ['DIFFSYNTH_MODEL_BASE_PATH']=str(self.root/'models')
|
| 54 |
+
os.environ['DIFFSYNTH_SKIP_DOWNLOAD']='true'
|
| 55 |
+
if not (self.root/'models/Wan2.2-TI2V-5B/Wan2.2_VAE.pth').is_file():raise FileNotFoundError('Missing bundled VAE.')
|
| 56 |
+
from fastwam.models.wan22.fastwam import FastWAM
|
| 57 |
+
m=cfg['model']
|
| 58 |
+
self.model=FastWAM.from_wan22_pretrained(
|
| 59 |
+
device=device,torch_dtype=torch.bfloat16,
|
| 60 |
+
model_id=m['model_id'],tokenizer_model_id=m['tokenizer_model_id'],
|
| 61 |
+
tokenizer_max_len=m['tokenizer_max_len'],load_text_encoder=False,
|
| 62 |
+
proprio_dim=m['proprio_dim'],redirect_common_files=False,
|
| 63 |
+
video_dit_config=m['video_dit_config'],action_dit_config=m['action_dit_config'],
|
| 64 |
+
skip_dit_load_from_pretrain=True,action_dit_pretrained_path=None,
|
| 65 |
+
mot_checkpoint_mixed_attn=False,compile_training_denoise=False,
|
| 66 |
+
video_train_shift=m['video_scheduler']['train_shift'],video_infer_shift=m['video_scheduler']['infer_shift'],
|
| 67 |
+
video_num_train_timesteps=m['video_scheduler']['num_train_timesteps'],
|
| 68 |
+
action_train_shift=m['action_scheduler']['train_shift'],action_infer_shift=m['action_scheduler']['infer_shift'],
|
| 69 |
+
action_num_train_timesteps=m['action_scheduler']['num_train_timesteps'])
|
| 70 |
+
checkpoint=torch.load(self.root/cfg['checkpoint'],map_location='cpu',weights_only=True,mmap=True)
|
| 71 |
+
if checkpoint['step']!=cfg['final_step']:raise ValueError('Checkpoint step mismatch.')
|
| 72 |
+
self.model.mot.load_state_dict(checkpoint['mot'],strict=True)
|
| 73 |
+
self.model.proprio_encoder.load_state_dict(checkpoint['proprio_encoder'],strict=True)
|
| 74 |
+
self.model.eval()
|
| 75 |
+
|
| 76 |
+
@torch.inference_mode()
|
| 77 |
+
def predict(self,images,state,steps=20,seed=None,predict_video=False):
|
| 78 |
+
state=np.asarray(state,dtype=np.float32)
|
| 79 |
+
if state.shape!=(14,):raise ValueError('state must be shape (14,), in the original physical units.')
|
| 80 |
+
if steps<1:raise ValueError('steps must be positive')
|
| 81 |
+
args=dict(prompt=None,input_image=pack_images(images),action_horizon=40,
|
| 82 |
+
proprio=torch.from_numpy(normalize(state,self.stats['state'])),
|
| 83 |
+
context=self.context['context'],context_mask=self.context['mask'],
|
| 84 |
+
num_inference_steps=steps,seed=seed,compile_action_infer=False)
|
| 85 |
+
if predict_video:
|
| 86 |
+
result=self.model.infer_joint(**args,num_video_frames=9,test_action_with_infer_action=False)
|
| 87 |
+
else:result=self.model.infer_action(**args)
|
| 88 |
+
normalized=result['action'].detach().float().cpu().numpy()
|
| 89 |
+
if normalized.shape!=(40,14):raise ValueError(f'Unexpected action shape {normalized.shape}')
|
| 90 |
+
output={'actions':denormalize(normalized,self.stats['actions']),'normalized_actions':normalized}
|
| 91 |
+
if predict_video:output['video']=result['video']
|
| 92 |
+
return output
|
| 93 |
+
|
| 94 |
+
def main():
|
| 95 |
+
p=argparse.ArgumentParser(description=__doc__)
|
| 96 |
+
p.add_argument('--bundle',type=Path,default=Path(__file__).parent)
|
| 97 |
+
p.add_argument('--input',type=Path,required=True,help='NPZ: head_left, wrist_left, wrist_right RGB uint8 arrays; state float32[14].')
|
| 98 |
+
p.add_argument('--output',type=Path,required=True)
|
| 99 |
+
p.add_argument('--device',default='cuda:0');p.add_argument('--steps',type=int,default=20);p.add_argument('--seed',type=int,default=None)
|
| 100 |
+
a=p.parse_args()
|
| 101 |
+
with np.load(a.input,allow_pickle=False) as data:
|
| 102 |
+
images={name:data[name] for name in CAMERAS};state=data['state']
|
| 103 |
+
result=Policy(a.bundle,a.device).predict(images,state,a.steps,a.seed)
|
| 104 |
+
np.savez(a.output,**result)
|
| 105 |
+
print(json.dumps({'output':str(a.output),'action_shape':list(result['actions'].shape),'action_semantics':'absolute motion_target.position'}))
|
| 106 |
+
|
| 107 |
+
if __name__=='__main__':main()
|
baseline/fastwam_zipper/inference_config.json
ADDED
|
@@ -0,0 +1,78 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"task": "zipper_pulling",
|
| 3 |
+
"training_job": "bifrost-2026102616142400-shouqx",
|
| 4 |
+
"final_step": 16380,
|
| 5 |
+
"checkpoint": "step_016380.pt",
|
| 6 |
+
"prompt": "Pull the zipper.",
|
| 7 |
+
"encoded_prompt": "A video recorded from a robot's point of view executing the following instruction: Pull the zipper.",
|
| 8 |
+
"fps": 20,
|
| 9 |
+
"action_horizon": 40,
|
| 10 |
+
"video_frames": 9,
|
| 11 |
+
"model": {
|
| 12 |
+
"model_id": "Wan2.2-TI2V-5B",
|
| 13 |
+
"tokenizer_model_id": "Wan2.1-T2V-1.3B",
|
| 14 |
+
"tokenizer_max_len": 128,
|
| 15 |
+
"load_text_encoder": false,
|
| 16 |
+
"proprio_dim": 14,
|
| 17 |
+
"redirect_common_files": false,
|
| 18 |
+
"mot_checkpoint_mixed_attn": false,
|
| 19 |
+
"compile_training_denoise": false,
|
| 20 |
+
"action_dit_pretrained_path": null,
|
| 21 |
+
"skip_dit_load_from_pretrain": true,
|
| 22 |
+
"video_dit_config": {
|
| 23 |
+
"has_image_input": false,
|
| 24 |
+
"patch_size": [
|
| 25 |
+
1,
|
| 26 |
+
2,
|
| 27 |
+
2
|
| 28 |
+
],
|
| 29 |
+
"in_dim": 48,
|
| 30 |
+
"hidden_dim": 3072,
|
| 31 |
+
"ffn_dim": 14336,
|
| 32 |
+
"freq_dim": 256,
|
| 33 |
+
"text_dim": 4096,
|
| 34 |
+
"out_dim": 48,
|
| 35 |
+
"num_heads": 24,
|
| 36 |
+
"attn_head_dim": 128,
|
| 37 |
+
"num_layers": 30,
|
| 38 |
+
"eps": 1e-06,
|
| 39 |
+
"seperated_timestep": true,
|
| 40 |
+
"require_clip_embedding": false,
|
| 41 |
+
"require_vae_embedding": false,
|
| 42 |
+
"fuse_vae_embedding_in_latents": true,
|
| 43 |
+
"use_gradient_checkpointing": false,
|
| 44 |
+
"video_attention_mask_mode": "first_frame_causal",
|
| 45 |
+
"action_conditioned": false,
|
| 46 |
+
"action_dim": 14,
|
| 47 |
+
"action_group_causal_mask_mode": "group_diagonal"
|
| 48 |
+
},
|
| 49 |
+
"action_dit_config": {
|
| 50 |
+
"action_dim": 14,
|
| 51 |
+
"hidden_dim": 1024,
|
| 52 |
+
"ffn_dim": 4096,
|
| 53 |
+
"num_heads": 24,
|
| 54 |
+
"attn_head_dim": 128,
|
| 55 |
+
"num_layers": 30,
|
| 56 |
+
"text_dim": 4096,
|
| 57 |
+
"freq_dim": 256,
|
| 58 |
+
"eps": 1e-06,
|
| 59 |
+
"use_gradient_checkpointing": false
|
| 60 |
+
},
|
| 61 |
+
"video_scheduler": {
|
| 62 |
+
"train_shift": 5.0,
|
| 63 |
+
"infer_shift": 5.0,
|
| 64 |
+
"num_train_timesteps": 1000
|
| 65 |
+
},
|
| 66 |
+
"action_scheduler": {
|
| 67 |
+
"train_shift": 1.0,
|
| 68 |
+
"infer_shift": 1.0,
|
| 69 |
+
"num_train_timesteps": 1000
|
| 70 |
+
},
|
| 71 |
+
"loss": {
|
| 72 |
+
"lambda_action": 1.0
|
| 73 |
+
}
|
| 74 |
+
},
|
| 75 |
+
"action_semantics": "absolute_original_motion_target_position",
|
| 76 |
+
"joint_order": "left_arm6,left_gripper,right_arm6,right_gripper",
|
| 77 |
+
"default_denoising_steps": 20
|
| 78 |
+
}
|
baseline/fastwam_zipper/inference_manifest.json
ADDED
|
@@ -0,0 +1,539 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"files": [
|
| 3 |
+
{
|
| 4 |
+
"path": "step_016380.pt",
|
| 5 |
+
"bytes": 12041813369,
|
| 6 |
+
"sha256": "5e574c8cc7bb5747852cd815cfed9896d8d0f1db83eee28a367bd21beb25f348"
|
| 7 |
+
},
|
| 8 |
+
{
|
| 9 |
+
"path": "README.md",
|
| 10 |
+
"bytes": 3521,
|
| 11 |
+
"sha256": "0fe1530fc1195d0f4ba393fb747a97485d762aefc4f26831ff7ae709ddceb641"
|
| 12 |
+
},
|
| 13 |
+
{
|
| 14 |
+
"path": "THIRD_PARTY.md",
|
| 15 |
+
"bytes": 497,
|
| 16 |
+
"sha256": "085e845844811f3cbf63a07766841a2af78ea80f337753353af868a1a243fc55"
|
| 17 |
+
},
|
| 18 |
+
{
|
| 19 |
+
"path": "context.pt",
|
| 20 |
+
"bytes": 1050977,
|
| 21 |
+
"sha256": "24c9c95ba634779ba7aeb33743ee38b3cf688805471889400a2e841fd4ea4ad1"
|
| 22 |
+
},
|
| 23 |
+
{
|
| 24 |
+
"path": "inference.py",
|
| 25 |
+
"bytes": 6194,
|
| 26 |
+
"sha256": "12e401a2400cc7fdb54560b08c30a1401ef3680261a7bee6fd5912443dadb961"
|
| 27 |
+
},
|
| 28 |
+
{
|
| 29 |
+
"path": "inference_config.json",
|
| 30 |
+
"bytes": 2206,
|
| 31 |
+
"sha256": "e207f5427b3c9a25fc9f1d61074138c6c04dd717f1d253794d2771345cd69f8f"
|
| 32 |
+
},
|
| 33 |
+
{
|
| 34 |
+
"path": "models/LICENSE-WAN-APACHE-2.0.txt",
|
| 35 |
+
"bytes": 11358,
|
| 36 |
+
"sha256": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30"
|
| 37 |
+
},
|
| 38 |
+
{
|
| 39 |
+
"path": "models/Wan2.2-TI2V-5B/Wan2.2_VAE.pth",
|
| 40 |
+
"bytes": 2818839170,
|
| 41 |
+
"sha256": "20eb789667fa5e60e7516bf509512f6cb61f01b0aa0695eadaea930c13892b36"
|
| 42 |
+
},
|
| 43 |
+
{
|
| 44 |
+
"path": "norm_stats.json",
|
| 45 |
+
"bytes": 3306,
|
| 46 |
+
"sha256": "10a5eed0657164c56f13efb0d99434824b201f07e450263f564c828944aa5231"
|
| 47 |
+
},
|
| 48 |
+
{
|
| 49 |
+
"path": "provenance.json",
|
| 50 |
+
"bytes": 563,
|
| 51 |
+
"sha256": "3858af2efaf9c8646d791137a5738db3cb082c7bcb74d9c6f92c2ba5b9573c9a"
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
"path": "requirements-inference.txt",
|
| 55 |
+
"bytes": 196,
|
| 56 |
+
"sha256": "87f014a6694ed6b262d730652c68878ea4297a420c6309711346891920a0d4dc"
|
| 57 |
+
},
|
| 58 |
+
{
|
| 59 |
+
"path": "source/LICENSE",
|
| 60 |
+
"bytes": 1210,
|
| 61 |
+
"sha256": "8379dddfbbb1154ad9cc57f008f1f8d4a8d26c3de99bd6b36aa4d8067888aa92"
|
| 62 |
+
},
|
| 63 |
+
{
|
| 64 |
+
"path": "source/README.md",
|
| 65 |
+
"bytes": 15889,
|
| 66 |
+
"sha256": "1fbad9ad9407d54a224864a10ec9576fa731e4a8f291bc9a7e28f545647fef2a"
|
| 67 |
+
},
|
| 68 |
+
{
|
| 69 |
+
"path": "source/configs/data/galaxea_r1_lite_3cam.yaml",
|
| 70 |
+
"bytes": 3145,
|
| 71 |
+
"sha256": "e83ab2ece05ccea8a352404a7cdd38b361325ecf720f64a79e94c6dccb36cc57"
|
| 72 |
+
},
|
| 73 |
+
{
|
| 74 |
+
"path": "source/configs/data/libero_2cam.yaml",
|
| 75 |
+
"bytes": 2448,
|
| 76 |
+
"sha256": "d2ba368bd906992a2f3ca6229da9a5abd58163e2e64a8c4569d012efef039008"
|
| 77 |
+
},
|
| 78 |
+
{
|
| 79 |
+
"path": "source/configs/data/libero_2cam_lerobot_v30.yaml",
|
| 80 |
+
"bytes": 1935,
|
| 81 |
+
"sha256": "9c243358de9e978f210d827569cf8bf074fec6b7a3d47b079e8827342562edc1"
|
| 82 |
+
},
|
| 83 |
+
{
|
| 84 |
+
"path": "source/configs/data/robotwin.yaml",
|
| 85 |
+
"bytes": 4110,
|
| 86 |
+
"sha256": "f1ba509fac21fc59d5a210d6f68bcf94640729ce00dce695b7ca6df24c7b5aa7"
|
| 87 |
+
},
|
| 88 |
+
{
|
| 89 |
+
"path": "source/configs/model/fastwam.yaml",
|
| 90 |
+
"bytes": 1536,
|
| 91 |
+
"sha256": "0c9378106366d35f99ea3a82ef30afdfbab9a1d330af3e6de6c94c4490d24868"
|
| 92 |
+
},
|
| 93 |
+
{
|
| 94 |
+
"path": "source/configs/model/fastwam_idm.yaml",
|
| 95 |
+
"bytes": 1567,
|
| 96 |
+
"sha256": "7c95ab549c677ade48c7664beb85202aae15ad8c86dba99fbe4b14f2047aeb76"
|
| 97 |
+
},
|
| 98 |
+
{
|
| 99 |
+
"path": "source/configs/model/fastwam_joint.yaml",
|
| 100 |
+
"bytes": 1510,
|
| 101 |
+
"sha256": "ab2a243f7452478b6ba779cc2507e08d503874e7e7ae6ddcc3f081055b0730ad"
|
| 102 |
+
},
|
| 103 |
+
{
|
| 104 |
+
"path": "source/configs/model/fastwam_optional_idm.yaml",
|
| 105 |
+
"bytes": 1607,
|
| 106 |
+
"sha256": "f5c01951f483d453f7a0ef01bc7026eb8c2922b32361230a99e0d09d898904db"
|
| 107 |
+
},
|
| 108 |
+
{
|
| 109 |
+
"path": "source/configs/sim_libero.yaml",
|
| 110 |
+
"bytes": 1082,
|
| 111 |
+
"sha256": "005c4da6d40abc2304ff6f9a06feca1204f330c1ac7c3919f807d1b58dfd016c"
|
| 112 |
+
},
|
| 113 |
+
{
|
| 114 |
+
"path": "source/configs/sim_robotwin.yaml",
|
| 115 |
+
"bytes": 975,
|
| 116 |
+
"sha256": "b96b579a8f243c935f1e17f8f3d6e9b7cb6bf2f2a6823272f626ecc33d68056f"
|
| 117 |
+
},
|
| 118 |
+
{
|
| 119 |
+
"path": "source/configs/task/galaxea_r1_lite_3cam_1e-4.yaml",
|
| 120 |
+
"bytes": 1263,
|
| 121 |
+
"sha256": "6553c8c22fafb7b3a563eb81fb310bf1b36d6b4d5ba7451c89b63d722a2543be"
|
| 122 |
+
},
|
| 123 |
+
{
|
| 124 |
+
"path": "source/configs/task/libero_idm_2cam224_1e-4.yaml",
|
| 125 |
+
"bytes": 409,
|
| 126 |
+
"sha256": "5df6cf62843806b0d9d2fb48b873945865bbfa74d135c58032651d54d6992c4e"
|
| 127 |
+
},
|
| 128 |
+
{
|
| 129 |
+
"path": "source/configs/task/libero_joint_2cam224_1e-4.yaml",
|
| 130 |
+
"bytes": 411,
|
| 131 |
+
"sha256": "1c46f34058c9c5363bab6814a0b86eb13be3a0077e944db7d39cf151da07636e"
|
| 132 |
+
},
|
| 133 |
+
{
|
| 134 |
+
"path": "source/configs/task/libero_optional_idm_2cam224_1e-4.yaml",
|
| 135 |
+
"bytes": 381,
|
| 136 |
+
"sha256": "367794491e728cba5fdb63929e69ab19f3c75b89eb6631754ae6a67d114d542c"
|
| 137 |
+
},
|
| 138 |
+
{
|
| 139 |
+
"path": "source/configs/task/libero_uncond_2cam224_1e-4.yaml",
|
| 140 |
+
"bytes": 405,
|
| 141 |
+
"sha256": "27bdc30545aa06bca81137690a241d469d5a67a433881976bd79da3d4171d1a0"
|
| 142 |
+
},
|
| 143 |
+
{
|
| 144 |
+
"path": "source/configs/task/robotwin_idm_3cam_384_1e-4.yaml",
|
| 145 |
+
"bytes": 405,
|
| 146 |
+
"sha256": "2e67c9f301105752962af825e57e5b46ef5e6ba4290e397e5efc7abb88652619"
|
| 147 |
+
},
|
| 148 |
+
{
|
| 149 |
+
"path": "source/configs/task/robotwin_joint_3cam_384_1e-4.yaml",
|
| 150 |
+
"bytes": 407,
|
| 151 |
+
"sha256": "daf4cb5246728ba60ae6903fbfd9d5e029d7172f30916b8143f81783450fa874"
|
| 152 |
+
},
|
| 153 |
+
{
|
| 154 |
+
"path": "source/configs/task/robotwin_uncond_3cam_384_1e-4.yaml",
|
| 155 |
+
"bytes": 401,
|
| 156 |
+
"sha256": "2f78a207415b452b05e0238f3e00eb9942d7960413076b5b3cdcf1ce09caef9f"
|
| 157 |
+
},
|
| 158 |
+
{
|
| 159 |
+
"path": "source/configs/train.yaml",
|
| 160 |
+
"bytes": 697,
|
| 161 |
+
"sha256": "42a60765bfdd842af01aacdde6066ce63be00698377fe7729110b795be684016"
|
| 162 |
+
},
|
| 163 |
+
{
|
| 164 |
+
"path": "source/pyproject.toml",
|
| 165 |
+
"bytes": 1049,
|
| 166 |
+
"sha256": "161e4be4e5a0c3f9888aa7b8b8416c45441eabf0aed4374ac193a08de02c043a"
|
| 167 |
+
},
|
| 168 |
+
{
|
| 169 |
+
"path": "source/scripts/__init__.py",
|
| 170 |
+
"bytes": 37,
|
| 171 |
+
"sha256": "5a71e51c716ea7579151f36996bebb1ff82c6fedd0c45c68f2918cf5812f1053"
|
| 172 |
+
},
|
| 173 |
+
{
|
| 174 |
+
"path": "source/scripts/accelerate_configs/accelerate_zero0.yaml",
|
| 175 |
+
"bytes": 489,
|
| 176 |
+
"sha256": "dacdf1cee3675246567f1f61e943c6762a4700b8b40c888af8fdf89773adc35b"
|
| 177 |
+
},
|
| 178 |
+
{
|
| 179 |
+
"path": "source/scripts/accelerate_configs/accelerate_zero1_ds.yaml",
|
| 180 |
+
"bytes": 496,
|
| 181 |
+
"sha256": "5b9e24f38c199fdff6bbd17981d86e2624f855f707660094c99f711efc8e0833"
|
| 182 |
+
},
|
| 183 |
+
{
|
| 184 |
+
"path": "source/scripts/accelerate_configs/accelerate_zero2_ds.yaml",
|
| 185 |
+
"bytes": 496,
|
| 186 |
+
"sha256": "3b838fbf44cd279613797e9ac3265f306806391963be593cb47787d38240a2e1"
|
| 187 |
+
},
|
| 188 |
+
{
|
| 189 |
+
"path": "source/scripts/check_galaxea_r1_lite_contract.py",
|
| 190 |
+
"bytes": 3820,
|
| 191 |
+
"sha256": "585200c8bea8e37a832381a00ec86cdb0fc3176840b35cd0df26e6700819c522"
|
| 192 |
+
},
|
| 193 |
+
{
|
| 194 |
+
"path": "source/scripts/check_galaxea_text_cache.py",
|
| 195 |
+
"bytes": 1749,
|
| 196 |
+
"sha256": "2ef7e6dcfc821b8a65ad75ca1af3e70d58100a2e129105d9dd0fcf389e1c7689"
|
| 197 |
+
},
|
| 198 |
+
{
|
| 199 |
+
"path": "source/scripts/dryrun_fastwam.py",
|
| 200 |
+
"bytes": 7168,
|
| 201 |
+
"sha256": "102e89470ae21392ab0e868f4df20eef19ee902cecebf4c0fcf9d937d6c29c50"
|
| 202 |
+
},
|
| 203 |
+
{
|
| 204 |
+
"path": "source/scripts/ds_configs/ds_zero1_config.json",
|
| 205 |
+
"bytes": 400,
|
| 206 |
+
"sha256": "8b65ebcb8de8c3c89113695a61783cdb97c2aa5003477f0ca743aa511bc68312"
|
| 207 |
+
},
|
| 208 |
+
{
|
| 209 |
+
"path": "source/scripts/ds_configs/ds_zero2_config.json",
|
| 210 |
+
"bytes": 428,
|
| 211 |
+
"sha256": "93928339afe3946e603128ed42aae1f44279f694e20be911527e6e02ab4f9c72"
|
| 212 |
+
},
|
| 213 |
+
{
|
| 214 |
+
"path": "source/scripts/precompute_text_embeds.py",
|
| 215 |
+
"bytes": 14454,
|
| 216 |
+
"sha256": "6be7ade7c56287ee640683a2dceedaf0bb5fc1b783802ca6a81fe27a53017edc"
|
| 217 |
+
},
|
| 218 |
+
{
|
| 219 |
+
"path": "source/scripts/prepare_galaxea_r1_lite_overlay.py",
|
| 220 |
+
"bytes": 5255,
|
| 221 |
+
"sha256": "dcab484f9105cde123d2b240d764082a7c165e2fc73af1948039bb837c51726d"
|
| 222 |
+
},
|
| 223 |
+
{
|
| 224 |
+
"path": "source/scripts/preprocess_action_dit_backbone.py",
|
| 225 |
+
"bytes": 9948,
|
| 226 |
+
"sha256": "0fb11f04756d71e6b1b1d26cdf9e5831dfead339da38aa25c02f171d7f484a94"
|
| 227 |
+
},
|
| 228 |
+
{
|
| 229 |
+
"path": "source/scripts/train.py",
|
| 230 |
+
"bytes": 358,
|
| 231 |
+
"sha256": "9fb7bff4bcc0eb47294b6815c547e8cf47c02532da4d58bcb072e1f18ba67ef8"
|
| 232 |
+
},
|
| 233 |
+
{
|
| 234 |
+
"path": "source/scripts/train_galaxea_r1_lite.sh",
|
| 235 |
+
"bytes": 6889,
|
| 236 |
+
"sha256": "d9e6ec7442afb4346d3a53ea97553262001fe84bed21c59a00a1c8a1e1dc912a"
|
| 237 |
+
},
|
| 238 |
+
{
|
| 239 |
+
"path": "source/scripts/train_zero1.sh",
|
| 240 |
+
"bytes": 3241,
|
| 241 |
+
"sha256": "0ee0031b65193953ab2c449201f9ff64574d93d7956a9d5fd6644317bdaf7654"
|
| 242 |
+
},
|
| 243 |
+
{
|
| 244 |
+
"path": "source/scripts/train_zero2.sh",
|
| 245 |
+
"bytes": 3235,
|
| 246 |
+
"sha256": "b75455a0c6519b6b7339ab6a3b4c8e6b6aa882c599a6029aff2d349620bdf43a"
|
| 247 |
+
},
|
| 248 |
+
{
|
| 249 |
+
"path": "source/src/fastwam/__init__.py",
|
| 250 |
+
"bytes": 23,
|
| 251 |
+
"sha256": "c4149c5ff0f9abd1517360daf761e692feb8570eb7a0caa2dfde52d4412c3e23"
|
| 252 |
+
},
|
| 253 |
+
{
|
| 254 |
+
"path": "source/src/fastwam/datasets/__init__.py",
|
| 255 |
+
"bytes": 0,
|
| 256 |
+
"sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
|
| 257 |
+
},
|
| 258 |
+
{
|
| 259 |
+
"path": "source/src/fastwam/datasets/dataset_utils.py",
|
| 260 |
+
"bytes": 3735,
|
| 261 |
+
"sha256": "3bc17590049c7cb54061acf8d731b07892e7f530c5d7eeac2613fa77a285a392"
|
| 262 |
+
},
|
| 263 |
+
{
|
| 264 |
+
"path": "source/src/fastwam/datasets/lerobot/base_lerobot_dataset.py",
|
| 265 |
+
"bytes": 19162,
|
| 266 |
+
"sha256": "c076b4028c5414d3c08199f1501698c3a4e15ac4fe09c6c7a67fa7f462a13b4a"
|
| 267 |
+
},
|
| 268 |
+
{
|
| 269 |
+
"path": "source/src/fastwam/datasets/lerobot/constants.py",
|
| 270 |
+
"bytes": 1915,
|
| 271 |
+
"sha256": "11ac9260341e1560c5eb0dd7acb79ed63c74180b8ab4113c27622ff213db417f"
|
| 272 |
+
},
|
| 273 |
+
{
|
| 274 |
+
"path": "source/src/fastwam/datasets/lerobot/lerobot/datasets/compute_stats.py",
|
| 275 |
+
"bytes": 7067,
|
| 276 |
+
"sha256": "7d7daab219332b5abdfd5ec67ccaf3502e8eb583a17a4e482c3f3d4f7a691e89"
|
| 277 |
+
},
|
| 278 |
+
{
|
| 279 |
+
"path": "source/src/fastwam/datasets/lerobot/lerobot/datasets/utils.py",
|
| 280 |
+
"bytes": 33439,
|
| 281 |
+
"sha256": "40f1b94e6ceb50765fb8dfa3d498887c2a0fc2eb49f09770c5f36190e7110c98"
|
| 282 |
+
},
|
| 283 |
+
{
|
| 284 |
+
"path": "source/src/fastwam/datasets/lerobot/lerobot/datasets/video_utils.py",
|
| 285 |
+
"bytes": 19421,
|
| 286 |
+
"sha256": "cb647a1e955c3279e39ce8d7af7a37ea7432bbd71534c183f9ceebc8557dd89f"
|
| 287 |
+
},
|
| 288 |
+
{
|
| 289 |
+
"path": "source/src/fastwam/datasets/lerobot/lerobot/lerobot_dataset.py",
|
| 290 |
+
"bytes": 55233,
|
| 291 |
+
"sha256": "a71b26a73a0e460198b8316db2acb77b43a719d4ab9706150e2ba53773ffda64"
|
| 292 |
+
},
|
| 293 |
+
{
|
| 294 |
+
"path": "source/src/fastwam/datasets/lerobot/processors/base_processor.py",
|
| 295 |
+
"bytes": 10811,
|
| 296 |
+
"sha256": "1f47bc85fee11e3d16714b2f1e3a2dc874f4609a1706a788617156d34b48b58e"
|
| 297 |
+
},
|
| 298 |
+
{
|
| 299 |
+
"path": "source/src/fastwam/datasets/lerobot/processors/fastwam_processor.py",
|
| 300 |
+
"bytes": 13458,
|
| 301 |
+
"sha256": "3aa0e275b3815b67785c6e38ac5fc400472f9d09ac18b2ab269ea77bf574ade4"
|
| 302 |
+
},
|
| 303 |
+
{
|
| 304 |
+
"path": "source/src/fastwam/datasets/lerobot/robot_video_dataset.py",
|
| 305 |
+
"bytes": 13058,
|
| 306 |
+
"sha256": "04f830e4c7dcc93be8e1d3514e502719aed94e62074364edbc1257e533a27de9"
|
| 307 |
+
},
|
| 308 |
+
{
|
| 309 |
+
"path": "source/src/fastwam/datasets/lerobot/transforms/action_state_merger.py",
|
| 310 |
+
"bytes": 2522,
|
| 311 |
+
"sha256": "669d6991415fc5eca44b32b129d15fee1e06e2f17d59d90350c252598033b427"
|
| 312 |
+
},
|
| 313 |
+
{
|
| 314 |
+
"path": "source/src/fastwam/datasets/lerobot/transforms/image.py",
|
| 315 |
+
"bytes": 725,
|
| 316 |
+
"sha256": "4d7ab61d0844918525244db653831f0f632730046b2714a6163ff57c441a16fa"
|
| 317 |
+
},
|
| 318 |
+
{
|
| 319 |
+
"path": "source/src/fastwam/datasets/lerobot/transforms/misc.py",
|
| 320 |
+
"bytes": 422,
|
| 321 |
+
"sha256": "8b505bd9e3a54ab2db7efcd64f0929d2e0303f5db1123de1074fd01fa9887e2b"
|
| 322 |
+
},
|
| 323 |
+
{
|
| 324 |
+
"path": "source/src/fastwam/datasets/lerobot/transforms/relative_action.py",
|
| 325 |
+
"bytes": 5736,
|
| 326 |
+
"sha256": "2b4f05297de6cd364e507b5874a752003a9dfb770a528ea73d2781c8a1a12059"
|
| 327 |
+
},
|
| 328 |
+
{
|
| 329 |
+
"path": "source/src/fastwam/datasets/lerobot/transforms/rotation.py",
|
| 330 |
+
"bytes": 5603,
|
| 331 |
+
"sha256": "0de34b252d60b956cda07c0c19baebbd3c8a28916face700c6664c5815e11e91"
|
| 332 |
+
},
|
| 333 |
+
{
|
| 334 |
+
"path": "source/src/fastwam/datasets/lerobot/utils/normalizer.py",
|
| 335 |
+
"bytes": 11274,
|
| 336 |
+
"sha256": "5ac89a646cd6395e98feaa832f8fdf7c7e00857080af3d9c15710e243c43ad43"
|
| 337 |
+
},
|
| 338 |
+
{
|
| 339 |
+
"path": "source/src/fastwam/datasets/lerobot/utils/rotation.py",
|
| 340 |
+
"bytes": 21110,
|
| 341 |
+
"sha256": "6169162f98faf02a3fe5201e2b8a1069fe299840d036ee7cad8af5fa8f561542"
|
| 342 |
+
},
|
| 343 |
+
{
|
| 344 |
+
"path": "source/src/fastwam/datasets/lerobot3/__init__.py",
|
| 345 |
+
"bytes": 389,
|
| 346 |
+
"sha256": "26f0c3a4afdbd54f723cab81d2288781844e61ee367fd88bc0a821c7929cb79f"
|
| 347 |
+
},
|
| 348 |
+
{
|
| 349 |
+
"path": "source/src/fastwam/datasets/lerobot3/base_lerobot_dataset.py",
|
| 350 |
+
"bytes": 389,
|
| 351 |
+
"sha256": "7dc96f86caf8239795a0dce0161e25969ac002ebf037dfb9695608c40d89d769"
|
| 352 |
+
},
|
| 353 |
+
{
|
| 354 |
+
"path": "source/src/fastwam/datasets/lerobot3/lerobot_dataset.py",
|
| 355 |
+
"bytes": 14998,
|
| 356 |
+
"sha256": "4acae53b736c47c1ed6cb42ad41a68cb0a64cf6578791c5f9e50a8dd9f5541e1"
|
| 357 |
+
},
|
| 358 |
+
{
|
| 359 |
+
"path": "source/src/fastwam/datasets/lerobot3/robot_video_dataset.py",
|
| 360 |
+
"bytes": 310,
|
| 361 |
+
"sha256": "62230146222dfe0719c43f72d9f8e913cd4998d8f68a3f9d09b9b8d3db7441d5"
|
| 362 |
+
},
|
| 363 |
+
{
|
| 364 |
+
"path": "source/src/fastwam/models/__init__.py",
|
| 365 |
+
"bytes": 35,
|
| 366 |
+
"sha256": "ae4c5733f4bbf7caf31df2ae6f789516f6a10cd84a321f7d0375a7034253ac6d"
|
| 367 |
+
},
|
| 368 |
+
{
|
| 369 |
+
"path": "source/src/fastwam/models/wan22/__init__.py",
|
| 370 |
+
"bytes": 0,
|
| 371 |
+
"sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
|
| 372 |
+
},
|
| 373 |
+
{
|
| 374 |
+
"path": "source/src/fastwam/models/wan22/action_dit.py",
|
| 375 |
+
"bytes": 14329,
|
| 376 |
+
"sha256": "631fdf598fbe06a95a41c1a27acd9de16f2187f3a2009e10690197527d18283b"
|
| 377 |
+
},
|
| 378 |
+
{
|
| 379 |
+
"path": "source/src/fastwam/models/wan22/fastwam.py",
|
| 380 |
+
"bytes": 54376,
|
| 381 |
+
"sha256": "154c339efd33fae119fe902d6efb91cde7475e3cfd0e7b50e606278c1c3eac09"
|
| 382 |
+
},
|
| 383 |
+
{
|
| 384 |
+
"path": "source/src/fastwam/models/wan22/fastwam_idm.py",
|
| 385 |
+
"bytes": 25404,
|
| 386 |
+
"sha256": "5ebdaabb42d4e6c0c6543760ad2ea9b3c8ef9415ebab44fbf845f73f52280bba"
|
| 387 |
+
},
|
| 388 |
+
{
|
| 389 |
+
"path": "source/src/fastwam/models/wan22/fastwam_joint.py",
|
| 390 |
+
"bytes": 10242,
|
| 391 |
+
"sha256": "a61aab79cfc607af02ab816849329a13c1dcd0bbcbf63c6584a345d8a67bb2b6"
|
| 392 |
+
},
|
| 393 |
+
{
|
| 394 |
+
"path": "source/src/fastwam/models/wan22/fastwam_optional_idm.py",
|
| 395 |
+
"bytes": 4420,
|
| 396 |
+
"sha256": "bd508293545895bae159a42998992cece9c1f1aca373e6f03521ea05f55749b2"
|
| 397 |
+
},
|
| 398 |
+
{
|
| 399 |
+
"path": "source/src/fastwam/models/wan22/helpers/__init__.py",
|
| 400 |
+
"bytes": 415,
|
| 401 |
+
"sha256": "f26116de229557a37a4812dc4ee75971f5f9e8b80240786ea4d0e5b0ff703cbf"
|
| 402 |
+
},
|
| 403 |
+
{
|
| 404 |
+
"path": "source/src/fastwam/models/wan22/helpers/gradient.py",
|
| 405 |
+
"bytes": 565,
|
| 406 |
+
"sha256": "ba5d8f7272eb029dc6cd2849ca99b70f6ad5abb838d21c818beb0590620dc793"
|
| 407 |
+
},
|
| 408 |
+
{
|
| 409 |
+
"path": "source/src/fastwam/models/wan22/helpers/io.py",
|
| 410 |
+
"bytes": 7751,
|
| 411 |
+
"sha256": "1c6c1bc8a975b47092912833194fce25e2e3ac886fbab72d73af3ff5a00b3c62"
|
| 412 |
+
},
|
| 413 |
+
{
|
| 414 |
+
"path": "source/src/fastwam/models/wan22/helpers/loader.py",
|
| 415 |
+
"bytes": 8279,
|
| 416 |
+
"sha256": "792f66bd0609c1365256aa24d43301340f81260c1651d89b04d2ae8224a08726"
|
| 417 |
+
},
|
| 418 |
+
{
|
| 419 |
+
"path": "source/src/fastwam/models/wan22/helpers/state_dict_converters.py",
|
| 420 |
+
"bytes": 5226,
|
| 421 |
+
"sha256": "81d213373e5056a8d8bf31768b9b04164138ea78d0b306e316334841499f4bf9"
|
| 422 |
+
},
|
| 423 |
+
{
|
| 424 |
+
"path": "source/src/fastwam/models/wan22/mot.py",
|
| 425 |
+
"bytes": 31234,
|
| 426 |
+
"sha256": "2383e4cfa00efa9cde889493e0b6c861bc7b17d24d731f5ff38d3ebf8ab7f541"
|
| 427 |
+
},
|
| 428 |
+
{
|
| 429 |
+
"path": "source/src/fastwam/models/wan22/schedulers/scheduler_continuous.py",
|
| 430 |
+
"bytes": 4068,
|
| 431 |
+
"sha256": "2ececce284f7b9fe876965fbf5e11ca7b69caaf693979266b0f63062a589f181"
|
| 432 |
+
},
|
| 433 |
+
{
|
| 434 |
+
"path": "source/src/fastwam/models/wan22/wan22.py",
|
| 435 |
+
"bytes": 16925,
|
| 436 |
+
"sha256": "b85349beeeb018d5be3f5b932bee3d640274d09c5e1e0fa0fa01e8ad0e02356e"
|
| 437 |
+
},
|
| 438 |
+
{
|
| 439 |
+
"path": "source/src/fastwam/models/wan22/wan_video_dit.py",
|
| 440 |
+
"bytes": 34764,
|
| 441 |
+
"sha256": "578712e83fac2815d3e3a3fd9134178028082deba361433122fe3c5764982dd6"
|
| 442 |
+
},
|
| 443 |
+
{
|
| 444 |
+
"path": "source/src/fastwam/models/wan22/wan_video_text_encoder.py",
|
| 445 |
+
"bytes": 11459,
|
| 446 |
+
"sha256": "aedf7f131556a39b99a273deec3a85d67f41b5367cef41f310f33a807ae60a87"
|
| 447 |
+
},
|
| 448 |
+
{
|
| 449 |
+
"path": "source/src/fastwam/models/wan22/wan_video_vae.py",
|
| 450 |
+
"bytes": 49604,
|
| 451 |
+
"sha256": "cb2b1b53f35a1a201c640ec52d12a7c4cb4ba1df322155addd3dc409e04dc07c"
|
| 452 |
+
},
|
| 453 |
+
{
|
| 454 |
+
"path": "source/src/fastwam/runtime.py",
|
| 455 |
+
"bytes": 22779,
|
| 456 |
+
"sha256": "230a814b7a42a3f74145513f5775a3c6cd56ffa244635c536b3d1f384c5f82d0"
|
| 457 |
+
},
|
| 458 |
+
{
|
| 459 |
+
"path": "source/src/fastwam/trainer.py",
|
| 460 |
+
"bytes": 37779,
|
| 461 |
+
"sha256": "00f9a8b7c7c222fe8572ec2a8f26ffe5fc5779a3fe07afd77369deab6253ddd3"
|
| 462 |
+
},
|
| 463 |
+
{
|
| 464 |
+
"path": "source/src/fastwam/utils/__init__.py",
|
| 465 |
+
"bytes": 96,
|
| 466 |
+
"sha256": "2893c05b99b17162941c70a41859c2808f8ae390be367a26cac7d5ab93ee197b"
|
| 467 |
+
},
|
| 468 |
+
{
|
| 469 |
+
"path": "source/src/fastwam/utils/config_resolvers.py",
|
| 470 |
+
"bytes": 2526,
|
| 471 |
+
"sha256": "01c48cbd42b86e636b70ccc3e193a81147162ba890777102b2c61086fa415340"
|
| 472 |
+
},
|
| 473 |
+
{
|
| 474 |
+
"path": "source/src/fastwam/utils/fs.py",
|
| 475 |
+
"bytes": 76,
|
| 476 |
+
"sha256": "69e6df236b43360428a613686c8f37c531c0d16a3a2c5b06a725c46b49d657c5"
|
| 477 |
+
},
|
| 478 |
+
{
|
| 479 |
+
"path": "source/src/fastwam/utils/logging_config.py",
|
| 480 |
+
"bytes": 4679,
|
| 481 |
+
"sha256": "cadc4f6ad487c3f37171a6defad36d23d9ab8bf262f73a461cd9ee9b03b757e7"
|
| 482 |
+
},
|
| 483 |
+
{
|
| 484 |
+
"path": "source/src/fastwam/utils/misc.py",
|
| 485 |
+
"bytes": 1223,
|
| 486 |
+
"sha256": "05aa51247936a8d3faac27f90b78b65ff83a688259bed55d114b6a617ced532d"
|
| 487 |
+
},
|
| 488 |
+
{
|
| 489 |
+
"path": "source/src/fastwam/utils/pytorch_utils.py",
|
| 490 |
+
"bytes": 5051,
|
| 491 |
+
"sha256": "24de921fe7305b51f94222f4163c6f81eb5e077e46272314d83eb44f9f4bde71"
|
| 492 |
+
},
|
| 493 |
+
{
|
| 494 |
+
"path": "source/src/fastwam/utils/samplers.py",
|
| 495 |
+
"bytes": 1339,
|
| 496 |
+
"sha256": "4d3bd9b2d498f4ad8f2cf9765d60c4bf5368fa6fa2a7d42d0db6210f66899652"
|
| 497 |
+
},
|
| 498 |
+
{
|
| 499 |
+
"path": "source/src/fastwam/utils/video_io.py",
|
| 500 |
+
"bytes": 819,
|
| 501 |
+
"sha256": "9e9898b021a44e87aa1b4e6d13e3755a9869a2e27a7eb53c72a6802808d5bd80"
|
| 502 |
+
},
|
| 503 |
+
{
|
| 504 |
+
"path": "source/src/fastwam/utils/video_metrics.py",
|
| 505 |
+
"bytes": 3446,
|
| 506 |
+
"sha256": "aaa37e458cc9e4bb79a17923b5ff5900c97b4309f7abd50db44cf76e9143ffd6"
|
| 507 |
+
},
|
| 508 |
+
{
|
| 509 |
+
"path": "source/src/fastwam.egg-info/PKG-INFO",
|
| 510 |
+
"bytes": 1161,
|
| 511 |
+
"sha256": "e129d204188c26725819c3994455d55762913c8345b8ae1b2fdabc469a1c3420"
|
| 512 |
+
},
|
| 513 |
+
{
|
| 514 |
+
"path": "source/src/fastwam.egg-info/SOURCES.txt",
|
| 515 |
+
"bytes": 2462,
|
| 516 |
+
"sha256": "dbb1f8a6580463d20f8bd783f10ba5b7a29e020ea86832f976691b855f250105"
|
| 517 |
+
},
|
| 518 |
+
{
|
| 519 |
+
"path": "source/src/fastwam.egg-info/dependency_links.txt",
|
| 520 |
+
"bytes": 1,
|
| 521 |
+
"sha256": "01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b"
|
| 522 |
+
},
|
| 523 |
+
{
|
| 524 |
+
"path": "source/src/fastwam.egg-info/requires.txt",
|
| 525 |
+
"bytes": 521,
|
| 526 |
+
"sha256": "c82af4bdebdaa050271d8eecd4aca96dede56d6a42bb2cf6db0372023503a1c8"
|
| 527 |
+
},
|
| 528 |
+
{
|
| 529 |
+
"path": "source/src/fastwam.egg-info/top_level.txt",
|
| 530 |
+
"bytes": 8,
|
| 531 |
+
"sha256": "8a42966a9145ab062768faad7503d31d40a654dcd024e8a8fb1885fe4bddeedb"
|
| 532 |
+
},
|
| 533 |
+
{
|
| 534 |
+
"path": "validation.json",
|
| 535 |
+
"bytes": 700,
|
| 536 |
+
"sha256": "95d3f67742d93369c73ed436ad412dab078ec8094d7dab2330a4a0f0cca1bc9c"
|
| 537 |
+
}
|
| 538 |
+
]
|
| 539 |
+
}
|
baseline/fastwam_zipper/models/LICENSE-WAN-APACHE-2.0.txt
ADDED
|
@@ -0,0 +1,202 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
Apache License
|
| 3 |
+
Version 2.0, January 2004
|
| 4 |
+
http://www.apache.org/licenses/
|
| 5 |
+
|
| 6 |
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
| 7 |
+
|
| 8 |
+
1. Definitions.
|
| 9 |
+
|
| 10 |
+
"License" shall mean the terms and conditions for use, reproduction,
|
| 11 |
+
and distribution as defined by Sections 1 through 9 of this document.
|
| 12 |
+
|
| 13 |
+
"Licensor" shall mean the copyright owner or entity authorized by
|
| 14 |
+
the copyright owner that is granting the License.
|
| 15 |
+
|
| 16 |
+
"Legal Entity" shall mean the union of the acting entity and all
|
| 17 |
+
other entities that control, are controlled by, or are under common
|
| 18 |
+
control with that entity. For the purposes of this definition,
|
| 19 |
+
"control" means (i) the power, direct or indirect, to cause the
|
| 20 |
+
direction or management of such entity, whether by contract or
|
| 21 |
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
| 22 |
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
| 23 |
+
|
| 24 |
+
"You" (or "Your") shall mean an individual or Legal Entity
|
| 25 |
+
exercising permissions granted by this License.
|
| 26 |
+
|
| 27 |
+
"Source" form shall mean the preferred form for making modifications,
|
| 28 |
+
including but not limited to software source code, documentation
|
| 29 |
+
source, and configuration files.
|
| 30 |
+
|
| 31 |
+
"Object" form shall mean any form resulting from mechanical
|
| 32 |
+
transformation or translation of a Source form, including but
|
| 33 |
+
not limited to compiled object code, generated documentation,
|
| 34 |
+
and conversions to other media types.
|
| 35 |
+
|
| 36 |
+
"Work" shall mean the work of authorship, whether in Source or
|
| 37 |
+
Object form, made available under the License, as indicated by a
|
| 38 |
+
copyright notice that is included in or attached to the work
|
| 39 |
+
(an example is provided in the Appendix below).
|
| 40 |
+
|
| 41 |
+
"Derivative Works" shall mean any work, whether in Source or Object
|
| 42 |
+
form, that is based on (or derived from) the Work and for which the
|
| 43 |
+
editorial revisions, annotations, elaborations, or other modifications
|
| 44 |
+
represent, as a whole, an original work of authorship. For the purposes
|
| 45 |
+
of this License, Derivative Works shall not include works that remain
|
| 46 |
+
separable from, or merely link (or bind by name) to the interfaces of,
|
| 47 |
+
the Work and Derivative Works thereof.
|
| 48 |
+
|
| 49 |
+
"Contribution" shall mean any work of authorship, including
|
| 50 |
+
the original version of the Work and any modifications or additions
|
| 51 |
+
to that Work or Derivative Works thereof, that is intentionally
|
| 52 |
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
| 53 |
+
or by an individual or Legal Entity authorized to submit on behalf of
|
| 54 |
+
the copyright owner. For the purposes of this definition, "submitted"
|
| 55 |
+
means any form of electronic, verbal, or written communication sent
|
| 56 |
+
to the Licensor or its representatives, including but not limited to
|
| 57 |
+
communication on electronic mailing lists, source code control systems,
|
| 58 |
+
and issue tracking systems that are managed by, or on behalf of, the
|
| 59 |
+
Licensor for the purpose of discussing and improving the Work, but
|
| 60 |
+
excluding communication that is conspicuously marked or otherwise
|
| 61 |
+
designated in writing by the copyright owner as "Not a Contribution."
|
| 62 |
+
|
| 63 |
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
| 64 |
+
on behalf of whom a Contribution has been received by Licensor and
|
| 65 |
+
subsequently incorporated within the Work.
|
| 66 |
+
|
| 67 |
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
| 68 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 69 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 70 |
+
copyright license to reproduce, prepare Derivative Works of,
|
| 71 |
+
publicly display, publicly perform, sublicense, and distribute the
|
| 72 |
+
Work and such Derivative Works in Source or Object form.
|
| 73 |
+
|
| 74 |
+
3. Grant of Patent License. Subject to the terms and conditions of
|
| 75 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 76 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 77 |
+
(except as stated in this section) patent license to make, have made,
|
| 78 |
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
| 79 |
+
where such license applies only to those patent claims licensable
|
| 80 |
+
by such Contributor that are necessarily infringed by their
|
| 81 |
+
Contribution(s) alone or by combination of their Contribution(s)
|
| 82 |
+
with the Work to which such Contribution(s) was submitted. If You
|
| 83 |
+
institute patent litigation against any entity (including a
|
| 84 |
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
| 85 |
+
or a Contribution incorporated within the Work constitutes direct
|
| 86 |
+
or contributory patent infringement, then any patent licenses
|
| 87 |
+
granted to You under this License for that Work shall terminate
|
| 88 |
+
as of the date such litigation is filed.
|
| 89 |
+
|
| 90 |
+
4. Redistribution. You may reproduce and distribute copies of the
|
| 91 |
+
Work or Derivative Works thereof in any medium, with or without
|
| 92 |
+
modifications, and in Source or Object form, provided that You
|
| 93 |
+
meet the following conditions:
|
| 94 |
+
|
| 95 |
+
(a) You must give any other recipients of the Work or
|
| 96 |
+
Derivative Works a copy of this License; and
|
| 97 |
+
|
| 98 |
+
(b) You must cause any modified files to carry prominent notices
|
| 99 |
+
stating that You changed the files; and
|
| 100 |
+
|
| 101 |
+
(c) You must retain, in the Source form of any Derivative Works
|
| 102 |
+
that You distribute, all copyright, patent, trademark, and
|
| 103 |
+
attribution notices from the Source form of the Work,
|
| 104 |
+
excluding those notices that do not pertain to any part of
|
| 105 |
+
the Derivative Works; and
|
| 106 |
+
|
| 107 |
+
(d) If the Work includes a "NOTICE" text file as part of its
|
| 108 |
+
distribution, then any Derivative Works that You distribute must
|
| 109 |
+
include a readable copy of the attribution notices contained
|
| 110 |
+
within such NOTICE file, excluding those notices that do not
|
| 111 |
+
pertain to any part of the Derivative Works, in at least one
|
| 112 |
+
of the following places: within a NOTICE text file distributed
|
| 113 |
+
as part of the Derivative Works; within the Source form or
|
| 114 |
+
documentation, if provided along with the Derivative Works; or,
|
| 115 |
+
within a display generated by the Derivative Works, if and
|
| 116 |
+
wherever such third-party notices normally appear. The contents
|
| 117 |
+
of the NOTICE file are for informational purposes only and
|
| 118 |
+
do not modify the License. You may add Your own attribution
|
| 119 |
+
notices within Derivative Works that You distribute, alongside
|
| 120 |
+
or as an addendum to the NOTICE text from the Work, provided
|
| 121 |
+
that such additional attribution notices cannot be construed
|
| 122 |
+
as modifying the License.
|
| 123 |
+
|
| 124 |
+
You may add Your own copyright statement to Your modifications and
|
| 125 |
+
may provide additional or different license terms and conditions
|
| 126 |
+
for use, reproduction, or distribution of Your modifications, or
|
| 127 |
+
for any such Derivative Works as a whole, provided Your use,
|
| 128 |
+
reproduction, and distribution of the Work otherwise complies with
|
| 129 |
+
the conditions stated in this License.
|
| 130 |
+
|
| 131 |
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
| 132 |
+
any Contribution intentionally submitted for inclusion in the Work
|
| 133 |
+
by You to the Licensor shall be under the terms and conditions of
|
| 134 |
+
this License, without any additional terms or conditions.
|
| 135 |
+
Notwithstanding the above, nothing herein shall supersede or modify
|
| 136 |
+
the terms of any separate license agreement you may have executed
|
| 137 |
+
with Licensor regarding such Contributions.
|
| 138 |
+
|
| 139 |
+
6. Trademarks. This License does not grant permission to use the trade
|
| 140 |
+
names, trademarks, service marks, or product names of the Licensor,
|
| 141 |
+
except as required for reasonable and customary use in describing the
|
| 142 |
+
origin of the Work and reproducing the content of the NOTICE file.
|
| 143 |
+
|
| 144 |
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
| 145 |
+
agreed to in writing, Licensor provides the Work (and each
|
| 146 |
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
| 147 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
| 148 |
+
implied, including, without limitation, any warranties or conditions
|
| 149 |
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
| 150 |
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
| 151 |
+
appropriateness of using or redistributing the Work and assume any
|
| 152 |
+
risks associated with Your exercise of permissions under this License.
|
| 153 |
+
|
| 154 |
+
8. Limitation of Liability. In no event and under no legal theory,
|
| 155 |
+
whether in tort (including negligence), contract, or otherwise,
|
| 156 |
+
unless required by applicable law (such as deliberate and grossly
|
| 157 |
+
negligent acts) or agreed to in writing, shall any Contributor be
|
| 158 |
+
liable to You for damages, including any direct, indirect, special,
|
| 159 |
+
incidental, or consequential damages of any character arising as a
|
| 160 |
+
result of this License or out of the use or inability to use the
|
| 161 |
+
Work (including but not limited to damages for loss of goodwill,
|
| 162 |
+
work stoppage, computer failure or malfunction, or any and all
|
| 163 |
+
other commercial damages or losses), even if such Contributor
|
| 164 |
+
has been advised of the possibility of such damages.
|
| 165 |
+
|
| 166 |
+
9. Accepting Warranty or Additional Liability. While redistributing
|
| 167 |
+
the Work or Derivative Works thereof, You may choose to offer,
|
| 168 |
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
| 169 |
+
or other liability obligations and/or rights consistent with this
|
| 170 |
+
License. However, in accepting such obligations, You may act only
|
| 171 |
+
on Your own behalf and on Your sole responsibility, not on behalf
|
| 172 |
+
of any other Contributor, and only if You agree to indemnify,
|
| 173 |
+
defend, and hold each Contributor harmless for any liability
|
| 174 |
+
incurred by, or claims asserted against, such Contributor by reason
|
| 175 |
+
of your accepting any such warranty or additional liability.
|
| 176 |
+
|
| 177 |
+
END OF TERMS AND CONDITIONS
|
| 178 |
+
|
| 179 |
+
APPENDIX: How to apply the Apache License to your work.
|
| 180 |
+
|
| 181 |
+
To apply the Apache License to your work, attach the following
|
| 182 |
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
| 183 |
+
replaced with your own identifying information. (Don't include
|
| 184 |
+
the brackets!) The text should be enclosed in the appropriate
|
| 185 |
+
comment syntax for the file format. We also recommend that a
|
| 186 |
+
file or class name and description of purpose be included on the
|
| 187 |
+
same "printed page" as the copyright notice for easier
|
| 188 |
+
identification within third-party archives.
|
| 189 |
+
|
| 190 |
+
Copyright [yyyy] [name of copyright owner]
|
| 191 |
+
|
| 192 |
+
Licensed under the Apache License, Version 2.0 (the "License");
|
| 193 |
+
you may not use this file except in compliance with the License.
|
| 194 |
+
You may obtain a copy of the License at
|
| 195 |
+
|
| 196 |
+
http://www.apache.org/licenses/LICENSE-2.0
|
| 197 |
+
|
| 198 |
+
Unless required by applicable law or agreed to in writing, software
|
| 199 |
+
distributed under the License is distributed on an "AS IS" BASIS,
|
| 200 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 201 |
+
See the License for the specific language governing permissions and
|
| 202 |
+
limitations under the License.
|
baseline/fastwam_zipper/models/Wan2.2-TI2V-5B/Wan2.2_VAE.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:20eb789667fa5e60e7516bf509512f6cb61f01b0aa0695eadaea930c13892b36
|
| 3 |
+
size 2818839170
|
baseline/fastwam_zipper/norm_stats.json
ADDED
|
@@ -0,0 +1,136 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"norm_stats": {
|
| 3 |
+
"state": {
|
| 4 |
+
"mean": [
|
| 5 |
+
-0.32184191477392804,
|
| 6 |
+
1.5207721980826827,
|
| 7 |
+
-0.7150242090298449,
|
| 8 |
+
-0.28362794490829607,
|
| 9 |
+
-0.0326558641526937,
|
| 10 |
+
-0.013098741542747098,
|
| 11 |
+
64.66151715104193,
|
| 12 |
+
0.21104983370396177,
|
| 13 |
+
1.7986193640856323,
|
| 14 |
+
-1.03603124575021,
|
| 15 |
+
-0.2070677500803398,
|
| 16 |
+
0.49687352638322985,
|
| 17 |
+
0.6771545717346843,
|
| 18 |
+
62.77788487973063
|
| 19 |
+
],
|
| 20 |
+
"std": [
|
| 21 |
+
0.3021247081483893,
|
| 22 |
+
1.0365802698172673,
|
| 23 |
+
0.6522747017660155,
|
| 24 |
+
0.4776692517096447,
|
| 25 |
+
0.17428160840573853,
|
| 26 |
+
0.21532609078838216,
|
| 27 |
+
43.57897586492507,
|
| 28 |
+
0.1853564774548537,
|
| 29 |
+
1.0606944989455414,
|
| 30 |
+
0.7407364588573183,
|
| 31 |
+
0.45290236233056375,
|
| 32 |
+
0.41556147349584566,
|
| 33 |
+
0.6096365556465444,
|
| 34 |
+
43.38786400775888
|
| 35 |
+
],
|
| 36 |
+
"q01": [
|
| 37 |
+
-0.9572340250015259,
|
| 38 |
+
0.2140425592660904,
|
| 39 |
+
-2.039149045944214,
|
| 40 |
+
-1.4893616437911987,
|
| 41 |
+
-0.5359574556350708,
|
| 42 |
+
-0.7127659320831299,
|
| 43 |
+
3.0947201251983643,
|
| 44 |
+
-0.0908510610461235,
|
| 45 |
+
0.21531914174556732,
|
| 46 |
+
-2.0451064109802246,
|
| 47 |
+
-1.52042555809021,
|
| 48 |
+
-0.1923404186964035,
|
| 49 |
+
-0.08297872543334961,
|
| 50 |
+
3.2300286293029785
|
| 51 |
+
],
|
| 52 |
+
"q99": [
|
| 53 |
+
0.10489361733198166,
|
| 54 |
+
3.0346808433532715,
|
| 55 |
+
-0.0019148936262354255,
|
| 56 |
+
0.6214893460273743,
|
| 57 |
+
0.31340426206588745,
|
| 58 |
+
0.7790850949287501,
|
| 59 |
+
100.67488861083984,
|
| 60 |
+
0.6489361524581909,
|
| 61 |
+
3.085106372833252,
|
| 62 |
+
-0.001276595750823617,
|
| 63 |
+
0.7149106299877201,
|
| 64 |
+
1.212978720664978,
|
| 65 |
+
1.943404197692871,
|
| 66 |
+
101.49044036865234
|
| 67 |
+
]
|
| 68 |
+
},
|
| 69 |
+
"actions": {
|
| 70 |
+
"mean": [
|
| 71 |
+
-0.3302394307611646,
|
| 72 |
+
1.3351088878676973,
|
| 73 |
+
-0.730087379438977,
|
| 74 |
+
-0.3051273035432853,
|
| 75 |
+
-0.05725393898553614,
|
| 76 |
+
-0.03795729405196406,
|
| 77 |
+
63.83399157797273,
|
| 78 |
+
0.1917094052418155,
|
| 79 |
+
1.6079172755673306,
|
| 80 |
+
-1.0522424038321934,
|
| 81 |
+
-0.22567802180755175,
|
| 82 |
+
0.4700642734741484,
|
| 83 |
+
0.6497894015838603,
|
| 84 |
+
59.816608312332825
|
| 85 |
+
],
|
| 86 |
+
"std": [
|
| 87 |
+
0.2976434695253568,
|
| 88 |
+
1.074422013023125,
|
| 89 |
+
0.6661020136243821,
|
| 90 |
+
0.4836939282841582,
|
| 91 |
+
0.17474814803096714,
|
| 92 |
+
0.2161063544717659,
|
| 93 |
+
47.573759196581726,
|
| 94 |
+
0.18222145953683355,
|
| 95 |
+
1.1005178847180743,
|
| 96 |
+
0.7551834604102067,
|
| 97 |
+
0.4582503537690994,
|
| 98 |
+
0.4172776054469184,
|
| 99 |
+
0.6124397993031185,
|
| 100 |
+
48.446447570698396
|
| 101 |
+
],
|
| 102 |
+
"q01": [
|
| 103 |
+
-0.9526020884513855,
|
| 104 |
+
0.0,
|
| 105 |
+
-2.0,
|
| 106 |
+
-1.5201749801635742,
|
| 107 |
+
-0.5614369511604309,
|
| 108 |
+
-0.7378447651863098,
|
| 109 |
+
0.0,
|
| 110 |
+
-0.12118448317050934,
|
| 111 |
+
0.0,
|
| 112 |
+
-2.0,
|
| 113 |
+
-1.5707999467849731,
|
| 114 |
+
-0.22856314480304718,
|
| 115 |
+
-0.13499030470848083,
|
| 116 |
+
0.0
|
| 117 |
+
],
|
| 118 |
+
"q99": [
|
| 119 |
+
0.10124272853136063,
|
| 120 |
+
3.0096702575683594,
|
| 121 |
+
0.0,
|
| 122 |
+
0.607456386089325,
|
| 123 |
+
0.29452431201934814,
|
| 124 |
+
0.754718542098999,
|
| 125 |
+
100.0,
|
| 126 |
+
0.6181942820549011,
|
| 127 |
+
3.0142722129821777,
|
| 128 |
+
0.0,
|
| 129 |
+
0.7148350477218628,
|
| 130 |
+
1.1888351440429688,
|
| 131 |
+
1.9190099239349365,
|
| 132 |
+
100.27293395996094
|
| 133 |
+
]
|
| 134 |
+
}
|
| 135 |
+
}
|
| 136 |
+
}
|
baseline/fastwam_zipper/provenance.json
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"training_job": "bifrost-2026102616142400-shouqx",
|
| 3 |
+
"task": "zipper_pulling",
|
| 4 |
+
"final_step": 16380,
|
| 5 |
+
"epochs": 30,
|
| 6 |
+
"filename": "step_016380.pt",
|
| 7 |
+
"bytes": 12041813369,
|
| 8 |
+
"sha256": "5e574c8cc7bb5747852cd815cfed9896d8d0f1db83eee28a367bd21beb25f348",
|
| 9 |
+
"checkpoint_keys": [
|
| 10 |
+
"mot",
|
| 11 |
+
"step",
|
| 12 |
+
"torch_dtype",
|
| 13 |
+
"proprio_encoder"
|
| 14 |
+
],
|
| 15 |
+
"torch_dtype": "torch.bfloat16",
|
| 16 |
+
"tensor_elements": 6020749518,
|
| 17 |
+
"training_status": "JOB_COMPLETE",
|
| 18 |
+
"uploaded_as": "original unchanged final weights",
|
| 19 |
+
"verified_utc": "2026-10-27T09:34:00.717440+00:00"
|
| 20 |
+
}
|
baseline/fastwam_zipper/requirements-inference.txt
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
torch==2.7.1
|
| 2 |
+
torchvision==0.22.1
|
| 3 |
+
numpy==2.2.6
|
| 4 |
+
pillow==12.0.0
|
| 5 |
+
einops==0.8.1
|
| 6 |
+
safetensors==0.5.3
|
| 7 |
+
transformers==4.49.0
|
| 8 |
+
regex==2025.11.3
|
| 9 |
+
imageio==2.37.0
|
| 10 |
+
imageio-ffmpeg==0.6.0
|
| 11 |
+
tqdm==4.68.3
|
| 12 |
+
rich==14.2.0
|
baseline/fastwam_zipper/source/LICENSE
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 The FastWAM Authors
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
Except for third-party code and files that retain their original license
|
| 16 |
+
headers, this repository is released under the MIT License.
|
| 17 |
+
|
| 18 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 19 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 20 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 21 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 22 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 23 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 24 |
+
SOFTWARE.
|
baseline/fastwam_zipper/source/README.md
ADDED
|
@@ -0,0 +1,437 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# FastWAM
|
| 2 |
+
|
| 3 |
+
Official codebase for **Fast-WAM: Do World Action Models Need Test-time Future Imagination?**
|
| 4 |
+
|
| 5 |
+
[](./README.md)
|
| 6 |
+
[](./README_zh.md)
|
| 7 |
+
|
| 8 |
+
[](https://arxiv.org/abs/2603.16666)
|
| 9 |
+
[](https://yuantianyuan01.github.io/FastWAM/)
|
| 10 |
+
[](https://huggingface.co/yuanty/fastwam)
|
| 11 |
+
[](https://huggingface.co/datasets/yuanty/LIBERO-fastwam)
|
| 12 |
+
[](https://huggingface.co/datasets/yuanty/robotwin2.0-fastwam)
|
| 13 |
+
|
| 14 |
+
This repository contains the training and evaluation code for FastWAM on LIBERO / RoboTwin.
|
| 15 |
+
|
| 16 |
+
## What's New
|
| 17 |
+
|
| 18 |
+
FastWAM is now faster, better suited to large-scale datasets, and more flexible
|
| 19 |
+
for research. This update brings substantially faster training and inference,
|
| 20 |
+
native LeRobot v3.0 support, and a new model that can switch between acting with
|
| 21 |
+
and without future imagination.
|
| 22 |
+
|
| 23 |
+
### ⚡ Approximately 2x faster inference
|
| 24 |
+
|
| 25 |
+
End-to-end FastWAM inference is now approximately **2x faster**, including text
|
| 26 |
+
encoding and VAE encoding:
|
| 27 |
+
|
| 28 |
+
- **NVIDIA H20:** 470 ms → 210 ms
|
| 29 |
+
- **NVIDIA RTX 4090:** 190 ms → 110 ms
|
| 30 |
+
|
| 31 |
+
The accelerated path is enabled by default for LIBERO with
|
| 32 |
+
`EVALUATION.compile_action_infer=true`. We gratefully acknowledge
|
| 33 |
+
[PR #43](https://github.com/yuantianyuan01/FastWAM/pull/43) for proposing the
|
| 34 |
+
optimization ideas that inspired this work. Existing checkpoints remain fully
|
| 35 |
+
compatible with the accelerated inference path.
|
| 36 |
+
|
| 37 |
+
### 🚀 Approximately 10% faster training
|
| 38 |
+
|
| 39 |
+
FastWAM training is approximately **10% faster on NVIDIA H20 GPUs**. The new
|
| 40 |
+
training path combines a compiled denoising core with batched VAE encoding and
|
| 41 |
+
a lightweight CUDA Graph backend. Enable denoising compilation with:
|
| 42 |
+
|
| 43 |
+
```bash
|
| 44 |
+
bash scripts/train_zero1.sh 8 task=libero_uncond_2cam224_1e-4 \
|
| 45 |
+
model.compile_training_denoise=true
|
| 46 |
+
```
|
| 47 |
+
|
| 48 |
+
Both cached text embeddings and on-the-fly T5 encoding are supported. The
|
| 49 |
+
latter skips text-cache preprocessing and is more convenient, at the cost of
|
| 50 |
+
approximately 10% lower training throughput.
|
| 51 |
+
|
| 52 |
+
### 📦 Native LeRobot 2.1 and 3.0 support
|
| 53 |
+
|
| 54 |
+
FastWAM now supports both **LeRobot 2.1 and LeRobot 3.0** datasets. LeRobot 3.0's
|
| 55 |
+
chunked parquet and video layout scales better to large datasets, with faster
|
| 56 |
+
data loading and dataset-statistics computation as the dataset grows.
|
| 57 |
+
|
| 58 |
+
Download the released LeRobot 3.0 LIBERO dataset from
|
| 59 |
+
[Hugging Face](https://huggingface.co/datasets/yuanty/LIBERO-fastwam) and select
|
| 60 |
+
the v3.0 data config:
|
| 61 |
+
|
| 62 |
+
```bash
|
| 63 |
+
huggingface-cli download yuanty/LIBERO-fastwam \
|
| 64 |
+
--repo-type dataset \
|
| 65 |
+
--include "lerobot_v30/**" \
|
| 66 |
+
--local-dir ./data
|
| 67 |
+
|
| 68 |
+
python scripts/train.py task=libero_uncond_2cam224_1e-4 \
|
| 69 |
+
data=libero_2cam_lerobot_v30
|
| 70 |
+
```
|
| 71 |
+
|
| 72 |
+
For another LeRobot 3.0 dataset, copy
|
| 73 |
+
`configs/data/libero_2cam_lerobot_v30.yaml`, update `train.dataset_dirs`, and
|
| 74 |
+
select the new config with `data=<config_name>`. Existing LeRobot 2.1 configs
|
| 75 |
+
continue to work unchanged.
|
| 76 |
+
|
| 77 |
+
### 🧠 Optional IDM: one model, two thinking modes
|
| 78 |
+
|
| 79 |
+
Optional IDM is a new FastWAM variant that supports **two inference modes in a
|
| 80 |
+
single model**:
|
| 81 |
+
|
| 82 |
+
- **IDM mode:** imagine the future video first, then predict actions.
|
| 83 |
+
- **First-frame mode (Fast-WAM):** skip test-time future imagination and predict
|
| 84 |
+
actions directly from the current observation.
|
| 85 |
+
|
| 86 |
+
Download the released Optional IDM checkpoint from
|
| 87 |
+
[Hugging Face](https://huggingface.co/yuanty/fastwam):
|
| 88 |
+
|
| 89 |
+
```bash
|
| 90 |
+
huggingface-cli download yuanty/fastwam \
|
| 91 |
+
libero_optional_idm_2cam224.pt \
|
| 92 |
+
libero_optional_idm_2cam224_dataset_stats.json \
|
| 93 |
+
--local-dir ./checkpoints/fastwam_release
|
| 94 |
+
```
|
| 95 |
+
|
| 96 |
+
Train the optional-IDM variant once:
|
| 97 |
+
|
| 98 |
+
```bash
|
| 99 |
+
bash scripts/train_zero1.sh 8 task=libero_optional_idm_2cam224_1e-4
|
| 100 |
+
```
|
| 101 |
+
|
| 102 |
+
Then choose either inference mode at evaluation time without retraining, making
|
| 103 |
+
it easy to study when future imagination helps:
|
| 104 |
+
|
| 105 |
+
```bash
|
| 106 |
+
python experiments/libero/run_libero_manager.py \
|
| 107 |
+
task=libero_optional_idm_2cam224_1e-4 \
|
| 108 |
+
ckpt=./checkpoints/fastwam_release/libero_optional_idm_2cam224.pt \
|
| 109 |
+
EVALUATION.dataset_stats_path=./checkpoints/fastwam_release/libero_optional_idm_2cam224_dataset_stats.json \
|
| 110 |
+
EVALUATION.sigma_shift=1.0 \
|
| 111 |
+
+EVALUATION.action_infer_mode=idm \
|
| 112 |
+
MULTIRUN.num_gpus=8
|
| 113 |
+
```
|
| 114 |
+
|
| 115 |
+
Replace `idm` with `first_frame` to use the Fast-WAM inference mode.
|
| 116 |
+
|
| 117 |
+
The released checkpoint, trained with action scheduler shift `1.0`, achieves
|
| 118 |
+
the following success rates on the full LIBERO benchmark (40 tasks, 50 episodes
|
| 119 |
+
per task):
|
| 120 |
+
|
| 121 |
+
| Inference mode | Spatial | Goal | Object | Long | Average |
|
| 122 |
+
| --- | ---: | ---: | ---: | ---: | ---: |
|
| 123 |
+
| IDM | 99.0% | 98.6% | 99.6% | 97.0% | **98.55%** |
|
| 124 |
+
| First-frame (Fast-WAM) | 98.2% | 97.8% | 99.2% | 95.8% | **97.75%** |
|
| 125 |
+
|
| 126 |
+
### Other improvements
|
| 127 |
+
|
| 128 |
+
- The action scheduler shift now defaults to `1.0` for both training and
|
| 129 |
+
evaluation; shifts from `1.0` to `3.0` perform similarly in our experiments.
|
| 130 |
+
When evaluating the original released checkpoints, set
|
| 131 |
+
`EVALUATION.sigma_shift=5.0` to reproduce the original setting.
|
| 132 |
+
- Upgraded LIBERO evaluation with persistent model workers, dynamic task
|
| 133 |
+
scheduling, bad-GPU quarantine, failure recovery, and resumable results.
|
| 134 |
+
- Optimized action-only inference for IDM and Optional IDM: `infer_action`
|
| 135 |
+
returns actions and video latents directly, while VAE decoding is performed
|
| 136 |
+
only by `infer_joint` when video output is requested, eliminating redundant
|
| 137 |
+
computation in action-only deployments.
|
| 138 |
+
|
| 139 |
+
## Index
|
| 140 |
+
|
| 141 |
+
- [File Structure](#file-structure)
|
| 142 |
+
- [Environment Setup](#environment-setup)
|
| 143 |
+
- [Model Preparation](#model-preparation)
|
| 144 |
+
- [Dataset Download](#dataset-download)
|
| 145 |
+
- [Inference with Released Checkpoints](#inference-with-released-checkpoints)
|
| 146 |
+
- [Training](#training)
|
| 147 |
+
- [Inference with Your Trained Checkpoints](#inference-with-your-trained-checkpoints)
|
| 148 |
+
- [Acknowledgements](#acknowledgements)
|
| 149 |
+
- [BibTeX](#bibtex)
|
| 150 |
+
|
| 151 |
+
## File Structure
|
| 152 |
+
|
| 153 |
+
```text
|
| 154 |
+
FastWAM/
|
| 155 |
+
├── configs/
|
| 156 |
+
│ ├── data/ # Dataset configs (LIBERO, RoboTwin, etc.)
|
| 157 |
+
│ ├── model/ # Model architecture and component configs
|
| 158 |
+
│ └── task/ # Task-level configs (training task names)
|
| 159 |
+
├── scripts/
|
| 160 |
+
│ ├── train.py
|
| 161 |
+
│ ├── train_zero1.sh # Deepspeed zero1 training entrypoint
|
| 162 |
+
│ ├── preprocess_action_dit_backbone.py # Preprocess ActionDiT backbone before training
|
| 163 |
+
│ └── precompute_text_embeds.py # Precompute T5 text embedding cache before training
|
| 164 |
+
├── experiments/
|
| 165 |
+
│ ├── libero/
|
| 166 |
+
│ │ └── run_libero_manager.py
|
| 167 |
+
│ └── robotwin/
|
| 168 |
+
│ └── run_robotwin_manager.py
|
| 169 |
+
├── src/fastwam/ # Core code
|
| 170 |
+
├── runs/ # Training outputs (ckpt, logs)
|
| 171 |
+
├── checkpoints/ # Pretrained or external checkpoints
|
| 172 |
+
├── data/ # Data directory
|
| 173 |
+
└── evaluate_results/ # Inference / evaluation results
|
| 174 |
+
```
|
| 175 |
+
|
| 176 |
+
## Environment Setup
|
| 177 |
+
|
| 178 |
+
```bash
|
| 179 |
+
conda create -n fastwam python=3.10 -y
|
| 180 |
+
conda activate fastwam
|
| 181 |
+
pip install -U pip
|
| 182 |
+
pip install torch==2.7.1+cu128 torchvision==0.22.1+cu128 --extra-index-url https://download.pytorch.org/whl/cu128
|
| 183 |
+
pip install -e .
|
| 184 |
+
```
|
| 185 |
+
|
| 186 |
+
## Model Preparation
|
| 187 |
+
|
| 188 |
+
This step is required before both training and inference.
|
| 189 |
+
|
| 190 |
+
Step 1: set the Wan model directory first (opional, default `./checkpoints`):
|
| 191 |
+
|
| 192 |
+
```bash
|
| 193 |
+
mkdir -p checkpoints
|
| 194 |
+
export DIFFSYNTH_MODEL_BASE_PATH="$(pwd)/checkpoints"
|
| 195 |
+
```
|
| 196 |
+
|
| 197 |
+
Step 2: pre-generate the ActionDiT backbone (interpolated from Wan22 DiT):
|
| 198 |
+
|
| 199 |
+
```bash
|
| 200 |
+
# uncond (fastwam)
|
| 201 |
+
python scripts/preprocess_action_dit_backbone.py \
|
| 202 |
+
--model-config configs/model/fastwam.yaml \
|
| 203 |
+
--output checkpoints/ActionDiT_linear_interp_Wan22_alphascale_1024hdim.pt \
|
| 204 |
+
--device cuda \
|
| 205 |
+
--dtype bfloat16
|
| 206 |
+
```
|
| 207 |
+
|
| 208 |
+
## Dataset Download
|
| 209 |
+
|
| 210 |
+
### LIBERO
|
| 211 |
+
|
| 212 |
+
The preprocessed LIBERO dataset used by Fast-WAM is available at:
|
| 213 |
+
|
| 214 |
+
- https://huggingface.co/datasets/yuanty/LIBERO-fastwam
|
| 215 |
+
|
| 216 |
+
Download all compressed files first, then extract them all:
|
| 217 |
+
|
| 218 |
+
```bash
|
| 219 |
+
mkdir -p data/libero_mujoco3.3.2
|
| 220 |
+
cd data/libero_mujoco3.3.2
|
| 221 |
+
|
| 222 |
+
# Run after downloading all 4 tar.gz files
|
| 223 |
+
for f in *.tar.gz; do
|
| 224 |
+
tar -xzf "$f"
|
| 225 |
+
done
|
| 226 |
+
```
|
| 227 |
+
|
| 228 |
+
The extracted directory structure should be:
|
| 229 |
+
|
| 230 |
+
```text
|
| 231 |
+
data/libero_mujoco3.3.2/
|
| 232 |
+
├── libero_10_no_noops_lerobot/
|
| 233 |
+
├── libero_goal_no_noops_lerobot/
|
| 234 |
+
├── libero_object_no_noops_lerobot/
|
| 235 |
+
└── libero_spatial_no_noops_lerobot/
|
| 236 |
+
```
|
| 237 |
+
|
| 238 |
+
### RoboTwin
|
| 239 |
+
|
| 240 |
+
The preprocessed RoboTwin dataset used by Fast-WAM is available at:
|
| 241 |
+
|
| 242 |
+
- https://huggingface.co/datasets/yuanty/robotwin2.0-fastwam
|
| 243 |
+
|
| 244 |
+
Download all split archive files first, then concatenate and extract:
|
| 245 |
+
|
| 246 |
+
```bash
|
| 247 |
+
mkdir -p data/robotwin2.0
|
| 248 |
+
cd data/robotwin2.0
|
| 249 |
+
|
| 250 |
+
# Run after downloading all robotwin2.0.tar.gz.part-* files
|
| 251 |
+
cat robotwin2.0.tar.gz.part-* | tar -xzf -
|
| 252 |
+
```
|
| 253 |
+
|
| 254 |
+
The extracted directory structure should be:
|
| 255 |
+
|
| 256 |
+
```text
|
| 257 |
+
data/robotwin2.0/
|
| 258 |
+
└── robotwin2.0/
|
| 259 |
+
├── data/
|
| 260 |
+
├── meta/
|
| 261 |
+
└── videos/
|
| 262 |
+
```
|
| 263 |
+
|
| 264 |
+
If you also keep:
|
| 265 |
+
|
| 266 |
+
```text
|
| 267 |
+
data/robotwin2.0/dataset_stats.json
|
| 268 |
+
```
|
| 269 |
+
|
| 270 |
+
in the root directory, it can be used directly as the statistics file for the current configs in this repo. You can also recompute it.
|
| 271 |
+
|
| 272 |
+
## Inference with Released Checkpoints
|
| 273 |
+
|
| 274 |
+
The released checkpoints and their corresponding dataset stats are available on [Hugging Face](https://huggingface.co/yuanty/fastwam).
|
| 275 |
+
|
| 276 |
+
Optional: download released checkpoints and dataset stats from Hugging Face:
|
| 277 |
+
|
| 278 |
+
```bash
|
| 279 |
+
pip install -U huggingface_hub
|
| 280 |
+
|
| 281 |
+
huggingface-cli download yuanty/fastwam \
|
| 282 |
+
libero_uncond_2cam224.pt \
|
| 283 |
+
libero_uncond_2cam224_dataset_stats.json \
|
| 284 |
+
libero_optional_idm_2cam224.pt \
|
| 285 |
+
libero_optional_idm_2cam224_dataset_stats.json \
|
| 286 |
+
robotwin_uncond_3cam_384.pt \
|
| 287 |
+
robotwin_uncond_3cam_384_dataset_stats.json \
|
| 288 |
+
--local-dir ./checkpoints/fastwam_release
|
| 289 |
+
```
|
| 290 |
+
|
| 291 |
+
After downloading, the local directory is expected to contain:
|
| 292 |
+
|
| 293 |
+
```text
|
| 294 |
+
checkpoints/fastwam_release/
|
| 295 |
+
├── libero_uncond_2cam224.pt
|
| 296 |
+
├── libero_uncond_2cam224_dataset_stats.json
|
| 297 |
+
├── libero_optional_idm_2cam224.pt
|
| 298 |
+
├── libero_optional_idm_2cam224_dataset_stats.json
|
| 299 |
+
├── robotwin_uncond_3cam_384.pt
|
| 300 |
+
└── robotwin_uncond_3cam_384_dataset_stats.json
|
| 301 |
+
```
|
| 302 |
+
|
| 303 |
+
Before running the `LIBERO` benchmark, install the official LIBERO environment first
|
| 304 |
+
from the [LIBERO repository](https://github.com/Lifelong-Robot-Learning/LIBERO).
|
| 305 |
+
Then run this final step:
|
| 306 |
+
|
| 307 |
+
```bash
|
| 308 |
+
pip install mujoco==3.3.2
|
| 309 |
+
```
|
| 310 |
+
|
| 311 |
+
The `mujoco` environment should ideally stay consistent with the LIBERO data version.
|
| 312 |
+
|
| 313 |
+
We have already copied the `RoboTwin` evaluation-related code into `third_party/RoboTwin`.
|
| 314 |
+
You still need to follow the official RoboTwin instructions from the
|
| 315 |
+
[RoboTwin repository](https://github.com/RoboTwin-Platform/RoboTwin) to finish environment installation and download the required assets, then create the policy symlink:
|
| 316 |
+
|
| 317 |
+
```bash
|
| 318 |
+
ln -sfn "$(pwd)/experiments/robotwin/fastwam_policy" "$(pwd)/third_party/RoboTwin/policy/fastwam_policy"
|
| 319 |
+
```
|
| 320 |
+
|
| 321 |
+
Optional: evaluate released LIBERO checkpoint:
|
| 322 |
+
|
| 323 |
+
The released `LIBERO` / `RoboTwin` evaluation managers default to `8` GPUs
|
| 324 |
+
(`MULTIRUN.num_gpus=8` in `configs/sim_libero.yaml` and `configs/sim_robotwin.yaml`).
|
| 325 |
+
If you want to evaluate with fewer GPUs, pass a smaller value such as
|
| 326 |
+
`MULTIRUN.num_gpus=4`.
|
| 327 |
+
|
| 328 |
+
```bash
|
| 329 |
+
python experiments/libero/run_libero_manager.py \
|
| 330 |
+
task=libero_uncond_2cam224_1e-4 \
|
| 331 |
+
ckpt=./checkpoints/fastwam_release/libero_uncond_2cam224.pt \
|
| 332 |
+
EVALUATION.dataset_stats_path=./checkpoints/fastwam_release/libero_uncond_2cam224_dataset_stats.json \
|
| 333 |
+
EVALUATION.sigma_shift=5.0 \
|
| 334 |
+
MULTIRUN.num_gpus=8
|
| 335 |
+
```
|
| 336 |
+
|
| 337 |
+
Optional: evaluate released RoboTwin checkpoint:
|
| 338 |
+
|
| 339 |
+
```bash
|
| 340 |
+
python experiments/robotwin/run_robotwin_manager.py \
|
| 341 |
+
task=robotwin_uncond_3cam_384_1e-4 \
|
| 342 |
+
ckpt=./checkpoints/fastwam_release/robotwin_uncond_3cam_384.pt \
|
| 343 |
+
EVALUATION.dataset_stats_path=./checkpoints/fastwam_release/robotwin_uncond_3cam_384_dataset_stats.json \
|
| 344 |
+
EVALUATION.sigma_shift=5.0 \
|
| 345 |
+
MULTIRUN.num_gpus=8
|
| 346 |
+
```
|
| 347 |
+
|
| 348 |
+
For faster RoboTwin evaluation, we have enabled `EVALUATION.skip_get_obs_within_replan=true` in [`configs/sim_robotwin.yaml`](./configs/sim_robotwin.yaml).
|
| 349 |
+
This skips RGB rendering while consecutively executing an action chunk within one replan window, which speeds up evaluation but makes the saved video look very low-FPS.
|
| 350 |
+
Set it to `false` if you want to save a fully rendered video.
|
| 351 |
+
|
| 352 |
+
**Note:** We evaluate with **unseen** instructions, following Motus. [Lingbot-VA](https://github.com/Robbyant/lingbot-va/blob/661d52a59dc634a650efcd10a79d06bbb17ea81f/evaluation/robotwin/eval_polict_client_openpi.py#L308) uses **seen** instructions instead. You can try `EVALUATION.instruction_type=seen` to use **seen** instructions, which should theoretically improve performance by one or two points.
|
| 353 |
+
|
| 354 |
+
## Training
|
| 355 |
+
|
| 356 |
+
### 1) Precompute T5 embedding cache before training
|
| 357 |
+
|
| 358 |
+
Use `scripts/precompute_text_embeds.py` to precompute embeddings for each training task:
|
| 359 |
+
|
| 360 |
+
```bash
|
| 361 |
+
# LIBERO
|
| 362 |
+
python scripts/precompute_text_embeds.py task=libero_uncond_2cam224_1e-4
|
| 363 |
+
|
| 364 |
+
# RoboTwin
|
| 365 |
+
python scripts/precompute_text_embeds.py task=robotwin_uncond_3cam_384_1e-4
|
| 366 |
+
```
|
| 367 |
+
|
| 368 |
+
For multi-GPU:
|
| 369 |
+
|
| 370 |
+
```bash
|
| 371 |
+
torchrun --standalone --nproc_per_node=8 scripts/precompute_text_embeds.py task=libero_uncond_2cam224_1e-4
|
| 372 |
+
```
|
| 373 |
+
|
| 374 |
+
### 2) Training (using `fastwam` as an example)
|
| 375 |
+
|
| 376 |
+
When running a new task for the first time, set `pretrained_norm_stats` in the corresponding `configs/data/*.yaml` to `null` first.
|
| 377 |
+
After one training run, a `dataset_stats.json` file will be generated in the current run directory (for example, `runs/{task_name}/{run_id}/dataset_stats.json`).
|
| 378 |
+
You can then update `pretrained_norm_stats` to that file path for subsequent runs.
|
| 379 |
+
|
| 380 |
+
```bash
|
| 381 |
+
# LIBERO
|
| 382 |
+
bash scripts/train_zero1.sh 8 task=libero_uncond_2cam224_1e-4
|
| 383 |
+
|
| 384 |
+
# RoboTwin
|
| 385 |
+
bash scripts/train_zero1.sh 8 task=robotwin_uncond_3cam_384_1e-4
|
| 386 |
+
```
|
| 387 |
+
|
| 388 |
+
For LIBERO, we train on a single node with 8 GPUs. For RoboTwin, we use 64 GPUs to accelerate training. You can try reducing the GPU count or training epochs.
|
| 389 |
+
|
| 390 |
+
## Inference with Your Trained Checkpoints
|
| 391 |
+
|
| 392 |
+
The `mujoco` environment should ideally stay consistent with the LIBERO data version. Then run LIBERO evaluation:
|
| 393 |
+
|
| 394 |
+
```bash
|
| 395 |
+
# LIBERO
|
| 396 |
+
python experiments/libero/run_libero_manager.py task={task_name} ckpt={ckpt_path}
|
| 397 |
+
```
|
| 398 |
+
|
| 399 |
+
We have already copied the `RoboTwin` evaluation-related code into `third_party/RoboTwin`.
|
| 400 |
+
You still need to follow the official RoboTwin instructions from the
|
| 401 |
+
[RoboTwin repository](https://github.com/RoboTwin-Platform/RoboTwin).
|
| 402 |
+
Finish installation and download the required assets, then create the policy symlink:
|
| 403 |
+
|
| 404 |
+
```bash
|
| 405 |
+
ln -sfn "$(pwd)/experiments/robotwin/fastwam_policy" "$(pwd)/third_party/RoboTwin/policy/fastwam_policy"
|
| 406 |
+
```
|
| 407 |
+
|
| 408 |
+
Then run RoboTwin evaluation:
|
| 409 |
+
|
| 410 |
+
```bash
|
| 411 |
+
python experiments/robotwin/run_robotwin_manager.py task={task_name} ckpt={ckpt_path}
|
| 412 |
+
```
|
| 413 |
+
|
| 414 |
+
Common `task_name` examples:
|
| 415 |
+
|
| 416 |
+
```text
|
| 417 |
+
libero_uncond_2cam224_1e-4
|
| 418 |
+
robotwin_uncond_3cam_384_1e-4
|
| 419 |
+
```
|
| 420 |
+
|
| 421 |
+
## Acknowledgements
|
| 422 |
+
|
| 423 |
+
The RoboTwin evaluation code in this repository is adapted from the official [RoboTwin repository](https://github.com/RoboTwin-Platform/RoboTwin). We thank the RoboTwin team for releasing their codebase and assets.
|
| 424 |
+
|
| 425 |
+
## BibTeX
|
| 426 |
+
|
| 427 |
+
If you find our work helpful, please consider citing:
|
| 428 |
+
|
| 429 |
+
```bibtex
|
| 430 |
+
@article{yuan2026fastwam,
|
| 431 |
+
title={Fast-WAM: Do World Action Models Need Test-time Future Imagination?},
|
| 432 |
+
author={Tianyuan Yuan and Zibin Dong and Yicheng Liu and Hang Zhao},
|
| 433 |
+
journal={arXiv preprint arXiv:2603.16666},
|
| 434 |
+
year={2026},
|
| 435 |
+
url={https://arxiv.org/abs/2603.16666}
|
| 436 |
+
}
|
| 437 |
+
```
|
baseline/fastwam_zipper/source/configs/data/galaxea_r1_lite_3cam.yaml
ADDED
|
@@ -0,0 +1,87 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
train:
|
| 2 |
+
_target_: fastwam.datasets.lerobot.robot_video_dataset.RobotVideoDataset
|
| 3 |
+
dataset_dirs:
|
| 4 |
+
- ${oc.env:GALAXEA_FASTWAM_TRAIN_DATA,/dataset_rc_o1_p1/qshou@xiaopeng.com/dataset/galaxea_real_baselines_derived/fastwam_lerobot_v2/train}
|
| 5 |
+
shape_meta: &shape_meta
|
| 6 |
+
images:
|
| 7 |
+
- key: head_left
|
| 8 |
+
raw_shape: [3, 480, 640]
|
| 9 |
+
shape: [3, 240, 320]
|
| 10 |
+
- key: wrist_left
|
| 11 |
+
raw_shape: [3, 480, 640]
|
| 12 |
+
shape: [3, 240, 320]
|
| 13 |
+
- key: wrist_right
|
| 14 |
+
raw_shape: [3, 480, 640]
|
| 15 |
+
shape: [3, 240, 320]
|
| 16 |
+
action:
|
| 17 |
+
- key: default
|
| 18 |
+
raw_shape: 14
|
| 19 |
+
shape: 14
|
| 20 |
+
state:
|
| 21 |
+
- key: default
|
| 22 |
+
raw_shape: 14
|
| 23 |
+
shape: 14
|
| 24 |
+
# 25 native 20 Hz steps give the shared 24-action horizon. Sampling every
|
| 25 |
+
# third image yields 9 video frames, satisfying Wan's 4n+1 contract.
|
| 26 |
+
num_frames: 25
|
| 27 |
+
global_sample_stride: 1
|
| 28 |
+
action_video_freq_ratio: 3
|
| 29 |
+
video_size: [384, 320]
|
| 30 |
+
camera_key: null
|
| 31 |
+
val_set_proportion: 0.0
|
| 32 |
+
is_training_set: true
|
| 33 |
+
skip_padding_as_possible: false
|
| 34 |
+
concat_multi_camera: robotwin
|
| 35 |
+
tolerance_s: 0.001
|
| 36 |
+
video_backend: pyav
|
| 37 |
+
processor: &processor
|
| 38 |
+
_target_: fastwam.datasets.lerobot.processors.fastwam_processor.FastWAMProcessor
|
| 39 |
+
shape_meta: ${data.train.shape_meta}
|
| 40 |
+
num_obs_steps: ${data.train.num_frames}
|
| 41 |
+
num_output_cameras: 3
|
| 42 |
+
action_output_dim: 14
|
| 43 |
+
proprio_output_dim: 14
|
| 44 |
+
action_state_transforms:
|
| 45 |
+
- _target_: fastwam.datasets.lerobot.transforms.relative_action.MaskedRelativeJointTransform
|
| 46 |
+
masks:
|
| 47 |
+
default: [true, true, true, true, true, true, false, true, true, true, true, true, true, false]
|
| 48 |
+
use_stepwise_action_norm: false
|
| 49 |
+
norm_default_mode: z-score
|
| 50 |
+
norm_exception_mode: null
|
| 51 |
+
action_state_merger:
|
| 52 |
+
_target_: fastwam.datasets.lerobot.transforms.action_state_merger.ConcatLeftAlign
|
| 53 |
+
train_transforms:
|
| 54 |
+
- _target_: fastwam.datasets.lerobot.transforms.image.ToTensor
|
| 55 |
+
- _target_: torchvision.transforms.Resize
|
| 56 |
+
size: [240, 320]
|
| 57 |
+
val_transforms:
|
| 58 |
+
- _target_: fastwam.datasets.lerobot.transforms.image.ToTensor
|
| 59 |
+
- _target_: torchvision.transforms.Resize
|
| 60 |
+
size: [240, 320]
|
| 61 |
+
text_embedding_cache_dir: ${oc.env:GALAXEA_FASTWAM_TEXT_CACHE,/dataset_rc_o1_p1/qshou@xiaopeng.com/dataset/galaxea_real_baselines_derived/fastwam_text_cache}
|
| 62 |
+
use_text_embed_cache: true
|
| 63 |
+
context_len: 128
|
| 64 |
+
|
| 65 |
+
val:
|
| 66 |
+
_target_: fastwam.datasets.lerobot.robot_video_dataset.RobotVideoDataset
|
| 67 |
+
dataset_dirs:
|
| 68 |
+
- ${oc.env:GALAXEA_FASTWAM_VAL_DATA,/dataset_rc_o1_p1/qshou@xiaopeng.com/dataset/galaxea_real_baselines_derived/fastwam_lerobot_v2/validation}
|
| 69 |
+
shape_meta: *shape_meta
|
| 70 |
+
num_frames: 25
|
| 71 |
+
global_sample_stride: 1
|
| 72 |
+
action_video_freq_ratio: 3
|
| 73 |
+
video_size: [384, 320]
|
| 74 |
+
camera_key: null
|
| 75 |
+
val_set_proportion: 0.0
|
| 76 |
+
is_training_set: false
|
| 77 |
+
skip_padding_as_possible: false
|
| 78 |
+
concat_multi_camera: robotwin
|
| 79 |
+
tolerance_s: 0.001
|
| 80 |
+
video_backend: pyav
|
| 81 |
+
processor:
|
| 82 |
+
<<: *processor
|
| 83 |
+
shape_meta: ${data.val.shape_meta}
|
| 84 |
+
num_obs_steps: ${data.val.num_frames}
|
| 85 |
+
text_embedding_cache_dir: ${data.train.text_embedding_cache_dir}
|
| 86 |
+
use_text_embed_cache: true
|
| 87 |
+
context_len: 128
|
baseline/fastwam_zipper/source/configs/data/libero_2cam.yaml
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
train:
|
| 2 |
+
_target_: fastwam.datasets.lerobot.robot_video_dataset.RobotVideoDataset
|
| 3 |
+
dataset_dirs:
|
| 4 |
+
- ./data/libero_mujoco3.3.2/libero_spatial_no_noops_lerobot
|
| 5 |
+
- ./data/libero_mujoco3.3.2/libero_object_no_noops_lerobot
|
| 6 |
+
- ./data/libero_mujoco3.3.2/libero_goal_no_noops_lerobot
|
| 7 |
+
- ./data/libero_mujoco3.3.2/libero_10_no_noops_lerobot
|
| 8 |
+
shape_meta:
|
| 9 |
+
images:
|
| 10 |
+
- key: image
|
| 11 |
+
# - key: observation.images.image
|
| 12 |
+
raw_shape: [3, 512, 512]
|
| 13 |
+
shape: [3, 224, 224]
|
| 14 |
+
- key: wrist_image
|
| 15 |
+
# - key: observation.images.wrist_image
|
| 16 |
+
raw_shape: [3, 512, 512]
|
| 17 |
+
shape: [3, 224, 224]
|
| 18 |
+
action:
|
| 19 |
+
- key: default
|
| 20 |
+
# - key: action
|
| 21 |
+
raw_shape: 7 # before transform
|
| 22 |
+
shape: 7 # after transform
|
| 23 |
+
state:
|
| 24 |
+
- key: default
|
| 25 |
+
# - key: observation.state
|
| 26 |
+
raw_shape: 8 # before transform
|
| 27 |
+
shape: 8 # after transform
|
| 28 |
+
num_frames: 33
|
| 29 |
+
global_sample_stride: 1
|
| 30 |
+
action_video_freq_ratio: 4 # 32 action, 9 video frames
|
| 31 |
+
video_size: [224, 448] # final resize video
|
| 32 |
+
camera_key: null
|
| 33 |
+
val_set_proportion: 0.0
|
| 34 |
+
is_training_set: true
|
| 35 |
+
skip_padding_as_possible: false
|
| 36 |
+
concat_multi_camera: "horizontal"
|
| 37 |
+
processor:
|
| 38 |
+
_target_: fastwam.datasets.lerobot.processors.fastwam_processor.FastWAMProcessor
|
| 39 |
+
shape_meta: ${data.train.shape_meta}
|
| 40 |
+
num_obs_steps: ${data.train.num_frames}
|
| 41 |
+
num_output_cameras: 2
|
| 42 |
+
action_output_dim: 7 # eef_pose (6) + gripper (1)
|
| 43 |
+
proprio_output_dim: 8 # eef_pose (6) + gripper (2)
|
| 44 |
+
delta_action_dim_mask:
|
| 45 |
+
default: [true, true, true, true, true, true, false] # eef poses are delta, gripper is not
|
| 46 |
+
|
| 47 |
+
# action & state normalization
|
| 48 |
+
action_state_transforms: null
|
| 49 |
+
# action_state_transforms:
|
| 50 |
+
# - _target_: fastwam.datasets.lerobot.transforms.libero.LiberoTransform
|
| 51 |
+
use_stepwise_action_norm: False
|
| 52 |
+
norm_default_mode: min/max
|
| 53 |
+
norm_exception_mode: null
|
| 54 |
+
|
| 55 |
+
action_state_merger:
|
| 56 |
+
_target_: fastwam.datasets.lerobot.transforms.action_state_merger.ConcatLeftAlign
|
| 57 |
+
|
| 58 |
+
train_transforms:
|
| 59 |
+
- _target_: fastwam.datasets.lerobot.transforms.image.ToTensor
|
| 60 |
+
- _target_: torchvision.transforms.Resize
|
| 61 |
+
size: [224, 224]
|
| 62 |
+
val_transforms:
|
| 63 |
+
- _target_: fastwam.datasets.lerobot.transforms.image.ToTensor
|
| 64 |
+
- _target_: torchvision.transforms.Resize
|
| 65 |
+
size: [224, 224]
|
| 66 |
+
text_embedding_cache_dir: ./data/text_embeds_cache/libero
|
| 67 |
+
use_text_embed_cache: true
|
| 68 |
+
context_len: 128
|
baseline/fastwam_zipper/source/configs/data/libero_2cam_lerobot_v30.yaml
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
train:
|
| 2 |
+
_target_: fastwam.datasets.lerobot3.robot_video_dataset.RobotVideoDataset
|
| 3 |
+
dataset_dirs:
|
| 4 |
+
- ./data/lerobot_v30/libero_spatial_no_noops_lerobot
|
| 5 |
+
- ./data/lerobot_v30/libero_object_no_noops_lerobot
|
| 6 |
+
- ./data/lerobot_v30/libero_goal_no_noops_lerobot
|
| 7 |
+
- ./data/lerobot_v30/libero_10_no_noops_lerobot
|
| 8 |
+
shape_meta:
|
| 9 |
+
images:
|
| 10 |
+
- key: image
|
| 11 |
+
raw_shape: [3, 512, 512]
|
| 12 |
+
shape: [3, 224, 224]
|
| 13 |
+
- key: wrist_image
|
| 14 |
+
raw_shape: [3, 512, 512]
|
| 15 |
+
shape: [3, 224, 224]
|
| 16 |
+
action:
|
| 17 |
+
- key: default
|
| 18 |
+
raw_shape: 7
|
| 19 |
+
shape: 7
|
| 20 |
+
state:
|
| 21 |
+
- key: default
|
| 22 |
+
raw_shape: 8
|
| 23 |
+
shape: 8
|
| 24 |
+
num_frames: 33
|
| 25 |
+
global_sample_stride: 1
|
| 26 |
+
action_video_freq_ratio: 4
|
| 27 |
+
video_size: [224, 448]
|
| 28 |
+
camera_key: null
|
| 29 |
+
val_set_proportion: 0.0
|
| 30 |
+
is_training_set: true
|
| 31 |
+
skip_padding_as_possible: false
|
| 32 |
+
concat_multi_camera: "horizontal"
|
| 33 |
+
processor:
|
| 34 |
+
_target_: fastwam.datasets.lerobot.processors.fastwam_processor.FastWAMProcessor
|
| 35 |
+
shape_meta: ${data.train.shape_meta}
|
| 36 |
+
num_obs_steps: ${data.train.num_frames}
|
| 37 |
+
num_output_cameras: 2
|
| 38 |
+
action_output_dim: 7
|
| 39 |
+
proprio_output_dim: 8
|
| 40 |
+
delta_action_dim_mask:
|
| 41 |
+
default: [true, true, true, true, true, true, false]
|
| 42 |
+
action_state_transforms: null
|
| 43 |
+
use_stepwise_action_norm: false
|
| 44 |
+
norm_default_mode: min/max
|
| 45 |
+
norm_exception_mode: null
|
| 46 |
+
action_state_merger:
|
| 47 |
+
_target_: fastwam.datasets.lerobot.transforms.action_state_merger.ConcatLeftAlign
|
| 48 |
+
train_transforms:
|
| 49 |
+
- _target_: fastwam.datasets.lerobot.transforms.image.ToTensor
|
| 50 |
+
- _target_: torchvision.transforms.Resize
|
| 51 |
+
size: [224, 224]
|
| 52 |
+
val_transforms:
|
| 53 |
+
- _target_: fastwam.datasets.lerobot.transforms.image.ToTensor
|
| 54 |
+
- _target_: torchvision.transforms.Resize
|
| 55 |
+
size: [224, 224]
|
| 56 |
+
text_embedding_cache_dir: ./data/text_embeds_cache/libero
|
| 57 |
+
use_text_embed_cache: true
|
| 58 |
+
context_len: 128
|
| 59 |
+
video_backend: pyav
|
baseline/fastwam_zipper/source/configs/data/robotwin.yaml
ADDED
|
@@ -0,0 +1,125 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
train:
|
| 2 |
+
_target_: fastwam.datasets.lerobot.robot_video_dataset.RobotVideoDataset
|
| 3 |
+
dataset_dirs:
|
| 4 |
+
- ./data/robotwin2.0/robotwin2.0
|
| 5 |
+
shape_meta:
|
| 6 |
+
images:
|
| 7 |
+
- key: cam_high
|
| 8 |
+
raw_shape: [3, 480, 640]
|
| 9 |
+
shape: [3, 240, 320]
|
| 10 |
+
- key: cam_left_wrist
|
| 11 |
+
raw_shape: [3, 480, 640]
|
| 12 |
+
shape: [3, 240, 320]
|
| 13 |
+
- key: cam_right_wrist
|
| 14 |
+
raw_shape: [3, 480, 640]
|
| 15 |
+
shape: [3, 240, 320]
|
| 16 |
+
action:
|
| 17 |
+
- key: default
|
| 18 |
+
raw_shape: 14 # before transform
|
| 19 |
+
shape: 14 # after transform
|
| 20 |
+
state:
|
| 21 |
+
- key: default
|
| 22 |
+
raw_shape: 14 # before transform
|
| 23 |
+
shape: 14 # after transform
|
| 24 |
+
num_frames: 33
|
| 25 |
+
global_sample_stride: 1
|
| 26 |
+
action_video_freq_ratio: 4 # 32 action, 9 video frames
|
| 27 |
+
video_size: [384, 320] # final resize video
|
| 28 |
+
camera_key: null
|
| 29 |
+
val_set_proportion: 0.01
|
| 30 |
+
is_training_set: true
|
| 31 |
+
pretrained_norm_stats: ./data/robotwin2.0/dataset_stats.json
|
| 32 |
+
skip_padding_as_possible: false
|
| 33 |
+
concat_multi_camera: "robotwin"
|
| 34 |
+
processor:
|
| 35 |
+
_target_: fastwam.datasets.lerobot.processors.fastwam_processor.FastWAMProcessor
|
| 36 |
+
shape_meta: ${data.train.shape_meta}
|
| 37 |
+
num_obs_steps: ${data.train.num_frames}
|
| 38 |
+
num_output_cameras: 3
|
| 39 |
+
action_output_dim: 14
|
| 40 |
+
proprio_output_dim: 14
|
| 41 |
+
|
| 42 |
+
# action & state normalization
|
| 43 |
+
action_state_transforms: null
|
| 44 |
+
# action_state_transforms:
|
| 45 |
+
# - _target_: fastwam.datasets.lerobot.transforms.libero.LiberoTransform
|
| 46 |
+
use_stepwise_action_norm: False
|
| 47 |
+
norm_default_mode: "z-score"
|
| 48 |
+
norm_exception_mode: null
|
| 49 |
+
|
| 50 |
+
action_state_merger:
|
| 51 |
+
_target_: fastwam.datasets.lerobot.transforms.action_state_merger.ConcatLeftAlign
|
| 52 |
+
|
| 53 |
+
train_transforms:
|
| 54 |
+
- _target_: fastwam.datasets.lerobot.transforms.image.ToTensor
|
| 55 |
+
- _target_: torchvision.transforms.Resize
|
| 56 |
+
size: [240, 320]
|
| 57 |
+
val_transforms:
|
| 58 |
+
- _target_: fastwam.datasets.lerobot.transforms.image.ToTensor
|
| 59 |
+
- _target_: torchvision.transforms.Resize
|
| 60 |
+
size: [240, 320]
|
| 61 |
+
text_embedding_cache_dir: ./data/text_embeds_cache/robotwin
|
| 62 |
+
context_len: 128
|
| 63 |
+
|
| 64 |
+
val:
|
| 65 |
+
_target_: fastwam.datasets.lerobot.robot_video_dataset.RobotVideoDataset
|
| 66 |
+
dataset_dirs:
|
| 67 |
+
- ./data/robotwin2.0/robotwin2.0/
|
| 68 |
+
shape_meta:
|
| 69 |
+
images:
|
| 70 |
+
- key: cam_high
|
| 71 |
+
raw_shape: [3, 480, 640]
|
| 72 |
+
shape: [3, 240, 320]
|
| 73 |
+
- key: cam_left_wrist
|
| 74 |
+
raw_shape: [3, 480, 640]
|
| 75 |
+
shape: [3, 240, 320]
|
| 76 |
+
- key: cam_right_wrist
|
| 77 |
+
raw_shape: [3, 480, 640]
|
| 78 |
+
shape: [3, 240, 320]
|
| 79 |
+
action:
|
| 80 |
+
- key: default
|
| 81 |
+
raw_shape: 14 # before transform
|
| 82 |
+
shape: 14 # after transform
|
| 83 |
+
state:
|
| 84 |
+
- key: default
|
| 85 |
+
raw_shape: 14 # before transform
|
| 86 |
+
shape: 14 # after transform
|
| 87 |
+
num_frames: 33
|
| 88 |
+
global_sample_stride: 1
|
| 89 |
+
action_video_freq_ratio: 4 # 32 action, 9 video frames
|
| 90 |
+
video_size: [384, 320] # final resize video
|
| 91 |
+
camera_key: null
|
| 92 |
+
val_set_proportion: 0.01
|
| 93 |
+
is_training_set: false
|
| 94 |
+
pretrained_norm_stats: ./data/robotwin2.0/dataset_stats.json
|
| 95 |
+
skip_padding_as_possible: false
|
| 96 |
+
concat_multi_camera: "robotwin"
|
| 97 |
+
processor:
|
| 98 |
+
_target_: fastwam.datasets.lerobot.processors.fastwam_processor.FastWAMProcessor
|
| 99 |
+
shape_meta: ${data.train.shape_meta}
|
| 100 |
+
num_obs_steps: ${data.train.num_frames}
|
| 101 |
+
num_output_cameras: 3
|
| 102 |
+
action_output_dim: 14
|
| 103 |
+
proprio_output_dim: 14
|
| 104 |
+
|
| 105 |
+
# action & state normalization
|
| 106 |
+
action_state_transforms: null
|
| 107 |
+
# action_state_transforms:
|
| 108 |
+
# - _target_: fastwam.datasets.lerobot.transforms.libero.LiberoTransform
|
| 109 |
+
use_stepwise_action_norm: False
|
| 110 |
+
norm_default_mode: "z-score"
|
| 111 |
+
norm_exception_mode: null
|
| 112 |
+
|
| 113 |
+
action_state_merger:
|
| 114 |
+
_target_: fastwam.datasets.lerobot.transforms.action_state_merger.ConcatLeftAlign
|
| 115 |
+
|
| 116 |
+
train_transforms:
|
| 117 |
+
- _target_: fastwam.datasets.lerobot.transforms.image.ToTensor
|
| 118 |
+
- _target_: torchvision.transforms.Resize
|
| 119 |
+
size: [240, 320]
|
| 120 |
+
val_transforms:
|
| 121 |
+
- _target_: fastwam.datasets.lerobot.transforms.image.ToTensor
|
| 122 |
+
- _target_: torchvision.transforms.Resize
|
| 123 |
+
size: [240, 320]
|
| 124 |
+
text_embedding_cache_dir: ./data/text_embeds_cache/robotwin
|
| 125 |
+
context_len: 128
|
baseline/fastwam_zipper/source/configs/model/fastwam.yaml
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
_target_: fastwam.runtime.create_fastwam
|
| 2 |
+
model_id: Wan-AI/Wan2.2-TI2V-5B
|
| 3 |
+
tokenizer_model_id: Wan-AI/Wan2.1-T2V-1.3B
|
| 4 |
+
tokenizer_max_len: 128
|
| 5 |
+
load_text_encoder: false
|
| 6 |
+
proprio_dim: ${data.train.processor.proprio_output_dim}
|
| 7 |
+
redirect_common_files: true
|
| 8 |
+
mot_checkpoint_mixed_attn: false
|
| 9 |
+
compile_training_denoise: false
|
| 10 |
+
action_dit_pretrained_path: checkpoints/ActionDiT_linear_interp_Wan22_alphascale_1024hdim.pt
|
| 11 |
+
skip_dit_load_from_pretrain: false
|
| 12 |
+
|
| 13 |
+
video_dit_config:
|
| 14 |
+
has_image_input: false
|
| 15 |
+
patch_size: [1, 2, 2]
|
| 16 |
+
in_dim: 48
|
| 17 |
+
hidden_dim: 3072
|
| 18 |
+
ffn_dim: 14336
|
| 19 |
+
freq_dim: 256
|
| 20 |
+
text_dim: 4096
|
| 21 |
+
out_dim: 48
|
| 22 |
+
num_heads: 24
|
| 23 |
+
attn_head_dim: 128
|
| 24 |
+
num_layers: 30
|
| 25 |
+
eps: 1.0e-06
|
| 26 |
+
seperated_timestep: true
|
| 27 |
+
require_clip_embedding: false
|
| 28 |
+
require_vae_embedding: false
|
| 29 |
+
fuse_vae_embedding_in_latents: true
|
| 30 |
+
use_gradient_checkpointing: ${model.mot_checkpoint_mixed_attn}
|
| 31 |
+
video_attention_mask_mode: "first_frame_causal" # default
|
| 32 |
+
action_conditioned: false
|
| 33 |
+
action_dim: ${data.train.processor.action_output_dim}
|
| 34 |
+
action_group_causal_mask_mode: "group_diagonal"
|
| 35 |
+
|
| 36 |
+
action_dit_config:
|
| 37 |
+
action_dim: ${data.train.processor.action_output_dim}
|
| 38 |
+
hidden_dim: 1024
|
| 39 |
+
ffn_dim: 4096
|
| 40 |
+
num_heads: 24
|
| 41 |
+
attn_head_dim: 128
|
| 42 |
+
num_layers: 30
|
| 43 |
+
text_dim: 4096
|
| 44 |
+
freq_dim: 256
|
| 45 |
+
eps: 1.0e-06
|
| 46 |
+
use_gradient_checkpointing: ${model.mot_checkpoint_mixed_attn}
|
| 47 |
+
|
| 48 |
+
video_scheduler:
|
| 49 |
+
train_shift: 5.0
|
| 50 |
+
infer_shift: 5.0
|
| 51 |
+
num_train_timesteps: 1000
|
| 52 |
+
|
| 53 |
+
action_scheduler:
|
| 54 |
+
train_shift: 1.0
|
| 55 |
+
infer_shift: 1.0
|
| 56 |
+
num_train_timesteps: 1000
|
| 57 |
+
|
| 58 |
+
loss:
|
| 59 |
+
lambda_action: 1.0
|
baseline/fastwam_zipper/source/configs/model/fastwam_idm.yaml
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
_target_: fastwam.runtime.create_fastwam_idm
|
| 2 |
+
model_id: Wan-AI/Wan2.2-TI2V-5B
|
| 3 |
+
tokenizer_model_id: Wan-AI/Wan2.1-T2V-1.3B
|
| 4 |
+
tokenizer_max_len: 128
|
| 5 |
+
load_text_encoder: false
|
| 6 |
+
proprio_dim: ${data.train.processor.proprio_output_dim}
|
| 7 |
+
redirect_common_files: true
|
| 8 |
+
mot_checkpoint_mixed_attn: false
|
| 9 |
+
compile_training_denoise: false
|
| 10 |
+
video_cond_noise_prob: 0.5
|
| 11 |
+
action_dit_pretrained_path: checkpoints/ActionDiT_linear_interp_Wan22_alphascale_1024hdim.pt
|
| 12 |
+
skip_dit_load_from_pretrain: false
|
| 13 |
+
|
| 14 |
+
video_dit_config:
|
| 15 |
+
has_image_input: false
|
| 16 |
+
patch_size: [1, 2, 2]
|
| 17 |
+
in_dim: 48
|
| 18 |
+
hidden_dim: 3072
|
| 19 |
+
ffn_dim: 14336
|
| 20 |
+
freq_dim: 256
|
| 21 |
+
text_dim: 4096
|
| 22 |
+
out_dim: 48
|
| 23 |
+
num_heads: 24
|
| 24 |
+
attn_head_dim: 128
|
| 25 |
+
num_layers: 30
|
| 26 |
+
eps: 1.0e-06
|
| 27 |
+
seperated_timestep: true
|
| 28 |
+
require_clip_embedding: false
|
| 29 |
+
require_vae_embedding: false
|
| 30 |
+
fuse_vae_embedding_in_latents: true
|
| 31 |
+
use_gradient_checkpointing: ${model.mot_checkpoint_mixed_attn}
|
| 32 |
+
video_attention_mask_mode: "first_frame_causal" # default
|
| 33 |
+
action_conditioned: false
|
| 34 |
+
action_dim: ${data.train.processor.action_output_dim}
|
| 35 |
+
action_group_causal_mask_mode: "group_diagonal"
|
| 36 |
+
|
| 37 |
+
action_dit_config:
|
| 38 |
+
action_dim: ${data.train.processor.action_output_dim}
|
| 39 |
+
hidden_dim: 1024
|
| 40 |
+
ffn_dim: 4096
|
| 41 |
+
num_heads: 24
|
| 42 |
+
attn_head_dim: 128
|
| 43 |
+
num_layers: 30
|
| 44 |
+
text_dim: 4096
|
| 45 |
+
freq_dim: 256
|
| 46 |
+
eps: 1.0e-06
|
| 47 |
+
use_gradient_checkpointing: ${model.mot_checkpoint_mixed_attn}
|
| 48 |
+
|
| 49 |
+
video_scheduler:
|
| 50 |
+
train_shift: 5.0
|
| 51 |
+
infer_shift: 5.0
|
| 52 |
+
num_train_timesteps: 1000
|
| 53 |
+
|
| 54 |
+
action_scheduler:
|
| 55 |
+
train_shift: 1.0
|
| 56 |
+
infer_shift: 1.0
|
| 57 |
+
num_train_timesteps: 1000
|
| 58 |
+
|
| 59 |
+
loss:
|
| 60 |
+
lambda_action: 1.0
|
baseline/fastwam_zipper/source/configs/model/fastwam_joint.yaml
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
_target_: fastwam.runtime.create_fastwam_joint
|
| 2 |
+
model_id: Wan-AI/Wan2.2-TI2V-5B
|
| 3 |
+
tokenizer_model_id: Wan-AI/Wan2.1-T2V-1.3B
|
| 4 |
+
tokenizer_max_len: 128
|
| 5 |
+
load_text_encoder: false
|
| 6 |
+
proprio_dim: ${data.train.processor.proprio_output_dim}
|
| 7 |
+
redirect_common_files: true
|
| 8 |
+
mot_checkpoint_mixed_attn: false
|
| 9 |
+
action_dit_pretrained_path: checkpoints/ActionDiT_linear_interp_Wan22_alphascale_1024hdim.pt
|
| 10 |
+
skip_dit_load_from_pretrain: false
|
| 11 |
+
|
| 12 |
+
video_dit_config:
|
| 13 |
+
has_image_input: false
|
| 14 |
+
patch_size: [1, 2, 2]
|
| 15 |
+
in_dim: 48
|
| 16 |
+
hidden_dim: 3072
|
| 17 |
+
ffn_dim: 14336
|
| 18 |
+
freq_dim: 256
|
| 19 |
+
text_dim: 4096
|
| 20 |
+
out_dim: 48
|
| 21 |
+
num_heads: 24
|
| 22 |
+
attn_head_dim: 128
|
| 23 |
+
num_layers: 30
|
| 24 |
+
eps: 1.0e-06
|
| 25 |
+
seperated_timestep: true
|
| 26 |
+
require_clip_embedding: false
|
| 27 |
+
require_vae_embedding: false
|
| 28 |
+
fuse_vae_embedding_in_latents: true
|
| 29 |
+
use_gradient_checkpointing: ${model.mot_checkpoint_mixed_attn}
|
| 30 |
+
video_attention_mask_mode: "first_frame_causal" # default
|
| 31 |
+
action_conditioned: false
|
| 32 |
+
action_dim: ${data.train.processor.action_output_dim}
|
| 33 |
+
action_group_causal_mask_mode: "group_diagonal"
|
| 34 |
+
|
| 35 |
+
action_dit_config:
|
| 36 |
+
action_dim: ${data.train.processor.action_output_dim}
|
| 37 |
+
hidden_dim: 1024
|
| 38 |
+
ffn_dim: 4096
|
| 39 |
+
num_heads: 24
|
| 40 |
+
attn_head_dim: 128
|
| 41 |
+
num_layers: 30
|
| 42 |
+
text_dim: 4096
|
| 43 |
+
freq_dim: 256
|
| 44 |
+
eps: 1.0e-06
|
| 45 |
+
use_gradient_checkpointing: ${model.mot_checkpoint_mixed_attn}
|
| 46 |
+
|
| 47 |
+
video_scheduler:
|
| 48 |
+
train_shift: 5.0
|
| 49 |
+
infer_shift: 5.0
|
| 50 |
+
num_train_timesteps: 1000
|
| 51 |
+
|
| 52 |
+
action_scheduler:
|
| 53 |
+
train_shift: 1.0
|
| 54 |
+
infer_shift: 1.0
|
| 55 |
+
num_train_timesteps: 1000
|
| 56 |
+
|
| 57 |
+
loss:
|
| 58 |
+
lambda_action: 1.0
|
baseline/fastwam_zipper/source/configs/model/fastwam_optional_idm.yaml
ADDED
|
@@ -0,0 +1,62 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
_target_: fastwam.runtime.create_fastwam_optional_idm
|
| 2 |
+
model_id: Wan-AI/Wan2.2-TI2V-5B
|
| 3 |
+
tokenizer_model_id: Wan-AI/Wan2.1-T2V-1.3B
|
| 4 |
+
tokenizer_max_len: 128
|
| 5 |
+
load_text_encoder: false
|
| 6 |
+
proprio_dim: ${data.train.processor.proprio_output_dim}
|
| 7 |
+
redirect_common_files: true
|
| 8 |
+
mot_checkpoint_mixed_attn: false
|
| 9 |
+
compile_training_denoise: false
|
| 10 |
+
action_idm_prob: 0.5
|
| 11 |
+
video_cond_noise_prob: 0.0
|
| 12 |
+
action_dit_pretrained_path: checkpoints/ActionDiT_linear_interp_Wan22_alphascale_1024hdim.pt
|
| 13 |
+
skip_dit_load_from_pretrain: false
|
| 14 |
+
|
| 15 |
+
video_dit_config:
|
| 16 |
+
has_image_input: false
|
| 17 |
+
patch_size: [1, 2, 2]
|
| 18 |
+
in_dim: 48
|
| 19 |
+
hidden_dim: 3072
|
| 20 |
+
ffn_dim: 14336
|
| 21 |
+
freq_dim: 256
|
| 22 |
+
text_dim: 4096
|
| 23 |
+
out_dim: 48
|
| 24 |
+
num_heads: 24
|
| 25 |
+
attn_head_dim: 128
|
| 26 |
+
num_layers: 30
|
| 27 |
+
eps: 1.0e-06
|
| 28 |
+
seperated_timestep: true
|
| 29 |
+
require_clip_embedding: false
|
| 30 |
+
require_vae_embedding: false
|
| 31 |
+
fuse_vae_embedding_in_latents: true
|
| 32 |
+
use_gradient_checkpointing: ${model.mot_checkpoint_mixed_attn}
|
| 33 |
+
video_attention_mask_mode: "first_frame_causal"
|
| 34 |
+
action_conditioned: false
|
| 35 |
+
action_dim: ${data.train.processor.action_output_dim}
|
| 36 |
+
action_group_causal_mask_mode: "group_diagonal"
|
| 37 |
+
|
| 38 |
+
action_dit_config:
|
| 39 |
+
action_dim: ${data.train.processor.action_output_dim}
|
| 40 |
+
hidden_dim: 1024
|
| 41 |
+
ffn_dim: 4096
|
| 42 |
+
num_heads: 24
|
| 43 |
+
attn_head_dim: 128
|
| 44 |
+
num_layers: 30
|
| 45 |
+
text_dim: 4096
|
| 46 |
+
freq_dim: 256
|
| 47 |
+
eps: 1.0e-06
|
| 48 |
+
use_gradient_checkpointing: ${model.mot_checkpoint_mixed_attn}
|
| 49 |
+
|
| 50 |
+
video_scheduler:
|
| 51 |
+
train_shift: 5.0
|
| 52 |
+
infer_shift: 5.0
|
| 53 |
+
num_train_timesteps: 1000
|
| 54 |
+
|
| 55 |
+
action_scheduler:
|
| 56 |
+
train_shift: 1.0
|
| 57 |
+
infer_shift: 1.0
|
| 58 |
+
num_train_timesteps: 1000
|
| 59 |
+
|
| 60 |
+
loss:
|
| 61 |
+
lambda_video: 1.0
|
| 62 |
+
lambda_action: 1.0
|
baseline/fastwam_zipper/source/configs/sim_libero.yaml
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# @package _global_
|
| 2 |
+
|
| 3 |
+
defaults:
|
| 4 |
+
- train
|
| 5 |
+
- override /task: libero_uncond_2cam224_1e-4
|
| 6 |
+
- _self_
|
| 7 |
+
|
| 8 |
+
ckpt: null
|
| 9 |
+
gpu_id: 0
|
| 10 |
+
|
| 11 |
+
model:
|
| 12 |
+
load_text_encoder: true
|
| 13 |
+
skip_dit_load_from_pretrain: true
|
| 14 |
+
action_dit_pretrained_path: null
|
| 15 |
+
|
| 16 |
+
EVALUATION:
|
| 17 |
+
task_suite_name: libero_spatial
|
| 18 |
+
task_id: 0
|
| 19 |
+
num_trials: 50
|
| 20 |
+
output_dir: ./evaluate_results/libero/${hydra:runtime.choices.task}/${now:%Y%m%d_%H%M%S}
|
| 21 |
+
|
| 22 |
+
# Runtime behavior
|
| 23 |
+
env_num: 1
|
| 24 |
+
num_steps_wait: 30
|
| 25 |
+
replan_steps: 10
|
| 26 |
+
binarize_gripper: true
|
| 27 |
+
use_action_ensembler: false
|
| 28 |
+
visualize_future_video: false
|
| 29 |
+
|
| 30 |
+
# Action inference
|
| 31 |
+
action_horizon: null
|
| 32 |
+
num_inference_steps: ${eval_num_inference_steps}
|
| 33 |
+
sigma_shift: null
|
| 34 |
+
text_cfg_scale: 1.0
|
| 35 |
+
negative_prompt: ""
|
| 36 |
+
rand_device: cpu
|
| 37 |
+
tiled: false
|
| 38 |
+
compile_action_infer: true
|
| 39 |
+
|
| 40 |
+
# Optional external inputs
|
| 41 |
+
dataset_stats_path: null
|
| 42 |
+
device: cuda
|
| 43 |
+
|
| 44 |
+
MULTIRUN:
|
| 45 |
+
create_only: false
|
| 46 |
+
task_file: null
|
| 47 |
+
task_suite_names:
|
| 48 |
+
- libero_10
|
| 49 |
+
- libero_goal
|
| 50 |
+
- libero_spatial
|
| 51 |
+
- libero_object
|
| 52 |
+
num_gpus: 8
|
| 53 |
+
|
| 54 |
+
hydra:
|
| 55 |
+
job:
|
| 56 |
+
chdir: false
|
| 57 |
+
run:
|
| 58 |
+
dir: .
|
| 59 |
+
output_subdir: null
|
baseline/fastwam_zipper/source/configs/sim_robotwin.yaml
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# @package _global_
|
| 2 |
+
|
| 3 |
+
defaults:
|
| 4 |
+
- train
|
| 5 |
+
- override /task: robotwin_uncond_3cam_384_1e-4
|
| 6 |
+
- _self_
|
| 7 |
+
|
| 8 |
+
ckpt: null
|
| 9 |
+
gpu_id: 0
|
| 10 |
+
|
| 11 |
+
model:
|
| 12 |
+
load_text_encoder: true
|
| 13 |
+
skip_dit_load_from_pretrain: true
|
| 14 |
+
action_dit_pretrained_path: null
|
| 15 |
+
|
| 16 |
+
EVALUATION:
|
| 17 |
+
robotwin_root: third_party/RoboTwin
|
| 18 |
+
policy_name: fastwam_policy
|
| 19 |
+
task_name: null
|
| 20 |
+
task_config: demo_randomized
|
| 21 |
+
instruction_type: unseen
|
| 22 |
+
eval_num_episodes: 100
|
| 23 |
+
output_dir: ./evaluate_results/robotwin/${hydra:runtime.choices.task}/${now:%Y%m%d_%H%M%S}
|
| 24 |
+
|
| 25 |
+
# Action inference
|
| 26 |
+
action_horizon: null
|
| 27 |
+
replan_steps: 24
|
| 28 |
+
num_inference_steps: ${eval_num_inference_steps}
|
| 29 |
+
sigma_shift: null
|
| 30 |
+
text_cfg_scale: 1.0
|
| 31 |
+
negative_prompt: ""
|
| 32 |
+
rand_device: cpu
|
| 33 |
+
tiled: false
|
| 34 |
+
timing_enabled: false
|
| 35 |
+
skip_get_obs_within_replan: true
|
| 36 |
+
|
| 37 |
+
# Runtime inputs
|
| 38 |
+
dataset_stats_path: null
|
| 39 |
+
device: cuda
|
| 40 |
+
|
| 41 |
+
MULTIRUN:
|
| 42 |
+
enabled: false
|
| 43 |
+
num_gpus: 8
|
| 44 |
+
max_tasks_per_gpu: 2
|
| 45 |
+
|
| 46 |
+
hydra:
|
| 47 |
+
job:
|
| 48 |
+
chdir: false
|
| 49 |
+
run:
|
| 50 |
+
dir: .
|
| 51 |
+
output_subdir: null
|
baseline/fastwam_zipper/source/configs/task/galaxea_r1_lite_3cam_1e-4.yaml
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# @package _global_
|
| 2 |
+
|
| 3 |
+
defaults:
|
| 4 |
+
- override /data: galaxea_r1_lite_3cam
|
| 5 |
+
- override /model: fastwam
|
| 6 |
+
- _self_
|
| 7 |
+
|
| 8 |
+
# Formal launcher: 8 samples/GPU x 8 GPUs x 2 accumulation = GBS 128.
|
| 9 |
+
batch_size: 8
|
| 10 |
+
num_workers: 8
|
| 11 |
+
gradient_accumulation_steps: 2
|
| 12 |
+
|
| 13 |
+
model:
|
| 14 |
+
# ModelConfig joins these ids to DIFFSYNTH_MODEL_BASE_PATH. The validated
|
| 15 |
+
# local snapshots are stored without a redundant `Wan-AI/` directory level.
|
| 16 |
+
model_id: ${oc.env:FASTWAM_WAN_MODEL_ID,Wan2.2-TI2V-5B}
|
| 17 |
+
tokenizer_model_id: ${oc.env:FASTWAM_TOKENIZER_MODEL_ID,Wan2.1-T2V-1.3B}
|
| 18 |
+
# The released RobotWin 3-camera checkpoint supplies both experts. Avoid a
|
| 19 |
+
# redundant 5B DiT load before the weight-only resume is applied.
|
| 20 |
+
skip_dit_load_from_pretrain: true
|
| 21 |
+
action_dit_pretrained_path: null
|
| 22 |
+
redirect_common_files: false
|
| 23 |
+
mot_checkpoint_mixed_attn: false
|
| 24 |
+
compile_training_denoise: false
|
| 25 |
+
|
| 26 |
+
learning_rate: 1.0e-4
|
| 27 |
+
lr_scheduler_type: cosine
|
| 28 |
+
num_epochs: 10
|
| 29 |
+
max_steps: null
|
| 30 |
+
log_every: 10
|
| 31 |
+
save_every: 1000
|
| 32 |
+
eval_every: 500
|
| 33 |
+
eval_num_inference_steps: 10
|
| 34 |
+
weight_decay: 1.0e-2
|
| 35 |
+
mixed_precision: bf16
|
| 36 |
+
max_grad_norm: 1.0
|
| 37 |
+
seed: 20261002
|
| 38 |
+
resume: null
|
| 39 |
+
|
| 40 |
+
wandb:
|
| 41 |
+
enabled: true
|
| 42 |
+
workspace: null
|
| 43 |
+
project: galaxea-real-baselines
|
| 44 |
+
name: fastwam-galaxea-r1-lite-3cam
|
| 45 |
+
group: galaxea-real-baselines-v1
|
| 46 |
+
mode: online
|
baseline/fastwam_zipper/source/configs/task/libero_idm_2cam224_1e-4.yaml
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# @package _global_
|
| 2 |
+
|
| 3 |
+
defaults:
|
| 4 |
+
- override /data: libero_2cam
|
| 5 |
+
- override /model: fastwam_idm
|
| 6 |
+
- _self_
|
| 7 |
+
|
| 8 |
+
# dataloading
|
| 9 |
+
batch_size: 16
|
| 10 |
+
num_workers: 8
|
| 11 |
+
|
| 12 |
+
model:
|
| 13 |
+
mot_checkpoint_mixed_attn: false
|
| 14 |
+
|
| 15 |
+
# scheduler
|
| 16 |
+
lr_scheduler_type: "cosine"
|
| 17 |
+
learning_rate: 1e-4
|
| 18 |
+
num_epochs: 10
|
| 19 |
+
max_steps: null
|
| 20 |
+
log_every: 10
|
| 21 |
+
save_every: 2000
|
| 22 |
+
eval_every: 200
|
| 23 |
+
|
| 24 |
+
# training
|
| 25 |
+
gradient_accumulation_steps: 1
|
| 26 |
+
weight_decay: 1e-2
|
| 27 |
+
resume: null
|
baseline/fastwam_zipper/source/configs/task/libero_joint_2cam224_1e-4.yaml
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# @package _global_
|
| 2 |
+
|
| 3 |
+
defaults:
|
| 4 |
+
- override /data: libero_2cam
|
| 5 |
+
- override /model: fastwam_joint
|
| 6 |
+
- _self_
|
| 7 |
+
|
| 8 |
+
# dataloading
|
| 9 |
+
batch_size: 16
|
| 10 |
+
num_workers: 8
|
| 11 |
+
|
| 12 |
+
model:
|
| 13 |
+
mot_checkpoint_mixed_attn: false
|
| 14 |
+
|
| 15 |
+
# scheduler
|
| 16 |
+
lr_scheduler_type: "cosine"
|
| 17 |
+
learning_rate: 1e-4
|
| 18 |
+
num_epochs: 10
|
| 19 |
+
max_steps: null
|
| 20 |
+
log_every: 10
|
| 21 |
+
save_every: 2000
|
| 22 |
+
eval_every: 200
|
| 23 |
+
|
| 24 |
+
# training
|
| 25 |
+
gradient_accumulation_steps: 1
|
| 26 |
+
weight_decay: 1e-2
|
| 27 |
+
resume: null
|
baseline/fastwam_zipper/source/configs/task/libero_optional_idm_2cam224_1e-4.yaml
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# @package _global_
|
| 2 |
+
|
| 3 |
+
defaults:
|
| 4 |
+
- override /data: libero_2cam
|
| 5 |
+
- override /model: fastwam_optional_idm
|
| 6 |
+
- _self_
|
| 7 |
+
|
| 8 |
+
batch_size: 16
|
| 9 |
+
num_workers: 8
|
| 10 |
+
|
| 11 |
+
model:
|
| 12 |
+
mot_checkpoint_mixed_attn: false
|
| 13 |
+
|
| 14 |
+
lr_scheduler_type: "cosine"
|
| 15 |
+
learning_rate: 1e-4
|
| 16 |
+
num_epochs: 10
|
| 17 |
+
max_steps: null
|
| 18 |
+
log_every: 10
|
| 19 |
+
save_every: 2000
|
| 20 |
+
eval_every: 200
|
| 21 |
+
|
| 22 |
+
gradient_accumulation_steps: 1
|
| 23 |
+
weight_decay: 1e-2
|
| 24 |
+
resume: null
|
baseline/fastwam_zipper/source/configs/task/libero_uncond_2cam224_1e-4.yaml
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# @package _global_
|
| 2 |
+
|
| 3 |
+
defaults:
|
| 4 |
+
- override /data: libero_2cam
|
| 5 |
+
- override /model: fastwam
|
| 6 |
+
- _self_
|
| 7 |
+
|
| 8 |
+
# dataloading
|
| 9 |
+
batch_size: 16
|
| 10 |
+
num_workers: 8
|
| 11 |
+
|
| 12 |
+
model:
|
| 13 |
+
mot_checkpoint_mixed_attn: false
|
| 14 |
+
|
| 15 |
+
# scheduler
|
| 16 |
+
lr_scheduler_type: "cosine"
|
| 17 |
+
learning_rate: 1e-4
|
| 18 |
+
num_epochs: 10
|
| 19 |
+
max_steps: null
|
| 20 |
+
log_every: 10
|
| 21 |
+
save_every: 2000
|
| 22 |
+
eval_every: 200
|
| 23 |
+
|
| 24 |
+
# training
|
| 25 |
+
gradient_accumulation_steps: 1
|
| 26 |
+
weight_decay: 1e-2
|
| 27 |
+
resume: null
|
baseline/fastwam_zipper/source/configs/task/robotwin_idm_3cam_384_1e-4.yaml
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# @package _global_
|
| 2 |
+
|
| 3 |
+
defaults:
|
| 4 |
+
- override /data: robotwin
|
| 5 |
+
- override /model: fastwam_idm
|
| 6 |
+
- _self_
|
| 7 |
+
|
| 8 |
+
# dataloading
|
| 9 |
+
batch_size: 16
|
| 10 |
+
num_workers: 8
|
| 11 |
+
|
| 12 |
+
model:
|
| 13 |
+
mot_checkpoint_mixed_attn: false
|
| 14 |
+
|
| 15 |
+
# scheduler
|
| 16 |
+
lr_scheduler_type: "cosine"
|
| 17 |
+
learning_rate: 1e-4
|
| 18 |
+
num_epochs: 5
|
| 19 |
+
max_steps: null
|
| 20 |
+
log_every: 10
|
| 21 |
+
save_every: 2500
|
| 22 |
+
eval_every: 500
|
| 23 |
+
|
| 24 |
+
# training
|
| 25 |
+
gradient_accumulation_steps: 1
|
| 26 |
+
weight_decay: 1e-2
|
| 27 |
+
resume: null
|
baseline/fastwam_zipper/source/configs/task/robotwin_joint_3cam_384_1e-4.yaml
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# @package _global_
|
| 2 |
+
|
| 3 |
+
defaults:
|
| 4 |
+
- override /data: robotwin
|
| 5 |
+
- override /model: fastwam_joint
|
| 6 |
+
- _self_
|
| 7 |
+
|
| 8 |
+
# dataloading
|
| 9 |
+
batch_size: 16
|
| 10 |
+
num_workers: 8
|
| 11 |
+
|
| 12 |
+
model:
|
| 13 |
+
mot_checkpoint_mixed_attn: false
|
| 14 |
+
|
| 15 |
+
# scheduler
|
| 16 |
+
lr_scheduler_type: "cosine"
|
| 17 |
+
learning_rate: 1e-4
|
| 18 |
+
num_epochs: 5
|
| 19 |
+
max_steps: null
|
| 20 |
+
log_every: 10
|
| 21 |
+
save_every: 2500
|
| 22 |
+
eval_every: 500
|
| 23 |
+
|
| 24 |
+
# training
|
| 25 |
+
gradient_accumulation_steps: 1
|
| 26 |
+
weight_decay: 1e-2
|
| 27 |
+
resume: null
|
baseline/fastwam_zipper/source/configs/task/robotwin_uncond_3cam_384_1e-4.yaml
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# @package _global_
|
| 2 |
+
|
| 3 |
+
defaults:
|
| 4 |
+
- override /data: robotwin
|
| 5 |
+
- override /model: fastwam
|
| 6 |
+
- _self_
|
| 7 |
+
|
| 8 |
+
# dataloading
|
| 9 |
+
batch_size: 16
|
| 10 |
+
num_workers: 8
|
| 11 |
+
|
| 12 |
+
model:
|
| 13 |
+
mot_checkpoint_mixed_attn: false
|
| 14 |
+
|
| 15 |
+
# scheduler
|
| 16 |
+
lr_scheduler_type: "cosine"
|
| 17 |
+
learning_rate: 1e-4
|
| 18 |
+
num_epochs: 5
|
| 19 |
+
max_steps: null
|
| 20 |
+
log_every: 10
|
| 21 |
+
save_every: 2500
|
| 22 |
+
eval_every: 500
|
| 23 |
+
|
| 24 |
+
# training
|
| 25 |
+
gradient_accumulation_steps: 1
|
| 26 |
+
weight_decay: 1e-2
|
| 27 |
+
resume: null
|
baseline/fastwam_zipper/source/configs/train.yaml
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# @package _global_
|
| 2 |
+
|
| 3 |
+
defaults:
|
| 4 |
+
- _self_
|
| 5 |
+
- data: null
|
| 6 |
+
- model: null
|
| 7 |
+
- task: null
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
output_dir: ./runs/train/${now:%Y-%m-%d}_${now:%H-%M-%S}
|
| 11 |
+
|
| 12 |
+
# dataloading
|
| 13 |
+
batch_size: 2
|
| 14 |
+
num_workers: 4
|
| 15 |
+
|
| 16 |
+
# scheduler
|
| 17 |
+
lr_scheduler_type: "cosine"
|
| 18 |
+
learning_rate: 1.0e-4
|
| 19 |
+
num_epochs: 1
|
| 20 |
+
max_steps: null
|
| 21 |
+
log_every: 10
|
| 22 |
+
save_every: 100
|
| 23 |
+
eval_every: 50
|
| 24 |
+
eval_num_inference_steps: 10
|
| 25 |
+
|
| 26 |
+
# training
|
| 27 |
+
gradient_accumulation_steps: 1
|
| 28 |
+
mixed_precision: "bf16" # one of: [no, fp16, bf16]
|
| 29 |
+
seed: 42
|
| 30 |
+
max_grad_norm: 1.0
|
| 31 |
+
weight_decay: 0.0
|
| 32 |
+
resume: null
|
| 33 |
+
|
| 34 |
+
# wandb
|
| 35 |
+
wandb:
|
| 36 |
+
enabled: false
|
| 37 |
+
workspace: null
|
| 38 |
+
project: fast-wam
|
| 39 |
+
name: train
|
| 40 |
+
group: null
|
| 41 |
+
mode: online
|
| 42 |
+
|
| 43 |
+
hydra:
|
| 44 |
+
job:
|
| 45 |
+
chdir: false
|
| 46 |
+
run:
|
| 47 |
+
dir: .
|
| 48 |
+
output_subdir: null
|
baseline/fastwam_zipper/source/pyproject.toml
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[build-system]
|
| 2 |
+
requires = ["setuptools>=68", "wheel"]
|
| 3 |
+
build-backend = "setuptools.build_meta"
|
| 4 |
+
|
| 5 |
+
[project]
|
| 6 |
+
name = "fastwam"
|
| 7 |
+
version = "0.1.0"
|
| 8 |
+
description = "FastWAM: standalone Wan2.2-TI2V-5B training/inference package"
|
| 9 |
+
requires-python = ">=3.10"
|
| 10 |
+
dependencies = [
|
| 11 |
+
"accelerate==1.12.0",
|
| 12 |
+
"av==15.1.0",
|
| 13 |
+
"boto3==1.35.99",
|
| 14 |
+
"datasets==4.8.5",
|
| 15 |
+
"deepspeed==0.18.7",
|
| 16 |
+
"einops==0.8.1",
|
| 17 |
+
"gitpython==3.1.45",
|
| 18 |
+
"huggingface-hub==0.29.2",
|
| 19 |
+
"hydra-core==1.3.2",
|
| 20 |
+
"imageio==2.37.0",
|
| 21 |
+
"imageio-ffmpeg==0.6.0",
|
| 22 |
+
"jsonlines==4.0.0",
|
| 23 |
+
"modelscope==1.34.0",
|
| 24 |
+
"numpy==2.2.6",
|
| 25 |
+
"omegaconf==2.3.0",
|
| 26 |
+
"packaging==25.0",
|
| 27 |
+
"pandas==2.2.3",
|
| 28 |
+
"pillow==12.0.0",
|
| 29 |
+
"pyarrow==24.0.0",
|
| 30 |
+
"regex==2025.11.3",
|
| 31 |
+
"rich==14.2.0",
|
| 32 |
+
"safetensors==0.5.3",
|
| 33 |
+
"termcolor==2.5.0",
|
| 34 |
+
"torch==2.7.1+cu128",
|
| 35 |
+
"torchcodec==0.4.0",
|
| 36 |
+
"torchvision==0.22.1+cu128",
|
| 37 |
+
"tqdm==4.68.3",
|
| 38 |
+
"transformers==4.49.0",
|
| 39 |
+
"typing-extensions==4.15.0",
|
| 40 |
+
"wandb==0.23.1",
|
| 41 |
+
]
|
| 42 |
+
|
| 43 |
+
[tool.setuptools.package-dir]
|
| 44 |
+
"" = "src"
|
| 45 |
+
|
| 46 |
+
[tool.setuptools.packages.find]
|
| 47 |
+
where = ["src"]
|
| 48 |
+
include = ["fastwam*"]
|
baseline/fastwam_zipper/source/scripts/__init__.py
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
"""CLI entry scripts for fastwam."""
|
baseline/fastwam_zipper/source/scripts/accelerate_configs/accelerate_zero0.yaml
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
compute_environment: LOCAL_MACHINE
|
| 2 |
+
debug: false
|
| 3 |
+
distributed_type: DEEPSPEED
|
| 4 |
+
deepspeed_config:
|
| 5 |
+
gradient_accumulation_steps: 1
|
| 6 |
+
offload_optimizer_device: none
|
| 7 |
+
offload_param_device: none
|
| 8 |
+
zero3_init_flag: false
|
| 9 |
+
zero_stage: 0
|
| 10 |
+
downcast_bf16: "no"
|
| 11 |
+
enable_cpu_affinity: false
|
| 12 |
+
machine_rank: 0
|
| 13 |
+
main_training_function: main
|
| 14 |
+
mixed_precision: "bf16"
|
| 15 |
+
num_machines: 1
|
| 16 |
+
num_processes: 1
|
| 17 |
+
rdzv_backend: static
|
| 18 |
+
same_network: true
|
| 19 |
+
tpu_env: []
|
| 20 |
+
tpu_use_cluster: false
|
| 21 |
+
tpu_use_sudo: false
|
| 22 |
+
use_cpu: false
|
baseline/fastwam_zipper/source/scripts/accelerate_configs/accelerate_zero1_ds.yaml
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
compute_environment: LOCAL_MACHINE
|
| 2 |
+
debug: false
|
| 3 |
+
distributed_type: DEEPSPEED
|
| 4 |
+
deepspeed_config:
|
| 5 |
+
deepspeed_config_file: scripts/ds_configs/ds_zero1_config.json
|
| 6 |
+
zero3_init_flag: false
|
| 7 |
+
|
| 8 |
+
# Keep this null when using deepspeed_config_file; precision is controlled by
|
| 9 |
+
# training config (Trainer -> Accelerator(mixed_precision=...)) and DS JSON.
|
| 10 |
+
mixed_precision: null
|
| 11 |
+
|
| 12 |
+
machine_rank: 0
|
| 13 |
+
main_training_function: main
|
| 14 |
+
num_machines: 1
|
| 15 |
+
num_processes: 1
|
| 16 |
+
rdzv_backend: static
|
| 17 |
+
same_network: true
|
| 18 |
+
use_cpu: false
|
baseline/fastwam_zipper/source/scripts/accelerate_configs/accelerate_zero2_ds.yaml
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
compute_environment: LOCAL_MACHINE
|
| 2 |
+
debug: false
|
| 3 |
+
distributed_type: DEEPSPEED
|
| 4 |
+
deepspeed_config:
|
| 5 |
+
deepspeed_config_file: scripts/ds_configs/ds_zero2_config.json
|
| 6 |
+
zero3_init_flag: false
|
| 7 |
+
|
| 8 |
+
# Keep this null when using deepspeed_config_file; precision is controlled by
|
| 9 |
+
# training config (Trainer -> Accelerator(mixed_precision=...)) and DS JSON.
|
| 10 |
+
mixed_precision: null
|
| 11 |
+
|
| 12 |
+
machine_rank: 0
|
| 13 |
+
main_training_function: main
|
| 14 |
+
num_machines: 1
|
| 15 |
+
num_processes: 1
|
| 16 |
+
rdzv_backend: static
|
| 17 |
+
same_network: true
|
| 18 |
+
use_cpu: false
|
baseline/fastwam_zipper/source/scripts/check_galaxea_r1_lite_contract.py
ADDED
|
@@ -0,0 +1,91 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Fail-closed checks for the shared Galaxea LeRobot-v2 FastWAM input."""
|
| 3 |
+
|
| 4 |
+
from __future__ import annotations
|
| 5 |
+
|
| 6 |
+
import argparse
|
| 7 |
+
import hashlib
|
| 8 |
+
import json
|
| 9 |
+
from pathlib import Path
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
EXPECTED_MANIFEST_SHA256 = "36dcbf10695ec1f6c79a90375f396e59e70ae7643595f51dbb3e1299765be5b2"
|
| 13 |
+
EXPECTED_IMAGE_KEYS = {
|
| 14 |
+
"observation.images.head_left",
|
| 15 |
+
"observation.images.wrist_left",
|
| 16 |
+
"observation.images.wrist_right",
|
| 17 |
+
}
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def read_json(path: Path) -> dict:
|
| 21 |
+
value = json.loads(path.read_text(encoding="utf-8"))
|
| 22 |
+
if not isinstance(value, dict):
|
| 23 |
+
raise TypeError(f"expected mapping: {path}")
|
| 24 |
+
return value
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
def check_pack(root: Path, split: str, expected_episodes: int) -> dict:
|
| 28 |
+
info = read_json(root / "meta/info.json")
|
| 29 |
+
qc = read_json(root / "meta/alignment_qc.json")
|
| 30 |
+
features = info.get("features", {})
|
| 31 |
+
if str(info.get("codebase_version")) != "v2.0":
|
| 32 |
+
raise ValueError(f"{split}: expected LeRobot v2.0")
|
| 33 |
+
if int(info.get("fps", -1)) != 20:
|
| 34 |
+
raise ValueError(f"{split}: expected 20 Hz")
|
| 35 |
+
if int(info.get("total_episodes", -1)) != expected_episodes:
|
| 36 |
+
raise ValueError(f"{split}: episode count mismatch")
|
| 37 |
+
if not EXPECTED_IMAGE_KEYS.issubset(features):
|
| 38 |
+
raise ValueError(f"{split}: missing camera features")
|
| 39 |
+
for key in ("observation.state", "action"):
|
| 40 |
+
if features.get(key, {}).get("shape") != [14]:
|
| 41 |
+
raise ValueError(f"{split}: {key} must be 14D")
|
| 42 |
+
if qc.get("split") != split:
|
| 43 |
+
raise ValueError(f"{split}: alignment report split mismatch")
|
| 44 |
+
if qc.get("pass") is not True or qc.get("strictly_causal") is not True:
|
| 45 |
+
raise ValueError(f"{split}: alignment QC did not pass")
|
| 46 |
+
if qc.get("all_required_timestamps_from_ros_header") is not True:
|
| 47 |
+
raise ValueError(f"{split}: timestamp fallback is forbidden")
|
| 48 |
+
if float(qc.get("max_source_minus_anchor_seconds", 1.0)) > 0.0:
|
| 49 |
+
raise ValueError(f"{split}: future selections are forbidden")
|
| 50 |
+
for episode in qc.get("episodes", []):
|
| 51 |
+
for stream in episode.get("streams", {}).values():
|
| 52 |
+
if int(stream.get("future_selected_frames", -1)) != 0:
|
| 53 |
+
raise ValueError(f"{split}: future-selected camera/state sample")
|
| 54 |
+
for side in episode.get("gripper_latching", {}).values():
|
| 55 |
+
if int(side.get("future_selected_frames", -1)) != 0:
|
| 56 |
+
raise ValueError(f"{split}: future-selected gripper sample")
|
| 57 |
+
return {
|
| 58 |
+
"root": str(root.resolve()),
|
| 59 |
+
"episodes": expected_episodes,
|
| 60 |
+
"frames": int(info["total_frames"]),
|
| 61 |
+
}
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def main() -> None:
|
| 65 |
+
parser = argparse.ArgumentParser()
|
| 66 |
+
parser.add_argument("--train", type=Path, required=True)
|
| 67 |
+
parser.add_argument("--validation", type=Path, required=True)
|
| 68 |
+
parser.add_argument("--manifest", type=Path, required=True)
|
| 69 |
+
parser.add_argument("--expected-train-episodes", type=int, default=511)
|
| 70 |
+
parser.add_argument("--expected-validation-episodes", type=int, default=57)
|
| 71 |
+
parser.add_argument("--allow-smoke-manifest", action="store_true")
|
| 72 |
+
args = parser.parse_args()
|
| 73 |
+
digest = hashlib.sha256(args.manifest.read_bytes()).hexdigest()
|
| 74 |
+
if digest != EXPECTED_MANIFEST_SHA256 and not args.allow_smoke_manifest:
|
| 75 |
+
raise ValueError(f"source manifest SHA256 mismatch: {digest}")
|
| 76 |
+
result = {
|
| 77 |
+
"status": "galaxea_fastwam_contract_ok",
|
| 78 |
+
"manifest_sha256": digest,
|
| 79 |
+
"action_horizon": 24,
|
| 80 |
+
"video_frames": 9,
|
| 81 |
+
"views": ["head_left", "wrist_left", "wrist_right"],
|
| 82 |
+
"train": check_pack(args.train, "train", args.expected_train_episodes),
|
| 83 |
+
"validation": check_pack(
|
| 84 |
+
args.validation, "validation", args.expected_validation_episodes
|
| 85 |
+
),
|
| 86 |
+
}
|
| 87 |
+
print(json.dumps(result, indent=2, sort_keys=True))
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
if __name__ == "__main__":
|
| 91 |
+
main()
|
baseline/fastwam_zipper/source/scripts/check_galaxea_text_cache.py
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Validate all fixed Galaxea prompt embeddings without loading UMT5."""
|
| 3 |
+
|
| 4 |
+
from __future__ import annotations
|
| 5 |
+
|
| 6 |
+
import argparse
|
| 7 |
+
import hashlib
|
| 8 |
+
import json
|
| 9 |
+
from pathlib import Path
|
| 10 |
+
|
| 11 |
+
import torch
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
PROMPT = "A video recorded from a robot's point of view executing the following instruction: {task}"
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
def main() -> None:
|
| 18 |
+
parser = argparse.ArgumentParser()
|
| 19 |
+
parser.add_argument("--train", type=Path, required=True)
|
| 20 |
+
parser.add_argument("--validation", type=Path, required=True)
|
| 21 |
+
parser.add_argument("--cache", type=Path, required=True)
|
| 22 |
+
args = parser.parse_args()
|
| 23 |
+
|
| 24 |
+
prompts: set[str] = set()
|
| 25 |
+
for dataset in (args.train, args.validation):
|
| 26 |
+
tasks = dataset / "meta/tasks.jsonl"
|
| 27 |
+
for line in tasks.read_text(encoding="utf-8").splitlines():
|
| 28 |
+
if line.strip():
|
| 29 |
+
prompts.add(PROMPT.format(task=json.loads(line)["task"]))
|
| 30 |
+
if len(prompts) != 5:
|
| 31 |
+
raise RuntimeError(f"expected five fixed task prompts, got {len(prompts)}")
|
| 32 |
+
|
| 33 |
+
for prompt in sorted(prompts):
|
| 34 |
+
digest = hashlib.sha256(prompt.encode("utf-8")).hexdigest()
|
| 35 |
+
path = args.cache / f"{digest}.t5_len128.wan22ti2v5b.pt"
|
| 36 |
+
payload = torch.load(path, map_location="cpu", weights_only=True)
|
| 37 |
+
context = payload["context"]
|
| 38 |
+
mask = payload["mask"]
|
| 39 |
+
if context.ndim != 2 or context.shape[0] != 128 or not torch.isfinite(context).all():
|
| 40 |
+
raise RuntimeError(f"invalid context tensor in {path}: {tuple(context.shape)}")
|
| 41 |
+
if mask.shape != (128,):
|
| 42 |
+
raise RuntimeError(f"invalid mask tensor in {path}: {tuple(mask.shape)}")
|
| 43 |
+
print(f"FASTWAM_TEXT_CACHE_OK prompts={len(prompts)} cache={args.cache.resolve()}")
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
if __name__ == "__main__":
|
| 47 |
+
main()
|
baseline/fastwam_zipper/source/scripts/dryrun_fastwam.py
ADDED
|
@@ -0,0 +1,227 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Benchmark FastWAM action inference with synthetic task-shaped inputs.
|
| 2 |
+
|
| 3 |
+
Example:
|
| 4 |
+
python scripts/dryrun_fastwam.py \
|
| 5 |
+
task=libero_uncond_2cam224_1e-4 \
|
| 6 |
+
ckpt=./checkpoints/fastwam_release/libero_uncond_2cam224.pt \
|
| 7 |
+
+DRYRUN.compile_action_infer=true \
|
| 8 |
+
+DRYRUN.warmup=2 +DRYRUN.iters=10
|
| 9 |
+
"""
|
| 10 |
+
|
| 11 |
+
from __future__ import annotations
|
| 12 |
+
|
| 13 |
+
import logging
|
| 14 |
+
import inspect
|
| 15 |
+
import statistics
|
| 16 |
+
import time
|
| 17 |
+
|
| 18 |
+
import hydra
|
| 19 |
+
import torch
|
| 20 |
+
from hydra.utils import instantiate
|
| 21 |
+
from omegaconf import DictConfig, OmegaConf
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
logger = logging.getLogger(__name__)
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
def _dryrun_config(cfg: DictConfig) -> DictConfig:
|
| 28 |
+
defaults = OmegaConf.create(
|
| 29 |
+
{
|
| 30 |
+
"warmup": 2,
|
| 31 |
+
"iters": 10,
|
| 32 |
+
"seed": 42,
|
| 33 |
+
"action_horizon": None,
|
| 34 |
+
"num_inference_steps": None,
|
| 35 |
+
"compile_action_infer": False,
|
| 36 |
+
"benchmark_both": False,
|
| 37 |
+
"use_random_context": False,
|
| 38 |
+
"action_infer_mode": "idm",
|
| 39 |
+
}
|
| 40 |
+
)
|
| 41 |
+
return OmegaConf.merge(defaults, cfg.get("DRYRUN", {}))
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def _model_dtype(mixed_precision: str) -> torch.dtype:
|
| 45 |
+
return {
|
| 46 |
+
"no": torch.float32,
|
| 47 |
+
"fp16": torch.float16,
|
| 48 |
+
"bf16": torch.bfloat16,
|
| 49 |
+
}[str(mixed_precision)]
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
def _timed_infer_with_output(
|
| 53 |
+
model,
|
| 54 |
+
kwargs: dict,
|
| 55 |
+
device: torch.device,
|
| 56 |
+
) -> tuple[float, torch.Tensor]:
|
| 57 |
+
torch.cuda.synchronize(device)
|
| 58 |
+
start = time.perf_counter()
|
| 59 |
+
with torch.no_grad():
|
| 60 |
+
output = model.infer_action(**kwargs)
|
| 61 |
+
torch.cuda.synchronize(device)
|
| 62 |
+
logger.info("action shape=%s", tuple(output["action"].shape))
|
| 63 |
+
return time.perf_counter() - start, output["action"].detach().clone()
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
def _timed_infer(model, kwargs: dict, device: torch.device) -> float:
|
| 67 |
+
elapsed, _ = _timed_infer_with_output(model, kwargs, device)
|
| 68 |
+
return elapsed
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
def _benchmark_mode(
|
| 72 |
+
model,
|
| 73 |
+
infer_kwargs: dict,
|
| 74 |
+
device: torch.device,
|
| 75 |
+
*,
|
| 76 |
+
label: str,
|
| 77 |
+
warmup: int,
|
| 78 |
+
iters: int,
|
| 79 |
+
) -> tuple[list[float], float]:
|
| 80 |
+
for index in range(warmup):
|
| 81 |
+
elapsed = _timed_infer(model, infer_kwargs, device)
|
| 82 |
+
logger.info("%s warmup %d/%d: %.4fs", label, index + 1, warmup, elapsed)
|
| 83 |
+
|
| 84 |
+
torch.cuda.reset_peak_memory_stats(device)
|
| 85 |
+
timings = []
|
| 86 |
+
for index in range(iters):
|
| 87 |
+
elapsed = _timed_infer(model, infer_kwargs, device)
|
| 88 |
+
timings.append(elapsed)
|
| 89 |
+
logger.info("%s iteration %d/%d: %.4fs", label, index + 1, iters, elapsed)
|
| 90 |
+
return timings, torch.cuda.max_memory_allocated(device) / 1024**3
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
@hydra.main(config_path="../configs", config_name="sim_libero.yaml", version_base="1.3")
|
| 94 |
+
def main(cfg: DictConfig) -> None:
|
| 95 |
+
logging.basicConfig(level=logging.INFO)
|
| 96 |
+
dryrun = _dryrun_config(cfg)
|
| 97 |
+
device = torch.device(str(cfg.EVALUATION.device))
|
| 98 |
+
model = instantiate(
|
| 99 |
+
cfg.model,
|
| 100 |
+
model_dtype=_model_dtype(cfg.mixed_precision),
|
| 101 |
+
device=str(device),
|
| 102 |
+
)
|
| 103 |
+
if cfg.ckpt is not None:
|
| 104 |
+
model.load_checkpoint(str(cfg.ckpt))
|
| 105 |
+
model.eval()
|
| 106 |
+
logger.info(
|
| 107 |
+
"device=%s gpu=%s dtype=%s",
|
| 108 |
+
device,
|
| 109 |
+
torch.cuda.get_device_name(device),
|
| 110 |
+
next(model.parameters()).dtype,
|
| 111 |
+
)
|
| 112 |
+
|
| 113 |
+
height, width = (int(value) for value in cfg.data.train.video_size)
|
| 114 |
+
action_horizon = (
|
| 115 |
+
int(cfg.data.train.num_frames) - 1
|
| 116 |
+
if dryrun.action_horizon is None
|
| 117 |
+
else int(dryrun.action_horizon)
|
| 118 |
+
)
|
| 119 |
+
num_inference_steps = (
|
| 120 |
+
int(cfg.eval_num_inference_steps)
|
| 121 |
+
if dryrun.num_inference_steps is None
|
| 122 |
+
else int(dryrun.num_inference_steps)
|
| 123 |
+
)
|
| 124 |
+
logger.info(
|
| 125 |
+
"action_horizon=%d num_inference_steps=%d warmup=%d iters=%d",
|
| 126 |
+
action_horizon,
|
| 127 |
+
num_inference_steps,
|
| 128 |
+
int(dryrun.warmup),
|
| 129 |
+
int(dryrun.iters),
|
| 130 |
+
)
|
| 131 |
+
torch.manual_seed(int(dryrun.seed))
|
| 132 |
+
input_image = torch.rand(1, 3, height, width) * 2 - 1
|
| 133 |
+
proprio_dim = int(cfg.data.train.processor.proprio_output_dim)
|
| 134 |
+
proprio = torch.randn(1, proprio_dim)
|
| 135 |
+
if bool(dryrun.use_random_context):
|
| 136 |
+
context = torch.randn(1, int(cfg.data.train.context_len), model.text_dim)
|
| 137 |
+
context_mask = torch.ones(context.shape[:2], dtype=torch.bool)
|
| 138 |
+
prompt = None
|
| 139 |
+
else:
|
| 140 |
+
context = None
|
| 141 |
+
context_mask = None
|
| 142 |
+
prompt = "pick up the object"
|
| 143 |
+
|
| 144 |
+
infer_kwargs = {
|
| 145 |
+
"prompt": prompt,
|
| 146 |
+
"input_image": input_image,
|
| 147 |
+
"action_horizon": action_horizon,
|
| 148 |
+
"proprio": proprio,
|
| 149 |
+
"context": context,
|
| 150 |
+
"context_mask": context_mask,
|
| 151 |
+
"num_inference_steps": num_inference_steps,
|
| 152 |
+
"seed": int(dryrun.seed),
|
| 153 |
+
"rand_device": str(cfg.EVALUATION.rand_device),
|
| 154 |
+
"compile_action_infer": bool(dryrun.compile_action_infer),
|
| 155 |
+
}
|
| 156 |
+
infer_parameters = inspect.signature(model.infer_action).parameters
|
| 157 |
+
if "num_video_frames" in infer_parameters:
|
| 158 |
+
infer_kwargs["num_video_frames"] = (
|
| 159 |
+
(int(cfg.data.train.num_frames) - 1)
|
| 160 |
+
// int(cfg.data.train.action_video_freq_ratio)
|
| 161 |
+
+ 1
|
| 162 |
+
)
|
| 163 |
+
if "action_infer_mode" in infer_parameters:
|
| 164 |
+
infer_kwargs["action_infer_mode"] = str(dryrun.action_infer_mode)
|
| 165 |
+
|
| 166 |
+
warmup = int(dryrun.warmup)
|
| 167 |
+
iters = int(dryrun.iters)
|
| 168 |
+
if bool(dryrun.benchmark_both):
|
| 169 |
+
infer_kwargs["compile_action_infer"] = False
|
| 170 |
+
eager_times, eager_peak = _benchmark_mode(
|
| 171 |
+
model,
|
| 172 |
+
infer_kwargs,
|
| 173 |
+
device,
|
| 174 |
+
label="eager",
|
| 175 |
+
warmup=warmup,
|
| 176 |
+
iters=iters,
|
| 177 |
+
)
|
| 178 |
+
|
| 179 |
+
infer_kwargs["compile_action_infer"] = True
|
| 180 |
+
cold_compile_time = _timed_infer(model, infer_kwargs, device)
|
| 181 |
+
logger.info("compile cold start: %.4fs", cold_compile_time)
|
| 182 |
+
compile_times, compile_peak = _benchmark_mode(
|
| 183 |
+
model,
|
| 184 |
+
infer_kwargs,
|
| 185 |
+
device,
|
| 186 |
+
label="compile",
|
| 187 |
+
warmup=warmup,
|
| 188 |
+
iters=iters,
|
| 189 |
+
)
|
| 190 |
+
eager_mean = statistics.mean(eager_times)
|
| 191 |
+
compile_mean = statistics.mean(compile_times)
|
| 192 |
+
logger.info(
|
| 193 |
+
"comparison | eager_mean=%.4fs compile_mean=%.4fs speedup=%.3fx "
|
| 194 |
+
"compile_cold_start=%.4fs eager_peak=%.3fGiB compile_peak=%.3fGiB",
|
| 195 |
+
eager_mean,
|
| 196 |
+
compile_mean,
|
| 197 |
+
eager_mean / compile_mean,
|
| 198 |
+
cold_compile_time,
|
| 199 |
+
eager_peak,
|
| 200 |
+
compile_peak,
|
| 201 |
+
)
|
| 202 |
+
return
|
| 203 |
+
|
| 204 |
+
torch.cuda.reset_peak_memory_stats(device)
|
| 205 |
+
for index in range(warmup):
|
| 206 |
+
elapsed = _timed_infer(model, infer_kwargs, device)
|
| 207 |
+
logger.info("warmup %d/%d: %.4fs", index + 1, warmup, elapsed)
|
| 208 |
+
|
| 209 |
+
torch.cuda.reset_peak_memory_stats(device)
|
| 210 |
+
timings = []
|
| 211 |
+
for index in range(iters):
|
| 212 |
+
elapsed = _timed_infer(model, infer_kwargs, device)
|
| 213 |
+
timings.append(elapsed)
|
| 214 |
+
logger.info("iteration %d/%d: %.4fs", index + 1, iters, elapsed)
|
| 215 |
+
peak_gib = torch.cuda.max_memory_allocated(device) / 1024**3
|
| 216 |
+
logger.info(
|
| 217 |
+
"mean=%.4fs median=%.4fs min=%.4fs max=%.4fs peak_allocated=%.3fGiB",
|
| 218 |
+
statistics.mean(timings),
|
| 219 |
+
statistics.median(timings),
|
| 220 |
+
min(timings),
|
| 221 |
+
max(timings),
|
| 222 |
+
peak_gib,
|
| 223 |
+
)
|
| 224 |
+
|
| 225 |
+
|
| 226 |
+
if __name__ == "__main__":
|
| 227 |
+
main()
|
baseline/fastwam_zipper/source/scripts/ds_configs/ds_zero1_config.json
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"train_batch_size": "auto",
|
| 3 |
+
"train_micro_batch_size_per_gpu": "auto",
|
| 4 |
+
"gradient_accumulation_steps": "auto",
|
| 5 |
+
"zero_optimization": {
|
| 6 |
+
"stage": 1,
|
| 7 |
+
"offload_optimizer": {
|
| 8 |
+
"device": "none"
|
| 9 |
+
},
|
| 10 |
+
"offload_param": {
|
| 11 |
+
"device": "none"
|
| 12 |
+
},
|
| 13 |
+
"overlap_comm": false,
|
| 14 |
+
"contiguous_gradients": false,
|
| 15 |
+
"reduce_bucket_size": 2e8,
|
| 16 |
+
"allgather_bucket_size": 2e8
|
| 17 |
+
}
|
| 18 |
+
}
|
baseline/fastwam_zipper/source/scripts/ds_configs/ds_zero2_config.json
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"train_batch_size": "auto",
|
| 3 |
+
"train_micro_batch_size_per_gpu": "auto",
|
| 4 |
+
"gradient_accumulation_steps": "auto",
|
| 5 |
+
"gradient_clipping": 1.0,
|
| 6 |
+
"zero_optimization": {
|
| 7 |
+
"stage": 2,
|
| 8 |
+
"offload_optimizer": {
|
| 9 |
+
"device": "none"
|
| 10 |
+
},
|
| 11 |
+
"offload_param": {
|
| 12 |
+
"device": "none"
|
| 13 |
+
},
|
| 14 |
+
"overlap_comm": false,
|
| 15 |
+
"contiguous_gradients": false,
|
| 16 |
+
"reduce_bucket_size": 2e8,
|
| 17 |
+
"allgather_bucket_size": 2e8
|
| 18 |
+
}
|
| 19 |
+
}
|
baseline/fastwam_zipper/source/scripts/precompute_text_embeds.py
ADDED
|
@@ -0,0 +1,389 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import hashlib
|
| 2 |
+
import json
|
| 3 |
+
import logging
|
| 4 |
+
import os
|
| 5 |
+
import re
|
| 6 |
+
import uuid
|
| 7 |
+
from pathlib import Path
|
| 8 |
+
from typing import Any
|
| 9 |
+
|
| 10 |
+
import hydra
|
| 11 |
+
import torch
|
| 12 |
+
import torch.distributed as dist
|
| 13 |
+
from omegaconf import DictConfig, ListConfig
|
| 14 |
+
from tqdm import tqdm
|
| 15 |
+
|
| 16 |
+
from fastwam.datasets.lerobot.robot_video_dataset import DEFAULT_PROMPT
|
| 17 |
+
from fastwam.models.wan22.helpers.loader import _load_registered_model, _resolve_configs
|
| 18 |
+
from fastwam.models.wan22.wan_video_text_encoder import HuggingfaceTokenizer
|
| 19 |
+
from fastwam.utils.config_resolvers import register_default_resolvers
|
| 20 |
+
from fastwam.utils.logging_config import get_logger, setup_logging
|
| 21 |
+
|
| 22 |
+
register_default_resolvers()
|
| 23 |
+
logger = get_logger(__name__)
|
| 24 |
+
|
| 25 |
+
DEFAULT_MODEL_ID = "Wan-AI/Wan2.2-TI2V-5B"
|
| 26 |
+
DEFAULT_TOKENIZER_MODEL_ID = "Wan-AI/Wan2.1-T2V-1.3B"
|
| 27 |
+
DEFAULT_CONTEXT_LEN = 128
|
| 28 |
+
DEFAULT_BATCH_SIZE = 16
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
def _init_distributed():
|
| 32 |
+
world_size = int(os.environ.get("WORLD_SIZE", "1"))
|
| 33 |
+
if world_size <= 1:
|
| 34 |
+
return False, 0, 1, 0
|
| 35 |
+
|
| 36 |
+
local_rank = int(os.environ.get("LOCAL_RANK", "0"))
|
| 37 |
+
backend = "nccl" if torch.cuda.is_available() else "gloo"
|
| 38 |
+
if torch.cuda.is_available():
|
| 39 |
+
torch.cuda.set_device(local_rank)
|
| 40 |
+
|
| 41 |
+
if not dist.is_initialized():
|
| 42 |
+
dist.init_process_group(backend=backend, init_method="env://")
|
| 43 |
+
|
| 44 |
+
return True, dist.get_rank(), dist.get_world_size(), local_rank
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def _to_bool(value: Any) -> bool:
|
| 48 |
+
if isinstance(value, bool):
|
| 49 |
+
return value
|
| 50 |
+
if isinstance(value, str):
|
| 51 |
+
text = value.strip().lower()
|
| 52 |
+
if text in {"1", "true", "yes", "y"}:
|
| 53 |
+
return True
|
| 54 |
+
if text in {"0", "false", "no", "n"}:
|
| 55 |
+
return False
|
| 56 |
+
raise ValueError(f"Cannot parse bool value: {value}")
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
def _iter_dataset_nodes(node: Any, path: str = "data"):
|
| 60 |
+
if isinstance(node, DictConfig):
|
| 61 |
+
if "dataset_dirs" in node and node.get("dataset_dirs") is not None:
|
| 62 |
+
yield path, node
|
| 63 |
+
for key, value in node.items():
|
| 64 |
+
yield from _iter_dataset_nodes(value, f"{path}.{key}")
|
| 65 |
+
elif isinstance(node, ListConfig):
|
| 66 |
+
for idx, value in enumerate(node):
|
| 67 |
+
yield from _iter_dataset_nodes(value, f"{path}[{idx}]")
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def _collect_dataset_settings(data_cfg: DictConfig):
|
| 71 |
+
dataset_dirs: list[str] = []
|
| 72 |
+
cache_dirs: list[Path] = []
|
| 73 |
+
context_lens = set()
|
| 74 |
+
|
| 75 |
+
for node_path, node in _iter_dataset_nodes(data_cfg, path="data"):
|
| 76 |
+
raw_dirs = node.get("dataset_dirs")
|
| 77 |
+
if raw_dirs is None:
|
| 78 |
+
continue
|
| 79 |
+
|
| 80 |
+
cache_dir = node.get("text_embedding_cache_dir")
|
| 81 |
+
if cache_dir is None or not str(cache_dir).strip():
|
| 82 |
+
raise ValueError(
|
| 83 |
+
f"Missing `text_embedding_cache_dir` for dataset node `{node_path}` "
|
| 84 |
+
"(this node defines `dataset_dirs`)."
|
| 85 |
+
)
|
| 86 |
+
|
| 87 |
+
for ds in raw_dirs:
|
| 88 |
+
ds_str = str(ds)
|
| 89 |
+
if ds_str not in dataset_dirs:
|
| 90 |
+
dataset_dirs.append(ds_str)
|
| 91 |
+
|
| 92 |
+
cache_dir_path = Path(str(cache_dir)).expanduser()
|
| 93 |
+
if cache_dir_path not in cache_dirs:
|
| 94 |
+
cache_dirs.append(cache_dir_path)
|
| 95 |
+
|
| 96 |
+
context_len = node.get("context_len")
|
| 97 |
+
if context_len is not None:
|
| 98 |
+
context_lens.add(int(context_len))
|
| 99 |
+
|
| 100 |
+
logger.info("Discovered dataset node `%s` with %d dataset_dirs.", node_path, len(raw_dirs))
|
| 101 |
+
|
| 102 |
+
return dataset_dirs, cache_dirs, context_lens
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
def _resolve_context_len(context_lens: set[int]) -> int:
|
| 106 |
+
if len(context_lens) != 1:
|
| 107 |
+
raise ValueError(
|
| 108 |
+
f"Found multiple context_len values in data config: {sorted(context_lens)}. "
|
| 109 |
+
"Please keep them consistent."
|
| 110 |
+
)
|
| 111 |
+
return next(iter(context_lens))
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
def _read_unique_prompts(dataset_dirs: list[str]) -> list[str]:
|
| 115 |
+
prompts: list[str] = []
|
| 116 |
+
seen = set()
|
| 117 |
+
total_task_rows = 0
|
| 118 |
+
|
| 119 |
+
for ds_dir in dataset_dirs:
|
| 120 |
+
tasks_path = Path(ds_dir) / "meta" / "tasks.jsonl"
|
| 121 |
+
if not tasks_path.exists():
|
| 122 |
+
raise FileNotFoundError(f"Missing tasks file: {tasks_path}")
|
| 123 |
+
|
| 124 |
+
with tasks_path.open("r", encoding="utf-8") as f:
|
| 125 |
+
for line_idx, line in enumerate(f, start=1):
|
| 126 |
+
line = line.strip()
|
| 127 |
+
if not line:
|
| 128 |
+
continue
|
| 129 |
+
record = json.loads(line)
|
| 130 |
+
if "task" not in record:
|
| 131 |
+
raise KeyError(f"Missing `task` field at {tasks_path}:{line_idx}")
|
| 132 |
+
task = str(record["task"])
|
| 133 |
+
prompt = DEFAULT_PROMPT.format(task=task)
|
| 134 |
+
total_task_rows += 1
|
| 135 |
+
if prompt not in seen:
|
| 136 |
+
seen.add(prompt)
|
| 137 |
+
prompts.append(prompt)
|
| 138 |
+
|
| 139 |
+
logger.info(
|
| 140 |
+
"Loaded %d task rows from %d datasets, deduplicated to %d prompts.",
|
| 141 |
+
total_task_rows,
|
| 142 |
+
len(dataset_dirs),
|
| 143 |
+
len(prompts),
|
| 144 |
+
)
|
| 145 |
+
return prompts
|
| 146 |
+
|
| 147 |
+
|
| 148 |
+
def _get_override_prompt(override_instruction: Any) -> str | None:
|
| 149 |
+
if override_instruction is None:
|
| 150 |
+
return None
|
| 151 |
+
task = str(override_instruction).strip()
|
| 152 |
+
if task == "":
|
| 153 |
+
return None
|
| 154 |
+
return DEFAULT_PROMPT.format(task=task)
|
| 155 |
+
|
| 156 |
+
|
| 157 |
+
def _model_id_to_enc_id(model_id: str) -> str:
|
| 158 |
+
base = str(model_id).split("/")[-1]
|
| 159 |
+
enc_id = re.sub(r"[^a-z0-9]+", "", base.lower())
|
| 160 |
+
return enc_id or "textenc"
|
| 161 |
+
|
| 162 |
+
|
| 163 |
+
def _atomic_torch_save(payload: dict[str, torch.Tensor], output_path: Path):
|
| 164 |
+
output_path.parent.mkdir(parents=True, exist_ok=True)
|
| 165 |
+
tmp_path = output_path.parent / f".{output_path.name}.tmp.{uuid.uuid4().hex}"
|
| 166 |
+
torch.save(payload, str(tmp_path))
|
| 167 |
+
os.replace(tmp_path, output_path)
|
| 168 |
+
|
| 169 |
+
|
| 170 |
+
@hydra.main(config_path="../configs", config_name="train", version_base="1.3")
|
| 171 |
+
def main(cfg: DictConfig):
|
| 172 |
+
setup_logging(log_level=logging.INFO)
|
| 173 |
+
|
| 174 |
+
is_distributed, rank, world_size, local_rank = _init_distributed()
|
| 175 |
+
if is_distributed and rank == 0:
|
| 176 |
+
logger.info("Distributed enabled: world_size=%d", world_size)
|
| 177 |
+
if (not is_distributed) and torch.cuda.is_available() and torch.cuda.device_count() > 1:
|
| 178 |
+
logger.info(
|
| 179 |
+
"Multi-GPU available. To use it, run: torchrun --standalone --nproc_per_node=%d scripts/precompute_text_embeds.py",
|
| 180 |
+
torch.cuda.device_count(),
|
| 181 |
+
)
|
| 182 |
+
|
| 183 |
+
overwrite = _to_bool(cfg.get("overwrite", True))
|
| 184 |
+
model_cfg = cfg.model
|
| 185 |
+
if model_cfg is None:
|
| 186 |
+
raise ValueError("`cfg.model` is required.")
|
| 187 |
+
if cfg.data is None:
|
| 188 |
+
raise ValueError("`cfg.data` is required.")
|
| 189 |
+
|
| 190 |
+
dataset_dirs, cache_dirs, context_lens = _collect_dataset_settings(cfg.data)
|
| 191 |
+
if not cache_dirs:
|
| 192 |
+
raise ValueError("No `text_embedding_cache_dir` found under `cfg.data`.")
|
| 193 |
+
|
| 194 |
+
context_len = _resolve_context_len(context_lens)
|
| 195 |
+
override_prompt = _get_override_prompt(cfg.get("override_instruction"))
|
| 196 |
+
if override_prompt is not None:
|
| 197 |
+
prompts = [override_prompt]
|
| 198 |
+
logger.info("Using override_instruction; skipping dataset scan and encoding exactly 1 prompt.")
|
| 199 |
+
else:
|
| 200 |
+
if not dataset_dirs:
|
| 201 |
+
raise ValueError("No `dataset_dirs` found under `cfg.data`.")
|
| 202 |
+
prompts = _read_unique_prompts(dataset_dirs)
|
| 203 |
+
if not prompts:
|
| 204 |
+
logger.warning("No prompts found from tasks.jsonl; nothing to do.")
|
| 205 |
+
return
|
| 206 |
+
|
| 207 |
+
if torch.cuda.is_available():
|
| 208 |
+
device = f"cuda:{local_rank}" if is_distributed else "cuda"
|
| 209 |
+
else:
|
| 210 |
+
device = "cpu"
|
| 211 |
+
torch_dtype = torch.bfloat16
|
| 212 |
+
model_id = str(model_cfg.get("model_id", DEFAULT_MODEL_ID))
|
| 213 |
+
tokenizer_model_id = str(model_cfg.get("tokenizer_model_id", DEFAULT_TOKENIZER_MODEL_ID))
|
| 214 |
+
redirect_common_files = bool(model_cfg.get("redirect_common_files", True))
|
| 215 |
+
enc_id = _model_id_to_enc_id(model_id)
|
| 216 |
+
|
| 217 |
+
logger.info(
|
| 218 |
+
"Preparing text encoder with model_id=%s tokenizer_model_id=%s device=%s dtype=%s context_len=%d overwrite=%s",
|
| 219 |
+
model_id,
|
| 220 |
+
tokenizer_model_id,
|
| 221 |
+
device,
|
| 222 |
+
torch_dtype,
|
| 223 |
+
context_len,
|
| 224 |
+
overwrite,
|
| 225 |
+
)
|
| 226 |
+
|
| 227 |
+
_, text_config, _, tokenizer_config = _resolve_configs(
|
| 228 |
+
model_id=model_id,
|
| 229 |
+
tokenizer_model_id=tokenizer_model_id,
|
| 230 |
+
redirect_common_files=redirect_common_files,
|
| 231 |
+
)
|
| 232 |
+
text_config.download_if_necessary()
|
| 233 |
+
tokenizer_config.download_if_necessary()
|
| 234 |
+
|
| 235 |
+
text_encoder = _load_registered_model(
|
| 236 |
+
text_config.path,
|
| 237 |
+
"wan_video_text_encoder",
|
| 238 |
+
torch_dtype=torch_dtype,
|
| 239 |
+
device=device,
|
| 240 |
+
).eval()
|
| 241 |
+
tokenizer = HuggingfaceTokenizer(
|
| 242 |
+
name=tokenizer_config.path,
|
| 243 |
+
seq_len=context_len,
|
| 244 |
+
clean="whitespace",
|
| 245 |
+
)
|
| 246 |
+
|
| 247 |
+
stats = {
|
| 248 |
+
str(cache_dir): {"new": 0, "overwrite": 0, "skip": 0}
|
| 249 |
+
for cache_dir in cache_dirs
|
| 250 |
+
}
|
| 251 |
+
|
| 252 |
+
prompts = prompts[rank::world_size] if is_distributed else prompts
|
| 253 |
+
|
| 254 |
+
if not overwrite:
|
| 255 |
+
fully_cached_local = 0
|
| 256 |
+
prompts_to_encode: list[str] = []
|
| 257 |
+
for prompt in prompts:
|
| 258 |
+
hashed = hashlib.sha256(prompt.encode("utf-8")).hexdigest()
|
| 259 |
+
filename = f"{hashed}.t5_len{context_len}.{enc_id}.pt"
|
| 260 |
+
fully_cached = True
|
| 261 |
+
for cache_dir in cache_dirs:
|
| 262 |
+
cache_path = cache_dir / filename
|
| 263 |
+
if not cache_path.exists():
|
| 264 |
+
fully_cached = False
|
| 265 |
+
break
|
| 266 |
+
if fully_cached:
|
| 267 |
+
fully_cached_local += 1
|
| 268 |
+
for cache_dir in cache_dirs:
|
| 269 |
+
stats[str(cache_dir)]["skip"] += 1
|
| 270 |
+
else:
|
| 271 |
+
prompts_to_encode.append(prompt)
|
| 272 |
+
|
| 273 |
+
prompts = prompts_to_encode
|
| 274 |
+
|
| 275 |
+
fully_cached_global = fully_cached_local
|
| 276 |
+
to_encode_global = len(prompts)
|
| 277 |
+
if is_distributed:
|
| 278 |
+
reduce_device = torch.device(device) if device.startswith("cuda") else torch.device("cpu")
|
| 279 |
+
count_tensor = torch.tensor([fully_cached_local, len(prompts)], device=reduce_device, dtype=torch.long)
|
| 280 |
+
dist.all_reduce(count_tensor, op=dist.ReduceOp.SUM)
|
| 281 |
+
fully_cached_global = int(count_tensor[0].item())
|
| 282 |
+
to_encode_global = int(count_tensor[1].item())
|
| 283 |
+
|
| 284 |
+
if (not is_distributed) or rank == 0:
|
| 285 |
+
logger.info(
|
| 286 |
+
"overwrite=false: fully cached prompts=%d, prompts to encode=%d",
|
| 287 |
+
fully_cached_global,
|
| 288 |
+
to_encode_global,
|
| 289 |
+
)
|
| 290 |
+
|
| 291 |
+
logger.info("Writing caches to %d directories.", len(cache_dirs))
|
| 292 |
+
prompts_encoded_local = len(prompts)
|
| 293 |
+
prompts_encoded_global = prompts_encoded_local
|
| 294 |
+
if is_distributed:
|
| 295 |
+
reduce_device = torch.device(device) if device.startswith("cuda") else torch.device("cpu")
|
| 296 |
+
count_tensor = torch.tensor([prompts_encoded_local], device=reduce_device, dtype=torch.long)
|
| 297 |
+
dist.all_reduce(count_tensor, op=dist.ReduceOp.SUM)
|
| 298 |
+
prompts_encoded_global = int(count_tensor.item())
|
| 299 |
+
|
| 300 |
+
over_length_prompts = 0
|
| 301 |
+
with tqdm(
|
| 302 |
+
total=len(prompts),
|
| 303 |
+
desc=f"Encoding prompts (rank {rank}/{world_size})" if is_distributed else "Encoding prompts",
|
| 304 |
+
unit="prompt",
|
| 305 |
+
dynamic_ncols=True,
|
| 306 |
+
disable=is_distributed and rank != 0,
|
| 307 |
+
) as pbar:
|
| 308 |
+
with torch.no_grad():
|
| 309 |
+
for start in range(0, len(prompts), DEFAULT_BATCH_SIZE):
|
| 310 |
+
batch_prompts = prompts[start : start + DEFAULT_BATCH_SIZE]
|
| 311 |
+
ids, mask = tokenizer(batch_prompts, return_mask=True, add_special_tokens=True)
|
| 312 |
+
ids = ids.to(device)
|
| 313 |
+
mask = mask.to(device=device, dtype=torch.bool)
|
| 314 |
+
over_length_prompts += int(mask.all(dim=1).sum().item())
|
| 315 |
+
context = text_encoder(ids, mask)
|
| 316 |
+
|
| 317 |
+
for i, prompt in enumerate(batch_prompts):
|
| 318 |
+
hashed = hashlib.sha256(prompt.encode("utf-8")).hexdigest()
|
| 319 |
+
context_i = context[i].detach().to(device="cpu", dtype=torch.bfloat16).contiguous()
|
| 320 |
+
mask_i = mask[i].detach().to(device="cpu", dtype=torch.bool).contiguous()
|
| 321 |
+
payload = {
|
| 322 |
+
"context": context_i,
|
| 323 |
+
"mask": mask_i,
|
| 324 |
+
}
|
| 325 |
+
|
| 326 |
+
for cache_dir in cache_dirs:
|
| 327 |
+
cache_path = cache_dir / f"{hashed}.t5_len{context_len}.{enc_id}.pt"
|
| 328 |
+
key = str(cache_dir)
|
| 329 |
+
if cache_path.exists() and not overwrite:
|
| 330 |
+
stats[key]["skip"] += 1
|
| 331 |
+
continue
|
| 332 |
+
|
| 333 |
+
if cache_path.exists():
|
| 334 |
+
stats[key]["overwrite"] += 1
|
| 335 |
+
else:
|
| 336 |
+
stats[key]["new"] += 1
|
| 337 |
+
|
| 338 |
+
_atomic_torch_save(payload, cache_path)
|
| 339 |
+
|
| 340 |
+
pbar.update(len(batch_prompts))
|
| 341 |
+
|
| 342 |
+
over_length_global = over_length_prompts
|
| 343 |
+
if is_distributed:
|
| 344 |
+
reduce_device = torch.device(device) if device.startswith("cuda") else torch.device("cpu")
|
| 345 |
+
over_tensor = torch.tensor([over_length_prompts], device=reduce_device, dtype=torch.long)
|
| 346 |
+
dist.all_reduce(over_tensor, op=dist.ReduceOp.SUM)
|
| 347 |
+
over_length_global = int(over_tensor.item())
|
| 348 |
+
|
| 349 |
+
counts_tensor = torch.tensor(
|
| 350 |
+
[
|
| 351 |
+
[stats[str(cache_dir)]["new"], stats[str(cache_dir)]["overwrite"], stats[str(cache_dir)]["skip"]]
|
| 352 |
+
for cache_dir in cache_dirs
|
| 353 |
+
],
|
| 354 |
+
device=reduce_device,
|
| 355 |
+
dtype=torch.long,
|
| 356 |
+
)
|
| 357 |
+
dist.all_reduce(counts_tensor, op=dist.ReduceOp.SUM)
|
| 358 |
+
if rank == 0:
|
| 359 |
+
for idx, cache_dir in enumerate(cache_dirs):
|
| 360 |
+
key = str(cache_dir)
|
| 361 |
+
stats[key]["new"] = int(counts_tensor[idx, 0].item())
|
| 362 |
+
stats[key]["overwrite"] = int(counts_tensor[idx, 1].item())
|
| 363 |
+
stats[key]["skip"] = int(counts_tensor[idx, 2].item())
|
| 364 |
+
|
| 365 |
+
if (not is_distributed) or rank == 0:
|
| 366 |
+
logger.info("Finished precomputing text embeddings.")
|
| 367 |
+
logger.info(
|
| 368 |
+
"Over-length prompts (mask all True, i.e. no padding after truncation/max_length=%d): %d/%d",
|
| 369 |
+
context_len,
|
| 370 |
+
over_length_global,
|
| 371 |
+
prompts_encoded_global,
|
| 372 |
+
)
|
| 373 |
+
for cache_dir in cache_dirs:
|
| 374 |
+
key = str(cache_dir)
|
| 375 |
+
logger.info(
|
| 376 |
+
"Cache dir: %s | new=%d overwrite=%d skip=%d",
|
| 377 |
+
key,
|
| 378 |
+
stats[key]["new"],
|
| 379 |
+
stats[key]["overwrite"],
|
| 380 |
+
stats[key]["skip"],
|
| 381 |
+
)
|
| 382 |
+
|
| 383 |
+
if is_distributed and dist.is_initialized():
|
| 384 |
+
dist.barrier()
|
| 385 |
+
dist.destroy_process_group()
|
| 386 |
+
|
| 387 |
+
|
| 388 |
+
if __name__ == "__main__":
|
| 389 |
+
main()
|
baseline/fastwam_zipper/source/scripts/prepare_galaxea_r1_lite_overlay.py
ADDED
|
@@ -0,0 +1,141 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Build a metadata-only FastWAM view of a canonical Galaxea LeRobot-v2 pack.
|
| 3 |
+
|
| 4 |
+
FastWAM vendors a LeRobot v2.1 loader which requires every statistics ``count``
|
| 5 |
+
to have shape ``[1]``. The shared converter stores an equivalent count once per
|
| 6 |
+
vector component. This script collapses only that redundant field in a private
|
| 7 |
+
overlay; parquet and video payloads remain read-only symlinks to the canonical
|
| 8 |
+
pack.
|
| 9 |
+
"""
|
| 10 |
+
|
| 11 |
+
from __future__ import annotations
|
| 12 |
+
|
| 13 |
+
import argparse
|
| 14 |
+
import hashlib
|
| 15 |
+
import json
|
| 16 |
+
import os
|
| 17 |
+
import shutil
|
| 18 |
+
import tempfile
|
| 19 |
+
from pathlib import Path
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
MARKER_NAME = "fastwam_overlay.json"
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def sha256(path: Path) -> str:
|
| 26 |
+
digest = hashlib.sha256()
|
| 27 |
+
with path.open("rb") as handle:
|
| 28 |
+
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
| 29 |
+
digest.update(chunk)
|
| 30 |
+
return digest.hexdigest()
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def metadata_hashes(meta: Path) -> dict[str, str]:
|
| 34 |
+
return {
|
| 35 |
+
path.relative_to(meta).as_posix(): sha256(path)
|
| 36 |
+
for path in sorted(meta.rglob("*"))
|
| 37 |
+
if path.is_file() and path.name != MARKER_NAME
|
| 38 |
+
}
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
def normalized_stats(source: Path) -> tuple[dict, list[str]]:
|
| 42 |
+
stats = json.loads(source.read_text(encoding="utf-8"))
|
| 43 |
+
changed: list[str] = []
|
| 44 |
+
for feature, values in stats.items():
|
| 45 |
+
if not isinstance(values, dict) or "count" not in values:
|
| 46 |
+
continue
|
| 47 |
+
count = values["count"]
|
| 48 |
+
if not isinstance(count, list) or not count:
|
| 49 |
+
raise ValueError(f"{source}: {feature}.count must be a nonempty list")
|
| 50 |
+
if len(count) == 1:
|
| 51 |
+
continue
|
| 52 |
+
if any(value != count[0] for value in count[1:]):
|
| 53 |
+
raise ValueError(f"{source}: {feature}.count is not redundant: {count}")
|
| 54 |
+
values["count"] = [count[0]]
|
| 55 |
+
changed.append(feature)
|
| 56 |
+
return stats, changed
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
def validate_existing(source: Path, destination: Path) -> bool:
|
| 60 |
+
marker_path = destination / "meta" / MARKER_NAME
|
| 61 |
+
if not marker_path.is_file():
|
| 62 |
+
return False
|
| 63 |
+
marker = json.loads(marker_path.read_text(encoding="utf-8"))
|
| 64 |
+
expected = {
|
| 65 |
+
"format": "fastwam-galaxea-metadata-overlay-v1",
|
| 66 |
+
"source": str(source.resolve()),
|
| 67 |
+
"source_meta_sha256": metadata_hashes(source / "meta"),
|
| 68 |
+
}
|
| 69 |
+
if any(marker.get(key) != value for key, value in expected.items()):
|
| 70 |
+
return False
|
| 71 |
+
for payload in ("data", "videos"):
|
| 72 |
+
link = destination / payload
|
| 73 |
+
if not link.is_symlink() or link.resolve() != (source / payload).resolve():
|
| 74 |
+
return False
|
| 75 |
+
return marker.get("overlay_meta_sha256") == metadata_hashes(destination / "meta")
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
def build_overlay(source: Path, destination: Path) -> dict:
|
| 79 |
+
source = source.resolve()
|
| 80 |
+
if not (source / "meta" / "stats.json").is_file():
|
| 81 |
+
raise FileNotFoundError(f"missing canonical metadata: {source / 'meta/stats.json'}")
|
| 82 |
+
for payload in ("data", "videos"):
|
| 83 |
+
if not (source / payload).exists():
|
| 84 |
+
raise FileNotFoundError(f"missing canonical payload: {source / payload}")
|
| 85 |
+
|
| 86 |
+
if destination.exists():
|
| 87 |
+
if validate_existing(source, destination):
|
| 88 |
+
return json.loads(
|
| 89 |
+
(destination / "meta" / MARKER_NAME).read_text(encoding="utf-8")
|
| 90 |
+
)
|
| 91 |
+
raise FileExistsError(
|
| 92 |
+
f"refusing stale or foreign overlay: {destination}; remove that exact "
|
| 93 |
+
"FastWAM overlay before rebuilding"
|
| 94 |
+
)
|
| 95 |
+
|
| 96 |
+
destination.parent.mkdir(parents=True, exist_ok=True)
|
| 97 |
+
staging = Path(
|
| 98 |
+
tempfile.mkdtemp(prefix=f".{destination.name}.tmp.", dir=destination.parent)
|
| 99 |
+
)
|
| 100 |
+
try:
|
| 101 |
+
shutil.copytree(source / "meta", staging / "meta")
|
| 102 |
+
stats, changed = normalized_stats(source / "meta" / "stats.json")
|
| 103 |
+
(staging / "meta" / "stats.json").write_text(
|
| 104 |
+
json.dumps(stats, indent=2, sort_keys=True) + "\n", encoding="utf-8"
|
| 105 |
+
)
|
| 106 |
+
for payload in ("data", "videos"):
|
| 107 |
+
os.symlink(source / payload, staging / payload, target_is_directory=True)
|
| 108 |
+
|
| 109 |
+
marker = {
|
| 110 |
+
"format": "fastwam-galaxea-metadata-overlay-v1",
|
| 111 |
+
"source": str(source),
|
| 112 |
+
"source_meta_sha256": metadata_hashes(source / "meta"),
|
| 113 |
+
"overlay_meta_sha256": metadata_hashes(staging / "meta"),
|
| 114 |
+
"transformation": {
|
| 115 |
+
"file": "meta/stats.json",
|
| 116 |
+
"operation": "collapse identical per-dimension count values to one scalar count",
|
| 117 |
+
"features": changed,
|
| 118 |
+
},
|
| 119 |
+
"payload_storage": "absolute read-only symlinks; no data or videos copied",
|
| 120 |
+
}
|
| 121 |
+
(staging / "meta" / MARKER_NAME).write_text(
|
| 122 |
+
json.dumps(marker, indent=2, sort_keys=True) + "\n", encoding="utf-8"
|
| 123 |
+
)
|
| 124 |
+
os.replace(staging, destination)
|
| 125 |
+
return marker
|
| 126 |
+
except BaseException:
|
| 127 |
+
shutil.rmtree(staging, ignore_errors=True)
|
| 128 |
+
raise
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
def main() -> None:
|
| 132 |
+
parser = argparse.ArgumentParser()
|
| 133 |
+
parser.add_argument("--source", type=Path, required=True)
|
| 134 |
+
parser.add_argument("--destination", type=Path, required=True)
|
| 135 |
+
args = parser.parse_args()
|
| 136 |
+
marker = build_overlay(args.source, args.destination)
|
| 137 |
+
print(json.dumps({"status": "fastwam_overlay_ok", **marker}, indent=2, sort_keys=True))
|
| 138 |
+
|
| 139 |
+
|
| 140 |
+
if __name__ == "__main__":
|
| 141 |
+
main()
|
baseline/fastwam_zipper/source/scripts/preprocess_action_dit_backbone.py
ADDED
|
@@ -0,0 +1,240 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import argparse
|
| 2 |
+
from pathlib import Path
|
| 3 |
+
from typing import Any
|
| 4 |
+
|
| 5 |
+
import torch
|
| 6 |
+
import torch.nn.functional as F
|
| 7 |
+
from omegaconf import OmegaConf
|
| 8 |
+
|
| 9 |
+
from fastwam.models.wan22.action_dit import ActionDiT
|
| 10 |
+
from fastwam.models.wan22.helpers.loader import load_wan22_ti2v_5b_components
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
def _parse_dtype(name: str) -> torch.dtype:
|
| 14 |
+
value = str(name).strip().lower()
|
| 15 |
+
if value == "float32":
|
| 16 |
+
return torch.float32
|
| 17 |
+
if value == "float16":
|
| 18 |
+
return torch.float16
|
| 19 |
+
if value == "bfloat16":
|
| 20 |
+
return torch.bfloat16
|
| 21 |
+
raise ValueError(f"Unsupported dtype: {name}. Expected one of: float32, float16, bfloat16.")
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def _parse_bool(name: str) -> bool:
|
| 25 |
+
value = str(name).strip().lower()
|
| 26 |
+
if value in {"1", "true", "yes", "y"}:
|
| 27 |
+
return True
|
| 28 |
+
if value in {"0", "false", "no", "n"}:
|
| 29 |
+
return False
|
| 30 |
+
raise ValueError(f"Cannot parse bool value: {name}")
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def _is_unresolved_interpolation(value: Any) -> bool:
|
| 34 |
+
return isinstance(value, str) and "${" in value and "}" in value
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def _resolve_from_video_cfg(value: Any, video_cfg: dict[str, Any]) -> Any:
|
| 38 |
+
if not _is_unresolved_interpolation(value):
|
| 39 |
+
return value
|
| 40 |
+
text = str(value).strip()
|
| 41 |
+
if not (text.startswith("${") and text.endswith("}")):
|
| 42 |
+
return value
|
| 43 |
+
expr = text[2:-1]
|
| 44 |
+
if not expr.startswith("video_dit_config."):
|
| 45 |
+
return value
|
| 46 |
+
key = expr.split(".", 1)[1]
|
| 47 |
+
if key not in video_cfg:
|
| 48 |
+
return value
|
| 49 |
+
resolved = video_cfg[key]
|
| 50 |
+
return value if _is_unresolved_interpolation(resolved) else resolved
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def _interpolate_last_dim(tensor: torch.Tensor, new_size: int) -> torch.Tensor:
|
| 54 |
+
if tensor.shape[-1] == new_size:
|
| 55 |
+
return tensor
|
| 56 |
+
flat = tensor.reshape(-1, 1, tensor.shape[-1]).to(torch.float32)
|
| 57 |
+
flat = F.interpolate(flat, size=new_size, mode="linear", align_corners=True)
|
| 58 |
+
return flat.reshape(*tensor.shape[:-1], new_size)
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
def _resize_tensor_to_shape(src: torch.Tensor, target_shape: tuple[int, ...]) -> torch.Tensor:
|
| 62 |
+
if tuple(src.shape) == tuple(target_shape):
|
| 63 |
+
return src
|
| 64 |
+
|
| 65 |
+
out = src.to(torch.float32)
|
| 66 |
+
while out.ndim < len(target_shape):
|
| 67 |
+
out = out.unsqueeze(0)
|
| 68 |
+
while out.ndim > len(target_shape):
|
| 69 |
+
if out.shape[0] != 1:
|
| 70 |
+
raise ValueError(
|
| 71 |
+
f"Cannot reduce tensor rank for resize: src shape={tuple(src.shape)}, target={target_shape}"
|
| 72 |
+
)
|
| 73 |
+
out = out.squeeze(0)
|
| 74 |
+
|
| 75 |
+
for dim, new_size in enumerate(target_shape):
|
| 76 |
+
current_size = out.shape[dim]
|
| 77 |
+
if current_size == new_size:
|
| 78 |
+
continue
|
| 79 |
+
# Permute the target dimension to the end for interpolation
|
| 80 |
+
perm = [i for i in range(out.ndim) if i != dim] + [dim]
|
| 81 |
+
# Construct inverse permutation to restore original order
|
| 82 |
+
inv_perm = [0] * out.ndim
|
| 83 |
+
for i, p in enumerate(perm):
|
| 84 |
+
inv_perm[p] = i
|
| 85 |
+
# Permute, interpolate, and restore original order
|
| 86 |
+
out_perm = out.permute(*perm).contiguous()
|
| 87 |
+
prefix_shape = out_perm.shape[:-1]
|
| 88 |
+
out_perm = _interpolate_last_dim(out_perm, new_size)
|
| 89 |
+
out_perm = out_perm.reshape(*prefix_shape, new_size)
|
| 90 |
+
out = out_perm.permute(*inv_perm).contiguous()
|
| 91 |
+
|
| 92 |
+
if tuple(out.shape) != tuple(target_shape):
|
| 93 |
+
raise ValueError(
|
| 94 |
+
f"Resize produced wrong shape for tensor. src={tuple(src.shape)}, target={target_shape}, got={tuple(out.shape)}"
|
| 95 |
+
)
|
| 96 |
+
return out.to(dtype=src.dtype)
|
| 97 |
+
|
| 98 |
+
|
| 99 |
+
def _load_model_config(path: Path) -> tuple[dict[str, Any], dict[str, Any]]:
|
| 100 |
+
cfg = OmegaConf.load(str(path))
|
| 101 |
+
if "video_dit_config" not in cfg or "action_dit_config" not in cfg:
|
| 102 |
+
raise ValueError(
|
| 103 |
+
f"`{path}` must contain both `video_dit_config` and `action_dit_config` at top level."
|
| 104 |
+
)
|
| 105 |
+
|
| 106 |
+
video_cfg = OmegaConf.to_container(cfg.video_dit_config, resolve=False)
|
| 107 |
+
action_cfg = OmegaConf.to_container(cfg.action_dit_config, resolve=False)
|
| 108 |
+
if not isinstance(video_cfg, dict) or not isinstance(action_cfg, dict):
|
| 109 |
+
raise ValueError("`video_dit_config` and `action_dit_config` must resolve to dicts.")
|
| 110 |
+
|
| 111 |
+
if _is_unresolved_interpolation(video_cfg.get("action_dim")):
|
| 112 |
+
print("[WARN] `video_dit_config.action_dim` is unresolved; defaulting to 7 for preprocessing.")
|
| 113 |
+
video_cfg["action_dim"] = 7
|
| 114 |
+
|
| 115 |
+
if _is_unresolved_interpolation(action_cfg.get("action_dim")):
|
| 116 |
+
print("[WARN] `action_dit_config.action_dim` is unresolved; defaulting to 7 for preprocessing.")
|
| 117 |
+
action_cfg["action_dim"] = 7
|
| 118 |
+
|
| 119 |
+
for key in ["num_heads", "attn_head_dim", "num_layers", "text_dim", "freq_dim"]:
|
| 120 |
+
action_cfg[key] = _resolve_from_video_cfg(action_cfg.get(key), video_cfg)
|
| 121 |
+
|
| 122 |
+
return video_cfg, action_cfg, cfg
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
def _require_int_config(cfg: dict[str, Any], key: str) -> int:
|
| 126 |
+
value = cfg.get(key)
|
| 127 |
+
if _is_unresolved_interpolation(value):
|
| 128 |
+
raise ValueError(f"`{key}` is unresolved interpolation: {value}")
|
| 129 |
+
return int(value)
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
def _require_float_config(cfg: dict[str, Any], key: str) -> float:
|
| 133 |
+
value = cfg.get(key)
|
| 134 |
+
if _is_unresolved_interpolation(value):
|
| 135 |
+
raise ValueError(f"`{key}` is unresolved interpolation: {value}")
|
| 136 |
+
return float(value)
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
def main() -> None:
|
| 140 |
+
parser = argparse.ArgumentParser(
|
| 141 |
+
description="Preprocess ActionDiT backbone weights from WanVideoDiT and save as .pt payload."
|
| 142 |
+
)
|
| 143 |
+
parser.add_argument("--model-config", required=True, help="Path to model yaml, e.g. configs/model/fastwam.yaml")
|
| 144 |
+
parser.add_argument("--output", required=True, help="Output .pt path for preprocessed ActionDiT backbone.")
|
| 145 |
+
parser.add_argument("--device", default="cpu", help="Device for loading model and preprocessing.")
|
| 146 |
+
parser.add_argument("--dtype", default="float32", choices=["float32", "float16", "bfloat16"])
|
| 147 |
+
parser.add_argument(
|
| 148 |
+
"--apply-alpha-scaling",
|
| 149 |
+
default="true",
|
| 150 |
+
help="Whether to apply alpha=sqrt(dv/da) when the last dimension is resized (true/false). Default: true.",
|
| 151 |
+
)
|
| 152 |
+
args = parser.parse_args()
|
| 153 |
+
|
| 154 |
+
model_config_path = Path(args.model_config)
|
| 155 |
+
output_path = Path(args.output)
|
| 156 |
+
output_path.parent.mkdir(parents=True, exist_ok=True)
|
| 157 |
+
apply_alpha_scaling = _parse_bool(args.apply_alpha_scaling)
|
| 158 |
+
|
| 159 |
+
video_cfg, action_cfg, cfg = _load_model_config(model_config_path)
|
| 160 |
+
torch_dtype = _parse_dtype(args.dtype)
|
| 161 |
+
redirect_common_files = _parse_bool(cfg.get("redirect_common_files", False))
|
| 162 |
+
|
| 163 |
+
int_fields = ["hidden_dim", "action_dim", "ffn_dim", "num_layers", "num_heads", "attn_head_dim", "text_dim", "freq_dim"]
|
| 164 |
+
for key in int_fields:
|
| 165 |
+
action_cfg[key] = _require_int_config(action_cfg, key)
|
| 166 |
+
action_cfg["eps"] = _require_float_config(action_cfg, "eps")
|
| 167 |
+
|
| 168 |
+
print(f"[INFO] Loaded model config from {model_config_path}. "
|
| 169 |
+
f"Preprocessing ActionDiT backbone with dtype={torch_dtype} on device={args.device}, "
|
| 170 |
+
f"apply_alpha_scaling={apply_alpha_scaling}.")
|
| 171 |
+
components = load_wan22_ti2v_5b_components(
|
| 172 |
+
device=args.device,
|
| 173 |
+
torch_dtype=torch_dtype,
|
| 174 |
+
model_id=cfg.get("model_id", "Wan-AI/Wan2.2-TI2V-5B"),
|
| 175 |
+
tokenizer_model_id=cfg.get("tokenizer_model_id", "Wan-AI/Wan2.1-T2V-1.3B"),
|
| 176 |
+
redirect_common_files=redirect_common_files,
|
| 177 |
+
dit_config=video_cfg,
|
| 178 |
+
)
|
| 179 |
+
video_expert = components.dit
|
| 180 |
+
|
| 181 |
+
action_expert = ActionDiT(**action_cfg).to(device=args.device, dtype=torch_dtype)
|
| 182 |
+
if int(action_cfg["num_heads"]) != int(video_expert.num_heads):
|
| 183 |
+
raise ValueError("ActionDiT `num_heads` must match video expert for MoT mixed attention.")
|
| 184 |
+
if int(action_cfg["attn_head_dim"]) != int(video_expert.attn_head_dim):
|
| 185 |
+
raise ValueError("ActionDiT `attn_head_dim` must match video expert for MoT mixed attention.")
|
| 186 |
+
if int(action_cfg["num_layers"]) != int(len(video_expert.blocks)):
|
| 187 |
+
raise ValueError("ActionDiT `num_layers` must match video expert.")
|
| 188 |
+
|
| 189 |
+
action_state = action_expert.state_dict()
|
| 190 |
+
video_state = video_expert.state_dict()
|
| 191 |
+
backbone_keys = ActionDiT.backbone_key_set(action_state.keys())
|
| 192 |
+
|
| 193 |
+
backbone_state_dict: dict[str, torch.Tensor] = {}
|
| 194 |
+
copied = 0
|
| 195 |
+
interpolated = 0
|
| 196 |
+
for key in sorted(backbone_keys):
|
| 197 |
+
if key not in video_state:
|
| 198 |
+
raise ValueError(f"Key `{key}` not found in video expert state dict.")
|
| 199 |
+
src = video_state[key]
|
| 200 |
+
target = action_state[key]
|
| 201 |
+
if tuple(src.shape) == tuple(target.shape):
|
| 202 |
+
value = src
|
| 203 |
+
copied += 1
|
| 204 |
+
else:
|
| 205 |
+
value = _resize_tensor_to_shape(src, tuple(target.shape))
|
| 206 |
+
if apply_alpha_scaling and src.ndim >= 2 and src.shape[-1] != target.shape[-1]:
|
| 207 |
+
alpha = (float(src.shape[-1]) / float(target.shape[-1])) ** 0.5
|
| 208 |
+
value = value.to(torch.float32) * alpha
|
| 209 |
+
interpolated += 1
|
| 210 |
+
backbone_state_dict[key] = value.detach().to(dtype=target.dtype, device="cpu").contiguous()
|
| 211 |
+
|
| 212 |
+
payload = {
|
| 213 |
+
"policy": {
|
| 214 |
+
"skip_prefixes": list(ActionDiT.ACTION_BACKBONE_SKIP_PREFIXES),
|
| 215 |
+
"alpha_scaling": bool(apply_alpha_scaling),
|
| 216 |
+
"interpolation": "sequential_1d_linear_align_corners_true",
|
| 217 |
+
},
|
| 218 |
+
"backbone_state_dict": backbone_state_dict,
|
| 219 |
+
"meta": {
|
| 220 |
+
"hidden_dim": int(action_cfg["hidden_dim"]),
|
| 221 |
+
"ffn_dim": int(action_cfg["ffn_dim"]),
|
| 222 |
+
"num_layers": int(action_cfg["num_layers"]),
|
| 223 |
+
"num_heads": int(action_cfg["num_heads"]),
|
| 224 |
+
"attn_head_dim": int(action_cfg["attn_head_dim"]),
|
| 225 |
+
"text_dim": int(action_cfg["text_dim"]),
|
| 226 |
+
"freq_dim": int(action_cfg["freq_dim"]),
|
| 227 |
+
"eps": float(action_cfg["eps"]),
|
| 228 |
+
},
|
| 229 |
+
}
|
| 230 |
+
torch.save(payload, str(output_path))
|
| 231 |
+
|
| 232 |
+
skipped = len(action_state) - len(backbone_keys)
|
| 233 |
+
print(
|
| 234 |
+
"[INFO] Saved ActionDiT backbone payload to "
|
| 235 |
+
f"{output_path} (copied={copied}, interpolated={interpolated}, skipped={skipped})."
|
| 236 |
+
)
|
| 237 |
+
|
| 238 |
+
|
| 239 |
+
if __name__ == "__main__":
|
| 240 |
+
main()
|
baseline/fastwam_zipper/source/scripts/train.py
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import hydra
|
| 2 |
+
from omegaconf import DictConfig
|
| 3 |
+
|
| 4 |
+
from fastwam.runtime import run_training
|
| 5 |
+
from fastwam.utils.config_resolvers import register_default_resolvers
|
| 6 |
+
|
| 7 |
+
register_default_resolvers()
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
@hydra.main(config_path="../configs", config_name="train", version_base="1.3")
|
| 11 |
+
def main(cfg: DictConfig):
|
| 12 |
+
run_training(cfg)
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
if __name__ == "__main__":
|
| 16 |
+
main()
|
baseline/fastwam_zipper/source/scripts/train_galaxea_r1_lite.sh
ADDED
|
@@ -0,0 +1,125 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
set -euo pipefail
|
| 3 |
+
|
| 4 |
+
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
| 5 |
+
RUNTIME_ROOT="${FASTWAM_RUNTIME_ROOT:-/code/fastwam-runtime-v1}"
|
| 6 |
+
PYTHON="${RUNTIME_ROOT}/bin/python"
|
| 7 |
+
NPROC_PER_NODE="${1:?usage: scripts/train_galaxea_r1_lite.sh NPROC_PER_NODE RUN_NAME [hydra overrides...]}"
|
| 8 |
+
RUN_NAME="${2:?usage: scripts/train_galaxea_r1_lite.sh NPROC_PER_NODE RUN_NAME [hydra overrides...]}"
|
| 9 |
+
shift 2
|
| 10 |
+
|
| 11 |
+
CANONICAL_TRAIN_DATA="${GALAXEA_CANONICAL_TRAIN_DATA:-/dataset_rc_o1_p1/qshou@xiaopeng.com/dataset/galaxea_real_baselines_derived/pi05_lerobot_v2/train}"
|
| 12 |
+
CANONICAL_VAL_DATA="${GALAXEA_CANONICAL_VAL_DATA:-/dataset_rc_o1_p1/qshou@xiaopeng.com/dataset/galaxea_real_baselines_derived/pi05_lerobot_v2/validation}"
|
| 13 |
+
TRAIN_DATA="${GALAXEA_FASTWAM_TRAIN_DATA:-/dataset_rc_o1_p1/qshou@xiaopeng.com/dataset/galaxea_real_baselines_derived/fastwam_lerobot_v2/train}"
|
| 14 |
+
VAL_DATA="${GALAXEA_FASTWAM_VAL_DATA:-/dataset_rc_o1_p1/qshou@xiaopeng.com/dataset/galaxea_real_baselines_derived/fastwam_lerobot_v2/validation}"
|
| 15 |
+
MANIFEST="${GALAXEA_MANIFEST:-${ROOT}/../reports/galaxea_baselines/source_protocol/episodes_final.jsonl}"
|
| 16 |
+
BASE_CKPT="${FASTWAM_BASE_CHECKPOINT:-/dataset_rc_o1_p1/qshou@xiaopeng.com/weights/real_world_baselines/fastwam/robotwin_uncond_3cam_384.pt}"
|
| 17 |
+
CKPT_MARKER="$(dirname "${BASE_CKPT}")/DOWNLOAD_VERIFIED.json"
|
| 18 |
+
RUN_ROOT="${GALAXEA_FASTWAM_RUN_ROOT:-/dataset_rc_o1_p1/qshou@xiaopeng.com/training_runs/galaxea_real_baselines/fastwam}"
|
| 19 |
+
OUTPUT_DIR="${RUN_ROOT}/${RUN_NAME}"
|
| 20 |
+
EXPECTED_CKPT_SIZE=12041813092
|
| 21 |
+
EXPECTED_CKPT_SHA256="776475b22566a791854ecf31cf3b50f25e7d8d94c343132ec16eb94994aa9e63"
|
| 22 |
+
EXPECTED_TRAIN_EPISODES="${GALAXEA_EXPECTED_TRAIN_EPISODES:-511}"
|
| 23 |
+
EXPECTED_VAL_EPISODES="${GALAXEA_EXPECTED_VAL_EPISODES:-57}"
|
| 24 |
+
NUM_EPOCHS="${NUM_EPOCHS:-10}"
|
| 25 |
+
PER_DEVICE_BATCH_SIZE="${PER_DEVICE_BATCH_SIZE:-8}"
|
| 26 |
+
GRADIENT_ACCUMULATION_STEPS="${GRADIENT_ACCUMULATION_STEPS:-2}"
|
| 27 |
+
MIN_FREE_GIB="${GALAXEA_RUN_MIN_FREE_GIB:-2048}"
|
| 28 |
+
|
| 29 |
+
[[ "${NPROC_PER_NODE}" =~ ^[1-9][0-9]*$ ]] || { echo "invalid process count" >&2; exit 2; }
|
| 30 |
+
[[ "${RUN_NAME}" =~ ^[A-Za-z0-9._-]+$ ]] || { echo "unsafe run name" >&2; exit 2; }
|
| 31 |
+
[[ "${NUM_EPOCHS}" =~ ^[1-9][0-9]*$ ]] || { echo "invalid NUM_EPOCHS" >&2; exit 2; }
|
| 32 |
+
[[ "${PER_DEVICE_BATCH_SIZE}" =~ ^[1-9][0-9]*$ ]] || { echo "invalid PER_DEVICE_BATCH_SIZE" >&2; exit 2; }
|
| 33 |
+
[[ "${GRADIENT_ACCUMULATION_STEPS}" =~ ^[1-9][0-9]*$ ]] || { echo "invalid GRADIENT_ACCUMULATION_STEPS" >&2; exit 2; }
|
| 34 |
+
[[ -x "${PYTHON}" ]] || { echo "missing FastWAM runtime: ${PYTHON}" >&2; exit 2; }
|
| 35 |
+
[[ -f "${BASE_CKPT}" ]] || { echo "missing FastWAM checkpoint: ${BASE_CKPT}" >&2; exit 2; }
|
| 36 |
+
[[ "$(stat -c %s "${BASE_CKPT}")" = "${EXPECTED_CKPT_SIZE}" ]] || { echo "FastWAM checkpoint size mismatch" >&2; exit 2; }
|
| 37 |
+
[[ "$(sha256sum "${BASE_CKPT}" | awk '{print $1}')" = "${EXPECTED_CKPT_SHA256}" ]] || { echo "FastWAM checkpoint SHA256 mismatch" >&2; exit 2; }
|
| 38 |
+
[[ -f "${CKPT_MARKER}" ]] || { echo "missing FastWAM checkpoint verification marker" >&2; exit 2; }
|
| 39 |
+
"${PYTHON}" - "${BASE_CKPT}" "${CKPT_MARKER}" <<'PY'
|
| 40 |
+
import json
|
| 41 |
+
import pathlib
|
| 42 |
+
import sys
|
| 43 |
+
|
| 44 |
+
checkpoint = pathlib.Path(sys.argv[1])
|
| 45 |
+
marker = json.loads(pathlib.Path(sys.argv[2]).read_text(encoding="utf-8"))
|
| 46 |
+
if marker.get("file") != checkpoint.name:
|
| 47 |
+
raise SystemExit("FastWAM checkpoint marker filename mismatch")
|
| 48 |
+
if int(marker.get("size_bytes", -1)) != checkpoint.stat().st_size:
|
| 49 |
+
raise SystemExit("FastWAM checkpoint marker size mismatch")
|
| 50 |
+
if marker.get("sha256") != "776475b22566a791854ecf31cf3b50f25e7d8d94c343132ec16eb94994aa9e63":
|
| 51 |
+
raise SystemExit("FastWAM checkpoint marker SHA256 mismatch")
|
| 52 |
+
PY
|
| 53 |
+
mkdir -p "${RUN_ROOT}"
|
| 54 |
+
available_kib="$(df -Pk "${RUN_ROOT}" | awk 'NR==2 {print $4}')"
|
| 55 |
+
required_kib="$((MIN_FREE_GIB * 1024 * 1024))"
|
| 56 |
+
(( available_kib >= required_kib )) || {
|
| 57 |
+
echo "insufficient run space: available_kib=${available_kib} required_kib=${required_kib}" >&2; exit 4;
|
| 58 |
+
}
|
| 59 |
+
|
| 60 |
+
resume_path="${BASE_CKPT}"
|
| 61 |
+
if [[ -e "${OUTPUT_DIR}" ]]; then
|
| 62 |
+
latest_state="$(find "${OUTPUT_DIR}/checkpoints/state" -mindepth 1 -maxdepth 1 -type d -name 'step_[0-9]*' -printf '%f\n' 2>/dev/null | sort | tail -n 1)"
|
| 63 |
+
[[ -n "${latest_state}" ]] || {
|
| 64 |
+
echo "existing output has no resumable state: ${OUTPUT_DIR}" >&2; exit 3;
|
| 65 |
+
}
|
| 66 |
+
resume_path="${OUTPUT_DIR}/checkpoints/state/${latest_state}"
|
| 67 |
+
fi
|
| 68 |
+
|
| 69 |
+
export GALAXEA_FASTWAM_TRAIN_DATA="${TRAIN_DATA}"
|
| 70 |
+
export GALAXEA_FASTWAM_VAL_DATA="${VAL_DATA}"
|
| 71 |
+
export DIFFSYNTH_MODEL_BASE_PATH="${DIFFSYNTH_MODEL_BASE_PATH:-/workspace-wl/shouqx@xiaopeng.com/model}"
|
| 72 |
+
export DIFFSYNTH_SKIP_DOWNLOAD=true
|
| 73 |
+
export PATH="${RUNTIME_ROOT}/bin:${PATH}"
|
| 74 |
+
export WANDB_MODE="${WANDB_MODE:-online}"
|
| 75 |
+
[[ "${WANDB_MODE}" != online || -n "${WANDB_API_KEY:-}" ]] || {
|
| 76 |
+
echo "WANDB_MODE=online requires WANDB_API_KEY" >&2; exit 2;
|
| 77 |
+
}
|
| 78 |
+
|
| 79 |
+
cd "${ROOT}"
|
| 80 |
+
"${PYTHON}" scripts/prepare_galaxea_r1_lite_overlay.py \
|
| 81 |
+
--source "${CANONICAL_TRAIN_DATA}" --destination "${TRAIN_DATA}"
|
| 82 |
+
"${PYTHON}" scripts/prepare_galaxea_r1_lite_overlay.py \
|
| 83 |
+
--source "${CANONICAL_VAL_DATA}" --destination "${VAL_DATA}"
|
| 84 |
+
"${PYTHON}" scripts/check_galaxea_r1_lite_contract.py \
|
| 85 |
+
--train "${TRAIN_DATA}" --validation "${VAL_DATA}" --manifest "${MANIFEST}" \
|
| 86 |
+
--expected-train-episodes "${EXPECTED_TRAIN_EPISODES}" \
|
| 87 |
+
--expected-validation-episodes "${EXPECTED_VAL_EPISODES}"
|
| 88 |
+
|
| 89 |
+
effective_global_batch_size="$((PER_DEVICE_BATCH_SIZE * NPROC_PER_NODE * GRADIENT_ACCUMULATION_STEPS))"
|
| 90 |
+
steps_per_epoch="$("${PYTHON}" "${ROOT}/../scripts/plan_galaxea_epochs.py" \
|
| 91 |
+
"${TRAIN_DATA}" --epochs "${NUM_EPOCHS}" \
|
| 92 |
+
--effective-global-batch-size "${effective_global_batch_size}" \
|
| 93 |
+
--field steps_per_epoch)"
|
| 94 |
+
|
| 95 |
+
if [[ "${FASTWAM_PREFLIGHT_ONLY:-0}" = "1" ]]; then
|
| 96 |
+
echo "GALEXEA_FASTWAM_PREFLIGHT_OK"
|
| 97 |
+
exit 0
|
| 98 |
+
fi
|
| 99 |
+
|
| 100 |
+
echo "GALAXEA_FASTWAM_LAUNCH gpus=${NPROC_PER_NODE} effective_gbs=${effective_global_batch_size} epochs=${NUM_EPOCHS} steps_per_epoch=${steps_per_epoch} dataset=${TRAIN_DATA}"
|
| 101 |
+
|
| 102 |
+
if [[ "${FASTWAM_SKIP_TEXT_PRECOMPUTE:-0}" == "1" ]]; then
|
| 103 |
+
"${PYTHON}" scripts/check_galaxea_text_cache.py \
|
| 104 |
+
--train "${TRAIN_DATA}" --validation "${VAL_DATA}" \
|
| 105 |
+
--cache "${GALAXEA_FASTWAM_TEXT_CACHE:-/dataset_rc_o1_p1/qshou@xiaopeng.com/dataset/galaxea_real_baselines_derived/fastwam_text_cache}"
|
| 106 |
+
else
|
| 107 |
+
"${PYTHON}" -m torch.distributed.run --standalone --nproc_per_node="${NPROC_PER_NODE}" \
|
| 108 |
+
scripts/precompute_text_embeds.py task=galaxea_r1_lite_3cam_1e-4 +overwrite=false
|
| 109 |
+
fi
|
| 110 |
+
|
| 111 |
+
RUN_ID="${RUN_NAME}" bash scripts/train_zero1.sh "${NPROC_PER_NODE}" \
|
| 112 |
+
task=galaxea_r1_lite_3cam_1e-4 \
|
| 113 |
+
"output_dir=${OUTPUT_DIR}" \
|
| 114 |
+
"resume=${resume_path}" \
|
| 115 |
+
"num_epochs=${NUM_EPOCHS}" \
|
| 116 |
+
"batch_size=${PER_DEVICE_BATCH_SIZE}" \
|
| 117 |
+
"gradient_accumulation_steps=${GRADIENT_ACCUMULATION_STEPS}" \
|
| 118 |
+
"save_every=${steps_per_epoch}" \
|
| 119 |
+
"eval_every=${steps_per_epoch}" \
|
| 120 |
+
"wandb.enabled=true" \
|
| 121 |
+
"wandb.mode=online" \
|
| 122 |
+
"wandb.project=galaxea-real-baselines" \
|
| 123 |
+
"wandb.group=${GALAXEA_WANDB_GROUP:-galaxea-real-baselines-10ep}" \
|
| 124 |
+
"wandb.name=${RUN_NAME}" \
|
| 125 |
+
"$@"
|
baseline/fastwam_zipper/source/scripts/train_zero1.sh
ADDED
|
@@ -0,0 +1,116 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
set -euo pipefail
|
| 3 |
+
|
| 4 |
+
NPROC_PER_NODE="${1:?Usage: bash scripts/train_zero1.sh <nproc_per_node> [hydra_overrides...]}"
|
| 5 |
+
shift
|
| 6 |
+
|
| 7 |
+
EXTRA_ARGS=("$@")
|
| 8 |
+
NUM_MACHINES="${NNODES:-1}"
|
| 9 |
+
MACHINE_RANK="${NODE_RANK:-0}"
|
| 10 |
+
MAIN_PROCESS_IP="${MASTER_ADDR:-127.0.0.1}"
|
| 11 |
+
MAIN_PROCESS_PORT="${MASTER_PORT:-29500}"
|
| 12 |
+
|
| 13 |
+
is_integer() {
|
| 14 |
+
[[ "${1}" =~ ^[0-9]+$ ]]
|
| 15 |
+
}
|
| 16 |
+
|
| 17 |
+
if ! is_integer "${NUM_MACHINES}" || ! is_integer "${MACHINE_RANK}"; then
|
| 18 |
+
echo "Error: NUM_MACHINES (${NUM_MACHINES}) and MACHINE_RANK (${MACHINE_RANK}) must be integers." >&2
|
| 19 |
+
exit 1
|
| 20 |
+
fi
|
| 21 |
+
|
| 22 |
+
extract_task_basename() {
|
| 23 |
+
local cfg="$1"
|
| 24 |
+
if [[ "${cfg}" == task/* ]]; then
|
| 25 |
+
local name="${cfg#task/}"
|
| 26 |
+
name="${name%.yaml}"
|
| 27 |
+
echo "${name}"
|
| 28 |
+
return 0
|
| 29 |
+
fi
|
| 30 |
+
return 1
|
| 31 |
+
}
|
| 32 |
+
|
| 33 |
+
TASK_BASENAME="train"
|
| 34 |
+
for ((i = 0; i < ${#EXTRA_ARGS[@]}; i++)); do
|
| 35 |
+
arg="${EXTRA_ARGS[$i]}"
|
| 36 |
+
case "${arg}" in
|
| 37 |
+
--config-name)
|
| 38 |
+
if ((i + 1 < ${#EXTRA_ARGS[@]})); then
|
| 39 |
+
next="${EXTRA_ARGS[$((i + 1))]}"
|
| 40 |
+
if parsed="$(extract_task_basename "${next}")"; then
|
| 41 |
+
TASK_BASENAME="${parsed}"
|
| 42 |
+
fi
|
| 43 |
+
fi
|
| 44 |
+
;;
|
| 45 |
+
--config-name=*)
|
| 46 |
+
cfg="${arg#--config-name=}"
|
| 47 |
+
if parsed="$(extract_task_basename "${cfg}")"; then
|
| 48 |
+
TASK_BASENAME="${parsed}"
|
| 49 |
+
fi
|
| 50 |
+
;;
|
| 51 |
+
task=*)
|
| 52 |
+
cfg="${arg#task=}"
|
| 53 |
+
cfg="${cfg%.yaml}"
|
| 54 |
+
TASK_BASENAME="${cfg}"
|
| 55 |
+
;;
|
| 56 |
+
esac
|
| 57 |
+
done
|
| 58 |
+
|
| 59 |
+
if [[ -z "${RUN_ID:-}" ]]; then
|
| 60 |
+
if (( NUM_MACHINES <= 1 )); then
|
| 61 |
+
RUN_ID="$(date +%Y-%m-%d_%H-%M-%S)"
|
| 62 |
+
else
|
| 63 |
+
RUN_ID_SYNC_TIMEOUT="${RUN_ID_SYNC_TIMEOUT:-180}"
|
| 64 |
+
RUN_ID_SYNC_PORT="${RUN_ID_SYNC_PORT:-$((MAIN_PROCESS_PORT + 11))}"
|
| 65 |
+
|
| 66 |
+
export RUN_ID_SYNC_HOST="${MAIN_PROCESS_IP}"
|
| 67 |
+
export RUN_ID_SYNC_PORT
|
| 68 |
+
export RUN_ID_SYNC_TIMEOUT
|
| 69 |
+
export RUN_ID_SYNC_MACHINE_RANK="${MACHINE_RANK}"
|
| 70 |
+
export RUN_ID_SYNC_NUM_MACHINES="${NUM_MACHINES}"
|
| 71 |
+
export RUN_ID_SYNC_TASK_BASENAME="${TASK_BASENAME}"
|
| 72 |
+
|
| 73 |
+
RUN_ID="$(
|
| 74 |
+
python - <<'PY'
|
| 75 |
+
import datetime
|
| 76 |
+
import os
|
| 77 |
+
from datetime import timedelta
|
| 78 |
+
|
| 79 |
+
import torch.distributed as dist
|
| 80 |
+
|
| 81 |
+
host = os.environ["RUN_ID_SYNC_HOST"]
|
| 82 |
+
port = int(os.environ["RUN_ID_SYNC_PORT"])
|
| 83 |
+
timeout_s = int(os.environ["RUN_ID_SYNC_TIMEOUT"])
|
| 84 |
+
machine_rank = int(os.environ["RUN_ID_SYNC_MACHINE_RANK"])
|
| 85 |
+
num_machines = int(os.environ["RUN_ID_SYNC_NUM_MACHINES"])
|
| 86 |
+
task_basename = os.environ.get("RUN_ID_SYNC_TASK_BASENAME", "train")
|
| 87 |
+
|
| 88 |
+
store = dist.TCPStore(
|
| 89 |
+
host_name=host,
|
| 90 |
+
port=port,
|
| 91 |
+
world_size=num_machines,
|
| 92 |
+
is_master=(machine_rank == 0),
|
| 93 |
+
timeout=timedelta(seconds=timeout_s),
|
| 94 |
+
)
|
| 95 |
+
key = f"run_id::{task_basename}"
|
| 96 |
+
if machine_rank == 0:
|
| 97 |
+
run_id = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
|
| 98 |
+
store.set(key, run_id)
|
| 99 |
+
run_id = store.get(key).decode("utf-8")
|
| 100 |
+
print(run_id)
|
| 101 |
+
PY
|
| 102 |
+
)"
|
| 103 |
+
|
| 104 |
+
echo "[run_id_sync] mode=tcpstore host=${RUN_ID_SYNC_HOST} port=${RUN_ID_SYNC_PORT} timeout_s=${RUN_ID_SYNC_TIMEOUT} run_id=${RUN_ID}"
|
| 105 |
+
fi
|
| 106 |
+
fi
|
| 107 |
+
|
| 108 |
+
echo "[launch] nproc_per_node=${NPROC_PER_NODE} num_machines=${NUM_MACHINES} machine_rank=${MACHINE_RANK} run_id=${RUN_ID}"
|
| 109 |
+
|
| 110 |
+
python -m accelerate.commands.launch \
|
| 111 |
+
--config_file scripts/accelerate_configs/accelerate_zero1_ds.yaml \
|
| 112 |
+
--num_processes "${NPROC_PER_NODE}" \
|
| 113 |
+
scripts/train.py \
|
| 114 |
+
"output_dir=./runs/${TASK_BASENAME}/${RUN_ID}" \
|
| 115 |
+
"wandb.name=${TASK_BASENAME}" \
|
| 116 |
+
"${EXTRA_ARGS[@]}"
|
baseline/fastwam_zipper/source/scripts/train_zero2.sh
ADDED
|
@@ -0,0 +1,116 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
set -euo pipefail
|
| 3 |
+
|
| 4 |
+
NPROC_PER_NODE="${1:?Usage: bash scripts/train.sh <nproc_per_node> [hydra_overrides...]}"
|
| 5 |
+
shift
|
| 6 |
+
|
| 7 |
+
EXTRA_ARGS=("$@")
|
| 8 |
+
NUM_MACHINES="${NNODES:-1}"
|
| 9 |
+
MACHINE_RANK="${NODE_RANK:-0}"
|
| 10 |
+
MAIN_PROCESS_IP="${MASTER_ADDR:-127.0.0.1}"
|
| 11 |
+
MAIN_PROCESS_PORT="${MASTER_PORT:-29500}"
|
| 12 |
+
|
| 13 |
+
is_integer() {
|
| 14 |
+
[[ "${1}" =~ ^[0-9]+$ ]]
|
| 15 |
+
}
|
| 16 |
+
|
| 17 |
+
if ! is_integer "${NUM_MACHINES}" || ! is_integer "${MACHINE_RANK}"; then
|
| 18 |
+
echo "Error: NUM_MACHINES (${NUM_MACHINES}) and MACHINE_RANK (${MACHINE_RANK}) must be integers." >&2
|
| 19 |
+
exit 1
|
| 20 |
+
fi
|
| 21 |
+
|
| 22 |
+
extract_task_basename() {
|
| 23 |
+
local cfg="$1"
|
| 24 |
+
if [[ "${cfg}" == task/* ]]; then
|
| 25 |
+
local name="${cfg#task/}"
|
| 26 |
+
name="${name%.yaml}"
|
| 27 |
+
echo "${name}"
|
| 28 |
+
return 0
|
| 29 |
+
fi
|
| 30 |
+
return 1
|
| 31 |
+
}
|
| 32 |
+
|
| 33 |
+
TASK_BASENAME="train"
|
| 34 |
+
for ((i = 0; i < ${#EXTRA_ARGS[@]}; i++)); do
|
| 35 |
+
arg="${EXTRA_ARGS[$i]}"
|
| 36 |
+
case "${arg}" in
|
| 37 |
+
--config-name)
|
| 38 |
+
if ((i + 1 < ${#EXTRA_ARGS[@]})); then
|
| 39 |
+
next="${EXTRA_ARGS[$((i + 1))]}"
|
| 40 |
+
if parsed="$(extract_task_basename "${next}")"; then
|
| 41 |
+
TASK_BASENAME="${parsed}"
|
| 42 |
+
fi
|
| 43 |
+
fi
|
| 44 |
+
;;
|
| 45 |
+
--config-name=*)
|
| 46 |
+
cfg="${arg#--config-name=}"
|
| 47 |
+
if parsed="$(extract_task_basename "${cfg}")"; then
|
| 48 |
+
TASK_BASENAME="${parsed}"
|
| 49 |
+
fi
|
| 50 |
+
;;
|
| 51 |
+
task=*)
|
| 52 |
+
cfg="${arg#task=}"
|
| 53 |
+
cfg="${cfg%.yaml}"
|
| 54 |
+
TASK_BASENAME="${cfg}"
|
| 55 |
+
;;
|
| 56 |
+
esac
|
| 57 |
+
done
|
| 58 |
+
|
| 59 |
+
if [[ -z "${RUN_ID:-}" ]]; then
|
| 60 |
+
if (( NUM_MACHINES <= 1 )); then
|
| 61 |
+
RUN_ID="$(date +%Y-%m-%d_%H-%M-%S)"
|
| 62 |
+
else
|
| 63 |
+
RUN_ID_SYNC_TIMEOUT="${RUN_ID_SYNC_TIMEOUT:-180}"
|
| 64 |
+
RUN_ID_SYNC_PORT="${RUN_ID_SYNC_PORT:-$((MAIN_PROCESS_PORT + 11))}"
|
| 65 |
+
|
| 66 |
+
export RUN_ID_SYNC_HOST="${MAIN_PROCESS_IP}"
|
| 67 |
+
export RUN_ID_SYNC_PORT
|
| 68 |
+
export RUN_ID_SYNC_TIMEOUT
|
| 69 |
+
export RUN_ID_SYNC_MACHINE_RANK="${MACHINE_RANK}"
|
| 70 |
+
export RUN_ID_SYNC_NUM_MACHINES="${NUM_MACHINES}"
|
| 71 |
+
export RUN_ID_SYNC_TASK_BASENAME="${TASK_BASENAME}"
|
| 72 |
+
|
| 73 |
+
RUN_ID="$(
|
| 74 |
+
python - <<'PY'
|
| 75 |
+
import datetime
|
| 76 |
+
import os
|
| 77 |
+
from datetime import timedelta
|
| 78 |
+
|
| 79 |
+
import torch.distributed as dist
|
| 80 |
+
|
| 81 |
+
host = os.environ["RUN_ID_SYNC_HOST"]
|
| 82 |
+
port = int(os.environ["RUN_ID_SYNC_PORT"])
|
| 83 |
+
timeout_s = int(os.environ["RUN_ID_SYNC_TIMEOUT"])
|
| 84 |
+
machine_rank = int(os.environ["RUN_ID_SYNC_MACHINE_RANK"])
|
| 85 |
+
num_machines = int(os.environ["RUN_ID_SYNC_NUM_MACHINES"])
|
| 86 |
+
task_basename = os.environ.get("RUN_ID_SYNC_TASK_BASENAME", "train")
|
| 87 |
+
|
| 88 |
+
store = dist.TCPStore(
|
| 89 |
+
host_name=host,
|
| 90 |
+
port=port,
|
| 91 |
+
world_size=num_machines,
|
| 92 |
+
is_master=(machine_rank == 0),
|
| 93 |
+
timeout=timedelta(seconds=timeout_s),
|
| 94 |
+
)
|
| 95 |
+
key = f"run_id::{task_basename}"
|
| 96 |
+
if machine_rank == 0:
|
| 97 |
+
run_id = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
|
| 98 |
+
store.set(key, run_id)
|
| 99 |
+
run_id = store.get(key).decode("utf-8")
|
| 100 |
+
print(run_id)
|
| 101 |
+
PY
|
| 102 |
+
)"
|
| 103 |
+
|
| 104 |
+
echo "[run_id_sync] mode=tcpstore host=${RUN_ID_SYNC_HOST} port=${RUN_ID_SYNC_PORT} timeout_s=${RUN_ID_SYNC_TIMEOUT} run_id=${RUN_ID}"
|
| 105 |
+
fi
|
| 106 |
+
fi
|
| 107 |
+
|
| 108 |
+
echo "[launch] nproc_per_node=${NPROC_PER_NODE} num_machines=${NUM_MACHINES} machine_rank=${MACHINE_RANK} run_id=${RUN_ID}"
|
| 109 |
+
|
| 110 |
+
python -m accelerate.commands.launch \
|
| 111 |
+
--config_file scripts/accelerate_configs/accelerate_zero2_ds.yaml \
|
| 112 |
+
--num_processes "${NPROC_PER_NODE}" \
|
| 113 |
+
scripts/train.py \
|
| 114 |
+
"output_dir=./runs/${TASK_BASENAME}/${RUN_ID}" \
|
| 115 |
+
"wandb.name=${TASK_BASENAME}" \
|
| 116 |
+
"${EXTRA_ARGS[@]}"
|
baseline/fastwam_zipper/source/src/fastwam.egg-info/PKG-INFO
ADDED
|
@@ -0,0 +1,37 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Metadata-Version: 2.4
|
| 2 |
+
Name: fastwam
|
| 3 |
+
Version: 0.1.0
|
| 4 |
+
Summary: FastWAM: standalone Wan2.2-TI2V-5B training/inference package
|
| 5 |
+
Requires-Python: >=3.10
|
| 6 |
+
License-File: LICENSE
|
| 7 |
+
Requires-Dist: accelerate==1.12.0
|
| 8 |
+
Requires-Dist: av==15.1.0
|
| 9 |
+
Requires-Dist: boto3==1.35.99
|
| 10 |
+
Requires-Dist: datasets==4.8.5
|
| 11 |
+
Requires-Dist: deepspeed==0.18.7
|
| 12 |
+
Requires-Dist: einops==0.8.1
|
| 13 |
+
Requires-Dist: gitpython==3.1.45
|
| 14 |
+
Requires-Dist: huggingface-hub==0.29.2
|
| 15 |
+
Requires-Dist: hydra-core==1.3.2
|
| 16 |
+
Requires-Dist: imageio==2.37.0
|
| 17 |
+
Requires-Dist: imageio-ffmpeg==0.6.0
|
| 18 |
+
Requires-Dist: jsonlines==4.0.0
|
| 19 |
+
Requires-Dist: modelscope==1.34.0
|
| 20 |
+
Requires-Dist: numpy==2.2.6
|
| 21 |
+
Requires-Dist: omegaconf==2.3.0
|
| 22 |
+
Requires-Dist: packaging==25.0
|
| 23 |
+
Requires-Dist: pandas==2.2.3
|
| 24 |
+
Requires-Dist: pillow==12.0.0
|
| 25 |
+
Requires-Dist: pyarrow==24.0.0
|
| 26 |
+
Requires-Dist: regex==2025.11.3
|
| 27 |
+
Requires-Dist: rich==14.2.0
|
| 28 |
+
Requires-Dist: safetensors==0.5.3
|
| 29 |
+
Requires-Dist: termcolor==2.5.0
|
| 30 |
+
Requires-Dist: torch==2.7.1+cu128
|
| 31 |
+
Requires-Dist: torchcodec==0.4.0
|
| 32 |
+
Requires-Dist: torchvision==0.22.1+cu128
|
| 33 |
+
Requires-Dist: tqdm==4.68.3
|
| 34 |
+
Requires-Dist: transformers==4.49.0
|
| 35 |
+
Requires-Dist: typing-extensions==4.15.0
|
| 36 |
+
Requires-Dist: wandb==0.23.1
|
| 37 |
+
Dynamic: license-file
|