OpenWAM-3sys / migration /run01-cpu-load.log
qiuly's picture
Add files using upload-large-folder tool
9020600 verified
Raw History Blame Contribute Delete
8.95 kB
INFO:openwam.deploy.model_loader:Loading checkpoint: /mnt/data/limingleyang/checkpoint_eval_cache/archive-validation-run01/checkpoint_step_10690.safetensors
INFO:openwam.deploy.model_loader:Using config-embedded component specs for video-backbone construction
INFO:openwam.deploy.model_loader:Using self-contained Reason1 artifacts from /mnt/data/limingleyang/checkpoint_eval_cache/archive-validation-run01/reason1 (clearing external text_encoder_path='/mnt/data/limingleyang/model/Cosmos-Reason1-7B')
INFO:openwam.deploy.model_loader:Rebuilding the VLM expert from /mnt/data/limingleyang/checkpoint_eval_cache/archive-validation-run01/vlm_expert; weights come from the checkpoint safetensors
INFO:megatron.core.msc_utils:The multistorageclient package is available.
INFO:megatron.core.distributed.fsdp.src.megatron_fsdp.param_and_grad_buffer:Detected Megatron Core, using Megatron-FSDP with Megatron.
INFO:megatron.core.distributed.fsdp.src.megatron_fsdp.megatron_fsdp:Detected Megatron Core, using Megatron-FSDP with Megatron.
/mnt/data/limingleyang/conda_envs/vlm-expert/lib/python3.12/site-packages/megatron/core/optimizer/clip_grads.py:32: UserWarning: Transformer Engine and Apex are not installed. Falling back to local implementations of multi_tensor_applier, multi_tensor_l2norm, and multi_tensor_scale
warnings.warn(
/mnt/data/limingleyang/conda_envs/vlm-expert/lib/python3.12/site-packages/megatron/core/hyper_comm_grid.py:20: UserWarning: absl.logging is not installed. Using logging.getLogger(__name__) instead. Please install absl.logging with `pip install absl-py` to use absl.logging.
warnings.warn(
INFO:openwam.model.video_backbone.cosmos_predict25.pipeline_builder:Cosmos DiT: skipping EMA bootstrap (checkpoint)
INFO:openwam.model.video_backbone.cosmos_predict25.pipeline_builder:Cosmos Reason1: building empty shell (deploy path; weights from unified safetensors, structural artifacts from /mnt/data/limingleyang/checkpoint_eval_cache/archive-validation-run01/reason1)
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
INFO:openwam.model.video_backbone.cosmos_predict25.text_encoder:Reason1LiveTextEncoder.from_empty ready (artifact_dir=/mnt/data/limingleyang/checkpoint_eval_cache/archive-validation-run01/reason1, dtype=torch.bfloat16, device=cpu)
INFO:openwam.model.video_backbone.cosmos_predict25.pipeline_builder:Cosmos VAE: building empty shell (deploy path; weights from unified safetensors)
INFO:openwam.deploy.model_loader:Architecture: tri_system_joint_self_attn (framework=tri_system variant=joint_self_attn)
INFO:openwam.deploy.model_loader:Loading all models with dtype=torch.bfloat16 (mixed_precision=bf16)
INFO:openwam.deploy.model_loader:[normalizer] Resolving deployment action normalizer from checkpoint dir: /mnt/data/limingleyang/checkpoint_eval_cache/archive-validation-run01
INFO:openwam.deploy.model_loader:[normalizer] Saved config: normalize_mode=min-max, action_mode=libero
INFO:openwam.deploy.model_loader:[normalizer] Found pre-computed stats file: /mnt/data/limingleyang/checkpoint_eval_cache/archive-validation-run01/normalization_stats.npy (exists ✓)
INFO:openwam.deploy.model_loader:[normalizer] Active: mode=min-max action_mode=libero dim=10 stats=/mnt/data/limingleyang/checkpoint_eval_cache/archive-validation-run01/normalization_stats.npy
INFO:openwam.deploy.model_loader:[normalizer/unify] ON: action 80D -> raw 10D; raw state 10D -> 80D unified; then unnormalize.
INFO:openwam.deploy.model_loader:Model loaded successfully on cpu
{
"status": "passed",
"time": "2026-09-22T12:58:36.207452+00:00",
"run": "sft/run01",
"device": "cpu",
"parameters": 13321370179,
"elapsed_seconds": 110.58072506636381,
"materialized": "/mnt/data/limingleyang/checkpoint_eval_cache/archive-validation-run01",
"index_sha256": "3df09de3ec6b922dea2e39a73b913bed7ea1e989aaa5ab5b136ae1999cd99250",
"repo_commit": "cc4bf0c6bd3e016723c0b4050cd93b7d5c827ef6",
"historical_weight_paths_blocked": [
"/mnt/data/limingleyang/openwam_checkpoints/2026-09-07_23-49-38",
"/mnt/data/limingleyang/openwam_checkpoints/2026-09-09_13-14-42",
"/mnt/data/limingleyang/openwam_checkpoints/2026-09-10_01-16-34",
"/mnt/data/limingleyang/openwam_checkpoints/2026-09-10_10-07-03",
"/mnt/data/limingleyang/openwam_checkpoints/2026-09-10_11-45-23",
"/mnt/data/limingleyang/openwam_checkpoints/2026-09-10_17-47-49",
"/mnt/data/limingleyang/openwam_checkpoints/2026-09-10_21-59-35",
"/mnt/data/limingleyang/openwam_checkpoints/2026-09-10_22-00-06",
"/mnt/data/limingleyang/openwam_checkpoints/2026-09-11_00-38-41",
"/mnt/data/limingleyang/openwam_checkpoints/2026-09-11_00-39-04",
"/mnt/data/limingleyang/openwam_checkpoints/run11_2sys_scratch_dinov3/2026-09-12_00-52-00",
"/mnt/data/limingleyang/openwam_checkpoints/2026-09-11_14-53-35",
"/mnt/data/limingleyang/openwam_checkpoints/2026-09-11_14-54-15",
"/mnt/data/limingleyang/openwam_checkpoints/2026-09-11_15-28-19",
"/mnt/data/limingleyang/openwam_checkpoints/run15_3sys_frozen_prefix7/2026-09-12_00-52-52",
"/mnt/data/limingleyang/openwam_checkpoints/run16_3sys_frozen_prefix14/2026-09-12_00-53-18",
"/mnt/data/limingleyang/openwam_checkpoints/run17_3sys_frozen_prefix21/2026-09-12_00-53-45",
"/mnt/data/limingleyang/openwam_checkpoints/run18_3sys_frozen_scratch_dinov3/2026-09-12_17-07-24",
"/mnt/data/limingleyang/openwam_checkpoints/run19_2sys_scratch_videorae_ft/2026-09-12_19-23-18",
"/mnt/data/limingleyang/openwam_checkpoints/run20_3sys_vlm_dino/2026-09-13_00-38-39",
"/mnt/data/limingleyang/openwam_checkpoints/run21_2sys_scratch_videorae_current/2026-09-13_07-29-21",
"/mnt/data/limingleyang/openwam_checkpoints/run22-2sys-scratch-dinov3-patch2/sft_libero",
"/mnt/data/limingleyang/openwam_checkpoints/run23-2sys-pretrain-dinov3-patch2/pretrain",
"/mnt/data/limingleyang/openwam_checkpoints/run23-2sys-pretrain-dinov3-patch2/sft_libero",
"/mnt/data/limingleyang/openwam_checkpoints/run24_3sys_frozen_scratch_vjepa_svae/2026-09-14_15-40-37",
"/mnt/data/limingleyang/openwam_checkpoints/run25_2sys_scratch_vjepa_svae/2026-09-15_12-05-32",
"/mnt/data/limingleyang/OpenWAM-private/outputs/run26-future-dit-dinov3-frozen/2026-09-15_01-09-52",
"/mnt/data/limingleyang/OpenWAM-private/outputs/run27-future-dit-dinov3-train-vlm/2026-09-15_01-16-59",
"/mnt/data/limingleyang/openwam_checkpoints/run28_3sys_frozen_scratch_vjepa_svae_patch2/2026-09-16_15-42-25",
"/mnt/data/limingleyang/openwam_checkpoints/run29_2sys_scratch_vjepa_svae_patch2/2026-09-15_19-19-16",
"/mnt/data/limingleyang/vjepa_target_variants/run30_3sys_vjepa_pairwise/policy",
"/mnt/data/limingleyang/vjepa_target_variants/run31_3sys_vjepa_future_only/policy",
"/mnt/data/limingleyang/vjepa_target_variants/run32_2sys_vjepa_pairwise/2026-09-16_13-55-43",
"/mnt/data/limingleyang/vjepa_target_variants/run33_2sys_vjepa_future_only/2026-09-16_15-29-52",
"/mnt/data/limingleyang/openwam_temporal_pipeline/run34_3sys_frozen_scratch_dinov3_svae_temporal/policy",
"/mnt/data/limingleyang/openwam_temporal_pipeline/run35_2sys_scratch_dinov3_svae_temporal/policy",
"/mnt/data/limingleyang/openwam_checkpoints/run36_3sys_frozen_scratch_dinov3_patch2/2026-09-17_15-39-44",
"/mnt/data/limingleyang/future_dit_vjepa_targets/run37-future-dit-vjepa-with-f0/policy",
"/mnt/data/limingleyang/future_dit_vjepa_targets/run38-future-dit-vjepa-without-f0/policy",
"/mnt/data/limingleyang/openwam_checkpoints/run39-3sys-pretrain-dinov3-patch2/pretrain",
"/mnt/data/limingleyang/openwam_checkpoints/run39-3sys-pretrain-dinov3-patch2/sft_libero",
"/mnt/data/limingleyang/openwam_checkpoints/run40-3sys-from-2sys-pretrain-dinov3-patch2/sft_libero",
"/mnt/data/limingleyang/openwam_svae/libero_dinov3_vitb16/svae48",
"/mnt/data/limingleyang/openwam_artifacts/run23-2sys-pretrain-dinov3-patch2/svae_train/svae48",
"/mnt/data/limingleyang/yfhu25/openwam_svae/libero_vjepa21_vitg384/svae48",
"/mnt/data/limingleyang/OpenWAM-private/artifacts/future_dit_libero_v1/svae48",
"/mnt/data/limingleyang/vjepa_target_variants/run30_3sys_vjepa_pairwise/svae48",
"/mnt/data/limingleyang/vjepa_target_variants/run31_3sys_vjepa_future_only/svae48",
"/mnt/data/limingleyang/openwam_temporal_pipeline/run34_3sys_frozen_scratch_dinov3_svae_temporal/svae48",
"/mnt/data/limingleyang/openwam_temporal_pipeline/run35_2sys_scratch_dinov3_svae_temporal/svae48",
"/mnt/data/limingleyang/future_dit_vjepa_targets/run37-future-dit-vjepa-with-f0/svae48",
"/mnt/data/limingleyang/future_dit_vjepa_targets/run38-future-dit-vjepa-without-f0/svae48",
"/mnt/data/limingleyang/openwam_videorae/run19_2sys_scratch_videorae_ft",
"/mnt/data/limingleyang/model",
"/mnt/data/limingleyang/models"
],
"network_model_downloads": "offline",
"forward_or_closed_loop_eval": false
}