Adam-YAN/lingbot-e5c3aug-8h200-s20000-code

LingBot-VLA v2 (6B) E5 + C3 augmentation — 8×H200, global batch 1024, 20000 steps (training code)

This repo holds the code and actual configs for the formal 8×H200 training run on ACRC: Slurm job 170648, output directory runs/formal8_170648, finished 2026-10-25. It does not contain weights, data or the full upstream lingbot-vla-v2.

• Weights: Adam-YAN/lingbot-vla-v2-6b-e5c3aug-8h200-gbs1024-s20000. Step 20000 is in checkpoints/step_20000/, step 12000 in checkpoints/step_12000/, and validation records in validation/.
• Initialization: the official base model robbyant/lingbot-vla-v2-6b at revision 11c703bf6a5c1f45b3b69168482da11fdbba53d7, trained fresh (not continued from an earlier E5 checkpoint).

Results

Full validation: 54500 clean windows, augmentation off, task-macro L1_fm.

step	L1
4000	0.03161
8000	0.03068
12000	0.02995
20000	0.03006

The run took 47 h 51 min of wall time at about 7.8 s per step. Each full validation took about 31 min.

Actual training configuration (run_config/attempt_04_lingbotvla_cli.yaml)

Item	Value
Hardware	8 × H200, single node 7GS9B74, FSDP2 full shard, BF16 mixed
Batch	micro 64 × GA 2 × 8 GPU = global 1024
Steps	max_steps 20000 (about 41 epochs; the user knowingly chose this over 2 epochs)
LR	1e-4 → 5e-5 cosine, no warmup, Muon (match_rms_adamw)
Saving	save_steps 2000, keeping the latest 2 checkpoints
Validation	at steps 4000 / 8000 / 12000 / 20000, run after the training process exits
Data loading	12 workers per GPU, visual_steps 1000000 (image logging off)
Augmentation	C3 recipe v2 (configs/augmentation/c3_recipe_v2.json), with SegFormer background masks served from a precomputed cache

Relative to the package's frozen plan (8 GPU, micro 8, global 64, 2 epochs), the user made these changes, all recorded in the run config:

• micro128 × GA1 became micro64 × GA2, because micro96 hung in rank-0 visualization. Global batch stayed at 1024.
• Image logging was turned off (config_update_visual_off.json). This does not change the training math.
• Loader workers were set to 12 per GPU (config_update_workers12.json).

Directory layout

• deployment/acrc8/ — the ACRC 8-GPU controller.
  • aug8_train.py is the chained controller. It runs, in order, the mask-cache check (live vs cache), the 8-rank smoke, and the formal run in the same allocation. The cost model decides whether there is enough time, and exit code 3 means a clean stop on budget.
  • acrc8_rank_entry.py is the per-rank entry.
  • mask_cache.py builds and verifies the mask cache, with the mask_*.sbatch jobs. The cache is keyed on the 256×256 input frame. A cache built on AMD CPUs never matches on Intel, which is why the *_intel scripts exist.
  • run4.sbatch / smoke8.sbatch are the Slurm entry points.
  • config_update_*.json are the parameter overrides listed above.
• deployment/finetune/ — the package's generic E5 training orchestration, i.e. the modules this run actually imports. e5_train.py launches third_party/lingbot-vla-v2/tasks/vla/train_lingbotvla.py through e5_rank_entry.py / e5_8gpu_rank_entry.py.
• scripts/ — augmented data (aug_data.py), augmentation validation, CPU preflight, cost measurement, offline/parallel validation.
• configs/ — the 8-GPU (acrc8_…_s20000.yaml) and 4-GPU fallback (acrc4_…_s40000.yaml) configs, the augmentation recipe, validation definitions, and robot config.
• third_party/lingbot-vla-v2/ — only the 5 files involved in C3 augmentation, plus the original Apache-2.0 LICENSE:
  • c3_augmentation.py and c3_recipe_v2.py (the augmentation implementation);
  • base_dataset.py, utils.py and arguments.py (where C3 hooks into the upstream code).
  • The rest of lingbot-vla-v2 is not redistributed. Use the version frozen by the package: source archive sha256 d640da3b6c9a5a21624bd140a00aa8a3fedc6399680c87f8d86b1fef6fd42151 (see package_meta/source_manifest.json).
• environment/ — the package's environment definition (Dockerfile, install.sh, requirements, source-freeze).
• setup/ — bootstrap.py and install_env.sh as used on ACRC. Gaps found on ACRC and filled:
  • utils3d installed from GitHub commit 3fab839f;
  • lingbotvla exposed to the venv as a source tree via .pth;
  • flash-attn from the official 2.8.3 cu12 / torch 2.8 wheel.
• run_config/ — the lingbotvla_cli.yaml actually in effect for each of the 4 attempts. attempt_04 is the one that finished.
• package_meta/ — the original package's README, AGENT_TASK, experiment.json (design-stage plan), dependencies.json (pinned revisions of the base model, tokenizer Qwen3-VL-4B and MoGe), and source_manifest.json.

Reproduction notes

• The source package is HF dataset ar-mine/e5-c3-evaluation-deployment, directory training/e5-c3-augmentation-2epoch-20261021/ (payload commit 90fb43d3…, manifest sha256 3dbc745c…). Download and verify it with setup/bootstrap.py. Data: 50 tasks, 2250 / 250 trajectories, 495287 / 54500 frames, split phase4_split_s7.
• Put torch.compile's inductor/triton cache and TMPDIR on node-local /tmp, not on a shared file system such as WekaFS. Concurrent tmp+rename on a shared file system makes ranks see FileNotFoundError transiently.
• A DCP resume requires the same number of loader workers as when the checkpoint was saved (torchdata assertion).
• Each full checkpoint is about 73 GiB (the FP32 hf_ckpt alone is 24 GiB). Plan disk space accordingly.

All of these files are copied byte-for-byte from the cluster package. Per-file sha256 values are in FILE_SHA256.txt.