EgoForesight-base

Pretrained VLA · post-training initialization

GitHub · Paper (arXiv) · Hugging Face · BibTeX

The pretrained EgoForesight base VLA for downstream policy learning.

This is a full native checkpoint, not an adapter or a standalone Transformers AutoModel repository. The retained framework is Qwen35MoT3 in the starVLA namespace. Pretraining scripts, logs, private data pipelines, and machine-specific paths are not part of this upload.

Evaluation

This repository is a post-training initialization checkpoint, not a benchmark-specific deployed policy. No standalone base success rate is reported. See the RoboCasa and RoboTwin checkpoint cards for their task-specific results. evaluation.json makes this scope explicit.

Model

Property	Value
Architecture	Qwen3.5-4B + DINOv3 ViT-B/16 + MoT3 / Foresight Forcing
Intended use	Downstream post-training / warm start
Weight selection	Foresight Forcing EMA already merged into `mot_model.*`
Action / state width	130 canonical coordinates
Predicted action chunk	30 timesteps
Saved flow inference default	4 steps
EEF representation	Relative, `camera_spatial`
Normalization	Multiple source-specific statistics; no automatic benchmark adapter
Export precision	Original supplied tensor precision; no repackaging cast

The model conditions on current visual observations, language, robot state and embodiment metadata. Observed future frames are training supervision, not deployment inputs. Downstream recipes define the embodiment, action layout, normalization, and action-chunk length appropriate to their own task.

The supplied export manifest reports 1,627 state-dict entries and 711 EMA MoT3 replacements. This repackaging preserves the weight file byte-for-byte. dataset_statistics.json is also unchanged; its keys are compatibility identifiers, not a distribution of the original datasets. Do not select an arbitrary source's statistics for a new robot or use canonical outputs as direct control commands.

Files

EgoForesight-base/
  checkpoints/pytorch_model.pt
  config.yaml
  dataset_statistics.json
  README.md
  CITATION.bib
  evaluation.json
  export_manifest.json
  validation.json
  SHA256SUMS
  requirements.txt
  LICENSE
  NOTICE.md
  licenses/
  .gitattributes
  .gitignore

No optimizer/scheduler state, separate EMA sidecar, raw datasets, training logs, or pretraining launchers are included. Use a downstream recipe rather than attempting to resume pretraining from this inference/warm-start package.

Load and serve

Set up the EgoForesight code release following docs/environment.md. requirements.txt matches the companion policy/training runtime; install the specified PyTorch/CUDA build before its CUDA extensions. Run from the EgoForesight code root:

export PYTHONPATH="$PWD"
export MODEL_DIR=/absolute/path/to/EgoForesight-base
export QWEN_MODEL=/absolute/path/to/Qwen3.5-4B
export DINO_REPO=/absolute/path/to/dinov3
export DINO_WEIGHTS=/absolute/path/to/dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth
export PRETRAINED_CHECKPOINT="$MODEL_DIR/checkpoints/pytorch_model.pt"
export DATA_ROOT=/absolute/path/to/prepared/robocasa-aligned

# Inspect the downstream launch command first; this does not start training.
DRY_RUN=1 NUM_GPUS=1 BATCH_SIZE=1 GLOBAL_BATCH_SIZE=1 \
  bash scripts/posttrain.sh configs/posttrain/robocasa.yaml

Use configs/posttrain/robotwin_full_no_mirror.yaml for RoboTwin full-data post-training, with the matching prepared dataset and runtime settings. Choose actual batch geometry and hardware before removing DRY_RUN=1. The downstream recipe, not this package's loading configuration, defines the new run's optimization and dataset. This is warm start, not optimizer resume.

For native model loading with the same asset environment variables:

import os
from pathlib import Path
import torch
from starVLA.model.framework.base_framework import baseframework

checkpoint = Path(os.environ["MODEL_DIR"]) / "checkpoints/pytorch_model.pt"
model = baseframework.from_pretrained(str(checkpoint))
model = model.to(dtype=torch.bfloat16, device="cuda").eval()
# Configure a matching downstream action-space adapter before robot deployment.

The native constructor still needs the official Qwen tokenizer/processor and DINOv3 assets, even though their model tensors are included in the full state dict. QWEN_MODEL defaults to Qwen/Qwen3.5-4B; DINO_REPO and DINO_WEIGHTS must be set. Keep config.yaml and dataset_statistics.json at the model root, alongside checkpoints/.

The EMA is already merged. If using a compatible policy server after configuring the appropriate action-space adapter, select --ema_teacher online; do not request a second EMA merge or a nonexistent sidecar. For ready-to-evaluate simulator policies, use the corresponding post-trained checkpoint instead.

Verification and provenance

export_manifest.json records the original input hashes, new checkpoint location, EMA provenance, and configuration cleanup. validation.json distinguishes the supplied historical export verification from the current byte-preserving packaging checks. No new base-model inference or simulator benchmark was run during repackaging.

Verify all downloaded files from the model directory:

sha256sum --check SHA256SUMS
# macOS alternative: shasum -a 256 --check SHA256SUMS

License and attribution

EgoForesight additions: Chubin Zhang, built on StarVLA. The original EgoForesight weights, configuration and documentation retain their MIT notice. Included Qwen3.5 and DINOv3 components retain their respective upstream terms; the root MIT license does not replace those terms. See NOTICE.md, LICENSE and the full license texts in licenses/.

Citation

If you find EgoForesight useful in your research, please cite:

@article{zhang2026egoforesight,
  title={EgoForesight: Effectively Scaling Policy Learning with Noisy Egocentric Actions},
  author={Zhang, Chubin and Wang, Changyuan and Wang, Yirui and Li, Ao and Ma, Shilin and Su, Yue and Zhou, Cai and Xu, Xiuwei and Zhang, Tianle and Tang, Yansong},
  journal={arXiv preprint arXiv:2610.31845},
  year={2026}
}