Yangyihui commited on
Commit
f5decbe
路
verified 路
1 Parent(s): 114270d

Clarify ScaleNav usage and fix page previews

Browse files
Files changed (3) hide show
  1. .gitattributes +1 -0
  2. README.md +53 -36
  3. replay.mp4 +3 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ replay.mp4 filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -11,6 +11,14 @@ tags:
11
  - video-diffusion
12
  - image-text-to-action
13
  - safetensors
 
 
 
 
 
 
 
 
14
  ---
15
 
16
  # ScaleNav
@@ -19,70 +27,65 @@ tags:
19
 
20
  [Paper](https://arxiv.org/abs/2610.25690) 路 [Code](https://github.com/MachEmbodied/ScaleNav) 路 [Data](https://huggingface.co/datasets/Yangyihui/ScaleNav-Data) 路 [Project page](https://zju3dv.github.io/ScaleNav/) 路 [Collection](https://huggingface.co/collections/Yangyihui/scalenav-6ad591e8ff62bb75ec75f645)
21
 
22
- ScaleNav learns navigation priors from Internet egocentric videos through
23
- action-free pretraining and transfers them to action generation for
24
- language-conditioned local navigation. The goal is to reach a language-specified
25
- target visible in the initial observation and stop at the instructed location.
26
 
27
- ## License
28
 
29
- The ScaleNav fine-tuned checkpoint is released under [CC BY-NC 4.0](LICENSE)
30
- for noncommercial use. When sharing it or adaptations, provide attribution,
31
- link to the license, and indicate any changes.
32
-
33
- ScaleNav modifies [Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B)
34
- through navigation pretraining, action alignment and joint fine-tuning. The
35
- CC BY-NC 4.0 terms apply to the ScaleNav contributions; upstream Wan materials
36
- retain their [Apache-2.0 license](LICENSE-WAN). The
37
- [code](https://github.com/MachEmbodied/ScaleNav/blob/main/LICENSE) is separately
38
- licensed under Apache-2.0, and [dataset components](https://huggingface.co/datasets/Yangyihui/ScaleNav-Data#sources-and-license)
39
- retain their respective terms.
40
 
41
  ## Download
42
 
43
- Install ScaleNav using the [installation guide](https://github.com/MachEmbodied/ScaleNav#installation),
44
- then run from the code repository root. This model repository is public;
45
- authentication is optional:
46
 
47
  ```bash
48
  python scripts/download_checkpoint.py --output checkpoints/scalenav
49
  ```
50
 
51
- The downloader uses the revision and file checksums in the
52
- [download manifest](https://github.com/MachEmbodied/ScaleNav/blob/main/docs/checkpoint-download-manifest.json),
53
- resumes interrupted downloads and verifies SHA-256.
 
 
 
 
 
54
 
55
- To download directly with the Hugging Face CLI:
 
56
 
57
  ```bash
58
  hf download Yangyihui/ScaleNav --revision 8808b3589142b52e3bf0719aa0ecff8f86715a4a --local-dir checkpoints/scalenav \
59
  --include 'transformer/*' ar_model.pt train_meta.pt SHA256SUMS.json
60
  ```
61
 
62
- Download the Wan VAE, text encoder and tokenizer using the
63
- [installation and checkpoint guide](https://github.com/MachEmbodied/ScaleNav#checkpoints).
64
 
65
  ## Inference
66
 
67
- Prepare the paired observation and its original instruction using the
68
- [single-image inference guide](https://github.com/MachEmbodied/ScaleNav/blob/main/docs/INFERENCE.md#single-image-inference), then run:
69
 
70
  ```bash
71
  awm check --checkpoint checkpoints/scalenav --sha256
72
  awm infer --checkpoint checkpoints/scalenav \
73
  --base-model checkpoints/Wan2.2-TI2V-5B \
74
- --image data/example/observation.png \
75
- --prompt "$(cat data/example/instruction.txt)" --seed 1065612570 \
76
  --output outputs/example
77
  ```
78
 
79
- This saves the nine-step action trajectory as `observation_trajectory.npy` and
80
- a trajectory visualization as `observation_trajectory.png`. Add `--decode-video`
81
- to also save nine predicted RGB frames as `observation_generated_raw.mp4`.
82
- Actions are per-frame forward/right/clockwise-yaw displacements in
83
- metres/metres/radians.
84
- Continuous navigation uses incoming camera frames through
85
- [`awm serve`](https://github.com/MachEmbodied/ScaleNav/blob/main/docs/INFERENCE.md).
 
86
 
87
  ## Model settings
88
 
@@ -115,3 +118,17 @@ The released checkpoint is the final joint-fine-tuning checkpoint at step 6500.
115
  It contains model weights and deployment metadata. Optimizer, scheduler and RNG
116
  state for exact training resumption are not included. The Wan VAE, text encoder
117
  and tokenizer are downloaded separately.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
  - video-diffusion
12
  - image-text-to-action
13
  - safetensors
14
+ library_name: scalenav
15
+ pipeline_tag: robotics
16
+ widget:
17
+ - example_title: 'ScaleNav: three real-robot navigation demos'
18
+ output:
19
+ url: replay.mp4
20
+ datasets:
21
+ - Yangyihui/ScaleNav-Data
22
  ---
23
 
24
  # ScaleNav
 
27
 
28
  [Paper](https://arxiv.org/abs/2610.25690) 路 [Code](https://github.com/MachEmbodied/ScaleNav) 路 [Data](https://huggingface.co/datasets/Yangyihui/ScaleNav-Data) 路 [Project page](https://zju3dv.github.io/ScaleNav/) 路 [Collection](https://huggingface.co/collections/Yangyihui/scalenav-6ad591e8ff62bb75ec75f645)
29
 
30
+ ScaleNav learns navigation priors through action-free pretraining on Internet
31
+ videos, then turns an RGB observation and a language instruction into robot actions.
32
+ It navigates to a target visible in the initial view and stops at the instructed location.
 
33
 
34
+ **5B parameters 路 10.02 GB checkpoint 路 [CC BY-NC 4.0](LICENSE)**
35
 
36
+ Watch the three real-robot demos in the video preview, or visit the
37
+ [project page](https://zju3dv.github.io/ScaleNav/#demos).
 
 
 
 
 
 
 
 
 
38
 
39
  ## Download
40
 
41
+ Complete the [installation](https://github.com/MachEmbodied/ScaleNav/blob/main/docs/INSTALL.md#environment),
42
+ then run from the ScaleNav code repository root:
 
43
 
44
  ```bash
45
  python scripts/download_checkpoint.py --output checkpoints/scalenav
46
  ```
47
 
48
+ The downloader resumes interrupted transfers and verifies the pinned
49
+ [checkpoint checksums](https://github.com/MachEmbodied/ScaleNav/blob/main/docs/checkpoint-download-manifest.json).
50
+ No HF login is required for this checkpoint.
51
+
52
+ Also download the **Wan VAE, text encoder and tokenizer** (14.20 GB) following
53
+ the [checkpoint setup](https://github.com/MachEmbodied/ScaleNav/blob/main/docs/INSTALL.md#checkpoints).
54
+ For inference, use a BF16-capable NVIDIA GPU; **24 GB GPU memory and 64 GB system RAM**
55
+ are recommended.
56
 
57
+ <details>
58
+ <summary>Download with the Hugging Face CLI</summary>
59
 
60
  ```bash
61
  hf download Yangyihui/ScaleNav --revision 8808b3589142b52e3bf0719aa0ecff8f86715a4a --local-dir checkpoints/scalenav \
62
  --include 'transformer/*' ar_model.pt train_meta.pt SHA256SUMS.json
63
  ```
64
 
65
+ </details>
 
66
 
67
  ## Inference
68
 
69
+ Use ScaleNav's `awm` CLI with your own image and an instruction describing a target
70
+ visible in it:
71
 
72
  ```bash
73
  awm check --checkpoint checkpoints/scalenav --sha256
74
  awm infer --checkpoint checkpoints/scalenav \
75
  --base-model checkpoints/Wan2.2-TI2V-5B \
76
+ --image /path/to/observation.png \
77
+ --prompt "Walk to the chair visible in the initial view." \
78
  --output outputs/example
79
  ```
80
 
81
+ Replace the image path and instruction with a matching pair. This saves nine
82
+ per-frame action deltas in `observation_trajectory.npy` and a visualization in
83
+ `observation_trajectory.png`. Actions are forward/right/clockwise-yaw displacements
84
+ in metres/metres/radians. Add `--decode-video` to save the nine predicted RGB frames.
85
+
86
+ For continuous navigation, use [`awm serve`](https://github.com/MachEmbodied/ScaleNav/blob/main/docs/INFERENCE.md#http-api).
87
+ See the [inference guide](https://github.com/MachEmbodied/ScaleNav/blob/main/docs/INFERENCE.md)
88
+ for paired examples, outputs and the robot adapter.
89
 
90
  ## Model settings
91
 
 
118
  It contains model weights and deployment metadata. Optimizer, scheduler and RNG
119
  state for exact training resumption are not included. The Wan VAE, text encoder
120
  and tokenizer are downloaded separately.
121
+
122
+ ## License
123
+
124
+ The ScaleNav fine-tuned checkpoint is released under [CC BY-NC 4.0](LICENSE)
125
+ for noncommercial use. When sharing it or adaptations, provide attribution,
126
+ link to the license, and indicate any changes.
127
+
128
+ ScaleNav modifies [Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B)
129
+ through navigation pretraining, action alignment and joint fine-tuning. The
130
+ CC BY-NC 4.0 terms apply to the ScaleNav contributions; upstream Wan materials
131
+ retain their [Apache-2.0 license](LICENSE-WAN). The
132
+ [code](https://github.com/MachEmbodied/ScaleNav/blob/main/LICENSE) is separately
133
+ licensed under Apache-2.0, and [dataset components](https://huggingface.co/datasets/Yangyihui/ScaleNav-Data#sources-and-license)
134
+ retain their respective terms.
replay.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e94499518683dad0c9a8ebfc6b8ceffc0c7c68f8bf5a6fe280fd6dc1c0803265
3
+ size 7621843