Align model card with the public ScaleNav code and project page
Browse files
README.md
CHANGED
|
@@ -15,52 +15,58 @@ tags:
|
|
| 15 |
|
| 16 |
**Scalable Internet Video Pretraining for Language-Conditioned Local Navigation**
|
| 17 |
|
| 18 |
-
[Code](https://github.com/
|
| 19 |
|
| 20 |
-
ScaleNav learns navigation priors from Internet videos through
|
| 21 |
-
pretraining
|
| 22 |
-
|
|
|
|
| 23 |
|
| 24 |
## Download
|
| 25 |
|
| 26 |
-
|
|
|
|
|
|
|
| 27 |
|
| 28 |
```bash
|
| 29 |
-
hf auth login
|
| 30 |
python scripts/download_checkpoint.py --output checkpoints/scalenav
|
| 31 |
```
|
| 32 |
|
| 33 |
-
The downloader uses
|
| 34 |
-
|
| 35 |
-
|
| 36 |
|
| 37 |
To download directly with the Hugging Face CLI:
|
| 38 |
|
| 39 |
```bash
|
| 40 |
-
hf download Yangyihui/ScaleNav --revision
|
| 41 |
--include 'transformer/*' ar_model.pt train_meta.pt SHA256SUMS.json
|
| 42 |
```
|
| 43 |
|
| 44 |
Download the Wan VAE, text encoder and tokenizer using the
|
| 45 |
-
[installation and checkpoint guide](https://github.com/
|
| 46 |
|
| 47 |
## Inference
|
| 48 |
|
| 49 |
Prepare the paired observation and its original instruction using the
|
| 50 |
-
[
|
| 51 |
|
| 52 |
```bash
|
| 53 |
awm check --checkpoint checkpoints/scalenav --sha256
|
| 54 |
awm infer --checkpoint checkpoints/scalenav \
|
| 55 |
--base-model checkpoints/Wan2.2-TI2V-5B \
|
| 56 |
--image data/example/observation.png \
|
| 57 |
-
--prompt "$(cat data/example/instruction.txt)" --
|
|
|
|
| 58 |
```
|
| 59 |
|
| 60 |
-
This
|
| 61 |
-
|
|
|
|
|
|
|
|
|
|
| 62 |
Continuous navigation uses incoming camera frames through
|
| 63 |
-
[`awm serve`](https://github.com/
|
| 64 |
|
| 65 |
## Model settings
|
| 66 |
|
|
@@ -85,6 +91,11 @@ Continuous navigation uses incoming camera frames through
|
|
| 85 |
| `SHA256SUMS.json` | File checksums |
|
| 86 |
|
| 87 |
Total checkpoint size: **10.02 GB**. The separate Wan components occupy about
|
| 88 |
-
14.2 GB. The code repository includes [training](https://github.com/
|
| 89 |
-
[evaluation](https://github.com/
|
| 90 |
-
and [trajectory visualization](https://github.com/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
|
| 16 |
**Scalable Internet Video Pretraining for Language-Conditioned Local Navigation**
|
| 17 |
|
| 18 |
+
[Code](https://github.com/MachEmbodied/ScaleNav) 路 [Data](https://huggingface.co/datasets/Yangyihui/ScaleNav-Data) 路 [Project page](https://zju3dv.github.io/ScaleNav/)
|
| 19 |
|
| 20 |
+
ScaleNav learns navigation priors from Internet egocentric videos through
|
| 21 |
+
action-free pretraining and transfers them to action generation for
|
| 22 |
+
language-conditioned local navigation. The goal is to reach a language-specified
|
| 23 |
+
target visible in the initial observation and stop at the instructed location.
|
| 24 |
|
| 25 |
## Download
|
| 26 |
|
| 27 |
+
Install ScaleNav using the [installation guide](https://github.com/MachEmbodied/ScaleNav#installation),
|
| 28 |
+
then run from the code repository root. This model repository is public;
|
| 29 |
+
authentication is optional:
|
| 30 |
|
| 31 |
```bash
|
|
|
|
| 32 |
python scripts/download_checkpoint.py --output checkpoints/scalenav
|
| 33 |
```
|
| 34 |
|
| 35 |
+
The downloader uses the revision and file checksums in the
|
| 36 |
+
[download manifest](https://github.com/MachEmbodied/ScaleNav/blob/main/docs/checkpoint-download-manifest.json),
|
| 37 |
+
resumes interrupted downloads and verifies SHA-256.
|
| 38 |
|
| 39 |
To download directly with the Hugging Face CLI:
|
| 40 |
|
| 41 |
```bash
|
| 42 |
+
hf download Yangyihui/ScaleNav --revision 8808b3589142b52e3bf0719aa0ecff8f86715a4a --local-dir checkpoints/scalenav \
|
| 43 |
--include 'transformer/*' ar_model.pt train_meta.pt SHA256SUMS.json
|
| 44 |
```
|
| 45 |
|
| 46 |
Download the Wan VAE, text encoder and tokenizer using the
|
| 47 |
+
[installation and checkpoint guide](https://github.com/MachEmbodied/ScaleNav#checkpoints).
|
| 48 |
|
| 49 |
## Inference
|
| 50 |
|
| 51 |
Prepare the paired observation and its original instruction using the
|
| 52 |
+
[single-image inference guide](https://github.com/MachEmbodied/ScaleNav/blob/main/docs/INFERENCE.md#single-image-inference), then run:
|
| 53 |
|
| 54 |
```bash
|
| 55 |
awm check --checkpoint checkpoints/scalenav --sha256
|
| 56 |
awm infer --checkpoint checkpoints/scalenav \
|
| 57 |
--base-model checkpoints/Wan2.2-TI2V-5B \
|
| 58 |
--image data/example/observation.png \
|
| 59 |
+
--prompt "$(cat data/example/instruction.txt)" --seed 1065612570 \
|
| 60 |
+
--output outputs/example
|
| 61 |
```
|
| 62 |
|
| 63 |
+
This saves the nine-step action trajectory as `observation_trajectory.npy` and
|
| 64 |
+
a trajectory visualization as `observation_trajectory.png`. Add `--decode-video`
|
| 65 |
+
to also save nine predicted RGB frames as `observation_generated_raw.mp4`.
|
| 66 |
+
Actions are per-frame forward/right/clockwise-yaw displacements in
|
| 67 |
+
metres/metres/radians.
|
| 68 |
Continuous navigation uses incoming camera frames through
|
| 69 |
+
[`awm serve`](https://github.com/MachEmbodied/ScaleNav/blob/main/docs/INFERENCE.md).
|
| 70 |
|
| 71 |
## Model settings
|
| 72 |
|
|
|
|
| 91 |
| `SHA256SUMS.json` | File checksums |
|
| 92 |
|
| 93 |
Total checkpoint size: **10.02 GB**. The separate Wan components occupy about
|
| 94 |
+
14.2 GB. The code repository includes [training](https://github.com/MachEmbodied/ScaleNav/blob/main/docs/TRAINING.md),
|
| 95 |
+
[evaluation](https://github.com/MachEmbodied/ScaleNav/blob/main/docs/EVALUATION.md)
|
| 96 |
+
and [trajectory visualization](https://github.com/MachEmbodied/ScaleNav/blob/main/docs/INFERENCE.md#outputs).
|
| 97 |
+
|
| 98 |
+
The released checkpoint is the final joint-fine-tuning checkpoint at step 6500.
|
| 99 |
+
It contains model weights and deployment metadata. Optimizer, scheduler and RNG
|
| 100 |
+
state for exact training resumption are not included. The Wan VAE, text encoder
|
| 101 |
+
and tokenizer are downloaded separately.
|