Robotics
reward-model
Ayleea929 commited on
Commit
e2bcfda
·
verified ·
1 Parent(s): 87f8154

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +26 -30
README.md CHANGED
@@ -10,42 +10,42 @@ tags:
10
 
11
  # NimCritic
12
 
13
- NimCritic 根据机器人视频和任务指令,逐帧估计任务进度与是否完成。
14
 
15
- [Project page](https://rimbot.com/en/research/nimcritic/) · [Paper](https://arxiv.org/abs/2610.32776)
16
 
17
- ## 目录里有什么
18
 
19
- | 路径 | 内容 |
20
  |---|---|
21
- | `infer_nimcritic.py` | 读帧、滑窗和命令行参数。 |
22
- | `nimcritic_model.py` | 模型:video DiT、value DiT、VAE、文本编码器。 |
23
- | `nimcritic_step_128000.pt` | step 128000。video expert 5.00B,value expert 1.02B,bfloat16。 |
24
- | `base/` | Wan2.2 VAE、UMT5-XXL 编码器和分词器。不含 Wan2.2 DiT:video expert 的完整权重已经在 `.pt` 里。 |
25
 
26
- `base/` 包含:
27
 
28
  - `DiffSynth-Studio/Wan-Series-Converted-Safetensors/Wan2.2_VAE.safetensors`
29
  - `DiffSynth-Studio/Wan-Series-Converted-Safetensors/models_t5_umt5-xxl-enc-bf16.safetensors`
30
- - `Wan-AI/Wan2.1-T2V-1.3B/google/umt5-xxl/`(分词器)
31
 
32
- 许可见 Apache-2.0。
33
 
34
- ## 环境
35
 
36
- Python 3.12+。在本目录安装推理依赖:
37
 
38
  ```bash
39
  uv sync
40
  ```
41
 
42
- 依赖写在 `pyproject.toml`:PyTorch、`transformers`(4.57,低于 4.58)、`safetensors`、`einops`、`opencv-python-headless`、`Pillow`、`numpy`、`tqdm`。
43
 
44
- 模型代码在 `nimcritic_model.py`,权重与 `base/` 在同目录。推理不访问其他路径,也不下载额外文件。
45
 
46
- ## 推理
47
 
48
- 在本目录执行。默认读取同目录的 `nimcritic_step_128000.pt` 与 `base/`。
49
 
50
  ```bash
51
  uv run python infer_nimcritic.py \
@@ -54,7 +54,7 @@ uv run python infer_nimcritic.py \
54
  --output result.json
55
  ```
56
 
57
- 输入为视频文件时,按容器帧率采样至 2Hz(`--sample-hz`,默认 2),并保留最后一帧。输入已是 2Hz 帧序列时使用 `--frames`,不再按 2Hz 重采样:
58
 
59
  ```bash
60
  uv run python infer_nimcritic.py \
@@ -63,27 +63,27 @@ uv run python infer_nimcritic.py \
63
  --output result.json
64
  ```
65
 
66
- `--base` 指向 `base/` 这一层,不是里面的 `DiffSynth-Studio/`。
67
 
68
- 默认口径:帧拉到 256×448(`stretch`),progress 是相对当前窗口条件帧的增量,写回前还原成 0–1,`rollout_stride=1`,`distance` 聚合,20 步,seed 42。和论文短任务表同一口径时加上 `--rollout-stride 2`。
69
 
70
- 每个参数的含义、可选项和默认值:
71
 
72
  ```bash
73
  uv run python infer_nimcritic.py --help
74
  ```
75
 
76
- 不加载权重、只检查抽帧和参数:
77
 
78
  ```bash
79
  uv run python infer_nimcritic.py --self-check
80
  ```
81
 
82
- GPU 推理建议预留不少于 24 GB 显存。CPU 推理通过 `--device cpu` 指定,耗时高于 GPU。
83
 
84
- 结果 JSON 中,`progress` 与 `success` 为逐帧的 0–1 数值。`sampling` 记录所采用的帧下标。
85
 
86
- ## 引用
87
 
88
  ```bibtex
89
  @article{xu2026nimcritic,
@@ -95,10 +95,6 @@ GPU 推理建议预留不少于 24 GB 显存。CPU 推理通过 `--device cpu`
95
  }
96
  ```
97
 
98
- 基准请同时引用 [LOOM][hf-benchmark]。
99
 
100
- [hf-model]: https://huggingface.co/RimbotAI/NimCritic
101
  [hf-benchmark]: https://huggingface.co/datasets/RimbotAI/LOOM
102
- [arxiv]: https://arxiv.org/abs/2610.32776
103
- [project]: https://rimbot.com/en/research/nimcritic/
104
- [demo]: https://critic.rimbot.com/
 
10
 
11
  # NimCritic
12
 
13
+ NimCritic estimates per-frame task progress and completion from a robot video and a task instruction.
14
 
15
+ [Project page](https://rimbot.com/en/research/nimcritic/) · [Paper](https://arxiv.org/abs/2610.32776) · [Benchmark][hf-benchmark] · [Demo](https://critic.rimbot.com/)
16
 
17
+ ## Contents
18
 
19
+ | Path | Contents |
20
  |---|---|
21
+ | `infer_nimcritic.py` | Frame loading, the sliding window, and the command-line arguments. |
22
+ | `nimcritic_model.py` | The model: video DiT, value DiT, VAE, and text encoder. |
23
+ | `nimcritic_step_128000.pt` | Step 128000. Video expert 5.00B, value expert 1.02B, bfloat16. |
24
+ | `base/` | Wan2.2 VAE, UMT5-XXL encoder, and tokenizer. The Wan2.2 DiT is not included: the full video-expert weights are already in the `.pt`. |
25
 
26
+ `base/` contains:
27
 
28
  - `DiffSynth-Studio/Wan-Series-Converted-Safetensors/Wan2.2_VAE.safetensors`
29
  - `DiffSynth-Studio/Wan-Series-Converted-Safetensors/models_t5_umt5-xxl-enc-bf16.safetensors`
30
+ - `Wan-AI/Wan2.1-T2V-1.3B/google/umt5-xxl/` (tokenizer)
31
 
32
+ License: Apache-2.0.
33
 
34
+ ## Environment
35
 
36
+ Python 3.12+. Install the inference dependencies in this directory:
37
 
38
  ```bash
39
  uv sync
40
  ```
41
 
42
+ Dependencies are listed in `pyproject.toml`: PyTorch, `transformers` (4.57, below 4.58), `safetensors`, `einops`, `opencv-python-headless`, `Pillow`, `numpy`, and `tqdm`.
43
 
44
+ The model code is `nimcritic_model.py`. The checkpoint and `base/` are in the same directory. Inference does not read any other path and does not download extra files.
45
 
46
+ ## Inference
47
 
48
+ Run from this directory. The defaults are `nimcritic_step_128000.pt` and `base/` next to the script.
49
 
50
  ```bash
51
  uv run python infer_nimcritic.py \
 
54
  --output result.json
55
  ```
56
 
57
+ A video file is sampled to 2 Hz from the container frame rate (`--sample-hz`, default 2), and the last frame is kept. A frame sequence that is already 2 Hz uses `--frames` and is not resampled:
58
 
59
  ```bash
60
  uv run python infer_nimcritic.py \
 
63
  --output result.json
64
  ```
65
 
66
+ `--base` is the `base/` directory, not the `DiffSynth-Studio/` directory inside it.
67
 
68
+ Defaults: frames are resized to 256×448 (`stretch`); progress is a delta relative to the condition frame of the current window and is mapped back to 0–1 before it is written; `rollout_stride=1`; `distance` aggregation; 20 steps; seed 42. Add `--rollout-stride 2` to match the short-task table in the paper.
69
 
70
+ Arguments, choices, and defaults:
71
 
72
  ```bash
73
  uv run python infer_nimcritic.py --help
74
  ```
75
 
76
+ Check frame sampling and arguments without loading weights:
77
 
78
  ```bash
79
  uv run python infer_nimcritic.py --self-check
80
  ```
81
 
82
+ Reserve at least 24 GB of GPU memory. CPU inference is `--device cpu` and is slower than the GPU.
83
 
84
+ In the result JSON, `progress` and `success` are per-frame values in 0–1. `sampling` records the frame indices that were used.
85
 
86
+ ## Citation
87
 
88
  ```bibtex
89
  @article{xu2026nimcritic,
 
95
  }
96
  ```
97
 
98
+ Cite [LOOM][hf-benchmark] as well when reporting benchmark numbers.
99
 
 
100
  [hf-benchmark]: https://huggingface.co/datasets/RimbotAI/LOOM