README.md

---
license: apache-2.0
base_model: Wan-AI/Wan2.2-TI2V-5B
datasets:
  - RimbotAI/LOOM
tags:
  - robotics
  - reward-model
---

# NimCritic

NimCritic estimates per-frame task progress and completion from a robot video and a task instruction.

[Project page](https://rimbot.com/en/research/nimcritic/) · [Paper](https://arxiv.org/abs/2610.32776) · [Benchmark][hf-benchmark] · [Demo](https://critic.rimbot.com/)

## Contents

| Path | Contents |
|---|---|
| `infer_nimcritic.py` | Frame loading, the sliding window, and the command-line arguments. |
| `nimcritic_model.py` | The model: video DiT, value DiT, VAE, and text encoder. |
| `nimcritic_step_128000.pt` | Step 128000. Video expert 5.00B, value expert 1.02B, bfloat16. |
| `base/` | Wan2.2 VAE, UMT5-XXL encoder, and tokenizer. The Wan2.2 DiT is not included: the full video-expert weights are already in the `.pt`. |

`base/` contains:

- `DiffSynth-Studio/Wan-Series-Converted-Safetensors/Wan2.2_VAE.safetensors`
- `DiffSynth-Studio/Wan-Series-Converted-Safetensors/models_t5_umt5-xxl-enc-bf16.safetensors`
- `Wan-AI/Wan2.1-T2V-1.3B/google/umt5-xxl/` (tokenizer)

License: Apache-2.0.

## Environment

Python 3.12+. Install the inference dependencies in this directory:

```bash
uv sync
```

Dependencies are listed in `pyproject.toml`: PyTorch, `transformers` (4.57, below 4.58), `safetensors`, `einops`, `opencv-python-headless`, `Pillow`, `numpy`, and `tqdm`.

The model code is `nimcritic_model.py`. The checkpoint and `base/` are in the same directory. Inference does not read any other path and does not download extra files.

## Inference

Run from this directory. The defaults are `nimcritic_step_128000.pt` and `base/` next to the script.

```bash
uv run python infer_nimcritic.py \
  --video clip.mp4 \
  --task "pick up the cup" \
  --output result.json
```

A video file is sampled to 2 Hz from the container frame rate (`--sample-hz`, default 2), and the last frame is kept. A frame sequence that is already 2 Hz uses `--frames` and is not resampled:

```bash
uv run python infer_nimcritic.py \
  --frames frames_dir/ \
  --task "pick up the cup" \
  --output result.json
```

`--base` is the `base/` directory, not the `DiffSynth-Studio/` directory inside it.

Defaults: frames are resized to 256×448 (`stretch`); progress is a delta relative to the condition frame of the current window and is mapped back to 0–1 before it is written; `rollout_stride=1`; `distance` aggregation; 20 steps; seed 42. Add `--rollout-stride 2` to match the short-task table in the paper.

Arguments, choices, and defaults:

```bash
uv run python infer_nimcritic.py --help
```

Check frame sampling and arguments without loading weights:

```bash
uv run python infer_nimcritic.py --self-check
```

Reserve at least 24 GB of GPU memory. CPU inference is `--device cpu` and is slower than the GPU.

In the result JSON, `progress` and `success` are per-frame values in 0–1. `sampling` records the frame indices that were used.

## Citation

```bibtex
@article{xu2026nimcritic,
  title   = {Predictive Long-Horizon Progress Modeling with Action-Aware Quantification},
  author  = {Xu, Tianchi and Zheng, Yalin and Zhu, Yangxin and Ren, Ruonan and Fu, Zheng and Wang, Yunlong and Zhong, Yuanxin},
  journal = {arXiv preprint arXiv:2610.32776},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.32776}
}
```

Cite [LOOM][hf-benchmark] as well when reporting benchmark numbers.

[hf-benchmark]: https://huggingface.co/datasets/RimbotAI/LOOM