FlowPilot-DST streaming ONNX (one frame per call, feature buffer) for small and dune; README

README.md

@@ -12,8 +12,9 @@ tags:
 # Visual Navigation Model Checkpoints
 
 Pretrained navigation policies trained with [VisNavKit](https://github.com/VAIL-UCLA/visnavkit).
-Each folder holds one model: the Lightning checkpoint, its ONNX export, the export metadata
-(shapes, anchor times, SHA-256 of both files, PyTorch/ONNX parity) and a sample input batch.
+Each folder holds one model: the Lightning checkpoint, its two ONNX exports (window and streaming),
+the export metadata (shapes, anchor times, SHA-256 of both files, PyTorch/ONNX parity) and a sample
+input batch.
 
 | Folder | VisNavKit config | Model | Params | Val top-1 ADE@1/2/4 s (m) | Val top-1 FDE (m) |
 | --- | --- | --- | --- | --- | --- |
@@ -32,6 +33,7 @@ PyTorch/ONNX parity is within 1e-5.
 <folder>/flowpilot_dst_<encoder>.onnx           # deployment graph
 <folder>/flowpilot_dst_<encoder>.metadata.json  # shapes, anchor times, hashes, parity
 <folder>/flowpilot_dst_<encoder>.inputs.npz     # traced sample inputs for a smoke run
+<folder>/flowpilot_dst_<encoder>_streaming.{onnx,metadata.json,inputs.npz}  # streaming graph, same weights
 ```
 
 ## Usage
@@ -53,7 +55,42 @@ Inputs are the last 20 frames (1, 20, 3, 216, 384) RGB in [0, 1], route patches,
 speed/yaw rate and action bounds. The full contract, including frame preparation, is in
 [FlowPilot-DST ONNX IO](https://github.com/VAIL-UCLA/visnavkit/blob/dev/docs/flowpilot_dst_onnx.md).
 
-Re-export from the checkpoint with VisNavKit:
+### Streaming ONNX
+
+`*_streaming.onnx` runs one frame per call. It caches each past frame's features (`[global image
+feature | route latent]`) in a buffer, encodes only the current frame, and decodes only the current
+plan. After 20 frames its decision equals the window graph's (checked at export: DST exact, DUNE
+to 5e-7 in ONNX Runtime).
+
+| input | shape | meaning |
+| --- | --- | --- |
+| `vision` | small (1, 2, 3, 216, 384): [previous, current]; dune (1, 1, 3, 216, 384) | the current frame; for small, the previous frame is all zeros on the first call |
+| `route_patch`, `goal`, `ego` | (1, 1, 80, 80), (1, 1, 3), (1, 1, 2) | the current frame's route, goal, `[v, w]` |
+| `action_bounds` | (1, 2, 5) | as in the window graph |
+| `feat_buffer` | (1, 19, F): F = 768 small, 2048 dune | the last call's `feat_buffer_out`; zeros at startup |
+| `buffer_mask` | (1, 19) | the last call's `buffer_mask_out`; zeros at startup |
+
+Outputs: `modes`, `probs`, `speed` as in the window graph, plus `feat_buffer_out`, `buffer_mask_out`.
+
+```python
+stem = "flowpilot-dst-dune/flowpilot_dst_dune_vitb14_streaming"
+session = ort.InferenceSession(hf_hub_download(repo, f"{stem}.onnx"))
+buf, mask = np.zeros((1, 19, 2048), np.float32), np.zeros((1, 19), np.float32)
+for frame, route, goal, ego in stream:  # 20 Hz, frames exactly 50 ms apart
+    modes, probs, speed, buf, mask = session.run(None, dict(
+        vision=frame[None, None], route_patch=route[None, None], goal=goal[None, None],
+        ego=ego[None, None], action_bounds=bounds, feat_buffer=buf, buffer_mask=mask))
+    plan = modes[0, 0]  # (80, 5)
+```
+
+- Feed `feat_buffer_out` and `buffer_mask_out` back unchanged.
+- Call once per frame at 20 Hz. Don't skip frames. Reset the buffer (and, for small, the previous
+  frame) to zeros after a gap or restart.
+- Keep one buffer per camera stream.
+- CPU latency per call (ONNX Runtime, loaded machine): small 1248 ms window vs 381 ms streaming;
+  dune 7537 ms vs 2021 ms.
+
+Re-export from the checkpoint with VisNavKit (add `streaming=true` for the streaming graph):
 
 ```bash
 uv run visnavkit-export-dst checkpoint=flowpilot_dst_fastvit_t12.ckpt output=flowpilot_dst_fastvit_t12.onnx

flowpilot-dst-dune/flowpilot_dst_dune_vitb14_streaming.inputs.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49ca7ba15671aef8d5a0a52f96e990e4be08b2cda827c9f5cd677e6f0534663b
+size 1178476

flowpilot-dst-dune/flowpilot_dst_dune_vitb14_streaming.metadata.json

@@ -0,0 +1,174 @@
+{
+  "exp_name": "flowpilot_dune_dst_clips1k",
+  "checkpoint": "logs/visnavkit/th7jpd4m/checkpoints/last.ckpt",
+  "checkpoint_sha256": "1380b49fd47aa428a2070a169f8a1eab287c9bb45ac41a49adb796f1253eab7f",
+  "onnx_sha256": "0504dace2478c3d89630ceafff31259d25869d013d1a0ab03d53078046708851",
+  "precision": "float32",
+  "opset": 17,
+  "input_shapes": {
+    "vision": [
+      1,
+      1,
+      3,
+      216,
+      384
+    ],
+    "route_patch": [
+      1,
+      1,
+      80,
+      80
+    ],
+    "goal": [
+      1,
+      1,
+      3
+    ],
+    "ego": [
+      1,
+      1,
+      2
+    ],
+    "action_bounds": [
+      1,
+      2,
+      5
+    ],
+    "feat_buffer": [
+      1,
+      19,
+      2048
+    ],
+    "buffer_mask": [
+      1,
+      19
+    ]
+  },
+  "output_shapes": {
+    "modes": [
+      1,
+      6,
+      80,
+      5
+    ],
+    "probs": [
+      1,
+      6
+    ],
+    "speed": [
+      1,
+      1
+    ],
+    "feat_buffer_out": [
+      1,
+      19,
+      2048
+    ],
+    "buffer_mask_out": [
+      1,
+      19
+    ]
+  },
+  "streaming": true,
+  "pose_fields": [
+    "x_m",
+    "y_m",
+    "yaw_rad",
+    "v_mps",
+    "w_radps"
+  ],
+  "target_times_s": [
+    0.05,
+    0.1,
+    0.15000000000000002,
+    0.2,
+    0.25,
+    0.30000000000000004,
+    0.35000000000000003,
+    0.4,
+    0.45,
+    0.5,
+    0.55,
+    0.6000000000000001,
+    0.65,
+    0.7000000000000001,
+    0.75,
+    0.8,
+    0.8500000000000001,
+    0.9,
+    0.9500000000000001,
+    1.0,
+    1.05,
+    1.1,
+    1.1500000000000001,
+    1.2000000000000002,
+    1.25,
+    1.3,
+    1.35,
+    1.4000000000000001,
+    1.4500000000000002,
+    1.5,
+    1.55,
+    1.6,
+    1.6500000000000001,
+    1.7000000000000002,
+    1.75,
+    1.8,
+    1.85,
+    1.9000000000000001,
+    1.9500000000000002,
+    2.0,
+    2.0500000000000003,
+    2.1,
+    2.15,
+    2.2,
+    2.25,
+    2.3000000000000003,
+    2.35,
+    2.4000000000000004,
+    2.45,
+    2.5,
+    2.5500000000000003,
+    2.6,
+    2.6500000000000004,
+    2.7,
+    2.75,
+    2.8000000000000003,
+    2.85,
+    2.9000000000000004,
+    2.95,
+    3.0,
+    3.0500000000000003,
+    3.1,
+    3.1500000000000004,
+    3.2,
+    3.25,
+    3.3000000000000003,
+    3.35,
+    3.4000000000000004,
+    3.45,
+    3.5,
+    3.5500000000000003,
+    3.6,
+    3.6500000000000004,
+    3.7,
+    3.75,
+    3.8000000000000003,
+    3.85,
+    3.9000000000000004,
+    3.95,
+    4.0
+  ],
+  "top_k": 6,
+  "noise": "zero",
+  "denoising_steps": 4,
+  "num_anchors": 64,
+  "parameters_total": 208170577,
+  "parity_max_abs_error": {
+    "modes": 1.430511474609375e-06,
+    "probs": 2.384185791015625e-07,
+    "speed": 0.0,
+    "feat_buffer_out": 4.410743713378906e-06,
+    "buffer_mask_out": 0.0
+  }
+}

flowpilot-dst-dune/flowpilot_dst_dune_vitb14_streaming.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0504dace2478c3d89630ceafff31259d25869d013d1a0ab03d53078046708851
+size 829900699

flowpilot-dst-small/flowpilot_dst_fastvit_t12_streaming.inputs.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb225613cd0d9d90e4c87350c7e730719a601501c2d416d2e1df83e6b755722c
+size 2076524

flowpilot-dst-small/flowpilot_dst_fastvit_t12_streaming.metadata.json

@@ -0,0 +1,174 @@
+{
+  "exp_name": "flowpilot_dst_clips1k",
+  "checkpoint": "logs/visnavkit/jaz0uw1a/checkpoints/last.ckpt",
+  "checkpoint_sha256": "a45c6cf724ad4b70675f344ef94b6e809268dfff33f78497b8e1a7b446fdf6b8",
+  "onnx_sha256": "f6d3e4e1b6cb90250e994bf66270f6887a10843fa2ccaebd896ee46d514c07c6",
+  "precision": "float32",
+  "opset": 17,
+  "input_shapes": {
+    "vision": [
+      1,
+      2,
+      3,
+      216,
+      384
+    ],
+    "route_patch": [
+      1,
+      1,
+      80,
+      80
+    ],
+    "goal": [
+      1,
+      1,
+      3
+    ],
+    "ego": [
+      1,
+      1,
+      2
+    ],
+    "action_bounds": [
+      1,
+      2,
+      5
+    ],
+    "feat_buffer": [
+      1,
+      19,
+      768
+    ],
+    "buffer_mask": [
+      1,
+      19
+    ]
+  },
+  "output_shapes": {
+    "modes": [
+      1,
+      6,
+      80,
+      5
+    ],
+    "probs": [
+      1,
+      6
+    ],
+    "speed": [
+      1,
+      1
+    ],
+    "feat_buffer_out": [
+      1,
+      19,
+      768
+    ],
+    "buffer_mask_out": [
+      1,
+      19
+    ]
+  },
+  "streaming": true,
+  "pose_fields": [
+    "x_m",
+    "y_m",
+    "yaw_rad",
+    "v_mps",
+    "w_radps"
+  ],
+  "target_times_s": [
+    0.05,
+    0.1,
+    0.15000000000000002,
+    0.2,
+    0.25,
+    0.30000000000000004,
+    0.35000000000000003,
+    0.4,
+    0.45,
+    0.5,
+    0.55,
+    0.6000000000000001,
+    0.65,
+    0.7000000000000001,
+    0.75,
+    0.8,
+    0.8500000000000001,
+    0.9,
+    0.9500000000000001,
+    1.0,
+    1.05,
+    1.1,
+    1.1500000000000001,
+    1.2000000000000002,
+    1.25,
+    1.3,
+    1.35,
+    1.4000000000000001,
+    1.4500000000000002,
+    1.5,
+    1.55,
+    1.6,
+    1.6500000000000001,
+    1.7000000000000002,
+    1.75,
+    1.8,
+    1.85,
+    1.9000000000000001,
+    1.9500000000000002,
+    2.0,
+    2.0500000000000003,
+    2.1,
+    2.15,
+    2.2,
+    2.25,
+    2.3000000000000003,
+    2.35,
+    2.4000000000000004,
+    2.45,
+    2.5,
+    2.5500000000000003,
+    2.6,
+    2.6500000000000004,
+    2.7,
+    2.75,
+    2.8000000000000003,
+    2.85,
+    2.9000000000000004,
+    2.95,
+    3.0,
+    3.0500000000000003,
+    3.1,
+    3.1500000000000004,
+    3.2,
+    3.25,
+    3.3000000000000003,
+    3.35,
+    3.4000000000000004,
+    3.45,
+    3.5,
+    3.5500000000000003,
+    3.6,
+    3.6500000000000004,
+    3.7,
+    3.75,
+    3.8000000000000003,
+    3.85,
+    3.9000000000000004,
+    3.95,
+    4.0
+  ],
+  "top_k": 6,
+  "noise": "zero",
+  "denoising_steps": 4,
+  "num_anchors": 64,
+  "parameters_total": 21336514,
+  "parity_max_abs_error": {
+    "modes": 3.3974647521972656e-06,
+    "probs": 3.3080577850341797e-06,
+    "speed": 7.450580596923828e-07,
+    "feat_buffer_out": 7.62939453125e-06,
+    "buffer_mask_out": 0.0
+  }
+}

flowpilot-dst-small/flowpilot_dst_fastvit_t12_streaming.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f6d3e4e1b6cb90250e994bf66270f6887a10843fa2ccaebd896ee46d514c07c6
+size 85721799