Publish four AutoE2E trajectory checkpoints
Browse filesPublish MLflow model versions 64-67 with immutable checkpoint hashes, per-model metadata, and KITScenes external overlay replay reports.
- README.md +55 -1
- SHA256SUMS +19 -0
- evaluations/overlay-replay-v1/index.json +320 -0
- evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-5/report.json +76 -0
- evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-7/report.json +76 -0
- evaluations/overlay-replay-v1/kitscenes-test/nuplan-epoch-4/report.json +76 -0
- evaluations/overlay-replay-v1/kitscenes-test/nuplan-epoch-5/report.json +76 -0
- evaluations/overlay-replay-v1/kitscenes-val/kitscenes-epoch-5/report.json +76 -0
- evaluations/overlay-replay-v1/kitscenes-val/kitscenes-epoch-7/report.json +76 -0
- evaluations/overlay-replay-v1/kitscenes-val/nuplan-epoch-4/report.json +76 -0
- evaluations/overlay-replay-v1/kitscenes-val/nuplan-epoch-5/report.json +76 -0
- models/kitscenes-epoch-5/checkpoint.pt +3 -0
- models/kitscenes-epoch-5/metadata.json +153 -0
- models/kitscenes-epoch-7/checkpoint.pt +3 -0
- models/kitscenes-epoch-7/metadata.json +153 -0
- models/nuplan-epoch-4/checkpoint.pt +3 -0
- models/nuplan-epoch-4/metadata.json +151 -0
- models/nuplan-epoch-5/checkpoint.pt +3 -0
- models/nuplan-epoch-5/metadata.json +151 -0
- release_manifest.json +40 -0
README.md
CHANGED
|
@@ -1,3 +1,57 @@
|
|
| 1 |
---
|
| 2 |
-
license:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
license: other
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
tags:
|
| 5 |
+
- autonomous-driving
|
| 6 |
+
- bevformer
|
| 7 |
+
- trajectory-planning
|
| 8 |
+
- nuplan
|
| 9 |
+
- kitscenes
|
| 10 |
---
|
| 11 |
+
|
| 12 |
+
# AutoE2E checkpoints
|
| 13 |
+
|
| 14 |
+
This repository contains four trajectory-planning checkpoints from the AutoE2E project. These are the exact PyTorch training checkpoints registered in MLflow, including model, optimizer, scheduler, configuration, metrics, and training state.
|
| 15 |
+
|
| 16 |
+
## Checkpoints
|
| 17 |
+
|
| 18 |
+
| Path | Training stage | Epoch | MLflow version | SHA-256 |
|
| 19 |
+
|---|---|---:|---:|---|
|
| 20 |
+
| `models/nuplan-epoch-4/checkpoint.pt` | nuPlan trajectory and route | 4 | 64 | `ed00e072471aae51b4cf48fa54c3fd9c03c61e0fd6e2978829cd9068dfd6da68` |
|
| 21 |
+
| `models/nuplan-epoch-5/checkpoint.pt` | nuPlan trajectory and route | 5 | 65 | `ca8b43d7a777d6fd9195bb253d1452f3cd645f7aab3bd0e36b1f74ffb31df29b` |
|
| 22 |
+
| `models/kitscenes-epoch-5/checkpoint.pt` | KITScenes fine-tuning | 5 | 66 | `120a21639d9767d512eea645be0a83b01476c9e0366a8b957ac51b7ead1bd762` |
|
| 23 |
+
| `models/kitscenes-epoch-7/checkpoint.pt` | KITScenes fine-tuning | 7 | 67 | `a1e6b1621018740485b3e1e43dda23713d913667eb39a65b77fcbf895dcbcefd` |
|
| 24 |
+
|
| 25 |
+
All checkpoints use the `bevformer_v2_t8_split_navigation_v5` configuration. The camera BEV starts from the official BEVFormer V2 R50 T8 initialization and is frozen during trajectory stages. World Model and Reasoning branches are disabled.
|
| 26 |
+
|
| 27 |
+
## External overlay replay
|
| 28 |
+
|
| 29 |
+
Each value is ADE / FDE in meters. KITScenes Val v3.5 contains 11,035 samples with Camera + HD Map + oracle post-hoc Route. KITScenes Test v1.0 contains 23,690 camera-only samples from a different scene population.
|
| 30 |
+
|
| 31 |
+
| Checkpoint | Val 1 s | Val 3 s | Val 5 s | Test 1 s | Test 3 s | Test 5 s |
|
| 32 |
+
|---|---:|---:|---:|---:|---:|---:|
|
| 33 |
+
| nuPlan Epoch 4 | 0.2030 / 0.4066 | 0.9503 / 2.4508 | 2.2919 / 6.2272 | 0.1916 / 0.3899 | 0.9181 / 2.3895 | 2.2840 / 6.4266 |
|
| 34 |
+
| nuPlan Epoch 5 | 0.1951 / 0.3779 | 0.9147 / 2.4226 | 2.2951 / 6.3885 | 0.1602 / 0.3134 | 0.7924 / 2.1621 | 2.1120 / 6.1751 |
|
| 35 |
+
| KITScenes Epoch 5 | 0.1472 / 0.2850 | 0.7449 / 2.0196 | 1.9405 / 5.5645 | 0.1463 / 0.3107 | 0.8263 / 2.2199 | 2.1042 / 5.9422 |
|
| 36 |
+
| KITScenes Epoch 7 | 0.1347 / 0.2831 | 0.8035 / 2.2127 | 2.0952 / 5.9509 | 0.1680 / 0.3754 | 0.9831 / 2.6242 | 2.4699 / 6.8964 |
|
| 37 |
+
|
| 38 |
+
The external evaluation is a deterministic replay of published control overlays. It does not re-run the model. Val and Test are different populations, so their difference is not evidence of a causal benefit from map or route inputs. The complete index and eight reports are under `evaluations/overlay-replay-v1/`.
|
| 39 |
+
|
| 40 |
+
## Loading
|
| 41 |
+
|
| 42 |
+
```python
|
| 43 |
+
import torch
|
| 44 |
+
|
| 45 |
+
checkpoint = torch.load("checkpoint.pt", map_location="cpu", weights_only=True)
|
| 46 |
+
model.load_state_dict(checkpoint["model_state_dict"], strict=True)
|
| 47 |
+
```
|
| 48 |
+
|
| 49 |
+
Use an AutoE2E source revision compatible with `bevformer_v2_t8_split_navigation_v5`. Each model directory contains `metadata.json`. `SHA256SUMS` verifies all published files.
|
| 50 |
+
|
| 51 |
+
## Intended use and limitations
|
| 52 |
+
|
| 53 |
+
These checkpoints are research artifacts for open-loop trajectory-planning evaluation and continued training. They are not validated for safety-critical deployment. The KITScenes Val route is reconstructed retrospectively from the logged future trajectory and contains oracle maneuver information. Distribution shift remains relevant.
|
| 54 |
+
|
| 55 |
+
## License notice
|
| 56 |
+
|
| 57 |
+
The AutoE2E source code is Apache-2.0. These checkpoint files include parameters initialized from the official BEVFormer V2 R50 T8 checkpoint. The upstream weight license was recorded as `NOASSERTION`, and the pretrained component was trained on nuScenes. Users are responsible for complying with the terms of the upstream weights and datasets. No broader checkpoint license is asserted here.
|
SHA256SUMS
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
ee3c2f95176c285f3649281a986849a872b0920f4d1c67cad627d4e6424214e0 README.md
|
| 2 |
+
be1cec5f77a7a12326070503dbcd1004c63f54450cf1151c763a215816d838bd evaluations/overlay-replay-v1/index.json
|
| 3 |
+
582dc108ce030756d5d1758dbe198d62b3355bf32a123af54be4e66704c50b8a evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-5/report.json
|
| 4 |
+
d01169d68266d4b7203e92a4d25ade2a3277357ff89cc26cdb202051a26016fc evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-7/report.json
|
| 5 |
+
5eeb68a4013a9777ba927030b73877a1789f5b4eb9e14f6da760bef7f9b42808 evaluations/overlay-replay-v1/kitscenes-test/nuplan-epoch-4/report.json
|
| 6 |
+
fc5b19beea416706461b535314a497d6a2b929d97668e8475638ae04a65221b7 evaluations/overlay-replay-v1/kitscenes-test/nuplan-epoch-5/report.json
|
| 7 |
+
a8f8777da510b832f49503d959f057df6ab298057542e3ec10fd26bd9cb0bf91 evaluations/overlay-replay-v1/kitscenes-val/kitscenes-epoch-5/report.json
|
| 8 |
+
832b9ec0648b997f5b1f825296269905e5ee3b13ae75918e0fdee5733dbd1818 evaluations/overlay-replay-v1/kitscenes-val/kitscenes-epoch-7/report.json
|
| 9 |
+
1ffde86560b8d9701c23c98b6b05b0de2611446a8d92f0fb90a08ccb884d1acb evaluations/overlay-replay-v1/kitscenes-val/nuplan-epoch-4/report.json
|
| 10 |
+
e45996448f95f193acfc225c37ff169fab64bb24b5ea7717928a20c426e7303b evaluations/overlay-replay-v1/kitscenes-val/nuplan-epoch-5/report.json
|
| 11 |
+
120a21639d9767d512eea645be0a83b01476c9e0366a8b957ac51b7ead1bd762 models/kitscenes-epoch-5/checkpoint.pt
|
| 12 |
+
7e576d547291522dd1be153408cea0cdaa280b18db340622b7f57688835a6d0c models/kitscenes-epoch-5/metadata.json
|
| 13 |
+
a1e6b1621018740485b3e1e43dda23713d913667eb39a65b77fcbf895dcbcefd models/kitscenes-epoch-7/checkpoint.pt
|
| 14 |
+
f6b4670c41462689c0a7b54c22acbea0c4cbd935d6e369feaea7a963921afb5b models/kitscenes-epoch-7/metadata.json
|
| 15 |
+
ed00e072471aae51b4cf48fa54c3fd9c03c61e0fd6e2978829cd9068dfd6da68 models/nuplan-epoch-4/checkpoint.pt
|
| 16 |
+
1199152e6c4493cfad16c5dff5053b8edc70db221f70cb57949862b4436eb640 models/nuplan-epoch-4/metadata.json
|
| 17 |
+
ca8b43d7a777d6fd9195bb253d1452f3cd645f7aab3bd0e36b1f74ffb31df29b models/nuplan-epoch-5/checkpoint.pt
|
| 18 |
+
feaa5ae17652c33bad0a4ec286efe9b487f621ca19e4c4ce428cd9d506967569 models/nuplan-epoch-5/metadata.json
|
| 19 |
+
c1c61990ed5cb0d66b36337d7d5ff67a46c37ddff41bd3266950507d747718ee release_manifest.json
|
evaluations/overlay-replay-v1/index.json
ADDED
|
@@ -0,0 +1,320 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"created_at": "2026-09-14T09:13:53Z",
|
| 3 |
+
"reports": [
|
| 4 |
+
{
|
| 5 |
+
"dataset": "kitscenes-val",
|
| 6 |
+
"dataset_version": "v3.5",
|
| 7 |
+
"model_artifact_id": "120a21639d9767d512eea645be0a83b01476c9e0366a8b957ac51b7ead1bd762",
|
| 8 |
+
"model_name": "KITScenes Trajectory Epoch 5",
|
| 9 |
+
"report_sha256": "a8f8777da510b832f49503d959f057df6ab298057542e3ec10fd26bd9cb0bf91",
|
| 10 |
+
"report_uri": "s3://auto-e2e-platform-artifacts-381491877296/direct-evaluations/overlay-replay-v1/dataset=kitscenes-val/version=v3.5/model=120a21639d9767d512eea645be0a83b01476c9e0366a8b957ac51b7ead1bd762/report.json",
|
| 11 |
+
"trajectory": {
|
| 12 |
+
"ade_1s_m": 0.14720280881581088,
|
| 13 |
+
"ade_1s_sample_count": 11035,
|
| 14 |
+
"ade_2s_m": 0.3729208092788606,
|
| 15 |
+
"ade_2s_sample_count": 11035,
|
| 16 |
+
"ade_3s_m": 0.744916487594762,
|
| 17 |
+
"ade_3s_sample_count": 11035,
|
| 18 |
+
"ade_5s_m": 1.9405047441156529,
|
| 19 |
+
"ade_5s_sample_count": 11035,
|
| 20 |
+
"ade_6p4s_m": null,
|
| 21 |
+
"ade_6p4s_sample_count": 0,
|
| 22 |
+
"fde_1s_m": 0.28501037315538935,
|
| 23 |
+
"fde_1s_sample_count": 11035,
|
| 24 |
+
"fde_2s_m": 0.9239302700556777,
|
| 25 |
+
"fde_2s_sample_count": 11035,
|
| 26 |
+
"fde_3s_m": 2.0195565863793923,
|
| 27 |
+
"fde_3s_sample_count": 11035,
|
| 28 |
+
"fde_5s_m": 5.564463390564478,
|
| 29 |
+
"fde_5s_sample_count": 11035,
|
| 30 |
+
"fde_6p4s_m": null,
|
| 31 |
+
"fde_6p4s_sample_count": 0,
|
| 32 |
+
"mean_abs_lateral_error_m": 0.9844258828682012,
|
| 33 |
+
"mean_abs_longitudinal_error_m": 1.402497202309407,
|
| 34 |
+
"nonfinite_prediction_count": 0,
|
| 35 |
+
"nonfinite_prediction_rate": 0.0,
|
| 36 |
+
"total_timestep_count": 706240,
|
| 37 |
+
"valid_horizon_coverage": 0.78125,
|
| 38 |
+
"valid_timestep_count": 551750
|
| 39 |
+
}
|
| 40 |
+
},
|
| 41 |
+
{
|
| 42 |
+
"dataset": "kitscenes-val",
|
| 43 |
+
"dataset_version": "v3.5",
|
| 44 |
+
"model_artifact_id": "a1e6b1621018740485b3e1e43dda23713d913667eb39a65b77fcbf895dcbcefd",
|
| 45 |
+
"model_name": "KITScenes Trajectory Epoch 7",
|
| 46 |
+
"report_sha256": "832b9ec0648b997f5b1f825296269905e5ee3b13ae75918e0fdee5733dbd1818",
|
| 47 |
+
"report_uri": "s3://auto-e2e-platform-artifacts-381491877296/direct-evaluations/overlay-replay-v1/dataset=kitscenes-val/version=v3.5/model=a1e6b1621018740485b3e1e43dda23713d913667eb39a65b77fcbf895dcbcefd/report.json",
|
| 48 |
+
"trajectory": {
|
| 49 |
+
"ade_1s_m": 0.13467869638959368,
|
| 50 |
+
"ade_1s_sample_count": 11035,
|
| 51 |
+
"ade_2s_m": 0.3886152812418257,
|
| 52 |
+
"ade_2s_sample_count": 11035,
|
| 53 |
+
"ade_3s_m": 0.8034680158738138,
|
| 54 |
+
"ade_3s_sample_count": 11035,
|
| 55 |
+
"ade_5s_m": 2.0951617386340757,
|
| 56 |
+
"ade_5s_sample_count": 11035,
|
| 57 |
+
"ade_6p4s_m": null,
|
| 58 |
+
"ade_6p4s_sample_count": 0,
|
| 59 |
+
"fde_1s_m": 0.2830596284286022,
|
| 60 |
+
"fde_1s_sample_count": 11035,
|
| 61 |
+
"fde_2s_m": 1.0101467772020263,
|
| 62 |
+
"fde_2s_sample_count": 11035,
|
| 63 |
+
"fde_3s_m": 2.212711069790632,
|
| 64 |
+
"fde_3s_sample_count": 11035,
|
| 65 |
+
"fde_5s_m": 5.9508777011937015,
|
| 66 |
+
"fde_5s_sample_count": 11035,
|
| 67 |
+
"fde_6p4s_m": null,
|
| 68 |
+
"fde_6p4s_sample_count": 0,
|
| 69 |
+
"mean_abs_lateral_error_m": 1.0219872821653013,
|
| 70 |
+
"mean_abs_longitudinal_error_m": 1.565857085189425,
|
| 71 |
+
"nonfinite_prediction_count": 0,
|
| 72 |
+
"nonfinite_prediction_rate": 0.0,
|
| 73 |
+
"total_timestep_count": 706240,
|
| 74 |
+
"valid_horizon_coverage": 0.78125,
|
| 75 |
+
"valid_timestep_count": 551750
|
| 76 |
+
}
|
| 77 |
+
},
|
| 78 |
+
{
|
| 79 |
+
"dataset": "kitscenes-val",
|
| 80 |
+
"dataset_version": "v3.5",
|
| 81 |
+
"model_artifact_id": "ca8b43d7a777d6fd9195bb253d1452f3cd645f7aab3bd0e36b1f74ffb31df29b",
|
| 82 |
+
"model_name": "NuPlan Trajectory Epoch 5",
|
| 83 |
+
"report_sha256": "e45996448f95f193acfc225c37ff169fab64bb24b5ea7717928a20c426e7303b",
|
| 84 |
+
"report_uri": "s3://auto-e2e-platform-artifacts-381491877296/direct-evaluations/overlay-replay-v1/dataset=kitscenes-val/version=v3.5/model=ca8b43d7a777d6fd9195bb253d1452f3cd645f7aab3bd0e36b1f74ffb31df29b/report.json",
|
| 85 |
+
"trajectory": {
|
| 86 |
+
"ade_1s_m": 0.19511089635939421,
|
| 87 |
+
"ade_1s_sample_count": 11035,
|
| 88 |
+
"ade_2s_m": 0.4722640826416731,
|
| 89 |
+
"ade_2s_sample_count": 11035,
|
| 90 |
+
"ade_3s_m": 0.9147207671270287,
|
| 91 |
+
"ade_3s_sample_count": 11035,
|
| 92 |
+
"ade_5s_m": 2.295116252818514,
|
| 93 |
+
"ade_5s_sample_count": 11035,
|
| 94 |
+
"ade_6p4s_m": null,
|
| 95 |
+
"ade_6p4s_sample_count": 0,
|
| 96 |
+
"fde_1s_m": 0.37794148166995123,
|
| 97 |
+
"fde_1s_sample_count": 11035,
|
| 98 |
+
"fde_2s_m": 1.1331487579302755,
|
| 99 |
+
"fde_2s_sample_count": 11035,
|
| 100 |
+
"fde_3s_m": 2.422637476977514,
|
| 101 |
+
"fde_3s_sample_count": 11035,
|
| 102 |
+
"fde_5s_m": 6.388465638260406,
|
| 103 |
+
"fde_5s_sample_count": 11035,
|
| 104 |
+
"fde_6p4s_m": null,
|
| 105 |
+
"fde_6p4s_sample_count": 0,
|
| 106 |
+
"mean_abs_lateral_error_m": 1.19114795667851,
|
| 107 |
+
"mean_abs_longitudinal_error_m": 1.6633371994012054,
|
| 108 |
+
"nonfinite_prediction_count": 0,
|
| 109 |
+
"nonfinite_prediction_rate": 0.0,
|
| 110 |
+
"total_timestep_count": 706240,
|
| 111 |
+
"valid_horizon_coverage": 0.78125,
|
| 112 |
+
"valid_timestep_count": 551750
|
| 113 |
+
}
|
| 114 |
+
},
|
| 115 |
+
{
|
| 116 |
+
"dataset": "kitscenes-val",
|
| 117 |
+
"dataset_version": "v3.5",
|
| 118 |
+
"model_artifact_id": "ed00e072471aae51b4cf48fa54c3fd9c03c61e0fd6e2978829cd9068dfd6da68",
|
| 119 |
+
"model_name": "NuPlan Trajectory Epoch 4",
|
| 120 |
+
"report_sha256": "1ffde86560b8d9701c23c98b6b05b0de2611446a8d92f0fb90a08ccb884d1acb",
|
| 121 |
+
"report_uri": "s3://auto-e2e-platform-artifacts-381491877296/direct-evaluations/overlay-replay-v1/dataset=kitscenes-val/version=v3.5/model=ed00e072471aae51b4cf48fa54c3fd9c03c61e0fd6e2978829cd9068dfd6da68/report.json",
|
| 122 |
+
"trajectory": {
|
| 123 |
+
"ade_1s_m": 0.20299996935773157,
|
| 124 |
+
"ade_1s_sample_count": 11035,
|
| 125 |
+
"ade_2s_m": 0.5018003104886625,
|
| 126 |
+
"ade_2s_sample_count": 11035,
|
| 127 |
+
"ade_3s_m": 0.950258079868937,
|
| 128 |
+
"ade_3s_sample_count": 11035,
|
| 129 |
+
"ade_5s_m": 2.291857949000488,
|
| 130 |
+
"ade_5s_sample_count": 11035,
|
| 131 |
+
"ade_6p4s_m": null,
|
| 132 |
+
"ade_6p4s_sample_count": 0,
|
| 133 |
+
"fde_1s_m": 0.40659580015215774,
|
| 134 |
+
"fde_1s_sample_count": 11035,
|
| 135 |
+
"fde_2s_m": 1.1934783774143614,
|
| 136 |
+
"fde_2s_sample_count": 11035,
|
| 137 |
+
"fde_3s_m": 2.450842261185858,
|
| 138 |
+
"fde_3s_sample_count": 11035,
|
| 139 |
+
"fde_5s_m": 6.22721162418439,
|
| 140 |
+
"fde_5s_sample_count": 11035,
|
| 141 |
+
"fde_6p4s_m": null,
|
| 142 |
+
"fde_6p4s_sample_count": 0,
|
| 143 |
+
"mean_abs_lateral_error_m": 1.2082210980047907,
|
| 144 |
+
"mean_abs_longitudinal_error_m": 1.6447808719447803,
|
| 145 |
+
"nonfinite_prediction_count": 0,
|
| 146 |
+
"nonfinite_prediction_rate": 0.0,
|
| 147 |
+
"total_timestep_count": 706240,
|
| 148 |
+
"valid_horizon_coverage": 0.78125,
|
| 149 |
+
"valid_timestep_count": 551750
|
| 150 |
+
}
|
| 151 |
+
},
|
| 152 |
+
{
|
| 153 |
+
"dataset": "kitscenes-test",
|
| 154 |
+
"dataset_version": "v1.0",
|
| 155 |
+
"model_artifact_id": "120a21639d9767d512eea645be0a83b01476c9e0366a8b957ac51b7ead1bd762",
|
| 156 |
+
"model_name": "KITScenes Trajectory Epoch 5",
|
| 157 |
+
"report_sha256": "582dc108ce030756d5d1758dbe198d62b3355bf32a123af54be4e66704c50b8a",
|
| 158 |
+
"report_uri": "s3://auto-e2e-platform-artifacts-381491877296/direct-evaluations/overlay-replay-v1/dataset=kitscenes-test/version=v1.0/model=120a21639d9767d512eea645be0a83b01476c9e0366a8b957ac51b7ead1bd762/report.json",
|
| 159 |
+
"trajectory": {
|
| 160 |
+
"ade_1s_m": 0.14632053498252978,
|
| 161 |
+
"ade_1s_sample_count": 23690,
|
| 162 |
+
"ade_2s_m": 0.4121187056069301,
|
| 163 |
+
"ade_2s_sample_count": 23690,
|
| 164 |
+
"ade_3s_m": 0.8263108995057848,
|
| 165 |
+
"ade_3s_sample_count": 23690,
|
| 166 |
+
"ade_5s_m": 2.104176420295984,
|
| 167 |
+
"ade_5s_sample_count": 23690,
|
| 168 |
+
"ade_6p4s_m": null,
|
| 169 |
+
"ade_6p4s_sample_count": 0,
|
| 170 |
+
"fde_1s_m": 0.3106528805343601,
|
| 171 |
+
"fde_1s_sample_count": 23690,
|
| 172 |
+
"fde_2s_m": 1.0453379525636344,
|
| 173 |
+
"fde_2s_sample_count": 23690,
|
| 174 |
+
"fde_3s_m": 2.219851167527027,
|
| 175 |
+
"fde_3s_sample_count": 23690,
|
| 176 |
+
"fde_5s_m": 5.942222944848886,
|
| 177 |
+
"fde_5s_sample_count": 23690,
|
| 178 |
+
"fde_6p4s_m": null,
|
| 179 |
+
"fde_6p4s_sample_count": 0,
|
| 180 |
+
"mean_abs_lateral_error_m": 1.2043875392879961,
|
| 181 |
+
"mean_abs_longitudinal_error_m": 1.4050545911622678,
|
| 182 |
+
"nonfinite_prediction_count": 0,
|
| 183 |
+
"nonfinite_prediction_rate": 0.0,
|
| 184 |
+
"total_timestep_count": 1516160,
|
| 185 |
+
"valid_horizon_coverage": 0.78125,
|
| 186 |
+
"valid_timestep_count": 1184500
|
| 187 |
+
}
|
| 188 |
+
},
|
| 189 |
+
{
|
| 190 |
+
"dataset": "kitscenes-test",
|
| 191 |
+
"dataset_version": "v1.0",
|
| 192 |
+
"model_artifact_id": "a1e6b1621018740485b3e1e43dda23713d913667eb39a65b77fcbf895dcbcefd",
|
| 193 |
+
"model_name": "KITScenes Trajectory Epoch 7",
|
| 194 |
+
"report_sha256": "d01169d68266d4b7203e92a4d25ade2a3277357ff89cc26cdb202051a26016fc",
|
| 195 |
+
"report_uri": "s3://auto-e2e-platform-artifacts-381491877296/direct-evaluations/overlay-replay-v1/dataset=kitscenes-test/version=v1.0/model=a1e6b1621018740485b3e1e43dda23713d913667eb39a65b77fcbf895dcbcefd/report.json",
|
| 196 |
+
"trajectory": {
|
| 197 |
+
"ade_1s_m": 0.1680068780791157,
|
| 198 |
+
"ade_1s_sample_count": 23690,
|
| 199 |
+
"ade_2s_m": 0.4923481277999561,
|
| 200 |
+
"ade_2s_sample_count": 23690,
|
| 201 |
+
"ade_3s_m": 0.9830703987751742,
|
| 202 |
+
"ade_3s_sample_count": 23690,
|
| 203 |
+
"ade_5s_m": 2.469892014634966,
|
| 204 |
+
"ade_5s_sample_count": 23690,
|
| 205 |
+
"ade_6p4s_m": null,
|
| 206 |
+
"ade_6p4s_sample_count": 0,
|
| 207 |
+
"fde_1s_m": 0.37537789674250205,
|
| 208 |
+
"fde_1s_sample_count": 23690,
|
| 209 |
+
"fde_2s_m": 1.2511962416817588,
|
| 210 |
+
"fde_2s_sample_count": 23690,
|
| 211 |
+
"fde_3s_m": 2.624177628524085,
|
| 212 |
+
"fde_3s_sample_count": 23690,
|
| 213 |
+
"fde_5s_m": 6.896411433552511,
|
| 214 |
+
"fde_5s_sample_count": 23690,
|
| 215 |
+
"fde_6p4s_m": null,
|
| 216 |
+
"fde_6p4s_sample_count": 0,
|
| 217 |
+
"mean_abs_lateral_error_m": 1.4730429718483722,
|
| 218 |
+
"mean_abs_longitudinal_error_m": 1.6598486258851333,
|
| 219 |
+
"nonfinite_prediction_count": 0,
|
| 220 |
+
"nonfinite_prediction_rate": 0.0,
|
| 221 |
+
"total_timestep_count": 1516160,
|
| 222 |
+
"valid_horizon_coverage": 0.78125,
|
| 223 |
+
"valid_timestep_count": 1184500
|
| 224 |
+
}
|
| 225 |
+
},
|
| 226 |
+
{
|
| 227 |
+
"dataset": "kitscenes-test",
|
| 228 |
+
"dataset_version": "v1.0",
|
| 229 |
+
"model_artifact_id": "ca8b43d7a777d6fd9195bb253d1452f3cd645f7aab3bd0e36b1f74ffb31df29b",
|
| 230 |
+
"model_name": "NuPlan Trajectory Epoch 5",
|
| 231 |
+
"report_sha256": "fc5b19beea416706461b535314a497d6a2b929d97668e8475638ae04a65221b7",
|
| 232 |
+
"report_uri": "s3://auto-e2e-platform-artifacts-381491877296/direct-evaluations/overlay-replay-v1/dataset=kitscenes-test/version=v1.0/model=ca8b43d7a777d6fd9195bb253d1452f3cd645f7aab3bd0e36b1f74ffb31df29b/report.json",
|
| 233 |
+
"trajectory": {
|
| 234 |
+
"ade_1s_m": 0.16017068059452302,
|
| 235 |
+
"ade_1s_sample_count": 23690,
|
| 236 |
+
"ade_2s_m": 0.3979342750750205,
|
| 237 |
+
"ade_2s_sample_count": 23690,
|
| 238 |
+
"ade_3s_m": 0.7923739682724033,
|
| 239 |
+
"ade_3s_sample_count": 23690,
|
| 240 |
+
"ade_5s_m": 2.1119916453019565,
|
| 241 |
+
"ade_5s_sample_count": 23690,
|
| 242 |
+
"ade_6p4s_m": null,
|
| 243 |
+
"ade_6p4s_sample_count": 0,
|
| 244 |
+
"fde_1s_m": 0.3134281782098752,
|
| 245 |
+
"fde_1s_sample_count": 23690,
|
| 246 |
+
"fde_2s_m": 0.9727075050905802,
|
| 247 |
+
"fde_2s_sample_count": 23690,
|
| 248 |
+
"fde_3s_m": 2.1621481640709304,
|
| 249 |
+
"fde_3s_sample_count": 23690,
|
| 250 |
+
"fde_5s_m": 6.175121158406895,
|
| 251 |
+
"fde_5s_sample_count": 23690,
|
| 252 |
+
"fde_6p4s_m": null,
|
| 253 |
+
"fde_6p4s_sample_count": 0,
|
| 254 |
+
"mean_abs_lateral_error_m": 1.0584339709055983,
|
| 255 |
+
"mean_abs_longitudinal_error_m": 1.5495691251398722,
|
| 256 |
+
"nonfinite_prediction_count": 0,
|
| 257 |
+
"nonfinite_prediction_rate": 0.0,
|
| 258 |
+
"total_timestep_count": 1516160,
|
| 259 |
+
"valid_horizon_coverage": 0.78125,
|
| 260 |
+
"valid_timestep_count": 1184500
|
| 261 |
+
}
|
| 262 |
+
},
|
| 263 |
+
{
|
| 264 |
+
"dataset": "kitscenes-test",
|
| 265 |
+
"dataset_version": "v1.0",
|
| 266 |
+
"model_artifact_id": "ed00e072471aae51b4cf48fa54c3fd9c03c61e0fd6e2978829cd9068dfd6da68",
|
| 267 |
+
"model_name": "NuPlan Trajectory Epoch 4",
|
| 268 |
+
"report_sha256": "5eeb68a4013a9777ba927030b73877a1789f5b4eb9e14f6da760bef7f9b42808",
|
| 269 |
+
"report_uri": "s3://auto-e2e-platform-artifacts-381491877296/direct-evaluations/overlay-replay-v1/dataset=kitscenes-test/version=v1.0/model=ed00e072471aae51b4cf48fa54c3fd9c03c61e0fd6e2978829cd9068dfd6da68/report.json",
|
| 270 |
+
"trajectory": {
|
| 271 |
+
"ade_1s_m": 0.1915564095529401,
|
| 272 |
+
"ade_1s_sample_count": 23690,
|
| 273 |
+
"ade_2s_m": 0.4816506897179045,
|
| 274 |
+
"ade_2s_sample_count": 23690,
|
| 275 |
+
"ade_3s_m": 0.9181210775291644,
|
| 276 |
+
"ade_3s_sample_count": 23690,
|
| 277 |
+
"ade_5s_m": 2.2840408150913345,
|
| 278 |
+
"ade_5s_sample_count": 23690,
|
| 279 |
+
"ade_6p4s_m": null,
|
| 280 |
+
"ade_6p4s_sample_count": 0,
|
| 281 |
+
"fde_1s_m": 0.3898781367591937,
|
| 282 |
+
"fde_1s_sample_count": 23690,
|
| 283 |
+
"fde_2s_m": 1.1514432707206264,
|
| 284 |
+
"fde_2s_sample_count": 23690,
|
| 285 |
+
"fde_3s_m": 2.38952477957317,
|
| 286 |
+
"fde_3s_sample_count": 23690,
|
| 287 |
+
"fde_5s_m": 6.42657723546563,
|
| 288 |
+
"fde_5s_sample_count": 23690,
|
| 289 |
+
"fde_6p4s_m": null,
|
| 290 |
+
"fde_6p4s_sample_count": 0,
|
| 291 |
+
"mean_abs_lateral_error_m": 1.2167123846906904,
|
| 292 |
+
"mean_abs_longitudinal_error_m": 1.6287253737215717,
|
| 293 |
+
"nonfinite_prediction_count": 0,
|
| 294 |
+
"nonfinite_prediction_rate": 0.0,
|
| 295 |
+
"total_timestep_count": 1516160,
|
| 296 |
+
"valid_horizon_coverage": 0.78125,
|
| 297 |
+
"valid_timestep_count": 1184500
|
| 298 |
+
}
|
| 299 |
+
}
|
| 300 |
+
],
|
| 301 |
+
"schema_version": "kitscenes_val_test_overlay_evaluation_index_v1",
|
| 302 |
+
"target_identities": {
|
| 303 |
+
"kitscenes-test": {
|
| 304 |
+
"sample_count": 23690,
|
| 305 |
+
"sample_tar_bytes_streamed": 12241868800,
|
| 306 |
+
"sample_uid_sha256": "9f7e5f0eae0582c3b439d29914cefb88e79b944ce1349125d8e9cdbc8bd891db",
|
| 307 |
+
"source_manifest_set_sha256": "ac291affe2c174d8c506f5564fc22989d8d627e01c90f2d17bed2c877a28528a",
|
| 308 |
+
"source_partition_count": 206,
|
| 309 |
+
"source_partition_list_sha256": "5aec3df6d997b94a6d782c9cd3ac7f704fc43cd5d8406b4cd7eb00d1f286b476"
|
| 310 |
+
},
|
| 311 |
+
"kitscenes-val": {
|
| 312 |
+
"sample_count": 11035,
|
| 313 |
+
"sample_tar_bytes_streamed": 7770572800,
|
| 314 |
+
"sample_uid_sha256": "00f824202c97b47ec74561d247e8e30686e5505eef2ad2bc0acba97e41ce9925",
|
| 315 |
+
"source_manifest_set_sha256": "5d4c0ffc7b0706cd917f1b1546cc557b63372839bd67677f8baeef59ad0981e2",
|
| 316 |
+
"source_partition_count": 117,
|
| 317 |
+
"source_partition_list_sha256": "a978e6a373677a15998f41032a850de26cc321271918094237dd71c6e0a51870"
|
| 318 |
+
}
|
| 319 |
+
}
|
| 320 |
+
}
|
evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-5/report.json
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"created_at": "2026-09-14T10:07:41Z",
|
| 3 |
+
"dataset": "kitscenes-test",
|
| 4 |
+
"dataset_manifest_sha256": "b5dbe72c8d2c5cf49f8d55c9b24a4bde9dca130bdb1816480f299a82b43f1a24",
|
| 5 |
+
"dataset_version": "v1.0",
|
| 6 |
+
"evaluation_role": "official_test_camera_only_missing_map_route",
|
| 7 |
+
"input_track": "camera_only_missing_map_route",
|
| 8 |
+
"limitations": [
|
| 9 |
+
"Trajectory metrics replay the published deterministic control overlay and do not rerun the model.",
|
| 10 |
+
"Route reconstruction, route counterfactual, and route input gradient metrics are not stored in trajectory overlay v5."
|
| 11 |
+
],
|
| 12 |
+
"metrics": {
|
| 13 |
+
"bev_segmentation": {
|
| 14 |
+
"available": false,
|
| 15 |
+
"reason": "not_replayed_from_trajectory_overlay"
|
| 16 |
+
},
|
| 17 |
+
"route": {
|
| 18 |
+
"available": false,
|
| 19 |
+
"reason": "route_counterfactuals_not_in_overlay_artifact"
|
| 20 |
+
},
|
| 21 |
+
"sample_count": 23690,
|
| 22 |
+
"schema_version": "reactive_multitask_evaluation_v1",
|
| 23 |
+
"trajectory": {
|
| 24 |
+
"ade_1s_m": 0.14632053498252978,
|
| 25 |
+
"ade_1s_sample_count": 23690,
|
| 26 |
+
"ade_2s_m": 0.4121187056069301,
|
| 27 |
+
"ade_2s_sample_count": 23690,
|
| 28 |
+
"ade_3s_m": 0.8263108995057848,
|
| 29 |
+
"ade_3s_sample_count": 23690,
|
| 30 |
+
"ade_5s_m": 2.104176420295984,
|
| 31 |
+
"ade_5s_sample_count": 23690,
|
| 32 |
+
"ade_6p4s_m": null,
|
| 33 |
+
"ade_6p4s_sample_count": 0,
|
| 34 |
+
"fde_1s_m": 0.3106528805343601,
|
| 35 |
+
"fde_1s_sample_count": 23690,
|
| 36 |
+
"fde_2s_m": 1.0453379525636344,
|
| 37 |
+
"fde_2s_sample_count": 23690,
|
| 38 |
+
"fde_3s_m": 2.219851167527027,
|
| 39 |
+
"fde_3s_sample_count": 23690,
|
| 40 |
+
"fde_5s_m": 5.942222944848886,
|
| 41 |
+
"fde_5s_sample_count": 23690,
|
| 42 |
+
"fde_6p4s_m": null,
|
| 43 |
+
"fde_6p4s_sample_count": 0,
|
| 44 |
+
"mean_abs_lateral_error_m": 1.2043875392879961,
|
| 45 |
+
"mean_abs_longitudinal_error_m": 1.4050545911622678,
|
| 46 |
+
"nonfinite_prediction_count": 0,
|
| 47 |
+
"nonfinite_prediction_rate": 0.0,
|
| 48 |
+
"total_timestep_count": 1516160,
|
| 49 |
+
"valid_horizon_coverage": 0.78125,
|
| 50 |
+
"valid_timestep_count": 1184500
|
| 51 |
+
}
|
| 52 |
+
},
|
| 53 |
+
"model_artifact_id": "120a21639d9767d512eea645be0a83b01476c9e0366a8b957ac51b7ead1bd762",
|
| 54 |
+
"model_name": "KITScenes Trajectory Epoch 5",
|
| 55 |
+
"overlay_object_count": 206,
|
| 56 |
+
"overlay_sample_count": 23690,
|
| 57 |
+
"overlay_schema": "v5",
|
| 58 |
+
"overlay_set_sha256": "a14fd102ff9f0ae19b6f6e50141cc3b7318ba5ad2d2e6093de38f470560ce32c",
|
| 59 |
+
"schema_version": "reactive_kitscenes_overlay_replay_evaluation_v1",
|
| 60 |
+
"source_split": "test",
|
| 61 |
+
"target_identity": {
|
| 62 |
+
"sample_count": 23690,
|
| 63 |
+
"sample_tar_bytes_streamed": 12241868800,
|
| 64 |
+
"sample_uid_sha256": "9f7e5f0eae0582c3b439d29914cefb88e79b944ce1349125d8e9cdbc8bd891db",
|
| 65 |
+
"source_manifest_set_sha256": "ac291affe2c174d8c506f5564fc22989d8d627e01c90f2d17bed2c877a28528a",
|
| 66 |
+
"source_partition_count": 206,
|
| 67 |
+
"source_partition_list_sha256": "5aec3df6d997b94a6d782c9cd3ac7f704fc43cd5d8406b4cd7eb00d1f286b476"
|
| 68 |
+
},
|
| 69 |
+
"target_sample_count": 23690,
|
| 70 |
+
"trajectory_inference_policy": {
|
| 71 |
+
"control_rollout": "semi_implicit_unicycle_v1",
|
| 72 |
+
"dt_seconds": 0.1,
|
| 73 |
+
"planner": "gru",
|
| 74 |
+
"stochastic_noise": false
|
| 75 |
+
}
|
| 76 |
+
}
|
evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-7/report.json
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"created_at": "2026-09-14T10:08:08Z",
|
| 3 |
+
"dataset": "kitscenes-test",
|
| 4 |
+
"dataset_manifest_sha256": "b5dbe72c8d2c5cf49f8d55c9b24a4bde9dca130bdb1816480f299a82b43f1a24",
|
| 5 |
+
"dataset_version": "v1.0",
|
| 6 |
+
"evaluation_role": "official_test_camera_only_missing_map_route",
|
| 7 |
+
"input_track": "camera_only_missing_map_route",
|
| 8 |
+
"limitations": [
|
| 9 |
+
"Trajectory metrics replay the published deterministic control overlay and do not rerun the model.",
|
| 10 |
+
"Route reconstruction, route counterfactual, and route input gradient metrics are not stored in trajectory overlay v5."
|
| 11 |
+
],
|
| 12 |
+
"metrics": {
|
| 13 |
+
"bev_segmentation": {
|
| 14 |
+
"available": false,
|
| 15 |
+
"reason": "not_replayed_from_trajectory_overlay"
|
| 16 |
+
},
|
| 17 |
+
"route": {
|
| 18 |
+
"available": false,
|
| 19 |
+
"reason": "route_counterfactuals_not_in_overlay_artifact"
|
| 20 |
+
},
|
| 21 |
+
"sample_count": 23690,
|
| 22 |
+
"schema_version": "reactive_multitask_evaluation_v1",
|
| 23 |
+
"trajectory": {
|
| 24 |
+
"ade_1s_m": 0.1680068780791157,
|
| 25 |
+
"ade_1s_sample_count": 23690,
|
| 26 |
+
"ade_2s_m": 0.4923481277999561,
|
| 27 |
+
"ade_2s_sample_count": 23690,
|
| 28 |
+
"ade_3s_m": 0.9830703987751742,
|
| 29 |
+
"ade_3s_sample_count": 23690,
|
| 30 |
+
"ade_5s_m": 2.469892014634966,
|
| 31 |
+
"ade_5s_sample_count": 23690,
|
| 32 |
+
"ade_6p4s_m": null,
|
| 33 |
+
"ade_6p4s_sample_count": 0,
|
| 34 |
+
"fde_1s_m": 0.37537789674250205,
|
| 35 |
+
"fde_1s_sample_count": 23690,
|
| 36 |
+
"fde_2s_m": 1.2511962416817588,
|
| 37 |
+
"fde_2s_sample_count": 23690,
|
| 38 |
+
"fde_3s_m": 2.624177628524085,
|
| 39 |
+
"fde_3s_sample_count": 23690,
|
| 40 |
+
"fde_5s_m": 6.896411433552511,
|
| 41 |
+
"fde_5s_sample_count": 23690,
|
| 42 |
+
"fde_6p4s_m": null,
|
| 43 |
+
"fde_6p4s_sample_count": 0,
|
| 44 |
+
"mean_abs_lateral_error_m": 1.4730429718483722,
|
| 45 |
+
"mean_abs_longitudinal_error_m": 1.6598486258851333,
|
| 46 |
+
"nonfinite_prediction_count": 0,
|
| 47 |
+
"nonfinite_prediction_rate": 0.0,
|
| 48 |
+
"total_timestep_count": 1516160,
|
| 49 |
+
"valid_horizon_coverage": 0.78125,
|
| 50 |
+
"valid_timestep_count": 1184500
|
| 51 |
+
}
|
| 52 |
+
},
|
| 53 |
+
"model_artifact_id": "a1e6b1621018740485b3e1e43dda23713d913667eb39a65b77fcbf895dcbcefd",
|
| 54 |
+
"model_name": "KITScenes Trajectory Epoch 7",
|
| 55 |
+
"overlay_object_count": 206,
|
| 56 |
+
"overlay_sample_count": 23690,
|
| 57 |
+
"overlay_schema": "v5",
|
| 58 |
+
"overlay_set_sha256": "850fda2cb17d89810a4ba9abadf5fb390583fce1b907861a3ec4de9ae23f3c38",
|
| 59 |
+
"schema_version": "reactive_kitscenes_overlay_replay_evaluation_v1",
|
| 60 |
+
"source_split": "test",
|
| 61 |
+
"target_identity": {
|
| 62 |
+
"sample_count": 23690,
|
| 63 |
+
"sample_tar_bytes_streamed": 12241868800,
|
| 64 |
+
"sample_uid_sha256": "9f7e5f0eae0582c3b439d29914cefb88e79b944ce1349125d8e9cdbc8bd891db",
|
| 65 |
+
"source_manifest_set_sha256": "ac291affe2c174d8c506f5564fc22989d8d627e01c90f2d17bed2c877a28528a",
|
| 66 |
+
"source_partition_count": 206,
|
| 67 |
+
"source_partition_list_sha256": "5aec3df6d997b94a6d782c9cd3ac7f704fc43cd5d8406b4cd7eb00d1f286b476"
|
| 68 |
+
},
|
| 69 |
+
"target_sample_count": 23690,
|
| 70 |
+
"trajectory_inference_policy": {
|
| 71 |
+
"control_rollout": "semi_implicit_unicycle_v1",
|
| 72 |
+
"dt_seconds": 0.1,
|
| 73 |
+
"planner": "gru",
|
| 74 |
+
"stochastic_noise": false
|
| 75 |
+
}
|
| 76 |
+
}
|
evaluations/overlay-replay-v1/kitscenes-test/nuplan-epoch-4/report.json
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"created_at": "2026-09-14T10:09:00Z",
|
| 3 |
+
"dataset": "kitscenes-test",
|
| 4 |
+
"dataset_manifest_sha256": "b5dbe72c8d2c5cf49f8d55c9b24a4bde9dca130bdb1816480f299a82b43f1a24",
|
| 5 |
+
"dataset_version": "v1.0",
|
| 6 |
+
"evaluation_role": "official_test_camera_only_missing_map_route",
|
| 7 |
+
"input_track": "camera_only_missing_map_route",
|
| 8 |
+
"limitations": [
|
| 9 |
+
"Trajectory metrics replay the published deterministic control overlay and do not rerun the model.",
|
| 10 |
+
"Route reconstruction, route counterfactual, and route input gradient metrics are not stored in trajectory overlay v5."
|
| 11 |
+
],
|
| 12 |
+
"metrics": {
|
| 13 |
+
"bev_segmentation": {
|
| 14 |
+
"available": false,
|
| 15 |
+
"reason": "not_replayed_from_trajectory_overlay"
|
| 16 |
+
},
|
| 17 |
+
"route": {
|
| 18 |
+
"available": false,
|
| 19 |
+
"reason": "route_counterfactuals_not_in_overlay_artifact"
|
| 20 |
+
},
|
| 21 |
+
"sample_count": 23690,
|
| 22 |
+
"schema_version": "reactive_multitask_evaluation_v1",
|
| 23 |
+
"trajectory": {
|
| 24 |
+
"ade_1s_m": 0.1915564095529401,
|
| 25 |
+
"ade_1s_sample_count": 23690,
|
| 26 |
+
"ade_2s_m": 0.4816506897179045,
|
| 27 |
+
"ade_2s_sample_count": 23690,
|
| 28 |
+
"ade_3s_m": 0.9181210775291644,
|
| 29 |
+
"ade_3s_sample_count": 23690,
|
| 30 |
+
"ade_5s_m": 2.2840408150913345,
|
| 31 |
+
"ade_5s_sample_count": 23690,
|
| 32 |
+
"ade_6p4s_m": null,
|
| 33 |
+
"ade_6p4s_sample_count": 0,
|
| 34 |
+
"fde_1s_m": 0.3898781367591937,
|
| 35 |
+
"fde_1s_sample_count": 23690,
|
| 36 |
+
"fde_2s_m": 1.1514432707206264,
|
| 37 |
+
"fde_2s_sample_count": 23690,
|
| 38 |
+
"fde_3s_m": 2.38952477957317,
|
| 39 |
+
"fde_3s_sample_count": 23690,
|
| 40 |
+
"fde_5s_m": 6.42657723546563,
|
| 41 |
+
"fde_5s_sample_count": 23690,
|
| 42 |
+
"fde_6p4s_m": null,
|
| 43 |
+
"fde_6p4s_sample_count": 0,
|
| 44 |
+
"mean_abs_lateral_error_m": 1.2167123846906904,
|
| 45 |
+
"mean_abs_longitudinal_error_m": 1.6287253737215717,
|
| 46 |
+
"nonfinite_prediction_count": 0,
|
| 47 |
+
"nonfinite_prediction_rate": 0.0,
|
| 48 |
+
"total_timestep_count": 1516160,
|
| 49 |
+
"valid_horizon_coverage": 0.78125,
|
| 50 |
+
"valid_timestep_count": 1184500
|
| 51 |
+
}
|
| 52 |
+
},
|
| 53 |
+
"model_artifact_id": "ed00e072471aae51b4cf48fa54c3fd9c03c61e0fd6e2978829cd9068dfd6da68",
|
| 54 |
+
"model_name": "NuPlan Trajectory Epoch 4",
|
| 55 |
+
"overlay_object_count": 206,
|
| 56 |
+
"overlay_sample_count": 23690,
|
| 57 |
+
"overlay_schema": "v5",
|
| 58 |
+
"overlay_set_sha256": "fc2202e59c30aa6bf7a7ff7f3afec24b8b29516cc1eb6083a80eacb41f9427c1",
|
| 59 |
+
"schema_version": "reactive_kitscenes_overlay_replay_evaluation_v1",
|
| 60 |
+
"source_split": "test",
|
| 61 |
+
"target_identity": {
|
| 62 |
+
"sample_count": 23690,
|
| 63 |
+
"sample_tar_bytes_streamed": 12241868800,
|
| 64 |
+
"sample_uid_sha256": "9f7e5f0eae0582c3b439d29914cefb88e79b944ce1349125d8e9cdbc8bd891db",
|
| 65 |
+
"source_manifest_set_sha256": "ac291affe2c174d8c506f5564fc22989d8d627e01c90f2d17bed2c877a28528a",
|
| 66 |
+
"source_partition_count": 206,
|
| 67 |
+
"source_partition_list_sha256": "5aec3df6d997b94a6d782c9cd3ac7f704fc43cd5d8406b4cd7eb00d1f286b476"
|
| 68 |
+
},
|
| 69 |
+
"target_sample_count": 23690,
|
| 70 |
+
"trajectory_inference_policy": {
|
| 71 |
+
"control_rollout": "semi_implicit_unicycle_v1",
|
| 72 |
+
"dt_seconds": 0.1,
|
| 73 |
+
"planner": "gru",
|
| 74 |
+
"stochastic_noise": false
|
| 75 |
+
}
|
| 76 |
+
}
|
evaluations/overlay-replay-v1/kitscenes-test/nuplan-epoch-5/report.json
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"created_at": "2026-09-14T10:08:35Z",
|
| 3 |
+
"dataset": "kitscenes-test",
|
| 4 |
+
"dataset_manifest_sha256": "b5dbe72c8d2c5cf49f8d55c9b24a4bde9dca130bdb1816480f299a82b43f1a24",
|
| 5 |
+
"dataset_version": "v1.0",
|
| 6 |
+
"evaluation_role": "official_test_camera_only_missing_map_route",
|
| 7 |
+
"input_track": "camera_only_missing_map_route",
|
| 8 |
+
"limitations": [
|
| 9 |
+
"Trajectory metrics replay the published deterministic control overlay and do not rerun the model.",
|
| 10 |
+
"Route reconstruction, route counterfactual, and route input gradient metrics are not stored in trajectory overlay v5."
|
| 11 |
+
],
|
| 12 |
+
"metrics": {
|
| 13 |
+
"bev_segmentation": {
|
| 14 |
+
"available": false,
|
| 15 |
+
"reason": "not_replayed_from_trajectory_overlay"
|
| 16 |
+
},
|
| 17 |
+
"route": {
|
| 18 |
+
"available": false,
|
| 19 |
+
"reason": "route_counterfactuals_not_in_overlay_artifact"
|
| 20 |
+
},
|
| 21 |
+
"sample_count": 23690,
|
| 22 |
+
"schema_version": "reactive_multitask_evaluation_v1",
|
| 23 |
+
"trajectory": {
|
| 24 |
+
"ade_1s_m": 0.16017068059452302,
|
| 25 |
+
"ade_1s_sample_count": 23690,
|
| 26 |
+
"ade_2s_m": 0.3979342750750205,
|
| 27 |
+
"ade_2s_sample_count": 23690,
|
| 28 |
+
"ade_3s_m": 0.7923739682724033,
|
| 29 |
+
"ade_3s_sample_count": 23690,
|
| 30 |
+
"ade_5s_m": 2.1119916453019565,
|
| 31 |
+
"ade_5s_sample_count": 23690,
|
| 32 |
+
"ade_6p4s_m": null,
|
| 33 |
+
"ade_6p4s_sample_count": 0,
|
| 34 |
+
"fde_1s_m": 0.3134281782098752,
|
| 35 |
+
"fde_1s_sample_count": 23690,
|
| 36 |
+
"fde_2s_m": 0.9727075050905802,
|
| 37 |
+
"fde_2s_sample_count": 23690,
|
| 38 |
+
"fde_3s_m": 2.1621481640709304,
|
| 39 |
+
"fde_3s_sample_count": 23690,
|
| 40 |
+
"fde_5s_m": 6.175121158406895,
|
| 41 |
+
"fde_5s_sample_count": 23690,
|
| 42 |
+
"fde_6p4s_m": null,
|
| 43 |
+
"fde_6p4s_sample_count": 0,
|
| 44 |
+
"mean_abs_lateral_error_m": 1.0584339709055983,
|
| 45 |
+
"mean_abs_longitudinal_error_m": 1.5495691251398722,
|
| 46 |
+
"nonfinite_prediction_count": 0,
|
| 47 |
+
"nonfinite_prediction_rate": 0.0,
|
| 48 |
+
"total_timestep_count": 1516160,
|
| 49 |
+
"valid_horizon_coverage": 0.78125,
|
| 50 |
+
"valid_timestep_count": 1184500
|
| 51 |
+
}
|
| 52 |
+
},
|
| 53 |
+
"model_artifact_id": "ca8b43d7a777d6fd9195bb253d1452f3cd645f7aab3bd0e36b1f74ffb31df29b",
|
| 54 |
+
"model_name": "NuPlan Trajectory Epoch 5",
|
| 55 |
+
"overlay_object_count": 206,
|
| 56 |
+
"overlay_sample_count": 23690,
|
| 57 |
+
"overlay_schema": "v5",
|
| 58 |
+
"overlay_set_sha256": "4e379136e08603ae8aece99e24e5269ed8283229c80b2844d120ec11aff09c36",
|
| 59 |
+
"schema_version": "reactive_kitscenes_overlay_replay_evaluation_v1",
|
| 60 |
+
"source_split": "test",
|
| 61 |
+
"target_identity": {
|
| 62 |
+
"sample_count": 23690,
|
| 63 |
+
"sample_tar_bytes_streamed": 12241868800,
|
| 64 |
+
"sample_uid_sha256": "9f7e5f0eae0582c3b439d29914cefb88e79b944ce1349125d8e9cdbc8bd891db",
|
| 65 |
+
"source_manifest_set_sha256": "ac291affe2c174d8c506f5564fc22989d8d627e01c90f2d17bed2c877a28528a",
|
| 66 |
+
"source_partition_count": 206,
|
| 67 |
+
"source_partition_list_sha256": "5aec3df6d997b94a6d782c9cd3ac7f704fc43cd5d8406b4cd7eb00d1f286b476"
|
| 68 |
+
},
|
| 69 |
+
"target_sample_count": 23690,
|
| 70 |
+
"trajectory_inference_policy": {
|
| 71 |
+
"control_rollout": "semi_implicit_unicycle_v1",
|
| 72 |
+
"dt_seconds": 0.1,
|
| 73 |
+
"planner": "gru",
|
| 74 |
+
"stochastic_noise": false
|
| 75 |
+
}
|
| 76 |
+
}
|
evaluations/overlay-replay-v1/kitscenes-val/kitscenes-epoch-5/report.json
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"created_at": "2026-09-14T09:16:51Z",
|
| 3 |
+
"dataset": "kitscenes-val",
|
| 4 |
+
"dataset_manifest_sha256": "4a12ede73949cf66fd01fdaee89766f1e534de705f783a2f65b372c142fa1d05",
|
| 5 |
+
"dataset_version": "v3.5",
|
| 6 |
+
"evaluation_role": "official_val_camera_map_route",
|
| 7 |
+
"input_track": "camera_map_route",
|
| 8 |
+
"limitations": [
|
| 9 |
+
"Trajectory metrics replay the published deterministic control overlay and do not rerun the model.",
|
| 10 |
+
"Route reconstruction, route counterfactual, and route input gradient metrics are not stored in trajectory overlay v5."
|
| 11 |
+
],
|
| 12 |
+
"metrics": {
|
| 13 |
+
"bev_segmentation": {
|
| 14 |
+
"available": false,
|
| 15 |
+
"reason": "not_replayed_from_trajectory_overlay"
|
| 16 |
+
},
|
| 17 |
+
"route": {
|
| 18 |
+
"available": false,
|
| 19 |
+
"reason": "route_counterfactuals_not_in_overlay_artifact"
|
| 20 |
+
},
|
| 21 |
+
"sample_count": 11035,
|
| 22 |
+
"schema_version": "reactive_multitask_evaluation_v1",
|
| 23 |
+
"trajectory": {
|
| 24 |
+
"ade_1s_m": 0.14720280881581088,
|
| 25 |
+
"ade_1s_sample_count": 11035,
|
| 26 |
+
"ade_2s_m": 0.3729208092788606,
|
| 27 |
+
"ade_2s_sample_count": 11035,
|
| 28 |
+
"ade_3s_m": 0.744916487594762,
|
| 29 |
+
"ade_3s_sample_count": 11035,
|
| 30 |
+
"ade_5s_m": 1.9405047441156529,
|
| 31 |
+
"ade_5s_sample_count": 11035,
|
| 32 |
+
"ade_6p4s_m": null,
|
| 33 |
+
"ade_6p4s_sample_count": 0,
|
| 34 |
+
"fde_1s_m": 0.28501037315538935,
|
| 35 |
+
"fde_1s_sample_count": 11035,
|
| 36 |
+
"fde_2s_m": 0.9239302700556777,
|
| 37 |
+
"fde_2s_sample_count": 11035,
|
| 38 |
+
"fde_3s_m": 2.0195565863793923,
|
| 39 |
+
"fde_3s_sample_count": 11035,
|
| 40 |
+
"fde_5s_m": 5.564463390564478,
|
| 41 |
+
"fde_5s_sample_count": 11035,
|
| 42 |
+
"fde_6p4s_m": null,
|
| 43 |
+
"fde_6p4s_sample_count": 0,
|
| 44 |
+
"mean_abs_lateral_error_m": 0.9844258828682012,
|
| 45 |
+
"mean_abs_longitudinal_error_m": 1.402497202309407,
|
| 46 |
+
"nonfinite_prediction_count": 0,
|
| 47 |
+
"nonfinite_prediction_rate": 0.0,
|
| 48 |
+
"total_timestep_count": 706240,
|
| 49 |
+
"valid_horizon_coverage": 0.78125,
|
| 50 |
+
"valid_timestep_count": 551750
|
| 51 |
+
}
|
| 52 |
+
},
|
| 53 |
+
"model_artifact_id": "120a21639d9767d512eea645be0a83b01476c9e0366a8b957ac51b7ead1bd762",
|
| 54 |
+
"model_name": "KITScenes Trajectory Epoch 5",
|
| 55 |
+
"overlay_object_count": 140,
|
| 56 |
+
"overlay_sample_count": 13525,
|
| 57 |
+
"overlay_schema": "v5",
|
| 58 |
+
"overlay_set_sha256": "d38f6b4b23cbeb94ac00f4ed2dfbf10bf80a9b19d8f8ee1aeeb849731fc407e7",
|
| 59 |
+
"schema_version": "reactive_kitscenes_overlay_replay_evaluation_v1",
|
| 60 |
+
"source_split": "val",
|
| 61 |
+
"target_identity": {
|
| 62 |
+
"sample_count": 11035,
|
| 63 |
+
"sample_tar_bytes_streamed": 7770572800,
|
| 64 |
+
"sample_uid_sha256": "00f824202c97b47ec74561d247e8e30686e5505eef2ad2bc0acba97e41ce9925",
|
| 65 |
+
"source_manifest_set_sha256": "5d4c0ffc7b0706cd917f1b1546cc557b63372839bd67677f8baeef59ad0981e2",
|
| 66 |
+
"source_partition_count": 117,
|
| 67 |
+
"source_partition_list_sha256": "a978e6a373677a15998f41032a850de26cc321271918094237dd71c6e0a51870"
|
| 68 |
+
},
|
| 69 |
+
"target_sample_count": 11035,
|
| 70 |
+
"trajectory_inference_policy": {
|
| 71 |
+
"control_rollout": "semi_implicit_unicycle_v1",
|
| 72 |
+
"dt_seconds": 0.1,
|
| 73 |
+
"planner": "gru",
|
| 74 |
+
"stochastic_noise": false
|
| 75 |
+
}
|
| 76 |
+
}
|
evaluations/overlay-replay-v1/kitscenes-val/kitscenes-epoch-7/report.json
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"created_at": "2026-09-14T09:34:08Z",
|
| 3 |
+
"dataset": "kitscenes-val",
|
| 4 |
+
"dataset_manifest_sha256": "4a12ede73949cf66fd01fdaee89766f1e534de705f783a2f65b372c142fa1d05",
|
| 5 |
+
"dataset_version": "v3.5",
|
| 6 |
+
"evaluation_role": "official_val_camera_map_route",
|
| 7 |
+
"input_track": "camera_map_route",
|
| 8 |
+
"limitations": [
|
| 9 |
+
"Trajectory metrics replay the published deterministic control overlay and do not rerun the model.",
|
| 10 |
+
"Route reconstruction, route counterfactual, and route input gradient metrics are not stored in trajectory overlay v5."
|
| 11 |
+
],
|
| 12 |
+
"metrics": {
|
| 13 |
+
"bev_segmentation": {
|
| 14 |
+
"available": false,
|
| 15 |
+
"reason": "not_replayed_from_trajectory_overlay"
|
| 16 |
+
},
|
| 17 |
+
"route": {
|
| 18 |
+
"available": false,
|
| 19 |
+
"reason": "route_counterfactuals_not_in_overlay_artifact"
|
| 20 |
+
},
|
| 21 |
+
"sample_count": 11035,
|
| 22 |
+
"schema_version": "reactive_multitask_evaluation_v1",
|
| 23 |
+
"trajectory": {
|
| 24 |
+
"ade_1s_m": 0.13467869638959368,
|
| 25 |
+
"ade_1s_sample_count": 11035,
|
| 26 |
+
"ade_2s_m": 0.3886152812418257,
|
| 27 |
+
"ade_2s_sample_count": 11035,
|
| 28 |
+
"ade_3s_m": 0.8034680158738138,
|
| 29 |
+
"ade_3s_sample_count": 11035,
|
| 30 |
+
"ade_5s_m": 2.0951617386340757,
|
| 31 |
+
"ade_5s_sample_count": 11035,
|
| 32 |
+
"ade_6p4s_m": null,
|
| 33 |
+
"ade_6p4s_sample_count": 0,
|
| 34 |
+
"fde_1s_m": 0.2830596284286022,
|
| 35 |
+
"fde_1s_sample_count": 11035,
|
| 36 |
+
"fde_2s_m": 1.0101467772020263,
|
| 37 |
+
"fde_2s_sample_count": 11035,
|
| 38 |
+
"fde_3s_m": 2.212711069790632,
|
| 39 |
+
"fde_3s_sample_count": 11035,
|
| 40 |
+
"fde_5s_m": 5.9508777011937015,
|
| 41 |
+
"fde_5s_sample_count": 11035,
|
| 42 |
+
"fde_6p4s_m": null,
|
| 43 |
+
"fde_6p4s_sample_count": 0,
|
| 44 |
+
"mean_abs_lateral_error_m": 1.0219872821653013,
|
| 45 |
+
"mean_abs_longitudinal_error_m": 1.565857085189425,
|
| 46 |
+
"nonfinite_prediction_count": 0,
|
| 47 |
+
"nonfinite_prediction_rate": 0.0,
|
| 48 |
+
"total_timestep_count": 706240,
|
| 49 |
+
"valid_horizon_coverage": 0.78125,
|
| 50 |
+
"valid_timestep_count": 551750
|
| 51 |
+
}
|
| 52 |
+
},
|
| 53 |
+
"model_artifact_id": "a1e6b1621018740485b3e1e43dda23713d913667eb39a65b77fcbf895dcbcefd",
|
| 54 |
+
"model_name": "KITScenes Trajectory Epoch 7",
|
| 55 |
+
"overlay_object_count": 140,
|
| 56 |
+
"overlay_sample_count": 13525,
|
| 57 |
+
"overlay_schema": "v5",
|
| 58 |
+
"overlay_set_sha256": "8886560702b7ba4cc22c3bb491bd814406029e9bae41d0a262f97ee8314d06b1",
|
| 59 |
+
"schema_version": "reactive_kitscenes_overlay_replay_evaluation_v1",
|
| 60 |
+
"source_split": "val",
|
| 61 |
+
"target_identity": {
|
| 62 |
+
"sample_count": 11035,
|
| 63 |
+
"sample_tar_bytes_streamed": 7770572800,
|
| 64 |
+
"sample_uid_sha256": "00f824202c97b47ec74561d247e8e30686e5505eef2ad2bc0acba97e41ce9925",
|
| 65 |
+
"source_manifest_set_sha256": "5d4c0ffc7b0706cd917f1b1546cc557b63372839bd67677f8baeef59ad0981e2",
|
| 66 |
+
"source_partition_count": 117,
|
| 67 |
+
"source_partition_list_sha256": "a978e6a373677a15998f41032a850de26cc321271918094237dd71c6e0a51870"
|
| 68 |
+
},
|
| 69 |
+
"target_sample_count": 11035,
|
| 70 |
+
"trajectory_inference_policy": {
|
| 71 |
+
"control_rollout": "semi_implicit_unicycle_v1",
|
| 72 |
+
"dt_seconds": 0.1,
|
| 73 |
+
"planner": "gru",
|
| 74 |
+
"stochastic_noise": false
|
| 75 |
+
}
|
| 76 |
+
}
|
evaluations/overlay-replay-v1/kitscenes-val/nuplan-epoch-4/report.json
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"created_at": "2026-09-14T10:02:44Z",
|
| 3 |
+
"dataset": "kitscenes-val",
|
| 4 |
+
"dataset_manifest_sha256": "4a12ede73949cf66fd01fdaee89766f1e534de705f783a2f65b372c142fa1d05",
|
| 5 |
+
"dataset_version": "v3.5",
|
| 6 |
+
"evaluation_role": "official_val_camera_map_route",
|
| 7 |
+
"input_track": "camera_map_route",
|
| 8 |
+
"limitations": [
|
| 9 |
+
"Trajectory metrics replay the published deterministic control overlay and do not rerun the model.",
|
| 10 |
+
"Route reconstruction, route counterfactual, and route input gradient metrics are not stored in trajectory overlay v5."
|
| 11 |
+
],
|
| 12 |
+
"metrics": {
|
| 13 |
+
"bev_segmentation": {
|
| 14 |
+
"available": false,
|
| 15 |
+
"reason": "not_replayed_from_trajectory_overlay"
|
| 16 |
+
},
|
| 17 |
+
"route": {
|
| 18 |
+
"available": false,
|
| 19 |
+
"reason": "route_counterfactuals_not_in_overlay_artifact"
|
| 20 |
+
},
|
| 21 |
+
"sample_count": 11035,
|
| 22 |
+
"schema_version": "reactive_multitask_evaluation_v1",
|
| 23 |
+
"trajectory": {
|
| 24 |
+
"ade_1s_m": 0.20299996935773157,
|
| 25 |
+
"ade_1s_sample_count": 11035,
|
| 26 |
+
"ade_2s_m": 0.5018003104886625,
|
| 27 |
+
"ade_2s_sample_count": 11035,
|
| 28 |
+
"ade_3s_m": 0.950258079868937,
|
| 29 |
+
"ade_3s_sample_count": 11035,
|
| 30 |
+
"ade_5s_m": 2.291857949000488,
|
| 31 |
+
"ade_5s_sample_count": 11035,
|
| 32 |
+
"ade_6p4s_m": null,
|
| 33 |
+
"ade_6p4s_sample_count": 0,
|
| 34 |
+
"fde_1s_m": 0.40659580015215774,
|
| 35 |
+
"fde_1s_sample_count": 11035,
|
| 36 |
+
"fde_2s_m": 1.1934783774143614,
|
| 37 |
+
"fde_2s_sample_count": 11035,
|
| 38 |
+
"fde_3s_m": 2.450842261185858,
|
| 39 |
+
"fde_3s_sample_count": 11035,
|
| 40 |
+
"fde_5s_m": 6.22721162418439,
|
| 41 |
+
"fde_5s_sample_count": 11035,
|
| 42 |
+
"fde_6p4s_m": null,
|
| 43 |
+
"fde_6p4s_sample_count": 0,
|
| 44 |
+
"mean_abs_lateral_error_m": 1.2082210980047907,
|
| 45 |
+
"mean_abs_longitudinal_error_m": 1.6447808719447803,
|
| 46 |
+
"nonfinite_prediction_count": 0,
|
| 47 |
+
"nonfinite_prediction_rate": 0.0,
|
| 48 |
+
"total_timestep_count": 706240,
|
| 49 |
+
"valid_horizon_coverage": 0.78125,
|
| 50 |
+
"valid_timestep_count": 551750
|
| 51 |
+
}
|
| 52 |
+
},
|
| 53 |
+
"model_artifact_id": "ed00e072471aae51b4cf48fa54c3fd9c03c61e0fd6e2978829cd9068dfd6da68",
|
| 54 |
+
"model_name": "NuPlan Trajectory Epoch 4",
|
| 55 |
+
"overlay_object_count": 140,
|
| 56 |
+
"overlay_sample_count": 13525,
|
| 57 |
+
"overlay_schema": "v5",
|
| 58 |
+
"overlay_set_sha256": "4a3f10790e6ad66a7c2aef5bdb121ca86f361f3e1bb86b5a5bd0abb056942c3f",
|
| 59 |
+
"schema_version": "reactive_kitscenes_overlay_replay_evaluation_v1",
|
| 60 |
+
"source_split": "val",
|
| 61 |
+
"target_identity": {
|
| 62 |
+
"sample_count": 11035,
|
| 63 |
+
"sample_tar_bytes_streamed": 7770572800,
|
| 64 |
+
"sample_uid_sha256": "00f824202c97b47ec74561d247e8e30686e5505eef2ad2bc0acba97e41ce9925",
|
| 65 |
+
"source_manifest_set_sha256": "5d4c0ffc7b0706cd917f1b1546cc557b63372839bd67677f8baeef59ad0981e2",
|
| 66 |
+
"source_partition_count": 117,
|
| 67 |
+
"source_partition_list_sha256": "a978e6a373677a15998f41032a850de26cc321271918094237dd71c6e0a51870"
|
| 68 |
+
},
|
| 69 |
+
"target_sample_count": 11035,
|
| 70 |
+
"trajectory_inference_policy": {
|
| 71 |
+
"control_rollout": "semi_implicit_unicycle_v1",
|
| 72 |
+
"dt_seconds": 0.1,
|
| 73 |
+
"planner": "gru",
|
| 74 |
+
"stochastic_noise": false
|
| 75 |
+
}
|
| 76 |
+
}
|
evaluations/overlay-replay-v1/kitscenes-val/nuplan-epoch-5/report.json
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"created_at": "2026-09-14T10:00:27Z",
|
| 3 |
+
"dataset": "kitscenes-val",
|
| 4 |
+
"dataset_manifest_sha256": "4a12ede73949cf66fd01fdaee89766f1e534de705f783a2f65b372c142fa1d05",
|
| 5 |
+
"dataset_version": "v3.5",
|
| 6 |
+
"evaluation_role": "official_val_camera_map_route",
|
| 7 |
+
"input_track": "camera_map_route",
|
| 8 |
+
"limitations": [
|
| 9 |
+
"Trajectory metrics replay the published deterministic control overlay and do not rerun the model.",
|
| 10 |
+
"Route reconstruction, route counterfactual, and route input gradient metrics are not stored in trajectory overlay v5."
|
| 11 |
+
],
|
| 12 |
+
"metrics": {
|
| 13 |
+
"bev_segmentation": {
|
| 14 |
+
"available": false,
|
| 15 |
+
"reason": "not_replayed_from_trajectory_overlay"
|
| 16 |
+
},
|
| 17 |
+
"route": {
|
| 18 |
+
"available": false,
|
| 19 |
+
"reason": "route_counterfactuals_not_in_overlay_artifact"
|
| 20 |
+
},
|
| 21 |
+
"sample_count": 11035,
|
| 22 |
+
"schema_version": "reactive_multitask_evaluation_v1",
|
| 23 |
+
"trajectory": {
|
| 24 |
+
"ade_1s_m": 0.19511089635939421,
|
| 25 |
+
"ade_1s_sample_count": 11035,
|
| 26 |
+
"ade_2s_m": 0.4722640826416731,
|
| 27 |
+
"ade_2s_sample_count": 11035,
|
| 28 |
+
"ade_3s_m": 0.9147207671270287,
|
| 29 |
+
"ade_3s_sample_count": 11035,
|
| 30 |
+
"ade_5s_m": 2.295116252818514,
|
| 31 |
+
"ade_5s_sample_count": 11035,
|
| 32 |
+
"ade_6p4s_m": null,
|
| 33 |
+
"ade_6p4s_sample_count": 0,
|
| 34 |
+
"fde_1s_m": 0.37794148166995123,
|
| 35 |
+
"fde_1s_sample_count": 11035,
|
| 36 |
+
"fde_2s_m": 1.1331487579302755,
|
| 37 |
+
"fde_2s_sample_count": 11035,
|
| 38 |
+
"fde_3s_m": 2.422637476977514,
|
| 39 |
+
"fde_3s_sample_count": 11035,
|
| 40 |
+
"fde_5s_m": 6.388465638260406,
|
| 41 |
+
"fde_5s_sample_count": 11035,
|
| 42 |
+
"fde_6p4s_m": null,
|
| 43 |
+
"fde_6p4s_sample_count": 0,
|
| 44 |
+
"mean_abs_lateral_error_m": 1.19114795667851,
|
| 45 |
+
"mean_abs_longitudinal_error_m": 1.6633371994012054,
|
| 46 |
+
"nonfinite_prediction_count": 0,
|
| 47 |
+
"nonfinite_prediction_rate": 0.0,
|
| 48 |
+
"total_timestep_count": 706240,
|
| 49 |
+
"valid_horizon_coverage": 0.78125,
|
| 50 |
+
"valid_timestep_count": 551750
|
| 51 |
+
}
|
| 52 |
+
},
|
| 53 |
+
"model_artifact_id": "ca8b43d7a777d6fd9195bb253d1452f3cd645f7aab3bd0e36b1f74ffb31df29b",
|
| 54 |
+
"model_name": "NuPlan Trajectory Epoch 5",
|
| 55 |
+
"overlay_object_count": 140,
|
| 56 |
+
"overlay_sample_count": 13525,
|
| 57 |
+
"overlay_schema": "v5",
|
| 58 |
+
"overlay_set_sha256": "036487860685027ef9e279b70417b47fa4165537daa50c6fde518e7ab6508f04",
|
| 59 |
+
"schema_version": "reactive_kitscenes_overlay_replay_evaluation_v1",
|
| 60 |
+
"source_split": "val",
|
| 61 |
+
"target_identity": {
|
| 62 |
+
"sample_count": 11035,
|
| 63 |
+
"sample_tar_bytes_streamed": 7770572800,
|
| 64 |
+
"sample_uid_sha256": "00f824202c97b47ec74561d247e8e30686e5505eef2ad2bc0acba97e41ce9925",
|
| 65 |
+
"source_manifest_set_sha256": "5d4c0ffc7b0706cd917f1b1546cc557b63372839bd67677f8baeef59ad0981e2",
|
| 66 |
+
"source_partition_count": 117,
|
| 67 |
+
"source_partition_list_sha256": "a978e6a373677a15998f41032a850de26cc321271918094237dd71c6e0a51870"
|
| 68 |
+
},
|
| 69 |
+
"target_sample_count": 11035,
|
| 70 |
+
"trajectory_inference_policy": {
|
| 71 |
+
"control_rollout": "semi_implicit_unicycle_v1",
|
| 72 |
+
"dt_seconds": 0.1,
|
| 73 |
+
"planner": "gru",
|
| 74 |
+
"stochastic_noise": false
|
| 75 |
+
}
|
| 76 |
+
}
|
models/kitscenes-epoch-5/checkpoint.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:120a21639d9767d512eea645be0a83b01476c9e0366a8b957ac51b7ead1bd762
|
| 3 |
+
size 334754024
|
models/kitscenes-epoch-5/metadata.json
ADDED
|
@@ -0,0 +1,153 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": {
|
| 3 |
+
"enable_bev_segmentation": true,
|
| 4 |
+
"enable_reasoning": false,
|
| 5 |
+
"enable_route_conditioning": true,
|
| 6 |
+
"enable_route_reconstruction": true,
|
| 7 |
+
"enable_world_model": false,
|
| 8 |
+
"freeze_bevformer": true,
|
| 9 |
+
"identifier": "bevformer_v2_t8_split_navigation_v5",
|
| 10 |
+
"num_views": 6
|
| 11 |
+
},
|
| 12 |
+
"checkpoint": {
|
| 13 |
+
"config_sha256": "03105a365c230c2d7c8e74f51b46afef3f626c79c417881661dd8c2267b9f2ee",
|
| 14 |
+
"contains": [
|
| 15 |
+
"config",
|
| 16 |
+
"config_sha256",
|
| 17 |
+
"epoch",
|
| 18 |
+
"metrics",
|
| 19 |
+
"metrics_sha256",
|
| 20 |
+
"model_state_dict",
|
| 21 |
+
"model_state_sha256",
|
| 22 |
+
"optimizer_state_dict",
|
| 23 |
+
"scheduler_state_dict",
|
| 24 |
+
"training_state"
|
| 25 |
+
],
|
| 26 |
+
"epoch": 5,
|
| 27 |
+
"format": "PyTorch training checkpoint",
|
| 28 |
+
"metrics_sha256": "5df5b910163252a3eed6712ed3ca62b5de74006b52a7402a88186104c98a349e",
|
| 29 |
+
"model_state_sha256": "3ff76c0b70eb3220e0de24439f24589565504a728da0d95555f71548753d2e47",
|
| 30 |
+
"path": "models/kitscenes-epoch-5/checkpoint.pt",
|
| 31 |
+
"sha256": "120a21639d9767d512eea645be0a83b01476c9e0366a8b957ac51b7ead1bd762",
|
| 32 |
+
"size_bytes": 334754024
|
| 33 |
+
},
|
| 34 |
+
"display_name": "KITScenes Epoch 5",
|
| 35 |
+
"evaluations": {
|
| 36 |
+
"external_overlay_replay_v1": {
|
| 37 |
+
"test": {
|
| 38 |
+
"dataset": "kitscenes-test",
|
| 39 |
+
"dataset_version": "v1.0",
|
| 40 |
+
"input_track": "camera_only_missing_map_route",
|
| 41 |
+
"model_reexecuted": false,
|
| 42 |
+
"report_path": "evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-5/report.json",
|
| 43 |
+
"report_sha256": "582dc108ce030756d5d1758dbe198d62b3355bf32a123af54be4e66704c50b8a",
|
| 44 |
+
"sample_count": 23690,
|
| 45 |
+
"trajectory": {
|
| 46 |
+
"ade_1s_m": 0.14632053498252978,
|
| 47 |
+
"ade_1s_sample_count": 23690,
|
| 48 |
+
"ade_2s_m": 0.4121187056069301,
|
| 49 |
+
"ade_2s_sample_count": 23690,
|
| 50 |
+
"ade_3s_m": 0.8263108995057848,
|
| 51 |
+
"ade_3s_sample_count": 23690,
|
| 52 |
+
"ade_5s_m": 2.104176420295984,
|
| 53 |
+
"ade_5s_sample_count": 23690,
|
| 54 |
+
"ade_6p4s_m": null,
|
| 55 |
+
"ade_6p4s_sample_count": 0,
|
| 56 |
+
"fde_1s_m": 0.3106528805343601,
|
| 57 |
+
"fde_1s_sample_count": 23690,
|
| 58 |
+
"fde_2s_m": 1.0453379525636344,
|
| 59 |
+
"fde_2s_sample_count": 23690,
|
| 60 |
+
"fde_3s_m": 2.219851167527027,
|
| 61 |
+
"fde_3s_sample_count": 23690,
|
| 62 |
+
"fde_5s_m": 5.942222944848886,
|
| 63 |
+
"fde_5s_sample_count": 23690,
|
| 64 |
+
"fde_6p4s_m": null,
|
| 65 |
+
"fde_6p4s_sample_count": 0,
|
| 66 |
+
"mean_abs_lateral_error_m": 1.2043875392879961,
|
| 67 |
+
"mean_abs_longitudinal_error_m": 1.4050545911622678,
|
| 68 |
+
"nonfinite_prediction_count": 0,
|
| 69 |
+
"nonfinite_prediction_rate": 0.0,
|
| 70 |
+
"total_timestep_count": 1516160,
|
| 71 |
+
"valid_horizon_coverage": 0.78125,
|
| 72 |
+
"valid_timestep_count": 1184500
|
| 73 |
+
}
|
| 74 |
+
},
|
| 75 |
+
"val": {
|
| 76 |
+
"dataset": "kitscenes-val",
|
| 77 |
+
"dataset_version": "v3.5",
|
| 78 |
+
"input_track": "camera_map_route",
|
| 79 |
+
"model_reexecuted": false,
|
| 80 |
+
"report_path": "evaluations/overlay-replay-v1/kitscenes-val/kitscenes-epoch-5/report.json",
|
| 81 |
+
"report_sha256": "a8f8777da510b832f49503d959f057df6ab298057542e3ec10fd26bd9cb0bf91",
|
| 82 |
+
"sample_count": 11035,
|
| 83 |
+
"trajectory": {
|
| 84 |
+
"ade_1s_m": 0.14720280881581088,
|
| 85 |
+
"ade_1s_sample_count": 11035,
|
| 86 |
+
"ade_2s_m": 0.3729208092788606,
|
| 87 |
+
"ade_2s_sample_count": 11035,
|
| 88 |
+
"ade_3s_m": 0.744916487594762,
|
| 89 |
+
"ade_3s_sample_count": 11035,
|
| 90 |
+
"ade_5s_m": 1.9405047441156529,
|
| 91 |
+
"ade_5s_sample_count": 11035,
|
| 92 |
+
"ade_6p4s_m": null,
|
| 93 |
+
"ade_6p4s_sample_count": 0,
|
| 94 |
+
"fde_1s_m": 0.28501037315538935,
|
| 95 |
+
"fde_1s_sample_count": 11035,
|
| 96 |
+
"fde_2s_m": 0.9239302700556777,
|
| 97 |
+
"fde_2s_sample_count": 11035,
|
| 98 |
+
"fde_3s_m": 2.0195565863793923,
|
| 99 |
+
"fde_3s_sample_count": 11035,
|
| 100 |
+
"fde_5s_m": 5.564463390564478,
|
| 101 |
+
"fde_5s_sample_count": 11035,
|
| 102 |
+
"fde_6p4s_m": null,
|
| 103 |
+
"fde_6p4s_sample_count": 0,
|
| 104 |
+
"mean_abs_lateral_error_m": 0.9844258828682012,
|
| 105 |
+
"mean_abs_longitudinal_error_m": 1.402497202309407,
|
| 106 |
+
"nonfinite_prediction_count": 0,
|
| 107 |
+
"nonfinite_prediction_rate": 0.0,
|
| 108 |
+
"total_timestep_count": 706240,
|
| 109 |
+
"valid_horizon_coverage": 0.78125,
|
| 110 |
+
"valid_timestep_count": 551750
|
| 111 |
+
}
|
| 112 |
+
}
|
| 113 |
+
},
|
| 114 |
+
"internal_validation": {
|
| 115 |
+
"ade_6p4s_m": 2.632596758054813,
|
| 116 |
+
"comfortable_rate": 0.7541884816753927,
|
| 117 |
+
"drivable_area_compliance_rate": 0.7758017015706806,
|
| 118 |
+
"drivable_area_success_rate": 0.4442408376963351,
|
| 119 |
+
"fde_6p4s_m": 7.556681459112317,
|
| 120 |
+
"route_corridor_compliance_rate": 0.39594963071109585,
|
| 121 |
+
"route_corridor_iou": 0.9991168210595887,
|
| 122 |
+
"route_corridor_success_rate": 0.06080384747509447
|
| 123 |
+
}
|
| 124 |
+
},
|
| 125 |
+
"license_notice": {
|
| 126 |
+
"checkpoint": "Other; see the repository model card.",
|
| 127 |
+
"pretrained_component": "Official BEVFormer V2 R50 T8 initialization; weight license recorded as NOASSERTION.",
|
| 128 |
+
"source_code": "Apache-2.0",
|
| 129 |
+
"training_data_notice": "The pretrained component was trained on nuScenes, whose terms may impose additional restrictions."
|
| 130 |
+
},
|
| 131 |
+
"limitations": [
|
| 132 |
+
"KITScenes Val uses an oracle post-hoc route reconstructed from the logged future trajectory.",
|
| 133 |
+
"KITScenes Val and Test are different scene populations and are not a causal map or route ablation.",
|
| 134 |
+
"External overlay replay evaluates published controls and does not re-run the model.",
|
| 135 |
+
"Confidence intervals and per-sample route counterfactuals are not included.",
|
| 136 |
+
"These open-loop checkpoints are not validated for safety-critical deployment."
|
| 137 |
+
],
|
| 138 |
+
"lineage": {
|
| 139 |
+
"parent": "nuPlan Epoch 5 checkpoint ca8b43d7a777",
|
| 140 |
+
"training_stage": "KITScenes fine-tuning with frozen camera BEV"
|
| 141 |
+
},
|
| 142 |
+
"mlflow_registry": {
|
| 143 |
+
"external_overlay_test_run_id": "6a6fe8ae954d495f9f0b4aab9f740e80",
|
| 144 |
+
"external_overlay_val_run_id": "dacc23b278a74932999333718e772fa5",
|
| 145 |
+
"internal_validation_run_id": "8e0c2d24d7094d6192eae58078728641",
|
| 146 |
+
"model_version": "66",
|
| 147 |
+
"registered_model_name": "auto-e2e-driving-policy"
|
| 148 |
+
},
|
| 149 |
+
"model_id": "kitscenes-epoch-5",
|
| 150 |
+
"published_at": "2026-09-16T02:26:06+00:00",
|
| 151 |
+
"repository": "AutowareFoundation/auto_e2e",
|
| 152 |
+
"schema_version": "auto_e2e_checkpoint_metadata_v1"
|
| 153 |
+
}
|
models/kitscenes-epoch-7/checkpoint.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a1e6b1621018740485b3e1e43dda23713d913667eb39a65b77fcbf895dcbcefd
|
| 3 |
+
size 334754024
|
models/kitscenes-epoch-7/metadata.json
ADDED
|
@@ -0,0 +1,153 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": {
|
| 3 |
+
"enable_bev_segmentation": true,
|
| 4 |
+
"enable_reasoning": false,
|
| 5 |
+
"enable_route_conditioning": true,
|
| 6 |
+
"enable_route_reconstruction": true,
|
| 7 |
+
"enable_world_model": false,
|
| 8 |
+
"freeze_bevformer": true,
|
| 9 |
+
"identifier": "bevformer_v2_t8_split_navigation_v5",
|
| 10 |
+
"num_views": 6
|
| 11 |
+
},
|
| 12 |
+
"checkpoint": {
|
| 13 |
+
"config_sha256": "03105a365c230c2d7c8e74f51b46afef3f626c79c417881661dd8c2267b9f2ee",
|
| 14 |
+
"contains": [
|
| 15 |
+
"config",
|
| 16 |
+
"config_sha256",
|
| 17 |
+
"epoch",
|
| 18 |
+
"metrics",
|
| 19 |
+
"metrics_sha256",
|
| 20 |
+
"model_state_dict",
|
| 21 |
+
"model_state_sha256",
|
| 22 |
+
"optimizer_state_dict",
|
| 23 |
+
"scheduler_state_dict",
|
| 24 |
+
"training_state"
|
| 25 |
+
],
|
| 26 |
+
"epoch": 7,
|
| 27 |
+
"format": "PyTorch training checkpoint",
|
| 28 |
+
"metrics_sha256": "40842a9483e223ab86d81b830b1c7fb136ce4ba274420860befd64d5d33b1fe1",
|
| 29 |
+
"model_state_sha256": "422601516e530c2c5793cd00f997abd4b708074117bdcdef0053a3620c651c81",
|
| 30 |
+
"path": "models/kitscenes-epoch-7/checkpoint.pt",
|
| 31 |
+
"sha256": "a1e6b1621018740485b3e1e43dda23713d913667eb39a65b77fcbf895dcbcefd",
|
| 32 |
+
"size_bytes": 334754024
|
| 33 |
+
},
|
| 34 |
+
"display_name": "KITScenes Epoch 7",
|
| 35 |
+
"evaluations": {
|
| 36 |
+
"external_overlay_replay_v1": {
|
| 37 |
+
"test": {
|
| 38 |
+
"dataset": "kitscenes-test",
|
| 39 |
+
"dataset_version": "v1.0",
|
| 40 |
+
"input_track": "camera_only_missing_map_route",
|
| 41 |
+
"model_reexecuted": false,
|
| 42 |
+
"report_path": "evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-7/report.json",
|
| 43 |
+
"report_sha256": "d01169d68266d4b7203e92a4d25ade2a3277357ff89cc26cdb202051a26016fc",
|
| 44 |
+
"sample_count": 23690,
|
| 45 |
+
"trajectory": {
|
| 46 |
+
"ade_1s_m": 0.1680068780791157,
|
| 47 |
+
"ade_1s_sample_count": 23690,
|
| 48 |
+
"ade_2s_m": 0.4923481277999561,
|
| 49 |
+
"ade_2s_sample_count": 23690,
|
| 50 |
+
"ade_3s_m": 0.9830703987751742,
|
| 51 |
+
"ade_3s_sample_count": 23690,
|
| 52 |
+
"ade_5s_m": 2.469892014634966,
|
| 53 |
+
"ade_5s_sample_count": 23690,
|
| 54 |
+
"ade_6p4s_m": null,
|
| 55 |
+
"ade_6p4s_sample_count": 0,
|
| 56 |
+
"fde_1s_m": 0.37537789674250205,
|
| 57 |
+
"fde_1s_sample_count": 23690,
|
| 58 |
+
"fde_2s_m": 1.2511962416817588,
|
| 59 |
+
"fde_2s_sample_count": 23690,
|
| 60 |
+
"fde_3s_m": 2.624177628524085,
|
| 61 |
+
"fde_3s_sample_count": 23690,
|
| 62 |
+
"fde_5s_m": 6.896411433552511,
|
| 63 |
+
"fde_5s_sample_count": 23690,
|
| 64 |
+
"fde_6p4s_m": null,
|
| 65 |
+
"fde_6p4s_sample_count": 0,
|
| 66 |
+
"mean_abs_lateral_error_m": 1.4730429718483722,
|
| 67 |
+
"mean_abs_longitudinal_error_m": 1.6598486258851333,
|
| 68 |
+
"nonfinite_prediction_count": 0,
|
| 69 |
+
"nonfinite_prediction_rate": 0.0,
|
| 70 |
+
"total_timestep_count": 1516160,
|
| 71 |
+
"valid_horizon_coverage": 0.78125,
|
| 72 |
+
"valid_timestep_count": 1184500
|
| 73 |
+
}
|
| 74 |
+
},
|
| 75 |
+
"val": {
|
| 76 |
+
"dataset": "kitscenes-val",
|
| 77 |
+
"dataset_version": "v3.5",
|
| 78 |
+
"input_track": "camera_map_route",
|
| 79 |
+
"model_reexecuted": false,
|
| 80 |
+
"report_path": "evaluations/overlay-replay-v1/kitscenes-val/kitscenes-epoch-7/report.json",
|
| 81 |
+
"report_sha256": "832b9ec0648b997f5b1f825296269905e5ee3b13ae75918e0fdee5733dbd1818",
|
| 82 |
+
"sample_count": 11035,
|
| 83 |
+
"trajectory": {
|
| 84 |
+
"ade_1s_m": 0.13467869638959368,
|
| 85 |
+
"ade_1s_sample_count": 11035,
|
| 86 |
+
"ade_2s_m": 0.3886152812418257,
|
| 87 |
+
"ade_2s_sample_count": 11035,
|
| 88 |
+
"ade_3s_m": 0.8034680158738138,
|
| 89 |
+
"ade_3s_sample_count": 11035,
|
| 90 |
+
"ade_5s_m": 2.0951617386340757,
|
| 91 |
+
"ade_5s_sample_count": 11035,
|
| 92 |
+
"ade_6p4s_m": null,
|
| 93 |
+
"ade_6p4s_sample_count": 0,
|
| 94 |
+
"fde_1s_m": 0.2830596284286022,
|
| 95 |
+
"fde_1s_sample_count": 11035,
|
| 96 |
+
"fde_2s_m": 1.0101467772020263,
|
| 97 |
+
"fde_2s_sample_count": 11035,
|
| 98 |
+
"fde_3s_m": 2.212711069790632,
|
| 99 |
+
"fde_3s_sample_count": 11035,
|
| 100 |
+
"fde_5s_m": 5.9508777011937015,
|
| 101 |
+
"fde_5s_sample_count": 11035,
|
| 102 |
+
"fde_6p4s_m": null,
|
| 103 |
+
"fde_6p4s_sample_count": 0,
|
| 104 |
+
"mean_abs_lateral_error_m": 1.0219872821653013,
|
| 105 |
+
"mean_abs_longitudinal_error_m": 1.565857085189425,
|
| 106 |
+
"nonfinite_prediction_count": 0,
|
| 107 |
+
"nonfinite_prediction_rate": 0.0,
|
| 108 |
+
"total_timestep_count": 706240,
|
| 109 |
+
"valid_horizon_coverage": 0.78125,
|
| 110 |
+
"valid_timestep_count": 551750
|
| 111 |
+
}
|
| 112 |
+
}
|
| 113 |
+
},
|
| 114 |
+
"internal_validation": {
|
| 115 |
+
"ade_6p4s_m": 2.754791687592786,
|
| 116 |
+
"comfortable_rate": 0.7465968586387435,
|
| 117 |
+
"drivable_area_compliance_rate": 0.7988219895287958,
|
| 118 |
+
"drivable_area_success_rate": 0.41387434554973823,
|
| 119 |
+
"fde_6p4s_m": 7.857446600512373,
|
| 120 |
+
"route_corridor_compliance_rate": 0.42360335795259363,
|
| 121 |
+
"route_corridor_iou": 0.9990024728541969,
|
| 122 |
+
"route_corridor_success_rate": 0.07626245276537272
|
| 123 |
+
}
|
| 124 |
+
},
|
| 125 |
+
"license_notice": {
|
| 126 |
+
"checkpoint": "Other; see the repository model card.",
|
| 127 |
+
"pretrained_component": "Official BEVFormer V2 R50 T8 initialization; weight license recorded as NOASSERTION.",
|
| 128 |
+
"source_code": "Apache-2.0",
|
| 129 |
+
"training_data_notice": "The pretrained component was trained on nuScenes, whose terms may impose additional restrictions."
|
| 130 |
+
},
|
| 131 |
+
"limitations": [
|
| 132 |
+
"KITScenes Val uses an oracle post-hoc route reconstructed from the logged future trajectory.",
|
| 133 |
+
"KITScenes Val and Test are different scene populations and are not a causal map or route ablation.",
|
| 134 |
+
"External overlay replay evaluates published controls and does not re-run the model.",
|
| 135 |
+
"Confidence intervals and per-sample route counterfactuals are not included.",
|
| 136 |
+
"These open-loop checkpoints are not validated for safety-critical deployment."
|
| 137 |
+
],
|
| 138 |
+
"lineage": {
|
| 139 |
+
"parent": "nuPlan Epoch 5 checkpoint ca8b43d7a777",
|
| 140 |
+
"training_stage": "KITScenes fine-tuning with frozen camera BEV"
|
| 141 |
+
},
|
| 142 |
+
"mlflow_registry": {
|
| 143 |
+
"external_overlay_test_run_id": "baa85b55d3cb46e9ba727df6979ca999",
|
| 144 |
+
"external_overlay_val_run_id": "730de8db5fb040f2bfb4c78c921913f1",
|
| 145 |
+
"internal_validation_run_id": "a313c95024b04cafa748832e5e532c5c",
|
| 146 |
+
"model_version": "67",
|
| 147 |
+
"registered_model_name": "auto-e2e-driving-policy"
|
| 148 |
+
},
|
| 149 |
+
"model_id": "kitscenes-epoch-7",
|
| 150 |
+
"published_at": "2026-09-16T02:26:06+00:00",
|
| 151 |
+
"repository": "AutowareFoundation/auto_e2e",
|
| 152 |
+
"schema_version": "auto_e2e_checkpoint_metadata_v1"
|
| 153 |
+
}
|
models/nuplan-epoch-4/checkpoint.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ed00e072471aae51b4cf48fa54c3fd9c03c61e0fd6e2978829cd9068dfd6da68
|
| 3 |
+
size 334907944
|
models/nuplan-epoch-4/metadata.json
ADDED
|
@@ -0,0 +1,151 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": {
|
| 3 |
+
"enable_bev_segmentation": true,
|
| 4 |
+
"enable_reasoning": false,
|
| 5 |
+
"enable_route_conditioning": true,
|
| 6 |
+
"enable_route_reconstruction": true,
|
| 7 |
+
"enable_world_model": false,
|
| 8 |
+
"freeze_bevformer": true,
|
| 9 |
+
"identifier": "bevformer_v2_t8_split_navigation_v5",
|
| 10 |
+
"num_views": 6
|
| 11 |
+
},
|
| 12 |
+
"checkpoint": {
|
| 13 |
+
"config_sha256": "e06fb3943e32bdab7c8b53f597ec216d5636591a73a7206045a465ffa49d1ea0",
|
| 14 |
+
"contains": [
|
| 15 |
+
"config",
|
| 16 |
+
"config_sha256",
|
| 17 |
+
"epoch",
|
| 18 |
+
"metrics",
|
| 19 |
+
"model_state_dict",
|
| 20 |
+
"model_state_sha256",
|
| 21 |
+
"optimizer_state_dict",
|
| 22 |
+
"scheduler_state_dict",
|
| 23 |
+
"training_state"
|
| 24 |
+
],
|
| 25 |
+
"epoch": 4,
|
| 26 |
+
"format": "PyTorch training checkpoint",
|
| 27 |
+
"metrics_sha256": null,
|
| 28 |
+
"model_state_sha256": "1a3f159c864e6098d633f8c68596934b7f4678fcaee5ec3d40ae8eec14e650c4",
|
| 29 |
+
"path": "models/nuplan-epoch-4/checkpoint.pt",
|
| 30 |
+
"sha256": "ed00e072471aae51b4cf48fa54c3fd9c03c61e0fd6e2978829cd9068dfd6da68",
|
| 31 |
+
"size_bytes": 334907944
|
| 32 |
+
},
|
| 33 |
+
"display_name": "nuPlan Epoch 4",
|
| 34 |
+
"evaluations": {
|
| 35 |
+
"external_overlay_replay_v1": {
|
| 36 |
+
"test": {
|
| 37 |
+
"dataset": "kitscenes-test",
|
| 38 |
+
"dataset_version": "v1.0",
|
| 39 |
+
"input_track": "camera_only_missing_map_route",
|
| 40 |
+
"model_reexecuted": false,
|
| 41 |
+
"report_path": "evaluations/overlay-replay-v1/kitscenes-test/nuplan-epoch-4/report.json",
|
| 42 |
+
"report_sha256": "5eeb68a4013a9777ba927030b73877a1789f5b4eb9e14f6da760bef7f9b42808",
|
| 43 |
+
"sample_count": 23690,
|
| 44 |
+
"trajectory": {
|
| 45 |
+
"ade_1s_m": 0.1915564095529401,
|
| 46 |
+
"ade_1s_sample_count": 23690,
|
| 47 |
+
"ade_2s_m": 0.4816506897179045,
|
| 48 |
+
"ade_2s_sample_count": 23690,
|
| 49 |
+
"ade_3s_m": 0.9181210775291644,
|
| 50 |
+
"ade_3s_sample_count": 23690,
|
| 51 |
+
"ade_5s_m": 2.2840408150913345,
|
| 52 |
+
"ade_5s_sample_count": 23690,
|
| 53 |
+
"ade_6p4s_m": null,
|
| 54 |
+
"ade_6p4s_sample_count": 0,
|
| 55 |
+
"fde_1s_m": 0.3898781367591937,
|
| 56 |
+
"fde_1s_sample_count": 23690,
|
| 57 |
+
"fde_2s_m": 1.1514432707206264,
|
| 58 |
+
"fde_2s_sample_count": 23690,
|
| 59 |
+
"fde_3s_m": 2.38952477957317,
|
| 60 |
+
"fde_3s_sample_count": 23690,
|
| 61 |
+
"fde_5s_m": 6.42657723546563,
|
| 62 |
+
"fde_5s_sample_count": 23690,
|
| 63 |
+
"fde_6p4s_m": null,
|
| 64 |
+
"fde_6p4s_sample_count": 0,
|
| 65 |
+
"mean_abs_lateral_error_m": 1.2167123846906904,
|
| 66 |
+
"mean_abs_longitudinal_error_m": 1.6287253737215717,
|
| 67 |
+
"nonfinite_prediction_count": 0,
|
| 68 |
+
"nonfinite_prediction_rate": 0.0,
|
| 69 |
+
"total_timestep_count": 1516160,
|
| 70 |
+
"valid_horizon_coverage": 0.78125,
|
| 71 |
+
"valid_timestep_count": 1184500
|
| 72 |
+
}
|
| 73 |
+
},
|
| 74 |
+
"val": {
|
| 75 |
+
"dataset": "kitscenes-val",
|
| 76 |
+
"dataset_version": "v3.5",
|
| 77 |
+
"input_track": "camera_map_route",
|
| 78 |
+
"model_reexecuted": false,
|
| 79 |
+
"report_path": "evaluations/overlay-replay-v1/kitscenes-val/nuplan-epoch-4/report.json",
|
| 80 |
+
"report_sha256": "1ffde86560b8d9701c23c98b6b05b0de2611446a8d92f0fb90a08ccb884d1acb",
|
| 81 |
+
"sample_count": 11035,
|
| 82 |
+
"trajectory": {
|
| 83 |
+
"ade_1s_m": 0.20299996935773157,
|
| 84 |
+
"ade_1s_sample_count": 11035,
|
| 85 |
+
"ade_2s_m": 0.5018003104886625,
|
| 86 |
+
"ade_2s_sample_count": 11035,
|
| 87 |
+
"ade_3s_m": 0.950258079868937,
|
| 88 |
+
"ade_3s_sample_count": 11035,
|
| 89 |
+
"ade_5s_m": 2.291857949000488,
|
| 90 |
+
"ade_5s_sample_count": 11035,
|
| 91 |
+
"ade_6p4s_m": null,
|
| 92 |
+
"ade_6p4s_sample_count": 0,
|
| 93 |
+
"fde_1s_m": 0.40659580015215774,
|
| 94 |
+
"fde_1s_sample_count": 11035,
|
| 95 |
+
"fde_2s_m": 1.1934783774143614,
|
| 96 |
+
"fde_2s_sample_count": 11035,
|
| 97 |
+
"fde_3s_m": 2.450842261185858,
|
| 98 |
+
"fde_3s_sample_count": 11035,
|
| 99 |
+
"fde_5s_m": 6.22721162418439,
|
| 100 |
+
"fde_5s_sample_count": 11035,
|
| 101 |
+
"fde_6p4s_m": null,
|
| 102 |
+
"fde_6p4s_sample_count": 0,
|
| 103 |
+
"mean_abs_lateral_error_m": 1.2082210980047907,
|
| 104 |
+
"mean_abs_longitudinal_error_m": 1.6447808719447803,
|
| 105 |
+
"nonfinite_prediction_count": 0,
|
| 106 |
+
"nonfinite_prediction_rate": 0.0,
|
| 107 |
+
"total_timestep_count": 706240,
|
| 108 |
+
"valid_horizon_coverage": 0.78125,
|
| 109 |
+
"valid_timestep_count": 551750
|
| 110 |
+
}
|
| 111 |
+
}
|
| 112 |
+
},
|
| 113 |
+
"internal_validation": {
|
| 114 |
+
"ade_6p4s_m": 1.2578079247687128,
|
| 115 |
+
"bev_dynamic_macro_average_precision": 0.006642145206117095,
|
| 116 |
+
"bev_static_macro_average_precision": 0.14298715599094136,
|
| 117 |
+
"bev_vehicle_average_precision": 0.017910564961292577,
|
| 118 |
+
"bev_vulnerable_road_user_average_precision": 0.0011931259995499048,
|
| 119 |
+
"fde_6p4s_m": 3.845009753582417,
|
| 120 |
+
"route_corridor_iou": 0.9911965199680612
|
| 121 |
+
}
|
| 122 |
+
},
|
| 123 |
+
"license_notice": {
|
| 124 |
+
"checkpoint": "Other; see the repository model card.",
|
| 125 |
+
"pretrained_component": "Official BEVFormer V2 R50 T8 initialization; weight license recorded as NOASSERTION.",
|
| 126 |
+
"source_code": "Apache-2.0",
|
| 127 |
+
"training_data_notice": "The pretrained component was trained on nuScenes, whose terms may impose additional restrictions."
|
| 128 |
+
},
|
| 129 |
+
"limitations": [
|
| 130 |
+
"KITScenes Val uses an oracle post-hoc route reconstructed from the logged future trajectory.",
|
| 131 |
+
"KITScenes Val and Test are different scene populations and are not a causal map or route ablation.",
|
| 132 |
+
"External overlay replay evaluates published controls and does not re-run the model.",
|
| 133 |
+
"Confidence intervals and per-sample route counterfactuals are not included.",
|
| 134 |
+
"These open-loop checkpoints are not validated for safety-critical deployment."
|
| 135 |
+
],
|
| 136 |
+
"lineage": {
|
| 137 |
+
"parent": "Official BEVFormer V2 R50 T8 initialization",
|
| 138 |
+
"training_stage": "nuPlan trajectory and route training with frozen camera BEV"
|
| 139 |
+
},
|
| 140 |
+
"mlflow_registry": {
|
| 141 |
+
"external_overlay_test_run_id": "cf984b0f7b8846a3919444ccdf3191cc",
|
| 142 |
+
"external_overlay_val_run_id": "ed3cf683849f4ddbaa55672502885eb1",
|
| 143 |
+
"internal_validation_run_id": "77a3ea44ea7440a682e6b2558c71f32f",
|
| 144 |
+
"model_version": "64",
|
| 145 |
+
"registered_model_name": "auto-e2e-driving-policy"
|
| 146 |
+
},
|
| 147 |
+
"model_id": "nuplan-epoch-4",
|
| 148 |
+
"published_at": "2026-09-16T02:26:06+00:00",
|
| 149 |
+
"repository": "AutowareFoundation/auto_e2e",
|
| 150 |
+
"schema_version": "auto_e2e_checkpoint_metadata_v1"
|
| 151 |
+
}
|
models/nuplan-epoch-5/checkpoint.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ca8b43d7a777d6fd9195bb253d1452f3cd645f7aab3bd0e36b1f74ffb31df29b
|
| 3 |
+
size 334907944
|
models/nuplan-epoch-5/metadata.json
ADDED
|
@@ -0,0 +1,151 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": {
|
| 3 |
+
"enable_bev_segmentation": true,
|
| 4 |
+
"enable_reasoning": false,
|
| 5 |
+
"enable_route_conditioning": true,
|
| 6 |
+
"enable_route_reconstruction": true,
|
| 7 |
+
"enable_world_model": false,
|
| 8 |
+
"freeze_bevformer": true,
|
| 9 |
+
"identifier": "bevformer_v2_t8_split_navigation_v5",
|
| 10 |
+
"num_views": 6
|
| 11 |
+
},
|
| 12 |
+
"checkpoint": {
|
| 13 |
+
"config_sha256": "e06fb3943e32bdab7c8b53f597ec216d5636591a73a7206045a465ffa49d1ea0",
|
| 14 |
+
"contains": [
|
| 15 |
+
"config",
|
| 16 |
+
"config_sha256",
|
| 17 |
+
"epoch",
|
| 18 |
+
"metrics",
|
| 19 |
+
"model_state_dict",
|
| 20 |
+
"model_state_sha256",
|
| 21 |
+
"optimizer_state_dict",
|
| 22 |
+
"scheduler_state_dict",
|
| 23 |
+
"training_state"
|
| 24 |
+
],
|
| 25 |
+
"epoch": 5,
|
| 26 |
+
"format": "PyTorch training checkpoint",
|
| 27 |
+
"metrics_sha256": null,
|
| 28 |
+
"model_state_sha256": "30ad1af5ab1ddfa14699f95751358608f929fb8c006f8f7e1347008179c90d9e",
|
| 29 |
+
"path": "models/nuplan-epoch-5/checkpoint.pt",
|
| 30 |
+
"sha256": "ca8b43d7a777d6fd9195bb253d1452f3cd645f7aab3bd0e36b1f74ffb31df29b",
|
| 31 |
+
"size_bytes": 334907944
|
| 32 |
+
},
|
| 33 |
+
"display_name": "nuPlan Epoch 5",
|
| 34 |
+
"evaluations": {
|
| 35 |
+
"external_overlay_replay_v1": {
|
| 36 |
+
"test": {
|
| 37 |
+
"dataset": "kitscenes-test",
|
| 38 |
+
"dataset_version": "v1.0",
|
| 39 |
+
"input_track": "camera_only_missing_map_route",
|
| 40 |
+
"model_reexecuted": false,
|
| 41 |
+
"report_path": "evaluations/overlay-replay-v1/kitscenes-test/nuplan-epoch-5/report.json",
|
| 42 |
+
"report_sha256": "fc5b19beea416706461b535314a497d6a2b929d97668e8475638ae04a65221b7",
|
| 43 |
+
"sample_count": 23690,
|
| 44 |
+
"trajectory": {
|
| 45 |
+
"ade_1s_m": 0.16017068059452302,
|
| 46 |
+
"ade_1s_sample_count": 23690,
|
| 47 |
+
"ade_2s_m": 0.3979342750750205,
|
| 48 |
+
"ade_2s_sample_count": 23690,
|
| 49 |
+
"ade_3s_m": 0.7923739682724033,
|
| 50 |
+
"ade_3s_sample_count": 23690,
|
| 51 |
+
"ade_5s_m": 2.1119916453019565,
|
| 52 |
+
"ade_5s_sample_count": 23690,
|
| 53 |
+
"ade_6p4s_m": null,
|
| 54 |
+
"ade_6p4s_sample_count": 0,
|
| 55 |
+
"fde_1s_m": 0.3134281782098752,
|
| 56 |
+
"fde_1s_sample_count": 23690,
|
| 57 |
+
"fde_2s_m": 0.9727075050905802,
|
| 58 |
+
"fde_2s_sample_count": 23690,
|
| 59 |
+
"fde_3s_m": 2.1621481640709304,
|
| 60 |
+
"fde_3s_sample_count": 23690,
|
| 61 |
+
"fde_5s_m": 6.175121158406895,
|
| 62 |
+
"fde_5s_sample_count": 23690,
|
| 63 |
+
"fde_6p4s_m": null,
|
| 64 |
+
"fde_6p4s_sample_count": 0,
|
| 65 |
+
"mean_abs_lateral_error_m": 1.0584339709055983,
|
| 66 |
+
"mean_abs_longitudinal_error_m": 1.5495691251398722,
|
| 67 |
+
"nonfinite_prediction_count": 0,
|
| 68 |
+
"nonfinite_prediction_rate": 0.0,
|
| 69 |
+
"total_timestep_count": 1516160,
|
| 70 |
+
"valid_horizon_coverage": 0.78125,
|
| 71 |
+
"valid_timestep_count": 1184500
|
| 72 |
+
}
|
| 73 |
+
},
|
| 74 |
+
"val": {
|
| 75 |
+
"dataset": "kitscenes-val",
|
| 76 |
+
"dataset_version": "v3.5",
|
| 77 |
+
"input_track": "camera_map_route",
|
| 78 |
+
"model_reexecuted": false,
|
| 79 |
+
"report_path": "evaluations/overlay-replay-v1/kitscenes-val/nuplan-epoch-5/report.json",
|
| 80 |
+
"report_sha256": "e45996448f95f193acfc225c37ff169fab64bb24b5ea7717928a20c426e7303b",
|
| 81 |
+
"sample_count": 11035,
|
| 82 |
+
"trajectory": {
|
| 83 |
+
"ade_1s_m": 0.19511089635939421,
|
| 84 |
+
"ade_1s_sample_count": 11035,
|
| 85 |
+
"ade_2s_m": 0.4722640826416731,
|
| 86 |
+
"ade_2s_sample_count": 11035,
|
| 87 |
+
"ade_3s_m": 0.9147207671270287,
|
| 88 |
+
"ade_3s_sample_count": 11035,
|
| 89 |
+
"ade_5s_m": 2.295116252818514,
|
| 90 |
+
"ade_5s_sample_count": 11035,
|
| 91 |
+
"ade_6p4s_m": null,
|
| 92 |
+
"ade_6p4s_sample_count": 0,
|
| 93 |
+
"fde_1s_m": 0.37794148166995123,
|
| 94 |
+
"fde_1s_sample_count": 11035,
|
| 95 |
+
"fde_2s_m": 1.1331487579302755,
|
| 96 |
+
"fde_2s_sample_count": 11035,
|
| 97 |
+
"fde_3s_m": 2.422637476977514,
|
| 98 |
+
"fde_3s_sample_count": 11035,
|
| 99 |
+
"fde_5s_m": 6.388465638260406,
|
| 100 |
+
"fde_5s_sample_count": 11035,
|
| 101 |
+
"fde_6p4s_m": null,
|
| 102 |
+
"fde_6p4s_sample_count": 0,
|
| 103 |
+
"mean_abs_lateral_error_m": 1.19114795667851,
|
| 104 |
+
"mean_abs_longitudinal_error_m": 1.6633371994012054,
|
| 105 |
+
"nonfinite_prediction_count": 0,
|
| 106 |
+
"nonfinite_prediction_rate": 0.0,
|
| 107 |
+
"total_timestep_count": 706240,
|
| 108 |
+
"valid_horizon_coverage": 0.78125,
|
| 109 |
+
"valid_timestep_count": 551750
|
| 110 |
+
}
|
| 111 |
+
}
|
| 112 |
+
},
|
| 113 |
+
"internal_validation": {
|
| 114 |
+
"ade_6p4s_m": 1.2667837238223,
|
| 115 |
+
"bev_dynamic_macro_average_precision": 0.006642125301720789,
|
| 116 |
+
"bev_static_macro_average_precision": 0.14298718688799092,
|
| 117 |
+
"bev_vehicle_average_precision": 0.017910500586346273,
|
| 118 |
+
"bev_vulnerable_road_user_average_precision": 0.0011931349457863422,
|
| 119 |
+
"fde_6p4s_m": 3.8982392455945956,
|
| 120 |
+
"route_corridor_iou": 0.9980338574990057
|
| 121 |
+
}
|
| 122 |
+
},
|
| 123 |
+
"license_notice": {
|
| 124 |
+
"checkpoint": "Other; see the repository model card.",
|
| 125 |
+
"pretrained_component": "Official BEVFormer V2 R50 T8 initialization; weight license recorded as NOASSERTION.",
|
| 126 |
+
"source_code": "Apache-2.0",
|
| 127 |
+
"training_data_notice": "The pretrained component was trained on nuScenes, whose terms may impose additional restrictions."
|
| 128 |
+
},
|
| 129 |
+
"limitations": [
|
| 130 |
+
"KITScenes Val uses an oracle post-hoc route reconstructed from the logged future trajectory.",
|
| 131 |
+
"KITScenes Val and Test are different scene populations and are not a causal map or route ablation.",
|
| 132 |
+
"External overlay replay evaluates published controls and does not re-run the model.",
|
| 133 |
+
"Confidence intervals and per-sample route counterfactuals are not included.",
|
| 134 |
+
"These open-loop checkpoints are not validated for safety-critical deployment."
|
| 135 |
+
],
|
| 136 |
+
"lineage": {
|
| 137 |
+
"parent": "Official BEVFormer V2 R50 T8 initialization",
|
| 138 |
+
"training_stage": "nuPlan trajectory and route training with frozen camera BEV"
|
| 139 |
+
},
|
| 140 |
+
"mlflow_registry": {
|
| 141 |
+
"external_overlay_test_run_id": "db91565f5e4741e48bf242e7b3e86a02",
|
| 142 |
+
"external_overlay_val_run_id": "0f9d11bd016448ac94d5095b7225e270",
|
| 143 |
+
"internal_validation_run_id": "3d419166635a44edacf8503fe94d4805",
|
| 144 |
+
"model_version": "65",
|
| 145 |
+
"registered_model_name": "auto-e2e-driving-policy"
|
| 146 |
+
},
|
| 147 |
+
"model_id": "nuplan-epoch-5",
|
| 148 |
+
"published_at": "2026-09-16T02:26:06+00:00",
|
| 149 |
+
"repository": "AutowareFoundation/auto_e2e",
|
| 150 |
+
"schema_version": "auto_e2e_checkpoint_metadata_v1"
|
| 151 |
+
}
|
release_manifest.json
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"external_overlay_replay_index_sha256": "be1cec5f77a7a12326070503dbcd1004c63f54450cf1151c763a215816d838bd",
|
| 3 |
+
"models": [
|
| 4 |
+
{
|
| 5 |
+
"checkpoint_path": "models/nuplan-epoch-4/checkpoint.pt",
|
| 6 |
+
"checkpoint_sha256": "ed00e072471aae51b4cf48fa54c3fd9c03c61e0fd6e2978829cd9068dfd6da68",
|
| 7 |
+
"checkpoint_size_bytes": 334907944,
|
| 8 |
+
"display_name": "nuPlan Epoch 4",
|
| 9 |
+
"mlflow_model_version": "64",
|
| 10 |
+
"model_id": "nuplan-epoch-4"
|
| 11 |
+
},
|
| 12 |
+
{
|
| 13 |
+
"checkpoint_path": "models/nuplan-epoch-5/checkpoint.pt",
|
| 14 |
+
"checkpoint_sha256": "ca8b43d7a777d6fd9195bb253d1452f3cd645f7aab3bd0e36b1f74ffb31df29b",
|
| 15 |
+
"checkpoint_size_bytes": 334907944,
|
| 16 |
+
"display_name": "nuPlan Epoch 5",
|
| 17 |
+
"mlflow_model_version": "65",
|
| 18 |
+
"model_id": "nuplan-epoch-5"
|
| 19 |
+
},
|
| 20 |
+
{
|
| 21 |
+
"checkpoint_path": "models/kitscenes-epoch-5/checkpoint.pt",
|
| 22 |
+
"checkpoint_sha256": "120a21639d9767d512eea645be0a83b01476c9e0366a8b957ac51b7ead1bd762",
|
| 23 |
+
"checkpoint_size_bytes": 334754024,
|
| 24 |
+
"display_name": "KITScenes Epoch 5",
|
| 25 |
+
"mlflow_model_version": "66",
|
| 26 |
+
"model_id": "kitscenes-epoch-5"
|
| 27 |
+
},
|
| 28 |
+
{
|
| 29 |
+
"checkpoint_path": "models/kitscenes-epoch-7/checkpoint.pt",
|
| 30 |
+
"checkpoint_sha256": "a1e6b1621018740485b3e1e43dda23713d913667eb39a65b77fcbf895dcbcefd",
|
| 31 |
+
"checkpoint_size_bytes": 334754024,
|
| 32 |
+
"display_name": "KITScenes Epoch 7",
|
| 33 |
+
"mlflow_model_version": "67",
|
| 34 |
+
"model_id": "kitscenes-epoch-7"
|
| 35 |
+
}
|
| 36 |
+
],
|
| 37 |
+
"published_at": "2026-09-16T02:26:06+00:00",
|
| 38 |
+
"repository": "AutowareFoundation/auto_e2e",
|
| 39 |
+
"schema_version": "auto_e2e_hf_release_manifest_v1"
|
| 40 |
+
}
|