Document experimental checkpoint evaluations
Browse filesAdd complete KITScenes Val v3.5 and Test v1.0 metrics and identify these artifacts as experimental contributor checkpoints rather than an official release.
- README.md +33 -9
- SHA256SUMS +1 -1
README.md
CHANGED
|
@@ -7,10 +7,15 @@ tags:
|
|
| 7 |
- trajectory-planning
|
| 8 |
- nuplan
|
| 9 |
- kitscenes
|
|
|
|
| 10 |
---
|
| 11 |
|
| 12 |
# AutoE2E checkpoints
|
| 13 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
This repository contains four trajectory-planning checkpoints from the AutoE2E project. These are the exact PyTorch training checkpoints registered in MLflow, including model, optimizer, scheduler, configuration, metrics, and training state.
|
| 15 |
|
| 16 |
## Checkpoints
|
|
@@ -24,19 +29,38 @@ This repository contains four trajectory-planning checkpoints from the AutoE2E p
|
|
| 24 |
|
| 25 |
All checkpoints use the `bevformer_v2_t8_split_navigation_v5` configuration. The camera BEV starts from the official BEVFormer V2 R50 T8 initialization and is frozen during trajectory stages. World Model and Reasoning branches are disabled.
|
| 26 |
|
| 27 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
|
| 29 |
-
|
| 30 |
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
|
| 36 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
|
| 38 |
The external evaluation is a deterministic replay of published control overlays. It does not re-run the model. Val and Test are different populations, so their difference is not evidence of a causal benefit from map or route inputs. The complete index and eight reports are under `evaluations/overlay-replay-v1/`.
|
| 39 |
|
|
|
|
|
|
|
| 40 |
## Loading
|
| 41 |
|
| 42 |
```python
|
|
@@ -50,7 +74,7 @@ Use an AutoE2E source revision compatible with `bevformer_v2_t8_split_navigation
|
|
| 50 |
|
| 51 |
## Intended use and limitations
|
| 52 |
|
| 53 |
-
These checkpoints are research artifacts for
|
| 54 |
|
| 55 |
## License notice
|
| 56 |
|
|
|
|
| 7 |
- trajectory-planning
|
| 8 |
- nuplan
|
| 9 |
- kitscenes
|
| 10 |
+
- experimental
|
| 11 |
---
|
| 12 |
|
| 13 |
# AutoE2E checkpoints
|
| 14 |
|
| 15 |
+
> Experimental contributor checkpoint notice
|
| 16 |
+
>
|
| 17 |
+
> These artifacts are not an official Autoware release. They are experimental checkpoints published for contributors to reproduce evaluation results, inspect model behavior, and continue research. They are not release-qualified or safety-qualified models.
|
| 18 |
+
|
| 19 |
This repository contains four trajectory-planning checkpoints from the AutoE2E project. These are the exact PyTorch training checkpoints registered in MLflow, including model, optimizer, scheduler, configuration, metrics, and training state.
|
| 20 |
|
| 21 |
## Checkpoints
|
|
|
|
| 29 |
|
| 30 |
All checkpoints use the `bevformer_v2_t8_split_navigation_v5` configuration. The camera BEV starts from the official BEVFormer V2 R50 T8 initialization and is frozen during trajectory stages. World Model and Reasoning branches are disabled.
|
| 31 |
|
| 32 |
+
## KITScenes Val v3.5
|
| 33 |
+
|
| 34 |
+
KITScenes Val v3.5 contains 11,035 samples. Evaluation uses Camera, HD Map, and an oracle Route reconstructed retrospectively from the logged future trajectory.
|
| 35 |
+
|
| 36 |
+
Each horizon value is ADE / FDE in meters.
|
| 37 |
+
|
| 38 |
+
| MLflow version | Model | 1 s | 2 s | 3 s | 5 s | Lateral error | Longitudinal error |
|
| 39 |
+
|---:|---|---:|---:|---:|---:|---:|---:|
|
| 40 |
+
| 64 | nuPlan Epoch 4 | 0.2030 / 0.4066 | 0.5018 / 1.1935 | 0.9503 / 2.4508 | 2.2919 / 6.2272 | 1.2082 | 1.6448 |
|
| 41 |
+
| 65 | nuPlan Epoch 5 | 0.1951 / 0.3779 | 0.4723 / 1.1331 | 0.9147 / 2.4226 | 2.2951 / 6.3885 | 1.1911 | 1.6633 |
|
| 42 |
+
| 66 | KITScenes Epoch 5 | 0.1472 / 0.2850 | 0.3729 / 0.9239 | 0.7449 / 2.0196 | 1.9405 / 5.5645 | 0.9844 | 1.4025 |
|
| 43 |
+
| 67 | KITScenes Epoch 7 | 0.1347 / 0.2831 | 0.3886 / 1.0101 | 0.8035 / 2.2127 | 2.0952 / 5.9509 | 1.0220 | 1.5659 |
|
| 44 |
|
| 45 |
+
## KITScenes Test v1.0
|
| 46 |
|
| 47 |
+
KITScenes Test v1.0 contains 23,690 samples. This evaluation is Camera-only and uses a different scene population from KITScenes Val.
|
| 48 |
+
|
| 49 |
+
Each horizon value is ADE / FDE in meters.
|
| 50 |
+
|
| 51 |
+
| MLflow version | Model | 1 s | 2 s | 3 s | 5 s | Lateral error | Longitudinal error |
|
| 52 |
+
|---:|---|---:|---:|---:|---:|---:|---:|
|
| 53 |
+
| 64 | nuPlan Epoch 4 | 0.1916 / 0.3899 | 0.4817 / 1.1514 | 0.9181 / 2.3895 | 2.2840 / 6.4266 | 1.2167 | 1.6287 |
|
| 54 |
+
| 65 | nuPlan Epoch 5 | 0.1602 / 0.3134 | 0.3979 / 0.9727 | 0.7924 / 2.1621 | 2.1120 / 6.1751 | 1.0584 | 1.5496 |
|
| 55 |
+
| 66 | KITScenes Epoch 5 | 0.1463 / 0.3107 | 0.4121 / 1.0453 | 0.8263 / 2.2199 | 2.1042 / 5.9422 | 1.2044 | 1.4051 |
|
| 56 |
+
| 67 | KITScenes Epoch 7 | 0.1680 / 0.3754 | 0.4923 / 1.2512 | 0.9831 / 2.6242 | 2.4699 / 6.8964 | 1.4730 | 1.6598 |
|
| 57 |
+
|
| 58 |
+
## Evaluation scope
|
| 59 |
|
| 60 |
The external evaluation is a deterministic replay of published control overlays. It does not re-run the model. Val and Test are different populations, so their difference is not evidence of a causal benefit from map or route inputs. The complete index and eight reports are under `evaluations/overlay-replay-v1/`.
|
| 61 |
|
| 62 |
+
The KITScenes Val route contains oracle maneuver information because it is reconstructed post hoc from the logged future trajectory. Val results must not be interpreted as online route-planning performance.
|
| 63 |
+
|
| 64 |
## Loading
|
| 65 |
|
| 66 |
```python
|
|
|
|
| 74 |
|
| 75 |
## Intended use and limitations
|
| 76 |
|
| 77 |
+
These checkpoints are experimental research artifacts for contributor evaluation and continued training. They are not an official Autoware release and are not validated for safety-critical deployment. Distribution shift remains relevant.
|
| 78 |
|
| 79 |
## License notice
|
| 80 |
|
SHA256SUMS
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
|
| 2 |
be1cec5f77a7a12326070503dbcd1004c63f54450cf1151c763a215816d838bd evaluations/overlay-replay-v1/index.json
|
| 3 |
582dc108ce030756d5d1758dbe198d62b3355bf32a123af54be4e66704c50b8a evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-5/report.json
|
| 4 |
d01169d68266d4b7203e92a4d25ade2a3277357ff89cc26cdb202051a26016fc evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-7/report.json
|
|
|
|
| 1 |
+
cfc1e20a29dca9538f74c67f847b61a38dfde528dbbf8001a324fc2c5b23e9de README.md
|
| 2 |
be1cec5f77a7a12326070503dbcd1004c63f54450cf1151c763a215816d838bd evaluations/overlay-replay-v1/index.json
|
| 3 |
582dc108ce030756d5d1758dbe198d62b3355bf32a123af54be4e66704c50b8a evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-5/report.json
|
| 4 |
d01169d68266d4b7203e92a4d25ade2a3277357ff89cc26cdb202051a26016fc evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-7/report.json
|