riita10069 commited on
Commit
437ae16
·
verified ·
1 Parent(s): c77e366

Document experimental checkpoint evaluations

Browse files

Add complete KITScenes Val v3.5 and Test v1.0 metrics and identify these artifacts as experimental contributor checkpoints rather than an official release.

Files changed (2) hide show
  1. README.md +33 -9
  2. SHA256SUMS +1 -1
README.md CHANGED
@@ -7,10 +7,15 @@ tags:
7
  - trajectory-planning
8
  - nuplan
9
  - kitscenes
 
10
  ---
11
 
12
  # AutoE2E checkpoints
13
 
 
 
 
 
14
  This repository contains four trajectory-planning checkpoints from the AutoE2E project. These are the exact PyTorch training checkpoints registered in MLflow, including model, optimizer, scheduler, configuration, metrics, and training state.
15
 
16
  ## Checkpoints
@@ -24,19 +29,38 @@ This repository contains four trajectory-planning checkpoints from the AutoE2E p
24
 
25
  All checkpoints use the `bevformer_v2_t8_split_navigation_v5` configuration. The camera BEV starts from the official BEVFormer V2 R50 T8 initialization and is frozen during trajectory stages. World Model and Reasoning branches are disabled.
26
 
27
- ## External overlay replay
 
 
 
 
 
 
 
 
 
 
 
28
 
29
- Each value is ADE / FDE in meters. KITScenes Val v3.5 contains 11,035 samples with Camera + HD Map + oracle post-hoc Route. KITScenes Test v1.0 contains 23,690 camera-only samples from a different scene population.
30
 
31
- | Checkpoint | Val 1 s | Val 3 s | Val 5 s | Test 1 s | Test 3 s | Test 5 s |
32
- |---|---:|---:|---:|---:|---:|---:|
33
- | nuPlan Epoch 4 | 0.2030 / 0.4066 | 0.9503 / 2.4508 | 2.2919 / 6.2272 | 0.1916 / 0.3899 | 0.9181 / 2.3895 | 2.2840 / 6.4266 |
34
- | nuPlan Epoch 5 | 0.1951 / 0.3779 | 0.9147 / 2.4226 | 2.2951 / 6.3885 | 0.1602 / 0.3134 | 0.7924 / 2.1621 | 2.1120 / 6.1751 |
35
- | KITScenes Epoch 5 | 0.1472 / 0.2850 | 0.7449 / 2.0196 | 1.9405 / 5.5645 | 0.1463 / 0.3107 | 0.8263 / 2.2199 | 2.1042 / 5.9422 |
36
- | KITScenes Epoch 7 | 0.1347 / 0.2831 | 0.8035 / 2.2127 | 2.0952 / 5.9509 | 0.1680 / 0.3754 | 0.9831 / 2.6242 | 2.4699 / 6.8964 |
 
 
 
 
 
 
37
 
38
  The external evaluation is a deterministic replay of published control overlays. It does not re-run the model. Val and Test are different populations, so their difference is not evidence of a causal benefit from map or route inputs. The complete index and eight reports are under `evaluations/overlay-replay-v1/`.
39
 
 
 
40
  ## Loading
41
 
42
  ```python
@@ -50,7 +74,7 @@ Use an AutoE2E source revision compatible with `bevformer_v2_t8_split_navigation
50
 
51
  ## Intended use and limitations
52
 
53
- These checkpoints are research artifacts for open-loop trajectory-planning evaluation and continued training. They are not validated for safety-critical deployment. The KITScenes Val route is reconstructed retrospectively from the logged future trajectory and contains oracle maneuver information. Distribution shift remains relevant.
54
 
55
  ## License notice
56
 
 
7
  - trajectory-planning
8
  - nuplan
9
  - kitscenes
10
+ - experimental
11
  ---
12
 
13
  # AutoE2E checkpoints
14
 
15
+ > Experimental contributor checkpoint notice
16
+ >
17
+ > These artifacts are not an official Autoware release. They are experimental checkpoints published for contributors to reproduce evaluation results, inspect model behavior, and continue research. They are not release-qualified or safety-qualified models.
18
+
19
  This repository contains four trajectory-planning checkpoints from the AutoE2E project. These are the exact PyTorch training checkpoints registered in MLflow, including model, optimizer, scheduler, configuration, metrics, and training state.
20
 
21
  ## Checkpoints
 
29
 
30
  All checkpoints use the `bevformer_v2_t8_split_navigation_v5` configuration. The camera BEV starts from the official BEVFormer V2 R50 T8 initialization and is frozen during trajectory stages. World Model and Reasoning branches are disabled.
31
 
32
+ ## KITScenes Val v3.5
33
+
34
+ KITScenes Val v3.5 contains 11,035 samples. Evaluation uses Camera, HD Map, and an oracle Route reconstructed retrospectively from the logged future trajectory.
35
+
36
+ Each horizon value is ADE / FDE in meters.
37
+
38
+ | MLflow version | Model | 1 s | 2 s | 3 s | 5 s | Lateral error | Longitudinal error |
39
+ |---:|---|---:|---:|---:|---:|---:|---:|
40
+ | 64 | nuPlan Epoch 4 | 0.2030 / 0.4066 | 0.5018 / 1.1935 | 0.9503 / 2.4508 | 2.2919 / 6.2272 | 1.2082 | 1.6448 |
41
+ | 65 | nuPlan Epoch 5 | 0.1951 / 0.3779 | 0.4723 / 1.1331 | 0.9147 / 2.4226 | 2.2951 / 6.3885 | 1.1911 | 1.6633 |
42
+ | 66 | KITScenes Epoch 5 | 0.1472 / 0.2850 | 0.3729 / 0.9239 | 0.7449 / 2.0196 | 1.9405 / 5.5645 | 0.9844 | 1.4025 |
43
+ | 67 | KITScenes Epoch 7 | 0.1347 / 0.2831 | 0.3886 / 1.0101 | 0.8035 / 2.2127 | 2.0952 / 5.9509 | 1.0220 | 1.5659 |
44
 
45
+ ## KITScenes Test v1.0
46
 
47
+ KITScenes Test v1.0 contains 23,690 samples. This evaluation is Camera-only and uses a different scene population from KITScenes Val.
48
+
49
+ Each horizon value is ADE / FDE in meters.
50
+
51
+ | MLflow version | Model | 1 s | 2 s | 3 s | 5 s | Lateral error | Longitudinal error |
52
+ |---:|---|---:|---:|---:|---:|---:|---:|
53
+ | 64 | nuPlan Epoch 4 | 0.1916 / 0.3899 | 0.4817 / 1.1514 | 0.9181 / 2.3895 | 2.2840 / 6.4266 | 1.2167 | 1.6287 |
54
+ | 65 | nuPlan Epoch 5 | 0.1602 / 0.3134 | 0.3979 / 0.9727 | 0.7924 / 2.1621 | 2.1120 / 6.1751 | 1.0584 | 1.5496 |
55
+ | 66 | KITScenes Epoch 5 | 0.1463 / 0.3107 | 0.4121 / 1.0453 | 0.8263 / 2.2199 | 2.1042 / 5.9422 | 1.2044 | 1.4051 |
56
+ | 67 | KITScenes Epoch 7 | 0.1680 / 0.3754 | 0.4923 / 1.2512 | 0.9831 / 2.6242 | 2.4699 / 6.8964 | 1.4730 | 1.6598 |
57
+
58
+ ## Evaluation scope
59
 
60
  The external evaluation is a deterministic replay of published control overlays. It does not re-run the model. Val and Test are different populations, so their difference is not evidence of a causal benefit from map or route inputs. The complete index and eight reports are under `evaluations/overlay-replay-v1/`.
61
 
62
+ The KITScenes Val route contains oracle maneuver information because it is reconstructed post hoc from the logged future trajectory. Val results must not be interpreted as online route-planning performance.
63
+
64
  ## Loading
65
 
66
  ```python
 
74
 
75
  ## Intended use and limitations
76
 
77
+ These checkpoints are experimental research artifacts for contributor evaluation and continued training. They are not an official Autoware release and are not validated for safety-critical deployment. Distribution shift remains relevant.
78
 
79
  ## License notice
80
 
SHA256SUMS CHANGED
@@ -1,4 +1,4 @@
1
- ee3c2f95176c285f3649281a986849a872b0920f4d1c67cad627d4e6424214e0 README.md
2
  be1cec5f77a7a12326070503dbcd1004c63f54450cf1151c763a215816d838bd evaluations/overlay-replay-v1/index.json
3
  582dc108ce030756d5d1758dbe198d62b3355bf32a123af54be4e66704c50b8a evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-5/report.json
4
  d01169d68266d4b7203e92a4d25ade2a3277357ff89cc26cdb202051a26016fc evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-7/report.json
 
1
+ cfc1e20a29dca9538f74c67f847b61a38dfde528dbbf8001a324fc2c5b23e9de README.md
2
  be1cec5f77a7a12326070503dbcd1004c63f54450cf1151c763a215816d838bd evaluations/overlay-replay-v1/index.json
3
  582dc108ce030756d5d1758dbe198d62b3355bf32a123af54be4e66704c50b8a evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-5/report.json
4
  d01169d68266d4b7203e92a4d25ade2a3277357ff89cc26cdb202051a26016fc evaluations/overlay-replay-v1/kitscenes-test/kitscenes-epoch-7/report.json