Add a Benchmarks section with the KITScenes 3 s ADE/FDE chart and table
#1
by Arseni10Lk - opened
- .gitattributes +1 -0
- README.md +19 -0
- assets/benchmark_kitscenes_3s.png +3 -0
.gitattributes
CHANGED
|
@@ -38,3 +38,4 @@ assets/prediction_kitscenes_epoch5_val_roundabout.jpg filter=lfs diff=lfs merge=
|
|
| 38 |
assets/prediction_nuplan_epoch5_test_forest_turn.jpg filter=lfs diff=lfs merge=lfs -text
|
| 39 |
assets/prediction_nuplan_epoch5_test_night_turn.jpg filter=lfs diff=lfs merge=lfs -text
|
| 40 |
replay.mp4 filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 38 |
assets/prediction_nuplan_epoch5_test_forest_turn.jpg filter=lfs diff=lfs merge=lfs -text
|
| 39 |
assets/prediction_nuplan_epoch5_test_night_turn.jpg filter=lfs diff=lfs merge=lfs -text
|
| 40 |
replay.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 41 |
+
assets/benchmark_kitscenes_3s.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -83,6 +83,25 @@ The v1.0 checkpoints contain the **Reactive model**, which is the part that driv
|
|
| 83 |
| Auxiliary heads | 8-class BEV segmentation and route reconstruction (training signals) |
|
| 84 |
| Parameters | 79.9 M |
|
| 85 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
## Results
|
| 87 |
|
| 88 |
All values are ADE / FDE in metres for a single predicted trajectory; lower is better. "Lateral" and "longitudinal" are the mean absolute errors across the valid horizon.
|
|
|
|
| 83 |
| Auxiliary heads | 8-class BEV segmentation and route reconstruction (training signals) |
|
| 84 |
| Parameters | 79.9 M |
|
| 85 |
|
| 86 |
+
## Benchmarks
|
| 87 |
+
|
| 88 |
+
<img src="https://huggingface.co/AutowareFoundation/auto_e2e/resolve/main/assets/benchmark_kitscenes_3s.png" alt="ADE and FDE at 3 s on KITScenes-Multimodal validation for eight open-weights driving models. The fine-tuned AutoE2E checkpoint is second, behind Qwen Drive 1.0." width="100%">
|
| 89 |
+
|
| 90 |
+
The fine-tuned AutoE2E checkpoint has the second-lowest error of the eight open-weights models we benchmarked on KITScenes-Multimodal validation. Only Qwen Drive 1.0 is ahead, and AutoE2E beats SimForge, Alpamayo 1.5 and METEOR with 79.9 M parameters. All models are scored open-loop: each one predicts the next 3 s from logged camera frames, and the prediction is compared with the path the human driver took. ADE is the average position error over those 3 s and FDE is the error at the 3 s point.
|
| 91 |
+
|
| 92 |
+
| Model | Checkpoint / setting | Evaluated on | ADE @ 3 s (m) | FDE @ 3 s (m) |
|
| 93 |
+
|---|---|---|---:|---:|
|
| 94 |
+
| Qwen Drive 1.0 | first of 6 trajectories | 202 keyframes | 0.565 | 1.397 |
|
| 95 |
+
| **AutoE2E fine-tuned** | [`kitscenes-epoch-5`](./models/kitscenes-epoch-5) | 202 keyframes | **0.744** | **1.980** |
|
| 96 |
+
| SimForge D1.5 | 7 camera slots (2 virtual telephoto), first of 6 trajectories | 202 keyframes | 0.831 | 2.209 |
|
| 97 |
+
| **AutoE2E** | [`nuplan-epoch-5`](./models/nuplan-epoch-5), zero-shot | 11,035 samples | **0.915** | **2.423** |
|
| 98 |
+
| Alpamayo 1.5 | first of 6 trajectories | 202 keyframes | 1.020 | 2.820 |
|
| 99 |
+
| VAD | Base stage-2, oracle driving command | 11,035 samples | 1.242 | 2.821 |
|
| 100 |
+
| METEOR | top-1 of 3 paths | 202 keyframes | 1.240 | 3.282 |
|
| 101 |
+
| Drive-JEPA | perception-based v2, 20 m route command | 11,035 samples | 2.266 | 4.774 |
|
| 102 |
+
|
| 103 |
+
The 11,035 samples are every frame of the 117 validation scenes, and the 202 keyframes are non-overlapping anchors taken from the same scenes. Inputs differ by model: AutoE2E also gets an HD map and the route, and VAD and Drive-JEPA get a driving command. [Issue #210](https://github.com/autowarefoundation/auto_e2e/issues/210) has each model's adapter, ground truth and results at the other horizons.
|
| 104 |
+
|
| 105 |
## Results
|
| 106 |
|
| 107 |
All values are ADE / FDE in metres for a single predicted trajectory; lower is better. "Lateral" and "longitudinal" are the mean absolute errors across the valid horizon.
|
assets/benchmark_kitscenes_3s.png
ADDED
|
Git LFS Details
|