Add a Benchmarks section with the KITScenes 3 s ADE/FDE chart and table

#1
by Arseni10Lk - opened
Files changed (3) hide show
  1. .gitattributes +1 -0
  2. README.md +19 -0
  3. assets/benchmark_kitscenes_3s.png +3 -0
.gitattributes CHANGED
@@ -38,3 +38,4 @@ assets/prediction_kitscenes_epoch5_val_roundabout.jpg filter=lfs diff=lfs merge=
38
  assets/prediction_nuplan_epoch5_test_forest_turn.jpg filter=lfs diff=lfs merge=lfs -text
39
  assets/prediction_nuplan_epoch5_test_night_turn.jpg filter=lfs diff=lfs merge=lfs -text
40
  replay.mp4 filter=lfs diff=lfs merge=lfs -text
 
 
38
  assets/prediction_nuplan_epoch5_test_forest_turn.jpg filter=lfs diff=lfs merge=lfs -text
39
  assets/prediction_nuplan_epoch5_test_night_turn.jpg filter=lfs diff=lfs merge=lfs -text
40
  replay.mp4 filter=lfs diff=lfs merge=lfs -text
41
+ assets/benchmark_kitscenes_3s.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -83,6 +83,25 @@ The v1.0 checkpoints contain the **Reactive model**, which is the part that driv
83
  | Auxiliary heads | 8-class BEV segmentation and route reconstruction (training signals) |
84
  | Parameters | 79.9 M |
85
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
86
  ## Results
87
 
88
  All values are ADE / FDE in metres for a single predicted trajectory; lower is better. "Lateral" and "longitudinal" are the mean absolute errors across the valid horizon.
 
83
  | Auxiliary heads | 8-class BEV segmentation and route reconstruction (training signals) |
84
  | Parameters | 79.9 M |
85
 
86
+ ## Benchmarks
87
+
88
+ <img src="https://huggingface.co/AutowareFoundation/auto_e2e/resolve/main/assets/benchmark_kitscenes_3s.png" alt="ADE and FDE at 3 s on KITScenes-Multimodal validation for eight open-weights driving models. The fine-tuned AutoE2E checkpoint is second, behind Qwen Drive 1.0." width="100%">
89
+
90
+ The fine-tuned AutoE2E checkpoint has the second-lowest error of the eight open-weights models we benchmarked on KITScenes-Multimodal validation. Only Qwen Drive 1.0 is ahead, and AutoE2E beats SimForge, Alpamayo 1.5 and METEOR with 79.9 M parameters. All models are scored open-loop: each one predicts the next 3 s from logged camera frames, and the prediction is compared with the path the human driver took. ADE is the average position error over those 3 s and FDE is the error at the 3 s point.
91
+
92
+ | Model | Checkpoint / setting | Evaluated on | ADE @ 3 s (m) | FDE @ 3 s (m) |
93
+ |---|---|---|---:|---:|
94
+ | Qwen Drive 1.0 | first of 6 trajectories | 202 keyframes | 0.565 | 1.397 |
95
+ | **AutoE2E fine-tuned** | [`kitscenes-epoch-5`](./models/kitscenes-epoch-5) | 202 keyframes | **0.744** | **1.980** |
96
+ | SimForge D1.5 | 7 camera slots (2 virtual telephoto), first of 6 trajectories | 202 keyframes | 0.831 | 2.209 |
97
+ | **AutoE2E** | [`nuplan-epoch-5`](./models/nuplan-epoch-5), zero-shot | 11,035 samples | **0.915** | **2.423** |
98
+ | Alpamayo 1.5 | first of 6 trajectories | 202 keyframes | 1.020 | 2.820 |
99
+ | VAD | Base stage-2, oracle driving command | 11,035 samples | 1.242 | 2.821 |
100
+ | METEOR | top-1 of 3 paths | 202 keyframes | 1.240 | 3.282 |
101
+ | Drive-JEPA | perception-based v2, 20 m route command | 11,035 samples | 2.266 | 4.774 |
102
+
103
+ The 11,035 samples are every frame of the 117 validation scenes, and the 202 keyframes are non-overlapping anchors taken from the same scenes. Inputs differ by model: AutoE2E also gets an HD map and the route, and VAD and Drive-JEPA get a driving command. [Issue #210](https://github.com/autowarefoundation/auto_e2e/issues/210) has each model's adapter, ground truth and results at the other horizons.
104
+
105
  ## Results
106
 
107
  All values are ADE / FDE in metres for a single predicted trajectory; lower is better. "Lateral" and "longitudinal" are the mean absolute errors across the valid horizon.
assets/benchmark_kitscenes_3s.png ADDED

Git LFS Details

  • SHA256: e930a87581789dfc004d15e00ac7c077a5d2f413f43eff58684ff212303825c9
  • Pointer size: 131 Bytes
  • Size of remote file: 143 kB