Hugging Face's logo

MLL-Lab
/
LAGEN-models

Robotics
PyTorch
Safetensors
vision-language-action
latency-sensitive-bench
multiple-checkpoints
Model card Files
xet
 Community
1
Caesarrr commited on
Commit
742eea3
·
verified ·
1 Parent(s): 12273c0

Index the verified thirty-case sim2real calibration release

Browse files
Files changed (3) hide show
  1. README.md +1 -0
  2. experiments/sim2real-calibration/README.md +18 -0
  3. experiments/sim2real-calibration/model_identity.json +224 -0
README.md CHANGED
@@ -13,6 +13,7 @@ Inference checkpoints, original configurations and model provenance.
13
 
14
  | Experiment | Entry |
15
  |---|---|
 
16
  | Visual history | [Visual history](https://huggingface.co/datasets/latency-sensitive-bench/benchmark-datasets/tree/main/experiments/visual-history) |
17
  | Latency in prompt | [Latency in prompt](https://huggingface.co/datasets/latency-sensitive-bench/benchmark-datasets/tree/main/experiments/latency-in-prompt) |
18
  | Latency transfer | [Latency transfer](https://huggingface.co/datasets/latency-sensitive-bench/benchmark-datasets/tree/main/experiments/latency-transfer) |
 
13
 
14
  | Experiment | Entry |
15
  |---|---|
16
+ | Sim2Real calibration | [30-case held-out calibration](https://huggingface.co/datasets/latency-sensitive-bench/benchmark-datasets/tree/7cf0da6324823aa98c8b4683c80f5dc75da0bb4f/experiments/sim2real-calibration) |
17
  | Visual history | [Visual history](https://huggingface.co/datasets/latency-sensitive-bench/benchmark-datasets/tree/main/experiments/visual-history) |
18
  | Latency in prompt | [Latency in prompt](https://huggingface.co/datasets/latency-sensitive-bench/benchmark-datasets/tree/main/experiments/latency-in-prompt) |
19
  | Latency transfer | [Latency transfer](https://huggingface.co/datasets/latency-sensitive-bench/benchmark-datasets/tree/main/experiments/latency-transfer) |
experiments/sim2real-calibration/README.md ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Sim2Real calibration policies
2
+
3
+ Six existing policies are shared across five inference GPU conditions.
4
+
5
+ | Task | Model | Existing bundle |
6
+ |---|---|---|
7
+ | flappy | qwenoft | [latency-aware/flappy/vla/starvla-qwenoft-h1/flappy-qwenoft-hist8-scratch-s42-v2](/latency-sensitive-bench/benchmark-models/tree/180da0effdaaaeacef218cf5fbaa4db41af67495/latency-aware/flappy/vla/starvla-qwenoft-h1/flappy-qwenoft-hist8-scratch-s42-v2) |
8
+ | flappy | qwengr00t | [latency-aware/flappy/vla/starvla-qwengr00t-h1/flappy-qwengr00t-hist8-scratch-s42-v2](/latency-sensitive-bench/benchmark-models/tree/180da0effdaaaeacef218cf5fbaa4db41af67495/latency-aware/flappy/vla/starvla-qwengr00t-h1/flappy-qwengr00t-hist8-scratch-s42-v2) |
9
+ | inverted_pendulum | qwenoft | [latency-aware/inverted-pendulum/vla/starvla-qwenoft-h1/inverted_pendulum_rgb_state_ip10_20260911T010114Z_openvla_native_sft_2k](/latency-sensitive-bench/benchmark-models/tree/180da0effdaaaeacef218cf5fbaa4db41af67495/latency-aware/inverted-pendulum/vla/starvla-qwenoft-h1/inverted_pendulum_rgb_state_ip10_20260911T010114Z_openvla_native_sft_2k) |
10
+ | inverted_pendulum | qwengr00t | [latency-aware/inverted-pendulum/vla/starvla-qwengr00t-h1/ip_profile_gr00t_h1_5fps_1h200_20260917](/latency-sensitive-bench/benchmark-models/tree/180da0effdaaaeacef218cf5fbaa4db41af67495/latency-aware/inverted-pendulum/vla/starvla-qwengr00t-h1/ip_profile_gr00t_h1_5fps_1h200_20260917) |
11
+ | mikasa_intercept_grab_fast | qwenoft | [latency-aware/mikasa-intercept-grab-fast/vla/starvla-qwenoft-h1/mikasa_qwenoft_latency_3090_2xh200_20260922_evalfix_20260923](/latency-sensitive-bench/benchmark-models/tree/180da0effdaaaeacef218cf5fbaa4db41af67495/latency-aware/mikasa-intercept-grab-fast/vla/starvla-qwenoft-h1/mikasa_qwenoft_latency_3090_2xh200_20260922_evalfix_20260923) |
12
+ | mikasa_intercept_grab_fast | qwengr00t | [latency-aware/mikasa-intercept-grab-fast/vla/starvla-qwengr00t-h1/mikasa_qwengr00t_latency_3090_2xh200_20260922_evalfix_20260923](/latency-sensitive-bench/benchmark-models/tree/180da0effdaaaeacef218cf5fbaa4db41af67495/latency-aware/mikasa-intercept-grab-fast/vla/starvla-qwengr00t-h1/mikasa_qwengr00t_latency_3090_2xh200_20260922_evalfix_20260923) |
13
+
14
+ [Experiment settings, checkpoints, 30 cases and results](https://huggingface.co/datasets/latency-sensitive-bench/benchmark-datasets/tree/7cf0da6324823aa98c8b4683c80f5dc75da0bb4f/experiments/sim2real-calibration)
15
+
16
+ Checkpoint, configuration, dataset statistics and policy metadata identities
17
+ match the experiment snapshot `180da0effdaaaeacef218cf5fbaa4db41af67495`.
18
+ Existing training provenance remains beside each bundle.
experiments/sim2real-calibration/model_identity.json ADDED
@@ -0,0 +1,224 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "models": [
3
+ {
4
+ "task": "flappy",
5
+ "model": "qwenoft",
6
+ "bundle": "latency-aware/flappy/vla/starvla-qwenoft-h1/flappy-qwenoft-hist8-scratch-s42-v2",
7
+ "model_repo": "latency-sensitive-bench/benchmark-models",
8
+ "model_revision": "180da0effdaaaeacef218cf5fbaa4db41af67495",
9
+ "current_revision": "365fe5402de3fb464d5da8da9f944b969c5801c4",
10
+ "checkpoint": "checkpoints/model.pt",
11
+ "required_files": [
12
+ {
13
+ "path": "latency-aware/flappy/vla/starvla-qwenoft-h1/flappy-qwenoft-hist8-scratch-s42-v2/checkpoints/model.pt",
14
+ "size": 9785137569,
15
+ "blob_id": "2febb4ed2f7e1393003dd0610eb67e89a3746a78",
16
+ "lfs_sha256": "c6c1f36ab709d596a9123d395377d43fa57d6e9e4b8399778f3b374ca501f883"
17
+ },
18
+ {
19
+ "path": "latency-aware/flappy/vla/starvla-qwenoft-h1/flappy-qwenoft-hist8-scratch-s42-v2/config.yaml",
20
+ "size": 786,
21
+ "blob_id": "35d7f0292768e50a19a4d1aaf09cdb0ea7e57b2a",
22
+ "lfs_sha256": null
23
+ },
24
+ {
25
+ "path": "latency-aware/flappy/vla/starvla-qwenoft-h1/flappy-qwenoft-hist8-scratch-s42-v2/dataset_statistics.json",
26
+ "size": 3298,
27
+ "blob_id": "ca1504715f087b3745857f2f63ff77f6b475f7a1",
28
+ "lfs_sha256": null
29
+ }
30
+ ]
31
+ },
32
+ {
33
+ "task": "flappy",
34
+ "model": "qwengr00t",
35
+ "bundle": "latency-aware/flappy/vla/starvla-qwengr00t-h1/flappy-qwengr00t-hist8-scratch-s42-v2",
36
+ "model_repo": "latency-sensitive-bench/benchmark-models",
37
+ "model_revision": "180da0effdaaaeacef218cf5fbaa4db41af67495",
38
+ "current_revision": "365fe5402de3fb464d5da8da9f944b969c5801c4",
39
+ "checkpoint": "checkpoints/model.pt",
40
+ "required_files": [
41
+ {
42
+ "path": "latency-aware/flappy/vla/starvla-qwengr00t-h1/flappy-qwengr00t-hist8-scratch-s42-v2/checkpoints/model.pt",
43
+ "size": 9976860963,
44
+ "blob_id": "ba87e2b3b59c8b8b66f2b75590a5d1de46def326",
45
+ "lfs_sha256": "5989ecd1e19c3ea4a4fc479fb90277f11b0cd4962173afa39ed22406daa1d939"
46
+ },
47
+ {
48
+ "path": "latency-aware/flappy/vla/starvla-qwengr00t-h1/flappy-qwengr00t-hist8-scratch-s42-v2/config.yaml",
49
+ "size": 1302,
50
+ "blob_id": "2618ada214493839b65433c338fe4eb5d8669cf7",
51
+ "lfs_sha256": null
52
+ },
53
+ {
54
+ "path": "latency-aware/flappy/vla/starvla-qwengr00t-h1/flappy-qwengr00t-hist8-scratch-s42-v2/dataset_statistics.json",
55
+ "size": 3193,
56
+ "blob_id": "f19778adda04585ff9ea336b4203e9693c6f79f8",
57
+ "lfs_sha256": null
58
+ }
59
+ ]
60
+ },
61
+ {
62
+ "task": "inverted_pendulum",
63
+ "model": "qwenoft",
64
+ "bundle": "latency-aware/inverted-pendulum/vla/starvla-qwenoft-h1/inverted_pendulum_rgb_state_ip10_20260911T010114Z_openvla_native_sft_2k",
65
+ "model_repo": "latency-sensitive-bench/benchmark-models",
66
+ "model_revision": "180da0effdaaaeacef218cf5fbaa4db41af67495",
67
+ "current_revision": "365fe5402de3fb464d5da8da9f944b969c5801c4",
68
+ "checkpoint": "checkpoints/model.pt",
69
+ "required_files": [
70
+ {
71
+ "path": "latency-aware/inverted-pendulum/vla/starvla-qwenoft-h1/inverted_pendulum_rgb_state_ip10_20260911T010114Z_openvla_native_sft_2k/checkpoints/model.pt",
72
+ "size": 9785039595,
73
+ "blob_id": "0a2990fcf3cf3f88ad5c45c2cfe3e9f749fef4c9",
74
+ "lfs_sha256": "b7cb1fd6ea38192d065b8379a6bcb6796b6eb23ca9c5d09df532a9ce7664bdec"
75
+ },
76
+ {
77
+ "path": "latency-aware/inverted-pendulum/vla/starvla-qwenoft-h1/inverted_pendulum_rgb_state_ip10_20260911T010114Z_openvla_native_sft_2k/config.full.yaml",
78
+ "size": 9157,
79
+ "blob_id": "72b4aaf22b1788cc3017b65a6b5481ea25f94d19",
80
+ "lfs_sha256": null
81
+ },
82
+ {
83
+ "path": "latency-aware/inverted-pendulum/vla/starvla-qwenoft-h1/inverted_pendulum_rgb_state_ip10_20260911T010114Z_openvla_native_sft_2k/config.yaml",
84
+ "size": 9157,
85
+ "blob_id": "72b4aaf22b1788cc3017b65a6b5481ea25f94d19",
86
+ "lfs_sha256": null
87
+ },
88
+ {
89
+ "path": "latency-aware/inverted-pendulum/vla/starvla-qwenoft-h1/inverted_pendulum_rgb_state_ip10_20260911T010114Z_openvla_native_sft_2k/dataset_statistics.json",
90
+ "size": 1558,
91
+ "blob_id": "f45ba5bc101d8a09ee7cbea7e0df8f8660026aaa",
92
+ "lfs_sha256": null
93
+ },
94
+ {
95
+ "path": "latency-aware/inverted-pendulum/vla/starvla-qwenoft-h1/inverted_pendulum_rgb_state_ip10_20260911T010114Z_openvla_native_sft_2k/latency_prompt_map.json",
96
+ "size": 812,
97
+ "blob_id": "f80dcdd6ce8a2b176d3b955311120de82bf568ee",
98
+ "lfs_sha256": null
99
+ },
100
+ {
101
+ "path": "latency-aware/inverted-pendulum/vla/starvla-qwenoft-h1/inverted_pendulum_rgb_state_ip10_20260911T010114Z_openvla_native_sft_2k/manifest.json",
102
+ "size": 3723,
103
+ "blob_id": "40902ccb30bcac6e1a2969d0f625da5a9d728f68",
104
+ "lfs_sha256": null
105
+ }
106
+ ]
107
+ },
108
+ {
109
+ "task": "inverted_pendulum",
110
+ "model": "qwengr00t",
111
+ "bundle": "latency-aware/inverted-pendulum/vla/starvla-qwengr00t-h1/ip_profile_gr00t_h1_5fps_1h200_20260917",
112
+ "model_repo": "latency-sensitive-bench/benchmark-models",
113
+ "model_revision": "180da0effdaaaeacef218cf5fbaa4db41af67495",
114
+ "current_revision": "365fe5402de3fb464d5da8da9f944b969c5801c4",
115
+ "checkpoint": "checkpoints/model.pt",
116
+ "required_files": [
117
+ {
118
+ "path": "latency-aware/inverted-pendulum/vla/starvla-qwengr00t-h1/ip_profile_gr00t_h1_5fps_1h200_20260917/checkpoints/model.pt",
119
+ "size": 9976830755,
120
+ "blob_id": "7cd96c6b65dc707337ac3c5630a0a23ae45db389",
121
+ "lfs_sha256": "9ec338c86369f59593c802c5c5b29e1c2956ee46a38eecac47e1788acf7fcdb8"
122
+ },
123
+ {
124
+ "path": "latency-aware/inverted-pendulum/vla/starvla-qwengr00t-h1/ip_profile_gr00t_h1_5fps_1h200_20260917/config.yaml",
125
+ "size": 2572,
126
+ "blob_id": "9164e5e50d61651f3aa1713d4d72cb0d5cdb235d",
127
+ "lfs_sha256": null
128
+ },
129
+ {
130
+ "path": "latency-aware/inverted-pendulum/vla/starvla-qwengr00t-h1/ip_profile_gr00t_h1_5fps_1h200_20260917/dataset_statistics.json",
131
+ "size": 1552,
132
+ "blob_id": "a7c48ee55eca3407bee5a9f4c7624edffe2293e7",
133
+ "lfs_sha256": null
134
+ },
135
+ {
136
+ "path": "latency-aware/inverted-pendulum/vla/starvla-qwengr00t-h1/ip_profile_gr00t_h1_5fps_1h200_20260917/latency_prompt_map.json",
137
+ "size": 405,
138
+ "blob_id": "092b2a6629205ca410268eafca3af6bafee294c9",
139
+ "lfs_sha256": null
140
+ },
141
+ {
142
+ "path": "latency-aware/inverted-pendulum/vla/starvla-qwengr00t-h1/ip_profile_gr00t_h1_5fps_1h200_20260917/manifest.json",
143
+ "size": 3257,
144
+ "blob_id": "2696c0f2db1ad5d69ea3e045e41539fe85183eb3",
145
+ "lfs_sha256": null
146
+ }
147
+ ]
148
+ },
149
+ {
150
+ "task": "mikasa_intercept_grab_fast",
151
+ "model": "qwenoft",
152
+ "bundle": "latency-aware/mikasa-intercept-grab-fast/vla/starvla-qwenoft-h1/mikasa_qwenoft_latency_3090_2xh200_20260922_evalfix_20260923",
153
+ "model_repo": "latency-sensitive-bench/benchmark-models",
154
+ "model_revision": "180da0effdaaaeacef218cf5fbaa4db41af67495",
155
+ "current_revision": "365fe5402de3fb464d5da8da9f944b969c5801c4",
156
+ "checkpoint": "checkpoints/model.pt",
157
+ "required_files": [
158
+ {
159
+ "path": "latency-aware/mikasa-intercept-grab-fast/vla/starvla-qwenoft-h1/mikasa_qwenoft_latency_3090_2xh200_20260922_evalfix_20260923/checkpoints/model.pt",
160
+ "size": 9785101729,
161
+ "blob_id": "830a92f8553b24301cb822838fbf930746716422",
162
+ "lfs_sha256": "c98a7895a68788343533ddf2e09a3a4aa5bc29207abee10e096cc1cd9dbad890"
163
+ },
164
+ {
165
+ "path": "latency-aware/mikasa-intercept-grab-fast/vla/starvla-qwenoft-h1/mikasa_qwenoft_latency_3090_2xh200_20260922_evalfix_20260923/config.yaml",
166
+ "size": 819,
167
+ "blob_id": "9e9403269dbe90e77e29f65ce00a79aa61896725",
168
+ "lfs_sha256": null
169
+ },
170
+ {
171
+ "path": "latency-aware/mikasa-intercept-grab-fast/vla/starvla-qwenoft-h1/mikasa_qwenoft_latency_3090_2xh200_20260922_evalfix_20260923/dataset_statistics.json",
172
+ "size": 2797,
173
+ "blob_id": "8912194b9a7663bd1c20eba30d7184971411584b",
174
+ "lfs_sha256": null
175
+ },
176
+ {
177
+ "path": "latency-aware/mikasa-intercept-grab-fast/vla/starvla-qwenoft-h1/mikasa_qwenoft_latency_3090_2xh200_20260922_evalfix_20260923/task_contract.json",
178
+ "size": 3018,
179
+ "blob_id": "6d4a98b8cf44e28d330c53c1ef4ced070a7a179f",
180
+ "lfs_sha256": null
181
+ }
182
+ ]
183
+ },
184
+ {
185
+ "task": "mikasa_intercept_grab_fast",
186
+ "model": "qwengr00t",
187
+ "bundle": "latency-aware/mikasa-intercept-grab-fast/vla/starvla-qwengr00t-h1/mikasa_qwengr00t_latency_3090_2xh200_20260922_evalfix_20260923",
188
+ "model_repo": "latency-sensitive-bench/benchmark-models",
189
+ "model_revision": "180da0effdaaaeacef218cf5fbaa4db41af67495",
190
+ "current_revision": "365fe5402de3fb464d5da8da9f944b969c5801c4",
191
+ "checkpoint": "checkpoints/model.pt",
192
+ "required_files": [
193
+ {
194
+ "path": "latency-aware/mikasa-intercept-grab-fast/vla/starvla-qwengr00t-h1/mikasa_qwengr00t_latency_3090_2xh200_20260922_evalfix_20260923/checkpoints/model.pt",
195
+ "size": 9976844067,
196
+ "blob_id": "5087af7557285aa10f98830eda33da970cb10b6d",
197
+ "lfs_sha256": "26e7217bde9e107c5005a2d0c06dfb223ca467e25cbb23d11a2e3b703bb325b9"
198
+ },
199
+ {
200
+ "path": "latency-aware/mikasa-intercept-grab-fast/vla/starvla-qwengr00t-h1/mikasa_qwengr00t_latency_3090_2xh200_20260922_evalfix_20260923/config.yaml",
201
+ "size": 1296,
202
+ "blob_id": "2e103f8111e7c4e219de0e76b22294678086ea96",
203
+ "lfs_sha256": null
204
+ },
205
+ {
206
+ "path": "latency-aware/mikasa-intercept-grab-fast/vla/starvla-qwengr00t-h1/mikasa_qwengr00t_latency_3090_2xh200_20260922_evalfix_20260923/dataset_statistics.json",
207
+ "size": 2840,
208
+ "blob_id": "0af125f37610cb6b94869b42007918dffe26be92",
209
+ "lfs_sha256": null
210
+ },
211
+ {
212
+ "path": "latency-aware/mikasa-intercept-grab-fast/vla/starvla-qwengr00t-h1/mikasa_qwengr00t_latency_3090_2xh200_20260922_evalfix_20260923/task_contract.json",
213
+ "size": 3061,
214
+ "blob_id": "b1219266b54f4af3d9f9caa363938fc1989e582c",
215
+ "lfs_sha256": null
216
+ }
217
+ ]
218
+ }
219
+ ],
220
+ "reused_weights": 6,
221
+ "original_revision": "180da0effdaaaeacef218cf5fbaa4db41af67495",
222
+ "current_weights_match": true,
223
+ "gpu_evaluation_performed": false
224
+ }