nikdevs commited on
Commit
ac9cf9b
·
verified ·
1 Parent(s): 9b17650

boot log tvfnt4hniqtnma

Browse files
Files changed (1) hide show
  1. logs/boot-tvfnt4hniqtnma.log +1160 -0
logs/boot-tvfnt4hniqtnma.log CHANGED
@@ -3134,3 +3134,1163 @@ worker-comfyui - Websocket receive timed out. Still waiting...
3134
  [DEBUG] Backend cuda selected for int8_linear
3135
  [DEBUG] Backend cuda selected for rms_rope_split_half_
3136
  [DEBUG] Backend cuda selected for int8_linear
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3137
  40%|████ | 8/20 [10:41<16:03, 80.26s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3138
  45%|████▌ | 9/20 [12:01<14:41, 80.15s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3139
  50%|█████ | 10/20 [13:21<13:20, 80.04s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3134
  [DEBUG] Backend cuda selected for int8_linear
3135
  [DEBUG] Backend cuda selected for rms_rope_split_half_
3136
  [DEBUG] Backend cuda selected for int8_linear
3137
+ [minimax] boot log shipped
3138
+ [DEBUG] Backend cuda selected for int8_linear
3139
+ [DEBUG] Backend cuda selected for int8_linear
3140
+ [DEBUG] Backend cuda selected for int8_linear
3141
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3142
+ [DEBUG] Backend cuda selected for int8_linear
3143
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3144
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3145
+ [DEBUG] Backend cuda selected for int8_linear
3146
+ [DEBUG] Backend cuda selected for int8_linear
3147
+ [DEBUG] Backend cuda selected for int8_linear
3148
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3149
+ [DEBUG] Backend cuda selected for int8_linear
3150
+ [DEBUG] Backend cuda selected for int8_linear
3151
+ [DEBUG] Backend cuda selected for int8_linear
3152
+ [DEBUG] Backend cuda selected for int8_linear
3153
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3154
+ [DEBUG] Backend cuda selected for int8_linear
3155
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3156
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3157
+ [DEBUG] Backend cuda selected for int8_linear
3158
+ [DEBUG] Backend cuda selected for int8_linear
3159
+ [DEBUG] Backend cuda selected for int8_linear
3160
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3161
+ [DEBUG] Backend cuda selected for int8_linear
3162
+ [DEBUG] Backend cuda selected for int8_linear
3163
+ [DEBUG] Backend cuda selected for int8_linear
3164
+ [DEBUG] Backend cuda selected for int8_linear
3165
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3166
+ [DEBUG] Backend cuda selected for int8_linear
3167
+ worker-comfyui - Websocket receive timed out. Still waiting...
3168
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3169
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3170
+ [DEBUG] Backend cuda selected for int8_linear
3171
+ [DEBUG] Backend cuda selected for int8_linear
3172
+ [DEBUG] Backend cuda selected for int8_linear
3173
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3174
+ [DEBUG] Backend cuda selected for int8_linear
3175
+ [DEBUG] Backend cuda selected for int8_linear
3176
+ [DEBUG] Backend cuda selected for int8_linear
3177
+ [DEBUG] Backend cuda selected for int8_linear
3178
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3179
+ [DEBUG] Backend cuda selected for int8_linear
3180
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3181
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3182
+ [DEBUG] Backend cuda selected for int8_linear
3183
+ [DEBUG] Backend cuda selected for int8_linear
3184
+ [DEBUG] Backend cuda selected for int8_linear
3185
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3186
+ [DEBUG] Backend cuda selected for int8_linear
3187
+ [DEBUG] Backend cuda selected for int8_linear
3188
+ [DEBUG] Backend cuda selected for int8_linear
3189
+ [DEBUG] Backend cuda selected for int8_linear
3190
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3191
+ [DEBUG] Backend cuda selected for int8_linear
3192
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3193
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3194
+ [DEBUG] Backend cuda selected for int8_linear
3195
+ [DEBUG] Backend cuda selected for int8_linear
3196
+ [DEBUG] Backend cuda selected for int8_linear
3197
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3198
+ [DEBUG] Backend cuda selected for int8_linear
3199
+ [DEBUG] Backend cuda selected for int8_linear
3200
+ [DEBUG] Backend cuda selected for int8_linear
3201
+ [DEBUG] Backend cuda selected for int8_linear
3202
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3203
+ [DEBUG] Backend cuda selected for int8_linear
3204
+ worker-comfyui - Websocket receive timed out. Still waiting...
3205
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3206
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3207
+ [DEBUG] Backend cuda selected for int8_linear
3208
+ [DEBUG] Backend cuda selected for int8_linear
3209
+ [DEBUG] Backend cuda selected for int8_linear
3210
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3211
+ [DEBUG] Backend cuda selected for int8_linear
3212
+ [DEBUG] Backend cuda selected for int8_linear
3213
+ [DEBUG] Backend cuda selected for int8_linear
3214
+ [DEBUG] Backend cuda selected for int8_linear
3215
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3216
+ [DEBUG] Backend cuda selected for int8_linear
3217
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3218
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3219
+ [DEBUG] Backend cuda selected for int8_linear
3220
+ [DEBUG] Backend cuda selected for int8_linear
3221
+ [DEBUG] Backend cuda selected for int8_linear
3222
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3223
+ [DEBUG] Backend cuda selected for int8_linear
3224
+ [DEBUG] Backend cuda selected for int8_linear
3225
+ [DEBUG] Backend cuda selected for int8_linear
3226
+ [DEBUG] Backend cuda selected for int8_linear
3227
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3228
+ [DEBUG] Backend cuda selected for int8_linear
3229
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3230
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3231
+ [DEBUG] Backend cuda selected for int8_linear
3232
+ [DEBUG] Backend cuda selected for int8_linear
3233
+ [DEBUG] Backend cuda selected for int8_linear
3234
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3235
+ [DEBUG] Backend cuda selected for int8_linear
3236
+ [DEBUG] Backend cuda selected for int8_linear
3237
+ [DEBUG] Backend cuda selected for int8_linear
3238
+ [DEBUG] Backend cuda selected for int8_linear
3239
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3240
+ [DEBUG] Backend cuda selected for int8_linear
3241
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3242
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3243
+ [DEBUG] Backend cuda selected for int8_linear
3244
+ [DEBUG] Backend cuda selected for int8_linear
3245
+ [DEBUG] Backend cuda selected for int8_linear
3246
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3247
+ [DEBUG] Backend cuda selected for int8_linear
3248
+ worker-comfyui - Websocket receive timed out. Still waiting...
3249
+ [DEBUG] Backend cuda selected for int8_linear
3250
+ [DEBUG] Backend cuda selected for int8_linear
3251
+ [DEBUG] Backend cuda selected for int8_linear
3252
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3253
+ [DEBUG] Backend cuda selected for int8_linear
3254
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3255
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3256
+ [DEBUG] Backend cuda selected for int8_linear
3257
+ [DEBUG] Backend cuda selected for int8_linear
3258
+ [DEBUG] Backend cuda selected for int8_linear
3259
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3260
+ [DEBUG] Backend cuda selected for int8_linear
3261
+ [DEBUG] Backend cuda selected for int8_linear
3262
+ [DEBUG] Backend cuda selected for int8_linear
3263
+ [DEBUG] Backend cuda selected for int8_linear
3264
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3265
+ [DEBUG] Backend cuda selected for int8_linear
3266
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3267
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3268
+ [DEBUG] Backend cuda selected for int8_linear
3269
+ [DEBUG] Backend cuda selected for int8_linear
3270
+ [DEBUG] Backend cuda selected for int8_linear
3271
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3272
+ [DEBUG] Backend cuda selected for int8_linear
3273
+ [DEBUG] Backend cuda selected for int8_linear
3274
+ [DEBUG] Backend cuda selected for int8_linear
3275
+ [DEBUG] Backend cuda selected for int8_linear
3276
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3277
+ [DEBUG] Backend cuda selected for int8_linear
3278
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3279
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3280
+ [DEBUG] Backend cuda selected for int8_linear
3281
+ [DEBUG] Backend cuda selected for int8_linear
3282
+ [DEBUG] Backend cuda selected for int8_linear
3283
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3284
+ [DEBUG] Backend cuda selected for int8_linear
3285
+ worker-comfyui - Websocket receive timed out. Still waiting...
3286
+ [DEBUG] Backend cuda selected for int8_linear
3287
+ [DEBUG] Backend cuda selected for int8_linear
3288
+ [DEBUG] Backend cuda selected for int8_linear
3289
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3290
+ [DEBUG] Backend cuda selected for int8_linear
3291
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3292
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3293
+ [DEBUG] Backend cuda selected for int8_linear
3294
+ [DEBUG] Backend cuda selected for int8_linear
3295
+ [DEBUG] Backend cuda selected for int8_linear
3296
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3297
+ [DEBUG] Backend cuda selected for int8_linear
3298
+ [DEBUG] Backend cuda selected for int8_linear
3299
+ [DEBUG] Backend cuda selected for int8_linear
3300
+ [DEBUG] Backend cuda selected for int8_linear
3301
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3302
+ [DEBUG] Backend cuda selected for int8_linear
3303
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3304
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3305
+ [DEBUG] Backend cuda selected for int8_linear
3306
+ [DEBUG] Backend cuda selected for int8_linear
3307
+ [DEBUG] Backend cuda selected for int8_linear
3308
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3309
+ [DEBUG] Backend cuda selected for int8_linear
3310
+ [DEBUG] Backend cuda selected for int8_linear
3311
+ [DEBUG] Backend cuda selected for int8_linear
3312
+ [DEBUG] Backend cuda selected for int8_linear
3313
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3314
+ [DEBUG] Backend cuda selected for int8_linear
3315
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3316
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3317
+ [DEBUG] Backend cuda selected for int8_linear
3318
+ [DEBUG] Backend cuda selected for int8_linear
3319
+ [DEBUG] Backend cuda selected for int8_linear
3320
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3321
+ [DEBUG] Backend cuda selected for int8_linear
3322
+ worker-comfyui - Websocket receive timed out. Still waiting...
3323
+ [DEBUG] Backend cuda selected for int8_linear
3324
+ [DEBUG] Backend cuda selected for int8_linear
3325
+ [DEBUG] Backend cuda selected for int8_linear
3326
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3327
+ [DEBUG] Backend cuda selected for int8_linear
3328
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3329
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3330
+ [DEBUG] Backend cuda selected for int8_linear
3331
+ [DEBUG] Backend cuda selected for int8_linear
3332
+ [DEBUG] Backend cuda selected for int8_linear
3333
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3334
+ [DEBUG] Backend cuda selected for int8_linear
3335
+ [DEBUG] Backend cuda selected for int8_linear
3336
+ [DEBUG] Backend cuda selected for int8_linear
3337
+ [DEBUG] Backend cuda selected for int8_linear
3338
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3339
+ [DEBUG] Backend cuda selected for int8_linear
3340
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3341
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3342
+ [DEBUG] Backend cuda selected for int8_linear
3343
+ [DEBUG] Backend cuda selected for int8_linear
3344
+ [DEBUG] Backend cuda selected for int8_linear
3345
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3346
+ [DEBUG] Backend cuda selected for int8_linear
3347
+ [DEBUG] Backend cuda selected for int8_linear
3348
+ [DEBUG] Backend cuda selected for int8_linear
3349
+ [DEBUG] Backend cuda selected for int8_linear
3350
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3351
+ [DEBUG] Backend cuda selected for int8_linear
3352
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3353
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3354
+ [DEBUG] Backend cuda selected for int8_linear
3355
+ [DEBUG] Backend cuda selected for int8_linear
3356
+ [DEBUG] Backend cuda selected for int8_linear
3357
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3358
+ [DEBUG] Backend cuda selected for int8_linear
3359
+ worker-comfyui - Websocket receive timed out. Still waiting...
3360
+ [DEBUG] Backend cuda selected for int8_linear
3361
+ [DEBUG] Backend cuda selected for int8_linear
3362
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=47294 MB
3363
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3364
+
3365
  40%|████ | 8/20 [10:41<16:03, 80.26s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
3366
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7d02f96a05f0: Actual Resident VRAM = 512 MB
3367
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cf4ab0e12b0: Actual Resident VRAM = 448 MB
3368
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cf476277d50: Actual Resident VRAM = 25888 MB
3369
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cfdee77f880: Actual Resident VRAM = 20000 MB
3370
+ [DEBUG] aimdo: /project/src/model-vbar.c:90:DEBUG:Total VRAM for VBARs: 46848 MB
3371
+ [DEBUG] Backend cuda selected for int8_linear
3372
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3373
+ [DEBUG] Backend cuda selected for int8_linear
3374
+ [DEBUG] Backend cuda selected for int8_linear
3375
+ [DEBUG] Backend cuda selected for int8_linear
3376
+ [DEBUG] Backend cuda selected for int8_linear
3377
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3378
+ [DEBUG] Backend cuda selected for int8_linear
3379
+ [DEBUG] Backend cuda selected for int8_linear
3380
+ [DEBUG] Backend cuda selected for int8_linear
3381
+ [DEBUG] Backend cuda selected for int8_linear
3382
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3383
+ [DEBUG] Backend cuda selected for int8_linear
3384
+ [DEBUG] Backend cuda selected for int8_linear
3385
+ [DEBUG] Backend cuda selected for int8_linear
3386
+ [DEBUG] Backend cuda selected for int8_linear
3387
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3388
+ [DEBUG] Backend cuda selected for int8_linear
3389
+ [DEBUG] Backend cuda selected for int8_linear
3390
+ [DEBUG] Backend cuda selected for int8_linear
3391
+ [DEBUG] Backend cuda selected for int8_linear
3392
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3393
+ [DEBUG] Backend cuda selected for int8_linear
3394
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3395
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3396
+ [DEBUG] Backend cuda selected for int8_linear
3397
+ [DEBUG] Backend cuda selected for int8_linear
3398
+ [DEBUG] Backend cuda selected for int8_linear
3399
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3400
+ [DEBUG] Backend cuda selected for int8_linear
3401
+ [DEBUG] Backend cuda selected for int8_linear
3402
+ [DEBUG] Backend cuda selected for int8_linear
3403
+ [DEBUG] Backend cuda selected for int8_linear
3404
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3405
+ [DEBUG] Backend cuda selected for int8_linear
3406
+ worker-comfyui - Websocket receive timed out. Still waiting...
3407
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3408
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3409
+ [DEBUG] Backend cuda selected for int8_linear
3410
+ [DEBUG] Backend cuda selected for int8_linear
3411
+ [DEBUG] Backend cuda selected for int8_linear
3412
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3413
+ [DEBUG] Backend cuda selected for int8_linear
3414
+ [DEBUG] Backend cuda selected for int8_linear
3415
+ [DEBUG] Backend cuda selected for int8_linear
3416
+ [DEBUG] Backend cuda selected for int8_linear
3417
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3418
+ [DEBUG] Backend cuda selected for int8_linear
3419
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3420
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3421
+ [DEBUG] Backend cuda selected for int8_linear
3422
+ [DEBUG] Backend cuda selected for int8_linear
3423
+ [DEBUG] Backend cuda selected for int8_linear
3424
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3425
+ [DEBUG] Backend cuda selected for int8_linear
3426
+ [DEBUG] Backend cuda selected for int8_linear
3427
+ [DEBUG] Backend cuda selected for int8_linear
3428
+ [DEBUG] Backend cuda selected for int8_linear
3429
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3430
+ [DEBUG] Backend cuda selected for int8_linear
3431
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3432
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3433
+ [DEBUG] Backend cuda selected for int8_linear
3434
+ [DEBUG] Backend cuda selected for int8_linear
3435
+ [DEBUG] Backend cuda selected for int8_linear
3436
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3437
+ [DEBUG] Backend cuda selected for int8_linear
3438
+ [DEBUG] Backend cuda selected for int8_linear
3439
+ [DEBUG] Backend cuda selected for int8_linear
3440
+ [DEBUG] Backend cuda selected for int8_linear
3441
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3442
+ [DEBUG] Backend cuda selected for int8_linear
3443
+ worker-comfyui - Websocket receive timed out. Still waiting...
3444
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3445
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3446
+ [DEBUG] Backend cuda selected for int8_linear
3447
+ [DEBUG] Backend cuda selected for int8_linear
3448
+ [DEBUG] Backend cuda selected for int8_linear
3449
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3450
+ [DEBUG] Backend cuda selected for int8_linear
3451
+ [DEBUG] Backend cuda selected for int8_linear
3452
+ [DEBUG] Backend cuda selected for int8_linear
3453
+ [DEBUG] Backend cuda selected for int8_linear
3454
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3455
+ [DEBUG] Backend cuda selected for int8_linear
3456
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3457
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3458
+ [DEBUG] Backend cuda selected for int8_linear
3459
+ [DEBUG] Backend cuda selected for int8_linear
3460
+ [DEBUG] Backend cuda selected for int8_linear
3461
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3462
+ [DEBUG] Backend cuda selected for int8_linear
3463
+ [DEBUG] Backend cuda selected for int8_linear
3464
+ [DEBUG] Backend cuda selected for int8_linear
3465
+ [DEBUG] Backend cuda selected for int8_linear
3466
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3467
+ [DEBUG] Backend cuda selected for int8_linear
3468
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3469
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3470
+ [DEBUG] Backend cuda selected for int8_linear
3471
+ [DEBUG] Backend cuda selected for int8_linear
3472
+ [DEBUG] Backend cuda selected for int8_linear
3473
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3474
+ [DEBUG] Backend cuda selected for int8_linear
3475
+ [DEBUG] Backend cuda selected for int8_linear
3476
+ [DEBUG] Backend cuda selected for int8_linear
3477
+ [DEBUG] Backend cuda selected for int8_linear
3478
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3479
+ [DEBUG] Backend cuda selected for int8_linear
3480
+ worker-comfyui - Websocket receive timed out. Still waiting...
3481
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3482
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3483
+ [DEBUG] Backend cuda selected for int8_linear
3484
+ [DEBUG] Backend cuda selected for int8_linear
3485
+ [DEBUG] Backend cuda selected for int8_linear
3486
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3487
+ [DEBUG] Backend cuda selected for int8_linear
3488
+ [DEBUG] Backend cuda selected for int8_linear
3489
+ [DEBUG] Backend cuda selected for int8_linear
3490
+ [DEBUG] Backend cuda selected for int8_linear
3491
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3492
+ [DEBUG] Backend cuda selected for int8_linear
3493
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3494
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3495
+ [DEBUG] Backend cuda selected for int8_linear
3496
+ [DEBUG] Backend cuda selected for int8_linear
3497
+ [DEBUG] Backend cuda selected for int8_linear
3498
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3499
+ [DEBUG] Backend cuda selected for int8_linear
3500
+ [DEBUG] Backend cuda selected for int8_linear
3501
+ [DEBUG] Backend cuda selected for int8_linear
3502
+ [DEBUG] Backend cuda selected for int8_linear
3503
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3504
+ [DEBUG] Backend cuda selected for int8_linear
3505
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3506
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3507
+ [DEBUG] Backend cuda selected for int8_linear
3508
+ [DEBUG] Backend cuda selected for int8_linear
3509
+ [DEBUG] Backend cuda selected for int8_linear
3510
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3511
+ [DEBUG] Backend cuda selected for int8_linear
3512
+ [DEBUG] Backend cuda selected for int8_linear
3513
+ [DEBUG] Backend cuda selected for int8_linear
3514
+ [DEBUG] Backend cuda selected for int8_linear
3515
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3516
+ [DEBUG] Backend cuda selected for int8_linear
3517
+ worker-comfyui - Websocket receive timed out. Still waiting...
3518
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3519
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3520
+ [DEBUG] Backend cuda selected for int8_linear
3521
+ [DEBUG] Backend cuda selected for int8_linear
3522
+ [DEBUG] Backend cuda selected for int8_linear
3523
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3524
+ [DEBUG] Backend cuda selected for int8_linear
3525
+ [DEBUG] Backend cuda selected for int8_linear
3526
+ [DEBUG] Backend cuda selected for int8_linear
3527
+ [DEBUG] Backend cuda selected for int8_linear
3528
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3529
+ [DEBUG] Backend cuda selected for int8_linear
3530
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3531
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3532
+ [DEBUG] Backend cuda selected for int8_linear
3533
+ [DEBUG] Backend cuda selected for int8_linear
3534
+ [DEBUG] Backend cuda selected for int8_linear
3535
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3536
+ [DEBUG] Backend cuda selected for int8_linear
3537
+ [DEBUG] Backend cuda selected for int8_linear
3538
+ [DEBUG] Backend cuda selected for int8_linear
3539
+ [DEBUG] Backend cuda selected for int8_linear
3540
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3541
+ [DEBUG] Backend cuda selected for int8_linear
3542
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3543
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3544
+ [DEBUG] Backend cuda selected for int8_linear
3545
+ [DEBUG] Backend cuda selected for int8_linear
3546
+ [DEBUG] Backend cuda selected for int8_linear
3547
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3548
+ [DEBUG] Backend cuda selected for int8_linear
3549
+ [DEBUG] Backend cuda selected for int8_linear
3550
+ [DEBUG] Backend cuda selected for int8_linear
3551
+ [DEBUG] Backend cuda selected for int8_linear
3552
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3553
+ [DEBUG] Backend cuda selected for int8_linear
3554
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3555
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3556
+ [DEBUG] Backend cuda selected for int8_linear
3557
+ [DEBUG] Backend cuda selected for int8_linear
3558
+ [DEBUG] Backend cuda selected for int8_linear
3559
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3560
+ [DEBUG] Backend cuda selected for int8_linear
3561
+ worker-comfyui - Websocket receive timed out. Still waiting...
3562
+ [DEBUG] Backend cuda selected for int8_linear
3563
+ [DEBUG] Backend cuda selected for int8_linear
3564
+ [DEBUG] Backend cuda selected for int8_linear
3565
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3566
+ [DEBUG] Backend cuda selected for int8_linear
3567
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3568
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3569
+ [DEBUG] Backend cuda selected for int8_linear
3570
+ [DEBUG] Backend cuda selected for int8_linear
3571
+ [DEBUG] Backend cuda selected for int8_linear
3572
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3573
+ [DEBUG] Backend cuda selected for int8_linear
3574
+ [DEBUG] Backend cuda selected for int8_linear
3575
+ [DEBUG] Backend cuda selected for int8_linear
3576
+ [DEBUG] Backend cuda selected for int8_linear
3577
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3578
+ [DEBUG] Backend cuda selected for int8_linear
3579
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3580
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3581
+ [DEBUG] Backend cuda selected for int8_linear
3582
+ [DEBUG] Backend cuda selected for int8_linear
3583
+ [DEBUG] Backend cuda selected for int8_linear
3584
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3585
+ [DEBUG] Backend cuda selected for int8_linear
3586
+ [DEBUG] Backend cuda selected for int8_linear
3587
+ [DEBUG] Backend cuda selected for int8_linear
3588
+ [DEBUG] Backend cuda selected for int8_linear
3589
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3590
+ [DEBUG] Backend cuda selected for int8_linear
3591
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3592
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3593
+ [DEBUG] Backend cuda selected for int8_linear
3594
+ [DEBUG] Backend cuda selected for int8_linear
3595
+ [DEBUG] Backend cuda selected for int8_linear
3596
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3597
+ [DEBUG] Backend cuda selected for int8_linear
3598
+ worker-comfyui - Websocket receive timed out. Still waiting...
3599
+ [DEBUG] Backend cuda selected for int8_linear
3600
+ [DEBUG] Backend cuda selected for int8_linear
3601
+ [DEBUG] Backend cuda selected for int8_linear
3602
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3603
+ [DEBUG] Backend cuda selected for int8_linear
3604
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3605
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3606
+ [DEBUG] Backend cuda selected for int8_linear
3607
+ [DEBUG] Backend cuda selected for int8_linear
3608
+ [DEBUG] Backend cuda selected for int8_linear
3609
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3610
+ [DEBUG] Backend cuda selected for int8_linear
3611
+ [DEBUG] Backend cuda selected for int8_linear
3612
+ [DEBUG] Backend cuda selected for int8_linear
3613
+ [DEBUG] Backend cuda selected for int8_linear
3614
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3615
+ [DEBUG] Backend cuda selected for int8_linear
3616
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3617
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3618
+ [DEBUG] Backend cuda selected for int8_linear
3619
+ [DEBUG] Backend cuda selected for int8_linear
3620
+ [DEBUG] Backend cuda selected for int8_linear
3621
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3622
+ [DEBUG] Backend cuda selected for int8_linear
3623
+ [DEBUG] Backend cuda selected for int8_linear
3624
+ [DEBUG] Backend cuda selected for int8_linear
3625
+ [DEBUG] Backend cuda selected for int8_linear
3626
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3627
+ [DEBUG] Backend cuda selected for int8_linear
3628
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3629
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3630
+ [DEBUG] Backend cuda selected for int8_linear
3631
+ [DEBUG] Backend cuda selected for int8_linear
3632
+ [DEBUG] Backend cuda selected for int8_linear
3633
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3634
+ [DEBUG] Backend cuda selected for int8_linear
3635
+ worker-comfyui - Websocket receive timed out. Still waiting...
3636
+ [DEBUG] Backend cuda selected for int8_linear
3637
+ [DEBUG] Backend cuda selected for int8_linear
3638
+ [DEBUG] Backend cuda selected for int8_linear
3639
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3640
+ [DEBUG] Backend cuda selected for int8_linear
3641
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3642
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3643
+ [DEBUG] Backend cuda selected for int8_linear
3644
+ [DEBUG] Backend cuda selected for int8_linear
3645
+ [DEBUG] Backend cuda selected for int8_linear
3646
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3647
+ [DEBUG] Backend cuda selected for int8_linear
3648
+ [DEBUG] Backend cuda selected for int8_linear
3649
+ [DEBUG] Backend cuda selected for int8_linear
3650
+ [DEBUG] Backend cuda selected for int8_linear
3651
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3652
+ [DEBUG] Backend cuda selected for int8_linear
3653
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3654
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3655
+ [DEBUG] Backend cuda selected for int8_linear
3656
+ [DEBUG] Backend cuda selected for int8_linear
3657
+ [DEBUG] Backend cuda selected for int8_linear
3658
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3659
+ [DEBUG] Backend cuda selected for int8_linear
3660
+ [DEBUG] Backend cuda selected for int8_linear
3661
+ [DEBUG] Backend cuda selected for int8_linear
3662
+ [DEBUG] Backend cuda selected for int8_linear
3663
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3664
+ [DEBUG] Backend cuda selected for int8_linear
3665
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3666
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3667
+ [DEBUG] Backend cuda selected for int8_linear
3668
+ [DEBUG] Backend cuda selected for int8_linear
3669
+ [DEBUG] Backend cuda selected for int8_linear
3670
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3671
+ [DEBUG] Backend cuda selected for int8_linear
3672
+ worker-comfyui - Websocket receive timed out. Still waiting...
3673
+ [DEBUG] Backend cuda selected for int8_linear
3674
+ [DEBUG] Backend cuda selected for int8_linear
3675
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=47294 MB
3676
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3677
+
3678
  45%|████▌ | 9/20 [12:01<14:41, 80.15s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
3679
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7d02f96a05f0: Actual Resident VRAM = 512 MB
3680
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cf4ab0e12b0: Actual Resident VRAM = 448 MB
3681
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cf476277d50: Actual Resident VRAM = 25888 MB
3682
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cfdee77f880: Actual Resident VRAM = 20000 MB
3683
+ [DEBUG] aimdo: /project/src/model-vbar.c:90:DEBUG:Total VRAM for VBARs: 46848 MB
3684
+ [DEBUG] Backend cuda selected for int8_linear
3685
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3686
+ [DEBUG] Backend cuda selected for int8_linear
3687
+ [DEBUG] Backend cuda selected for int8_linear
3688
+ [DEBUG] Backend cuda selected for int8_linear
3689
+ [DEBUG] Backend cuda selected for int8_linear
3690
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3691
+ [DEBUG] Backend cuda selected for int8_linear
3692
+ [DEBUG] Backend cuda selected for int8_linear
3693
+ [DEBUG] Backend cuda selected for int8_linear
3694
+ [DEBUG] Backend cuda selected for int8_linear
3695
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3696
+ [DEBUG] Backend cuda selected for int8_linear
3697
+ [DEBUG] Backend cuda selected for int8_linear
3698
+ [DEBUG] Backend cuda selected for int8_linear
3699
+ [DEBUG] Backend cuda selected for int8_linear
3700
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3701
+ [DEBUG] Backend cuda selected for int8_linear
3702
+ [DEBUG] Backend cuda selected for int8_linear
3703
+ [DEBUG] Backend cuda selected for int8_linear
3704
+ [DEBUG] Backend cuda selected for int8_linear
3705
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3706
+ [DEBUG] Backend cuda selected for int8_linear
3707
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3708
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3709
+ [DEBUG] Backend cuda selected for int8_linear
3710
+ [DEBUG] Backend cuda selected for int8_linear
3711
+ [DEBUG] Backend cuda selected for int8_linear
3712
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3713
+ [DEBUG] Backend cuda selected for int8_linear
3714
+ [DEBUG] Backend cuda selected for int8_linear
3715
+ [DEBUG] Backend cuda selected for int8_linear
3716
+ [DEBUG] Backend cuda selected for int8_linear
3717
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3718
+ [DEBUG] Backend cuda selected for int8_linear
3719
+ worker-comfyui - Websocket receive timed out. Still waiting...
3720
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3721
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3722
+ [DEBUG] Backend cuda selected for int8_linear
3723
+ [DEBUG] Backend cuda selected for int8_linear
3724
+ [DEBUG] Backend cuda selected for int8_linear
3725
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3726
+ [DEBUG] Backend cuda selected for int8_linear
3727
+ [DEBUG] Backend cuda selected for int8_linear
3728
+ [DEBUG] Backend cuda selected for int8_linear
3729
+ [DEBUG] Backend cuda selected for int8_linear
3730
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3731
+ [DEBUG] Backend cuda selected for int8_linear
3732
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3733
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3734
+ [DEBUG] Backend cuda selected for int8_linear
3735
+ [DEBUG] Backend cuda selected for int8_linear
3736
+ [DEBUG] Backend cuda selected for int8_linear
3737
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3738
+ [DEBUG] Backend cuda selected for int8_linear
3739
+ [DEBUG] Backend cuda selected for int8_linear
3740
+ [DEBUG] Backend cuda selected for int8_linear
3741
+ [DEBUG] Backend cuda selected for int8_linear
3742
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3743
+ [DEBUG] Backend cuda selected for int8_linear
3744
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3745
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3746
+ [DEBUG] Backend cuda selected for int8_linear
3747
+ [DEBUG] Backend cuda selected for int8_linear
3748
+ [DEBUG] Backend cuda selected for int8_linear
3749
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3750
+ [DEBUG] Backend cuda selected for int8_linear
3751
+ [DEBUG] Backend cuda selected for int8_linear
3752
+ [DEBUG] Backend cuda selected for int8_linear
3753
+ [DEBUG] Backend cuda selected for int8_linear
3754
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3755
+ [DEBUG] Backend cuda selected for int8_linear
3756
+ worker-comfyui - Websocket receive timed out. Still waiting...
3757
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3758
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3759
+ [DEBUG] Backend cuda selected for int8_linear
3760
+ [DEBUG] Backend cuda selected for int8_linear
3761
+ [DEBUG] Backend cuda selected for int8_linear
3762
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3763
+ [DEBUG] Backend cuda selected for int8_linear
3764
+ [DEBUG] Backend cuda selected for int8_linear
3765
+ [DEBUG] Backend cuda selected for int8_linear
3766
+ [DEBUG] Backend cuda selected for int8_linear
3767
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3768
+ [DEBUG] Backend cuda selected for int8_linear
3769
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3770
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3771
+ [DEBUG] Backend cuda selected for int8_linear
3772
+ [DEBUG] Backend cuda selected for int8_linear
3773
+ [DEBUG] Backend cuda selected for int8_linear
3774
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3775
+ [DEBUG] Backend cuda selected for int8_linear
3776
+ [DEBUG] Backend cuda selected for int8_linear
3777
+ [DEBUG] Backend cuda selected for int8_linear
3778
+ [DEBUG] Backend cuda selected for int8_linear
3779
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3780
+ [DEBUG] Backend cuda selected for int8_linear
3781
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3782
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3783
+ [DEBUG] Backend cuda selected for int8_linear
3784
+ [DEBUG] Backend cuda selected for int8_linear
3785
+ [DEBUG] Backend cuda selected for int8_linear
3786
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3787
+ [DEBUG] Backend cuda selected for int8_linear
3788
+ [DEBUG] Backend cuda selected for int8_linear
3789
+ [DEBUG] Backend cuda selected for int8_linear
3790
+ [DEBUG] Backend cuda selected for int8_linear
3791
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3792
+ [DEBUG] Backend cuda selected for int8_linear
3793
+ worker-comfyui - Websocket receive timed out. Still waiting...
3794
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3795
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3796
+ [DEBUG] Backend cuda selected for int8_linear
3797
+ [DEBUG] Backend cuda selected for int8_linear
3798
+ [DEBUG] Backend cuda selected for int8_linear
3799
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3800
+ [DEBUG] Backend cuda selected for int8_linear
3801
+ [DEBUG] Backend cuda selected for int8_linear
3802
+ [DEBUG] Backend cuda selected for int8_linear
3803
+ [DEBUG] Backend cuda selected for int8_linear
3804
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3805
+ [DEBUG] Backend cuda selected for int8_linear
3806
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3807
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3808
+ [DEBUG] Backend cuda selected for int8_linear
3809
+ [DEBUG] Backend cuda selected for int8_linear
3810
+ [DEBUG] Backend cuda selected for int8_linear
3811
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3812
+ [DEBUG] Backend cuda selected for int8_linear
3813
+ [DEBUG] Backend cuda selected for int8_linear
3814
+ [DEBUG] Backend cuda selected for int8_linear
3815
+ [DEBUG] Backend cuda selected for int8_linear
3816
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3817
+ [DEBUG] Backend cuda selected for int8_linear
3818
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3819
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3820
+ [DEBUG] Backend cuda selected for int8_linear
3821
+ [DEBUG] Backend cuda selected for int8_linear
3822
+ [DEBUG] Backend cuda selected for int8_linear
3823
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3824
+ [DEBUG] Backend cuda selected for int8_linear
3825
+ [DEBUG] Backend cuda selected for int8_linear
3826
+ [DEBUG] Backend cuda selected for int8_linear
3827
+ [DEBUG] Backend cuda selected for int8_linear
3828
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3829
+ [DEBUG] Backend cuda selected for int8_linear
3830
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3831
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3832
+ [DEBUG] Backend cuda selected for int8_linear
3833
+ [DEBUG] Backend cuda selected for int8_linear
3834
+ worker-comfyui - Websocket receive timed out. Still waiting...
3835
+ [DEBUG] Backend cuda selected for int8_linear
3836
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3837
+ [DEBUG] Backend cuda selected for int8_linear
3838
+ [DEBUG] Backend cuda selected for int8_linear
3839
+ [DEBUG] Backend cuda selected for int8_linear
3840
+ [DEBUG] Backend cuda selected for int8_linear
3841
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3842
+ [DEBUG] Backend cuda selected for int8_linear
3843
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3844
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3845
+ [DEBUG] Backend cuda selected for int8_linear
3846
+ [DEBUG] Backend cuda selected for int8_linear
3847
+ [DEBUG] Backend cuda selected for int8_linear
3848
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3849
+ [DEBUG] Backend cuda selected for int8_linear
3850
+ [DEBUG] Backend cuda selected for int8_linear
3851
+ [DEBUG] Backend cuda selected for int8_linear
3852
+ [DEBUG] Backend cuda selected for int8_linear
3853
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3854
+ [DEBUG] Backend cuda selected for int8_linear
3855
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3856
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3857
+ [DEBUG] Backend cuda selected for int8_linear
3858
+ [DEBUG] Backend cuda selected for int8_linear
3859
+ [DEBUG] Backend cuda selected for int8_linear
3860
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3861
+ [DEBUG] Backend cuda selected for int8_linear
3862
+ [DEBUG] Backend cuda selected for int8_linear
3863
+ [DEBUG] Backend cuda selected for int8_linear
3864
+ [DEBUG] Backend cuda selected for int8_linear
3865
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3866
+ [DEBUG] Backend cuda selected for int8_linear
3867
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3868
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3869
+ [DEBUG] Backend cuda selected for int8_linear
3870
+ [DEBUG] Backend cuda selected for int8_linear
3871
+ [DEBUG] Backend cuda selected for int8_linear
3872
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3873
+ [DEBUG] Backend cuda selected for int8_linear
3874
+ worker-comfyui - Websocket receive timed out. Still waiting...
3875
+ [DEBUG] Backend cuda selected for int8_linear
3876
+ [DEBUG] Backend cuda selected for int8_linear
3877
+ [DEBUG] Backend cuda selected for int8_linear
3878
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3879
+ [DEBUG] Backend cuda selected for int8_linear
3880
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3881
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3882
+ [DEBUG] Backend cuda selected for int8_linear
3883
+ [DEBUG] Backend cuda selected for int8_linear
3884
+ [DEBUG] Backend cuda selected for int8_linear
3885
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3886
+ [DEBUG] Backend cuda selected for int8_linear
3887
+ [DEBUG] Backend cuda selected for int8_linear
3888
+ [DEBUG] Backend cuda selected for int8_linear
3889
+ [DEBUG] Backend cuda selected for int8_linear
3890
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3891
+ [DEBUG] Backend cuda selected for int8_linear
3892
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3893
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3894
+ [DEBUG] Backend cuda selected for int8_linear
3895
+ [DEBUG] Backend cuda selected for int8_linear
3896
+ [DEBUG] Backend cuda selected for int8_linear
3897
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3898
+ [DEBUG] Backend cuda selected for int8_linear
3899
+ [DEBUG] Backend cuda selected for int8_linear
3900
+ [DEBUG] Backend cuda selected for int8_linear
3901
+ [DEBUG] Backend cuda selected for int8_linear
3902
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3903
+ [DEBUG] Backend cuda selected for int8_linear
3904
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3905
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3906
+ [DEBUG] Backend cuda selected for int8_linear
3907
+ [DEBUG] Backend cuda selected for int8_linear
3908
+ [DEBUG] Backend cuda selected for int8_linear
3909
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3910
+ [DEBUG] Backend cuda selected for int8_linear
3911
+ worker-comfyui - Websocket receive timed out. Still waiting...
3912
+ [DEBUG] Backend cuda selected for int8_linear
3913
+ [DEBUG] Backend cuda selected for int8_linear
3914
+ [DEBUG] Backend cuda selected for int8_linear
3915
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3916
+ [DEBUG] Backend cuda selected for int8_linear
3917
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3918
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3919
+ [DEBUG] Backend cuda selected for int8_linear
3920
+ [DEBUG] Backend cuda selected for int8_linear
3921
+ [DEBUG] Backend cuda selected for int8_linear
3922
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3923
+ [DEBUG] Backend cuda selected for int8_linear
3924
+ [DEBUG] Backend cuda selected for int8_linear
3925
+ [DEBUG] Backend cuda selected for int8_linear
3926
+ [DEBUG] Backend cuda selected for int8_linear
3927
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3928
+ [DEBUG] Backend cuda selected for int8_linear
3929
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3930
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3931
+ [DEBUG] Backend cuda selected for int8_linear
3932
+ [DEBUG] Backend cuda selected for int8_linear
3933
+ [DEBUG] Backend cuda selected for int8_linear
3934
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3935
+ [DEBUG] Backend cuda selected for int8_linear
3936
+ [DEBUG] Backend cuda selected for int8_linear
3937
+ [DEBUG] Backend cuda selected for int8_linear
3938
+ [DEBUG] Backend cuda selected for int8_linear
3939
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3940
+ [DEBUG] Backend cuda selected for int8_linear
3941
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3942
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3943
+ [DEBUG] Backend cuda selected for int8_linear
3944
+ [DEBUG] Backend cuda selected for int8_linear
3945
+ [DEBUG] Backend cuda selected for int8_linear
3946
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3947
+ [DEBUG] Backend cuda selected for int8_linear
3948
+ worker-comfyui - Websocket receive timed out. Still waiting...
3949
+ [DEBUG] Backend cuda selected for int8_linear
3950
+ [DEBUG] Backend cuda selected for int8_linear
3951
+ [DEBUG] Backend cuda selected for int8_linear
3952
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3953
+ [DEBUG] Backend cuda selected for int8_linear
3954
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3955
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3956
+ [DEBUG] Backend cuda selected for int8_linear
3957
+ [DEBUG] Backend cuda selected for int8_linear
3958
+ [DEBUG] Backend cuda selected for int8_linear
3959
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3960
+ [DEBUG] Backend cuda selected for int8_linear
3961
+ [DEBUG] Backend cuda selected for int8_linear
3962
+ [DEBUG] Backend cuda selected for int8_linear
3963
+ [DEBUG] Backend cuda selected for int8_linear
3964
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3965
+ [DEBUG] Backend cuda selected for int8_linear
3966
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3967
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3968
+ [DEBUG] Backend cuda selected for int8_linear
3969
+ [DEBUG] Backend cuda selected for int8_linear
3970
+ [DEBUG] Backend cuda selected for int8_linear
3971
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3972
+ [DEBUG] Backend cuda selected for int8_linear
3973
+ [DEBUG] Backend cuda selected for int8_linear
3974
+ [DEBUG] Backend cuda selected for int8_linear
3975
+ [DEBUG] Backend cuda selected for int8_linear
3976
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3977
+ [DEBUG] Backend cuda selected for int8_linear
3978
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3979
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3980
+ [DEBUG] Backend cuda selected for int8_linear
3981
+ [DEBUG] Backend cuda selected for int8_linear
3982
+ [DEBUG] Backend cuda selected for int8_linear
3983
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3984
+ [DEBUG] Backend cuda selected for int8_linear
3985
+ [DEBUG] Backend cuda selected for int8_linear
3986
+ [DEBUG] Backend cuda selected for int8_linear
3987
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=47294 MB
3988
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3989
+
3990
  50%|█████ | 10/20 [13:21<13:20, 80.04s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
3991
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7d02f96a05f0: Actual Resident VRAM = 512 MB
3992
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cf4ab0e12b0: Actual Resident VRAM = 448 MB
3993
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cf476277d50: Actual Resident VRAM = 25888 MB
3994
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cfdee77f880: Actual Resident VRAM = 20000 MB
3995
+ [DEBUG] aimdo: /project/src/model-vbar.c:90:DEBUG:Total VRAM for VBARs: 46848 MB
3996
+ [DEBUG] Backend cuda selected for int8_linear
3997
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3998
+ [DEBUG] Backend cuda selected for int8_linear
3999
+ [DEBUG] Backend cuda selected for int8_linear
4000
+ [DEBUG] Backend cuda selected for int8_linear
4001
+ [DEBUG] Backend cuda selected for int8_linear
4002
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4003
+ [DEBUG] Backend cuda selected for int8_linear
4004
+ [DEBUG] Backend cuda selected for int8_linear
4005
+ [DEBUG] Backend cuda selected for int8_linear
4006
+ [DEBUG] Backend cuda selected for int8_linear
4007
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4008
+ [DEBUG] Backend cuda selected for int8_linear
4009
+ [DEBUG] Backend cuda selected for int8_linear
4010
+ [DEBUG] Backend cuda selected for int8_linear
4011
+ [DEBUG] Backend cuda selected for int8_linear
4012
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4013
+ [DEBUG] Backend cuda selected for int8_linear
4014
+ [DEBUG] Backend cuda selected for int8_linear
4015
+ [DEBUG] Backend cuda selected for int8_linear
4016
+ [DEBUG] Backend cuda selected for int8_linear
4017
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4018
+ [DEBUG] Backend cuda selected for int8_linear
4019
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4020
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4021
+ [DEBUG] Backend cuda selected for int8_linear
4022
+ [DEBUG] Backend cuda selected for int8_linear
4023
+ [DEBUG] Backend cuda selected for int8_linear
4024
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4025
+ [DEBUG] Backend cuda selected for int8_linear
4026
+ [DEBUG] Backend cuda selected for int8_linear
4027
+ [DEBUG] Backend cuda selected for int8_linear
4028
+ [DEBUG] Backend cuda selected for int8_linear
4029
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4030
+ [DEBUG] Backend cuda selected for int8_linear
4031
+ worker-comfyui - Websocket receive timed out. Still waiting...
4032
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4033
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4034
+ [DEBUG] Backend cuda selected for int8_linear
4035
+ [DEBUG] Backend cuda selected for int8_linear
4036
+ [DEBUG] Backend cuda selected for int8_linear
4037
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4038
+ [DEBUG] Backend cuda selected for int8_linear
4039
+ [DEBUG] Backend cuda selected for int8_linear
4040
+ [DEBUG] Backend cuda selected for int8_linear
4041
+ [DEBUG] Backend cuda selected for int8_linear
4042
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4043
+ [DEBUG] Backend cuda selected for int8_linear
4044
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4045
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4046
+ [DEBUG] Backend cuda selected for int8_linear
4047
+ [DEBUG] Backend cuda selected for int8_linear
4048
+ [DEBUG] Backend cuda selected for int8_linear
4049
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4050
+ [DEBUG] Backend cuda selected for int8_linear
4051
+ [DEBUG] Backend cuda selected for int8_linear
4052
+ [DEBUG] Backend cuda selected for int8_linear
4053
+ [DEBUG] Backend cuda selected for int8_linear
4054
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4055
+ [DEBUG] Backend cuda selected for int8_linear
4056
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4057
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4058
+ [DEBUG] Backend cuda selected for int8_linear
4059
+ [DEBUG] Backend cuda selected for int8_linear
4060
+ [DEBUG] Backend cuda selected for int8_linear
4061
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4062
+ [DEBUG] Backend cuda selected for int8_linear
4063
+ [DEBUG] Backend cuda selected for int8_linear
4064
+ [DEBUG] Backend cuda selected for int8_linear
4065
+ [DEBUG] Backend cuda selected for int8_linear
4066
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4067
+ [DEBUG] Backend cuda selected for int8_linear
4068
+ worker-comfyui - Websocket receive timed out. Still waiting...
4069
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4070
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4071
+ [DEBUG] Backend cuda selected for int8_linear
4072
+ [DEBUG] Backend cuda selected for int8_linear
4073
+ [DEBUG] Backend cuda selected for int8_linear
4074
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4075
+ [DEBUG] Backend cuda selected for int8_linear
4076
+ [DEBUG] Backend cuda selected for int8_linear
4077
+ [DEBUG] Backend cuda selected for int8_linear
4078
+ [DEBUG] Backend cuda selected for int8_linear
4079
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4080
+ [DEBUG] Backend cuda selected for int8_linear
4081
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4082
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4083
+ [DEBUG] Backend cuda selected for int8_linear
4084
+ [DEBUG] Backend cuda selected for int8_linear
4085
+ [DEBUG] Backend cuda selected for int8_linear
4086
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4087
+ [DEBUG] Backend cuda selected for int8_linear
4088
+ [DEBUG] Backend cuda selected for int8_linear
4089
+ [DEBUG] Backend cuda selected for int8_linear
4090
+ [DEBUG] Backend cuda selected for int8_linear
4091
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4092
+ [DEBUG] Backend cuda selected for int8_linear
4093
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4094
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4095
+ [DEBUG] Backend cuda selected for int8_linear
4096
+ [DEBUG] Backend cuda selected for int8_linear
4097
+ [DEBUG] Backend cuda selected for int8_linear
4098
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4099
+ [DEBUG] Backend cuda selected for int8_linear
4100
+ [DEBUG] Backend cuda selected for int8_linear
4101
+ [DEBUG] Backend cuda selected for int8_linear
4102
+ [DEBUG] Backend cuda selected for int8_linear
4103
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4104
+ [DEBUG] Backend cuda selected for int8_linear
4105
+ worker-comfyui - Websocket receive timed out. Still waiting...
4106
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4107
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4108
+ [DEBUG] Backend cuda selected for int8_linear
4109
+ [DEBUG] Backend cuda selected for int8_linear
4110
+ [DEBUG] Backend cuda selected for int8_linear
4111
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4112
+ [DEBUG] Backend cuda selected for int8_linear
4113
+ [DEBUG] Backend cuda selected for int8_linear
4114
+ [DEBUG] Backend cuda selected for int8_linear
4115
+ [DEBUG] Backend cuda selected for int8_linear
4116
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4117
+ [DEBUG] Backend cuda selected for int8_linear
4118
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4119
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4120
+ [DEBUG] Backend cuda selected for int8_linear
4121
+ [DEBUG] Backend cuda selected for int8_linear
4122
+ [DEBUG] Backend cuda selected for int8_linear
4123
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4124
+ [DEBUG] Backend cuda selected for int8_linear
4125
+ [DEBUG] Backend cuda selected for int8_linear
4126
+ [DEBUG] Backend cuda selected for int8_linear
4127
+ [DEBUG] Backend cuda selected for int8_linear
4128
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4129
+ [DEBUG] Backend cuda selected for int8_linear
4130
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4131
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4132
+ [DEBUG] Backend cuda selected for int8_linear
4133
+ [DEBUG] Backend cuda selected for int8_linear
4134
+ [DEBUG] Backend cuda selected for int8_linear
4135
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4136
+ [DEBUG] Backend cuda selected for int8_linear
4137
+ [DEBUG] Backend cuda selected for int8_linear
4138
+ [DEBUG] Backend cuda selected for int8_linear
4139
+ [DEBUG] Backend cuda selected for int8_linear
4140
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4141
+ [DEBUG] Backend cuda selected for int8_linear
4142
+ worker-comfyui - Websocket receive timed out. Still waiting...
4143
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4144
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4145
+ [DEBUG] Backend cuda selected for int8_linear
4146
+ [DEBUG] Backend cuda selected for int8_linear
4147
+ [DEBUG] Backend cuda selected for int8_linear
4148
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4149
+ [DEBUG] Backend cuda selected for int8_linear
4150
+ [DEBUG] Backend cuda selected for int8_linear
4151
+ [DEBUG] Backend cuda selected for int8_linear
4152
+ [DEBUG] Backend cuda selected for int8_linear
4153
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4154
+ [DEBUG] Backend cuda selected for int8_linear
4155
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4156
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4157
+ [DEBUG] Backend cuda selected for int8_linear
4158
+ [DEBUG] Backend cuda selected for int8_linear
4159
+ [DEBUG] Backend cuda selected for int8_linear
4160
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4161
+ [DEBUG] Backend cuda selected for int8_linear
4162
+ [DEBUG] Backend cuda selected for int8_linear
4163
+ [DEBUG] Backend cuda selected for int8_linear
4164
+ [DEBUG] Backend cuda selected for int8_linear
4165
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4166
+ [DEBUG] Backend cuda selected for int8_linear
4167
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4168
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4169
+ [DEBUG] Backend cuda selected for int8_linear
4170
+ [DEBUG] Backend cuda selected for int8_linear
4171
+ [DEBUG] Backend cuda selected for int8_linear
4172
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4173
+ [DEBUG] Backend cuda selected for int8_linear
4174
+ [DEBUG] Backend cuda selected for int8_linear
4175
+ [DEBUG] Backend cuda selected for int8_linear
4176
+ [DEBUG] Backend cuda selected for int8_linear
4177
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4178
+ [DEBUG] Backend cuda selected for int8_linear
4179
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4180
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4181
+ [DEBUG] Backend cuda selected for int8_linear
4182
+ [DEBUG] Backend cuda selected for int8_linear
4183
+ [DEBUG] Backend cuda selected for int8_linear
4184
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4185
+ [DEBUG] Backend cuda selected for int8_linear
4186
+ worker-comfyui - Websocket receive timed out. Still waiting...
4187
+ [DEBUG] Backend cuda selected for int8_linear
4188
+ [DEBUG] Backend cuda selected for int8_linear
4189
+ [DEBUG] Backend cuda selected for int8_linear
4190
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4191
+ [DEBUG] Backend cuda selected for int8_linear
4192
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4193
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4194
+ [DEBUG] Backend cuda selected for int8_linear
4195
+ [DEBUG] Backend cuda selected for int8_linear
4196
+ [DEBUG] Backend cuda selected for int8_linear
4197
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4198
+ [DEBUG] Backend cuda selected for int8_linear
4199
+ [DEBUG] Backend cuda selected for int8_linear
4200
+ [DEBUG] Backend cuda selected for int8_linear
4201
+ [DEBUG] Backend cuda selected for int8_linear
4202
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4203
+ [DEBUG] Backend cuda selected for int8_linear
4204
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4205
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4206
+ [DEBUG] Backend cuda selected for int8_linear
4207
+ [DEBUG] Backend cuda selected for int8_linear
4208
+ [DEBUG] Backend cuda selected for int8_linear
4209
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4210
+ [DEBUG] Backend cuda selected for int8_linear
4211
+ [DEBUG] Backend cuda selected for int8_linear
4212
+ [DEBUG] Backend cuda selected for int8_linear
4213
+ [DEBUG] Backend cuda selected for int8_linear
4214
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4215
+ [DEBUG] Backend cuda selected for int8_linear
4216
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4217
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4218
+ [DEBUG] Backend cuda selected for int8_linear
4219
+ [DEBUG] Backend cuda selected for int8_linear
4220
+ [DEBUG] Backend cuda selected for int8_linear
4221
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4222
+ [DEBUG] Backend cuda selected for int8_linear
4223
+ worker-comfyui - Websocket receive timed out. Still waiting...
4224
+ [DEBUG] Backend cuda selected for int8_linear
4225
+ [DEBUG] Backend cuda selected for int8_linear
4226
+ [DEBUG] Backend cuda selected for int8_linear
4227
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4228
+ [DEBUG] Backend cuda selected for int8_linear
4229
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4230
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4231
+ [DEBUG] Backend cuda selected for int8_linear
4232
+ [DEBUG] Backend cuda selected for int8_linear
4233
+ [DEBUG] Backend cuda selected for int8_linear
4234
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4235
+ [DEBUG] Backend cuda selected for int8_linear
4236
+ [DEBUG] Backend cuda selected for int8_linear
4237
+ [DEBUG] Backend cuda selected for int8_linear
4238
+ [DEBUG] Backend cuda selected for int8_linear
4239
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4240
+ [DEBUG] Backend cuda selected for int8_linear
4241
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4242
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4243
+ [DEBUG] Backend cuda selected for int8_linear
4244
+ [DEBUG] Backend cuda selected for int8_linear
4245
+ [DEBUG] Backend cuda selected for int8_linear
4246
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4247
+ [DEBUG] Backend cuda selected for int8_linear
4248
+ [DEBUG] Backend cuda selected for int8_linear
4249
+ [DEBUG] Backend cuda selected for int8_linear
4250
+ [DEBUG] Backend cuda selected for int8_linear
4251
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4252
+ [DEBUG] Backend cuda selected for int8_linear
4253
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4254
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4255
+ [DEBUG] Backend cuda selected for int8_linear
4256
+ [DEBUG] Backend cuda selected for int8_linear
4257
+ [DEBUG] Backend cuda selected for int8_linear
4258
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4259
+ [DEBUG] Backend cuda selected for int8_linear
4260
+ worker-comfyui - Websocket receive timed out. Still waiting...
4261
+ [DEBUG] Backend cuda selected for int8_linear
4262
+ [DEBUG] Backend cuda selected for int8_linear
4263
+ [DEBUG] Backend cuda selected for int8_linear
4264
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4265
+ [DEBUG] Backend cuda selected for int8_linear
4266
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4267
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4268
+ [DEBUG] Backend cuda selected for int8_linear
4269
+ [DEBUG] Backend cuda selected for int8_linear
4270
+ [DEBUG] Backend cuda selected for int8_linear
4271
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4272
+ [DEBUG] Backend cuda selected for int8_linear
4273
+ [DEBUG] Backend cuda selected for int8_linear
4274
+ [DEBUG] Backend cuda selected for int8_linear
4275
+ [DEBUG] Backend cuda selected for int8_linear
4276
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4277
+ [DEBUG] Backend cuda selected for int8_linear
4278
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4279
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4280
+ [DEBUG] Backend cuda selected for int8_linear
4281
+ [DEBUG] Backend cuda selected for int8_linear
4282
+ [DEBUG] Backend cuda selected for int8_linear
4283
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4284
+ [DEBUG] Backend cuda selected for int8_linear
4285
+ [DEBUG] Backend cuda selected for int8_linear
4286
+ [DEBUG] Backend cuda selected for int8_linear
4287
+ [DEBUG] Backend cuda selected for int8_linear
4288
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4289
+ [DEBUG] Backend cuda selected for int8_linear
4290
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
4291
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
4292
+ [DEBUG] Backend cuda selected for int8_linear
4293
+ [DEBUG] Backend cuda selected for int8_linear
4294
+ [DEBUG] Backend cuda selected for int8_linear
4295
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
4296
+ [DEBUG] Backend cuda selected for int8_linear
4297
+ worker-comfyui - Websocket receive timed out. Still waiting...
4298
+ [DEBUG] Backend cuda selected for int8_linear
4299
+ [DEBUG] Backend cuda selected for int8_linear