nikdevs commited on
Commit
cc38278
·
verified ·
1 Parent(s): d50c57f

boot log irv8h7gfnhyzfd

Browse files
Files changed (1) hide show
  1. logs/boot-irv8h7gfnhyzfd.log +1153 -0
logs/boot-irv8h7gfnhyzfd.log CHANGED
@@ -2233,3 +2233,1156 @@ worker-comfyui - Websocket receive timed out. Still waiting...
2233
  [DEBUG] Backend cuda selected for int8_linear
2234
  [DEBUG] Backend cuda selected for int8_linear
2235
  [DEBUG] Backend cuda selected for int8_linear
 
 
 
 
2236
  15%|█▌ | 3/20 [04:08<23:25, 82.69s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2237
  20%|██ | 4/20 [05:30<22:02, 82.66s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2238
  25%|██▌ | 5/20 [06:53<20:39, 82.64s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2239
  30%|███ | 6/20 [08:16<19:16, 82.62s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2233
  [DEBUG] Backend cuda selected for int8_linear
2234
  [DEBUG] Backend cuda selected for int8_linear
2235
  [DEBUG] Backend cuda selected for int8_linear
2236
+ [minimax] boot log shipped
2237
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=47090 MB
2238
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2239
+
2240
  15%|█▌ | 3/20 [04:08<23:25, 82.69s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
2241
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x76234b183180: Actual Resident VRAM = 224 MB
2242
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x761e02feb1c0: Actual Resident VRAM = 448 MB
2243
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7614c1d62f50: Actual Resident VRAM = 25888 MB
2244
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x76255e390150: Actual Resident VRAM = 20000 MB
2245
+ [DEBUG] aimdo: /project/src/model-vbar.c:90:DEBUG:Total VRAM for VBARs: 46560 MB
2246
+ [DEBUG] Backend cuda selected for int8_linear
2247
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2248
+ [DEBUG] Backend cuda selected for int8_linear
2249
+ [DEBUG] Backend cuda selected for int8_linear
2250
+ [DEBUG] Backend cuda selected for int8_linear
2251
+ [DEBUG] Backend cuda selected for int8_linear
2252
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2253
+ [DEBUG] Backend cuda selected for int8_linear
2254
+ [DEBUG] Backend cuda selected for int8_linear
2255
+ [DEBUG] Backend cuda selected for int8_linear
2256
+ [DEBUG] Backend cuda selected for int8_linear
2257
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2258
+ [DEBUG] Backend cuda selected for int8_linear
2259
+ [DEBUG] Backend cuda selected for int8_linear
2260
+ [DEBUG] Backend cuda selected for int8_linear
2261
+ [DEBUG] Backend cuda selected for int8_linear
2262
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2263
+ [DEBUG] Backend cuda selected for int8_linear
2264
+ [DEBUG] Backend cuda selected for int8_linear
2265
+ [DEBUG] Backend cuda selected for int8_linear
2266
+ [DEBUG] Backend cuda selected for int8_linear
2267
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2268
+ [DEBUG] Backend cuda selected for int8_linear
2269
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2270
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2271
+ [DEBUG] Backend cuda selected for int8_linear
2272
+ [DEBUG] Backend cuda selected for int8_linear
2273
+ [DEBUG] Backend cuda selected for int8_linear
2274
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2275
+ [DEBUG] Backend cuda selected for int8_linear
2276
+ [DEBUG] Backend cuda selected for int8_linear
2277
+ [DEBUG] Backend cuda selected for int8_linear
2278
+ [DEBUG] Backend cuda selected for int8_linear
2279
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2280
+ [DEBUG] Backend cuda selected for int8_linear
2281
+ worker-comfyui - Websocket receive timed out. Still waiting...
2282
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2283
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2284
+ [DEBUG] Backend cuda selected for int8_linear
2285
+ [DEBUG] Backend cuda selected for int8_linear
2286
+ [DEBUG] Backend cuda selected for int8_linear
2287
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2288
+ [DEBUG] Backend cuda selected for int8_linear
2289
+ [DEBUG] Backend cuda selected for int8_linear
2290
+ [DEBUG] Backend cuda selected for int8_linear
2291
+ [DEBUG] Backend cuda selected for int8_linear
2292
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2293
+ [DEBUG] Backend cuda selected for int8_linear
2294
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2295
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2296
+ [DEBUG] Backend cuda selected for int8_linear
2297
+ [DEBUG] Backend cuda selected for int8_linear
2298
+ [DEBUG] Backend cuda selected for int8_linear
2299
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2300
+ [DEBUG] Backend cuda selected for int8_linear
2301
+ [DEBUG] Backend cuda selected for int8_linear
2302
+ [DEBUG] Backend cuda selected for int8_linear
2303
+ [DEBUG] Backend cuda selected for int8_linear
2304
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2305
+ [DEBUG] Backend cuda selected for int8_linear
2306
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2307
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2308
+ [DEBUG] Backend cuda selected for int8_linear
2309
+ [DEBUG] Backend cuda selected for int8_linear
2310
+ [DEBUG] Backend cuda selected for int8_linear
2311
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2312
+ [DEBUG] Backend cuda selected for int8_linear
2313
+ [DEBUG] Backend cuda selected for int8_linear
2314
+ [DEBUG] Backend cuda selected for int8_linear
2315
+ [DEBUG] Backend cuda selected for int8_linear
2316
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2317
+ [DEBUG] Backend cuda selected for int8_linear
2318
+ worker-comfyui - Websocket receive timed out. Still waiting...
2319
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2320
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2321
+ [DEBUG] Backend cuda selected for int8_linear
2322
+ [DEBUG] Backend cuda selected for int8_linear
2323
+ [DEBUG] Backend cuda selected for int8_linear
2324
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2325
+ [DEBUG] Backend cuda selected for int8_linear
2326
+ [DEBUG] Backend cuda selected for int8_linear
2327
+ [DEBUG] Backend cuda selected for int8_linear
2328
+ [DEBUG] Backend cuda selected for int8_linear
2329
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2330
+ [DEBUG] Backend cuda selected for int8_linear
2331
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2332
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2333
+ [DEBUG] Backend cuda selected for int8_linear
2334
+ [DEBUG] Backend cuda selected for int8_linear
2335
+ [DEBUG] Backend cuda selected for int8_linear
2336
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2337
+ [DEBUG] Backend cuda selected for int8_linear
2338
+ [DEBUG] Backend cuda selected for int8_linear
2339
+ [DEBUG] Backend cuda selected for int8_linear
2340
+ [DEBUG] Backend cuda selected for int8_linear
2341
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2342
+ [DEBUG] Backend cuda selected for int8_linear
2343
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2344
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2345
+ [DEBUG] Backend cuda selected for int8_linear
2346
+ [DEBUG] Backend cuda selected for int8_linear
2347
+ [DEBUG] Backend cuda selected for int8_linear
2348
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2349
+ [DEBUG] Backend cuda selected for int8_linear
2350
+ [DEBUG] Backend cuda selected for int8_linear
2351
+ [DEBUG] Backend cuda selected for int8_linear
2352
+ [DEBUG] Backend cuda selected for int8_linear
2353
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2354
+ [DEBUG] Backend cuda selected for int8_linear
2355
+ worker-comfyui - Websocket receive timed out. Still waiting...
2356
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2357
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2358
+ [DEBUG] Backend cuda selected for int8_linear
2359
+ [DEBUG] Backend cuda selected for int8_linear
2360
+ [DEBUG] Backend cuda selected for int8_linear
2361
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2362
+ [DEBUG] Backend cuda selected for int8_linear
2363
+ [DEBUG] Backend cuda selected for int8_linear
2364
+ [DEBUG] Backend cuda selected for int8_linear
2365
+ [DEBUG] Backend cuda selected for int8_linear
2366
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2367
+ [DEBUG] Backend cuda selected for int8_linear
2368
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2369
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2370
+ [DEBUG] Backend cuda selected for int8_linear
2371
+ [DEBUG] Backend cuda selected for int8_linear
2372
+ [DEBUG] Backend cuda selected for int8_linear
2373
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2374
+ [DEBUG] Backend cuda selected for int8_linear
2375
+ [DEBUG] Backend cuda selected for int8_linear
2376
+ [DEBUG] Backend cuda selected for int8_linear
2377
+ [DEBUG] Backend cuda selected for int8_linear
2378
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2379
+ [DEBUG] Backend cuda selected for int8_linear
2380
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2381
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2382
+ [DEBUG] Backend cuda selected for int8_linear
2383
+ [DEBUG] Backend cuda selected for int8_linear
2384
+ [DEBUG] Backend cuda selected for int8_linear
2385
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2386
+ [DEBUG] Backend cuda selected for int8_linear
2387
+ [DEBUG] Backend cuda selected for int8_linear
2388
+ [DEBUG] Backend cuda selected for int8_linear
2389
+ [DEBUG] Backend cuda selected for int8_linear
2390
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2391
+ [DEBUG] Backend cuda selected for int8_linear
2392
+ worker-comfyui - Websocket receive timed out. Still waiting...
2393
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2394
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2395
+ [DEBUG] Backend cuda selected for int8_linear
2396
+ [DEBUG] Backend cuda selected for int8_linear
2397
+ [DEBUG] Backend cuda selected for int8_linear
2398
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2399
+ [DEBUG] Backend cuda selected for int8_linear
2400
+ [DEBUG] Backend cuda selected for int8_linear
2401
+ [DEBUG] Backend cuda selected for int8_linear
2402
+ [DEBUG] Backend cuda selected for int8_linear
2403
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2404
+ [DEBUG] Backend cuda selected for int8_linear
2405
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2406
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2407
+ [DEBUG] Backend cuda selected for int8_linear
2408
+ [DEBUG] Backend cuda selected for int8_linear
2409
+ [DEBUG] Backend cuda selected for int8_linear
2410
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2411
+ [DEBUG] Backend cuda selected for int8_linear
2412
+ [DEBUG] Backend cuda selected for int8_linear
2413
+ [DEBUG] Backend cuda selected for int8_linear
2414
+ [DEBUG] Backend cuda selected for int8_linear
2415
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2416
+ [DEBUG] Backend cuda selected for int8_linear
2417
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2418
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2419
+ [DEBUG] Backend cuda selected for int8_linear
2420
+ [DEBUG] Backend cuda selected for int8_linear
2421
+ [DEBUG] Backend cuda selected for int8_linear
2422
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2423
+ [DEBUG] Backend cuda selected for int8_linear
2424
+ [DEBUG] Backend cuda selected for int8_linear
2425
+ [DEBUG] Backend cuda selected for int8_linear
2426
+ [DEBUG] Backend cuda selected for int8_linear
2427
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2428
+ [DEBUG] Backend cuda selected for int8_linear
2429
+ worker-comfyui - Websocket receive timed out. Still waiting...
2430
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2431
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2432
+ [DEBUG] Backend cuda selected for int8_linear
2433
+ [DEBUG] Backend cuda selected for int8_linear
2434
+ [DEBUG] Backend cuda selected for int8_linear
2435
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2436
+ [DEBUG] Backend cuda selected for int8_linear
2437
+ [DEBUG] Backend cuda selected for int8_linear
2438
+ [DEBUG] Backend cuda selected for int8_linear
2439
+ [DEBUG] Backend cuda selected for int8_linear
2440
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2441
+ [DEBUG] Backend cuda selected for int8_linear
2442
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2443
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2444
+ [DEBUG] Backend cuda selected for int8_linear
2445
+ [DEBUG] Backend cuda selected for int8_linear
2446
+ [DEBUG] Backend cuda selected for int8_linear
2447
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2448
+ [DEBUG] Backend cuda selected for int8_linear
2449
+ [DEBUG] Backend cuda selected for int8_linear
2450
+ [DEBUG] Backend cuda selected for int8_linear
2451
+ [DEBUG] Backend cuda selected for int8_linear
2452
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2453
+ [DEBUG] Backend cuda selected for int8_linear
2454
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2455
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2456
+ [DEBUG] Backend cuda selected for int8_linear
2457
+ [DEBUG] Backend cuda selected for int8_linear
2458
+ [DEBUG] Backend cuda selected for int8_linear
2459
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2460
+ [DEBUG] Backend cuda selected for int8_linear
2461
+ [DEBUG] Backend cuda selected for int8_linear
2462
+ [DEBUG] Backend cuda selected for int8_linear
2463
+ [DEBUG] Backend cuda selected for int8_linear
2464
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2465
+ [DEBUG] Backend cuda selected for int8_linear
2466
+ worker-comfyui - Websocket receive timed out. Still waiting...
2467
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2468
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2469
+ [DEBUG] Backend cuda selected for int8_linear
2470
+ [DEBUG] Backend cuda selected for int8_linear
2471
+ [DEBUG] Backend cuda selected for int8_linear
2472
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2473
+ [DEBUG] Backend cuda selected for int8_linear
2474
+ [DEBUG] Backend cuda selected for int8_linear
2475
+ [DEBUG] Backend cuda selected for int8_linear
2476
+ [DEBUG] Backend cuda selected for int8_linear
2477
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2478
+ [DEBUG] Backend cuda selected for int8_linear
2479
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2480
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2481
+ [DEBUG] Backend cuda selected for int8_linear
2482
+ [DEBUG] Backend cuda selected for int8_linear
2483
+ [DEBUG] Backend cuda selected for int8_linear
2484
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2485
+ [DEBUG] Backend cuda selected for int8_linear
2486
+ [DEBUG] Backend cuda selected for int8_linear
2487
+ [DEBUG] Backend cuda selected for int8_linear
2488
+ [DEBUG] Backend cuda selected for int8_linear
2489
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2490
+ [DEBUG] Backend cuda selected for int8_linear
2491
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2492
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2493
+ [DEBUG] Backend cuda selected for int8_linear
2494
+ [DEBUG] Backend cuda selected for int8_linear
2495
+ [DEBUG] Backend cuda selected for int8_linear
2496
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2497
+ [DEBUG] Backend cuda selected for int8_linear
2498
+ [DEBUG] Backend cuda selected for int8_linear
2499
+ [DEBUG] Backend cuda selected for int8_linear
2500
+ [DEBUG] Backend cuda selected for int8_linear
2501
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2502
+ [DEBUG] Backend cuda selected for int8_linear
2503
+ worker-comfyui - Websocket receive timed out. Still waiting...
2504
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2505
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2506
+ [DEBUG] Backend cuda selected for int8_linear
2507
+ [DEBUG] Backend cuda selected for int8_linear
2508
+ [DEBUG] Backend cuda selected for int8_linear
2509
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2510
+ [DEBUG] Backend cuda selected for int8_linear
2511
+ [DEBUG] Backend cuda selected for int8_linear
2512
+ [DEBUG] Backend cuda selected for int8_linear
2513
+ [DEBUG] Backend cuda selected for int8_linear
2514
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2515
+ [DEBUG] Backend cuda selected for int8_linear
2516
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2517
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2518
+ [DEBUG] Backend cuda selected for int8_linear
2519
+ [DEBUG] Backend cuda selected for int8_linear
2520
+ [DEBUG] Backend cuda selected for int8_linear
2521
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2522
+ [DEBUG] Backend cuda selected for int8_linear
2523
+ [DEBUG] Backend cuda selected for int8_linear
2524
+ [DEBUG] Backend cuda selected for int8_linear
2525
+ [DEBUG] Backend cuda selected for int8_linear
2526
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2527
+ [DEBUG] Backend cuda selected for int8_linear
2528
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2529
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2530
+ [DEBUG] Backend cuda selected for int8_linear
2531
+ [DEBUG] Backend cuda selected for int8_linear
2532
+ [DEBUG] Backend cuda selected for int8_linear
2533
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2534
+ [DEBUG] Backend cuda selected for int8_linear
2535
+ [DEBUG] Backend cuda selected for int8_linear
2536
+ [DEBUG] Backend cuda selected for int8_linear
2537
+ [DEBUG] Backend cuda selected for int8_linear
2538
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2539
+ [DEBUG] Backend cuda selected for int8_linear
2540
+ worker-comfyui - Websocket receive timed out. Still waiting...
2541
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2542
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2543
+ [DEBUG] Backend cuda selected for int8_linear
2544
+ [DEBUG] Backend cuda selected for int8_linear
2545
+ [DEBUG] Backend cuda selected for int8_linear
2546
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2547
+ [DEBUG] Backend cuda selected for int8_linear
2548
+ [DEBUG] Backend cuda selected for int8_linear
2549
+ [DEBUG] Backend cuda selected for int8_linear
2550
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=47090 MB
2551
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2552
+
2553
  20%|██ | 4/20 [05:30<22:02, 82.66s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
2554
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x76234b183180: Actual Resident VRAM = 224 MB
2555
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x761e02feb1c0: Actual Resident VRAM = 448 MB
2556
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7614c1d62f50: Actual Resident VRAM = 25888 MB
2557
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x76255e390150: Actual Resident VRAM = 20000 MB
2558
+ [DEBUG] aimdo: /project/src/model-vbar.c:90:DEBUG:Total VRAM for VBARs: 46560 MB
2559
+ [DEBUG] Backend cuda selected for int8_linear
2560
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2561
+ [DEBUG] Backend cuda selected for int8_linear
2562
+ [DEBUG] Backend cuda selected for int8_linear
2563
+ [DEBUG] Backend cuda selected for int8_linear
2564
+ [DEBUG] Backend cuda selected for int8_linear
2565
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2566
+ [DEBUG] Backend cuda selected for int8_linear
2567
+ [DEBUG] Backend cuda selected for int8_linear
2568
+ [DEBUG] Backend cuda selected for int8_linear
2569
+ [DEBUG] Backend cuda selected for int8_linear
2570
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2571
+ [DEBUG] Backend cuda selected for int8_linear
2572
+ [DEBUG] Backend cuda selected for int8_linear
2573
+ [DEBUG] Backend cuda selected for int8_linear
2574
+ [DEBUG] Backend cuda selected for int8_linear
2575
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2576
+ [DEBUG] Backend cuda selected for int8_linear
2577
+ [DEBUG] Backend cuda selected for int8_linear
2578
+ [DEBUG] Backend cuda selected for int8_linear
2579
+ [DEBUG] Backend cuda selected for int8_linear
2580
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2581
+ [DEBUG] Backend cuda selected for int8_linear
2582
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2583
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2584
+ [DEBUG] Backend cuda selected for int8_linear
2585
+ [DEBUG] Backend cuda selected for int8_linear
2586
+ [DEBUG] Backend cuda selected for int8_linear
2587
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2588
+ [DEBUG] Backend cuda selected for int8_linear
2589
+ [DEBUG] Backend cuda selected for int8_linear
2590
+ [DEBUG] Backend cuda selected for int8_linear
2591
+ [DEBUG] Backend cuda selected for int8_linear
2592
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2593
+ [DEBUG] Backend cuda selected for int8_linear
2594
+ worker-comfyui - Websocket receive timed out. Still waiting...
2595
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2596
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2597
+ [DEBUG] Backend cuda selected for int8_linear
2598
+ [DEBUG] Backend cuda selected for int8_linear
2599
+ [DEBUG] Backend cuda selected for int8_linear
2600
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2601
+ [DEBUG] Backend cuda selected for int8_linear
2602
+ [DEBUG] Backend cuda selected for int8_linear
2603
+ [DEBUG] Backend cuda selected for int8_linear
2604
+ [DEBUG] Backend cuda selected for int8_linear
2605
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2606
+ [DEBUG] Backend cuda selected for int8_linear
2607
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2608
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2609
+ [DEBUG] Backend cuda selected for int8_linear
2610
+ [DEBUG] Backend cuda selected for int8_linear
2611
+ [DEBUG] Backend cuda selected for int8_linear
2612
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2613
+ [DEBUG] Backend cuda selected for int8_linear
2614
+ [DEBUG] Backend cuda selected for int8_linear
2615
+ [DEBUG] Backend cuda selected for int8_linear
2616
+ [DEBUG] Backend cuda selected for int8_linear
2617
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2618
+ [DEBUG] Backend cuda selected for int8_linear
2619
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2620
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2621
+ [DEBUG] Backend cuda selected for int8_linear
2622
+ [DEBUG] Backend cuda selected for int8_linear
2623
+ [DEBUG] Backend cuda selected for int8_linear
2624
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2625
+ [DEBUG] Backend cuda selected for int8_linear
2626
+ [DEBUG] Backend cuda selected for int8_linear
2627
+ [DEBUG] Backend cuda selected for int8_linear
2628
+ [DEBUG] Backend cuda selected for int8_linear
2629
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2630
+ [DEBUG] Backend cuda selected for int8_linear
2631
+ worker-comfyui - Websocket receive timed out. Still waiting...
2632
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2633
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2634
+ [DEBUG] Backend cuda selected for int8_linear
2635
+ [DEBUG] Backend cuda selected for int8_linear
2636
+ [DEBUG] Backend cuda selected for int8_linear
2637
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2638
+ [DEBUG] Backend cuda selected for int8_linear
2639
+ [DEBUG] Backend cuda selected for int8_linear
2640
+ [DEBUG] Backend cuda selected for int8_linear
2641
+ [DEBUG] Backend cuda selected for int8_linear
2642
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2643
+ [DEBUG] Backend cuda selected for int8_linear
2644
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2645
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2646
+ [DEBUG] Backend cuda selected for int8_linear
2647
+ [DEBUG] Backend cuda selected for int8_linear
2648
+ [DEBUG] Backend cuda selected for int8_linear
2649
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2650
+ [DEBUG] Backend cuda selected for int8_linear
2651
+ [DEBUG] Backend cuda selected for int8_linear
2652
+ [DEBUG] Backend cuda selected for int8_linear
2653
+ [DEBUG] Backend cuda selected for int8_linear
2654
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2655
+ [DEBUG] Backend cuda selected for int8_linear
2656
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2657
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2658
+ [DEBUG] Backend cuda selected for int8_linear
2659
+ [DEBUG] Backend cuda selected for int8_linear
2660
+ [DEBUG] Backend cuda selected for int8_linear
2661
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2662
+ [DEBUG] Backend cuda selected for int8_linear
2663
+ [DEBUG] Backend cuda selected for int8_linear
2664
+ [DEBUG] Backend cuda selected for int8_linear
2665
+ [DEBUG] Backend cuda selected for int8_linear
2666
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2667
+ [DEBUG] Backend cuda selected for int8_linear
2668
+ worker-comfyui - Websocket receive timed out. Still waiting...
2669
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2670
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2671
+ [DEBUG] Backend cuda selected for int8_linear
2672
+ [DEBUG] Backend cuda selected for int8_linear
2673
+ [DEBUG] Backend cuda selected for int8_linear
2674
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2675
+ [DEBUG] Backend cuda selected for int8_linear
2676
+ [DEBUG] Backend cuda selected for int8_linear
2677
+ [DEBUG] Backend cuda selected for int8_linear
2678
+ [DEBUG] Backend cuda selected for int8_linear
2679
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2680
+ [DEBUG] Backend cuda selected for int8_linear
2681
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2682
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2683
+ [DEBUG] Backend cuda selected for int8_linear
2684
+ [DEBUG] Backend cuda selected for int8_linear
2685
+ [DEBUG] Backend cuda selected for int8_linear
2686
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2687
+ [DEBUG] Backend cuda selected for int8_linear
2688
+ [DEBUG] Backend cuda selected for int8_linear
2689
+ [DEBUG] Backend cuda selected for int8_linear
2690
+ [DEBUG] Backend cuda selected for int8_linear
2691
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2692
+ [DEBUG] Backend cuda selected for int8_linear
2693
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2694
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2695
+ [DEBUG] Backend cuda selected for int8_linear
2696
+ [DEBUG] Backend cuda selected for int8_linear
2697
+ [DEBUG] Backend cuda selected for int8_linear
2698
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2699
+ [DEBUG] Backend cuda selected for int8_linear
2700
+ [DEBUG] Backend cuda selected for int8_linear
2701
+ [DEBUG] Backend cuda selected for int8_linear
2702
+ [DEBUG] Backend cuda selected for int8_linear
2703
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2704
+ [DEBUG] Backend cuda selected for int8_linear
2705
+ worker-comfyui - Websocket receive timed out. Still waiting...
2706
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2707
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2708
+ [DEBUG] Backend cuda selected for int8_linear
2709
+ [DEBUG] Backend cuda selected for int8_linear
2710
+ [DEBUG] Backend cuda selected for int8_linear
2711
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2712
+ [DEBUG] Backend cuda selected for int8_linear
2713
+ [DEBUG] Backend cuda selected for int8_linear
2714
+ [DEBUG] Backend cuda selected for int8_linear
2715
+ [DEBUG] Backend cuda selected for int8_linear
2716
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2717
+ [DEBUG] Backend cuda selected for int8_linear
2718
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2719
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2720
+ [DEBUG] Backend cuda selected for int8_linear
2721
+ [DEBUG] Backend cuda selected for int8_linear
2722
+ [DEBUG] Backend cuda selected for int8_linear
2723
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2724
+ [DEBUG] Backend cuda selected for int8_linear
2725
+ [DEBUG] Backend cuda selected for int8_linear
2726
+ [DEBUG] Backend cuda selected for int8_linear
2727
+ [DEBUG] Backend cuda selected for int8_linear
2728
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2729
+ [DEBUG] Backend cuda selected for int8_linear
2730
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2731
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2732
+ [DEBUG] Backend cuda selected for int8_linear
2733
+ [DEBUG] Backend cuda selected for int8_linear
2734
+ [DEBUG] Backend cuda selected for int8_linear
2735
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2736
+ [DEBUG] Backend cuda selected for int8_linear
2737
+ [DEBUG] Backend cuda selected for int8_linear
2738
+ [DEBUG] Backend cuda selected for int8_linear
2739
+ [DEBUG] Backend cuda selected for int8_linear
2740
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2741
+ [DEBUG] Backend cuda selected for int8_linear
2742
+ worker-comfyui - Websocket receive timed out. Still waiting...
2743
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2744
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2745
+ [DEBUG] Backend cuda selected for int8_linear
2746
+ [DEBUG] Backend cuda selected for int8_linear
2747
+ [DEBUG] Backend cuda selected for int8_linear
2748
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2749
+ [DEBUG] Backend cuda selected for int8_linear
2750
+ [DEBUG] Backend cuda selected for int8_linear
2751
+ [DEBUG] Backend cuda selected for int8_linear
2752
+ [DEBUG] Backend cuda selected for int8_linear
2753
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2754
+ [DEBUG] Backend cuda selected for int8_linear
2755
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2756
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2757
+ [DEBUG] Backend cuda selected for int8_linear
2758
+ [DEBUG] Backend cuda selected for int8_linear
2759
+ [DEBUG] Backend cuda selected for int8_linear
2760
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2761
+ [DEBUG] Backend cuda selected for int8_linear
2762
+ [DEBUG] Backend cuda selected for int8_linear
2763
+ [DEBUG] Backend cuda selected for int8_linear
2764
+ [DEBUG] Backend cuda selected for int8_linear
2765
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2766
+ [DEBUG] Backend cuda selected for int8_linear
2767
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2768
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2769
+ [DEBUG] Backend cuda selected for int8_linear
2770
+ [DEBUG] Backend cuda selected for int8_linear
2771
+ [DEBUG] Backend cuda selected for int8_linear
2772
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2773
+ [DEBUG] Backend cuda selected for int8_linear
2774
+ [DEBUG] Backend cuda selected for int8_linear
2775
+ [DEBUG] Backend cuda selected for int8_linear
2776
+ [DEBUG] Backend cuda selected for int8_linear
2777
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2778
+ [DEBUG] Backend cuda selected for int8_linear
2779
+ worker-comfyui - Websocket receive timed out. Still waiting...
2780
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2781
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2782
+ [DEBUG] Backend cuda selected for int8_linear
2783
+ [DEBUG] Backend cuda selected for int8_linear
2784
+ [DEBUG] Backend cuda selected for int8_linear
2785
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2786
+ [DEBUG] Backend cuda selected for int8_linear
2787
+ [DEBUG] Backend cuda selected for int8_linear
2788
+ [DEBUG] Backend cuda selected for int8_linear
2789
+ [DEBUG] Backend cuda selected for int8_linear
2790
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2791
+ [DEBUG] Backend cuda selected for int8_linear
2792
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2793
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2794
+ [DEBUG] Backend cuda selected for int8_linear
2795
+ [DEBUG] Backend cuda selected for int8_linear
2796
+ [DEBUG] Backend cuda selected for int8_linear
2797
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2798
+ [DEBUG] Backend cuda selected for int8_linear
2799
+ [DEBUG] Backend cuda selected for int8_linear
2800
+ [DEBUG] Backend cuda selected for int8_linear
2801
+ [DEBUG] Backend cuda selected for int8_linear
2802
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2803
+ [DEBUG] Backend cuda selected for int8_linear
2804
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2805
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2806
+ [DEBUG] Backend cuda selected for int8_linear
2807
+ [DEBUG] Backend cuda selected for int8_linear
2808
+ [DEBUG] Backend cuda selected for int8_linear
2809
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2810
+ [DEBUG] Backend cuda selected for int8_linear
2811
+ [DEBUG] Backend cuda selected for int8_linear
2812
+ [DEBUG] Backend cuda selected for int8_linear
2813
+ [DEBUG] Backend cuda selected for int8_linear
2814
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2815
+ [DEBUG] Backend cuda selected for int8_linear
2816
+ worker-comfyui - Websocket receive timed out. Still waiting...
2817
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2818
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2819
+ [DEBUG] Backend cuda selected for int8_linear
2820
+ [DEBUG] Backend cuda selected for int8_linear
2821
+ [DEBUG] Backend cuda selected for int8_linear
2822
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2823
+ [DEBUG] Backend cuda selected for int8_linear
2824
+ [DEBUG] Backend cuda selected for int8_linear
2825
+ [DEBUG] Backend cuda selected for int8_linear
2826
+ [DEBUG] Backend cuda selected for int8_linear
2827
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2828
+ [DEBUG] Backend cuda selected for int8_linear
2829
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2830
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2831
+ [DEBUG] Backend cuda selected for int8_linear
2832
+ [DEBUG] Backend cuda selected for int8_linear
2833
+ [DEBUG] Backend cuda selected for int8_linear
2834
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2835
+ [DEBUG] Backend cuda selected for int8_linear
2836
+ [DEBUG] Backend cuda selected for int8_linear
2837
+ [DEBUG] Backend cuda selected for int8_linear
2838
+ [DEBUG] Backend cuda selected for int8_linear
2839
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2840
+ [DEBUG] Backend cuda selected for int8_linear
2841
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2842
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2843
+ [DEBUG] Backend cuda selected for int8_linear
2844
+ [DEBUG] Backend cuda selected for int8_linear
2845
+ [DEBUG] Backend cuda selected for int8_linear
2846
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2847
+ [DEBUG] Backend cuda selected for int8_linear
2848
+ [DEBUG] Backend cuda selected for int8_linear
2849
+ [DEBUG] Backend cuda selected for int8_linear
2850
+ [DEBUG] Backend cuda selected for int8_linear
2851
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2852
+ [DEBUG] Backend cuda selected for int8_linear
2853
+ worker-comfyui - Websocket receive timed out. Still waiting...
2854
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2855
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2856
+ [DEBUG] Backend cuda selected for int8_linear
2857
+ [DEBUG] Backend cuda selected for int8_linear
2858
+ [DEBUG] Backend cuda selected for int8_linear
2859
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2860
+ [DEBUG] Backend cuda selected for int8_linear
2861
+ [DEBUG] Backend cuda selected for int8_linear
2862
+ [DEBUG] Backend cuda selected for int8_linear
2863
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=47090 MB
2864
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2865
+
2866
  25%|██▌ | 5/20 [06:53<20:39, 82.64s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
2867
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x76234b183180: Actual Resident VRAM = 224 MB
2868
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x761e02feb1c0: Actual Resident VRAM = 448 MB
2869
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7614c1d62f50: Actual Resident VRAM = 25888 MB
2870
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x76255e390150: Actual Resident VRAM = 20000 MB
2871
+ [DEBUG] aimdo: /project/src/model-vbar.c:90:DEBUG:Total VRAM for VBARs: 46560 MB
2872
+ [DEBUG] Backend cuda selected for int8_linear
2873
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2874
+ [DEBUG] Backend cuda selected for int8_linear
2875
+ [DEBUG] Backend cuda selected for int8_linear
2876
+ [DEBUG] Backend cuda selected for int8_linear
2877
+ [DEBUG] Backend cuda selected for int8_linear
2878
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2879
+ [DEBUG] Backend cuda selected for int8_linear
2880
+ [DEBUG] Backend cuda selected for int8_linear
2881
+ [DEBUG] Backend cuda selected for int8_linear
2882
+ [DEBUG] Backend cuda selected for int8_linear
2883
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2884
+ [DEBUG] Backend cuda selected for int8_linear
2885
+ [DEBUG] Backend cuda selected for int8_linear
2886
+ [DEBUG] Backend cuda selected for int8_linear
2887
+ [DEBUG] Backend cuda selected for int8_linear
2888
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2889
+ [DEBUG] Backend cuda selected for int8_linear
2890
+ [DEBUG] Backend cuda selected for int8_linear
2891
+ [DEBUG] Backend cuda selected for int8_linear
2892
+ [DEBUG] Backend cuda selected for int8_linear
2893
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2894
+ [DEBUG] Backend cuda selected for int8_linear
2895
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2896
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2897
+ [DEBUG] Backend cuda selected for int8_linear
2898
+ [DEBUG] Backend cuda selected for int8_linear
2899
+ [DEBUG] Backend cuda selected for int8_linear
2900
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2901
+ [DEBUG] Backend cuda selected for int8_linear
2902
+ [DEBUG] Backend cuda selected for int8_linear
2903
+ [DEBUG] Backend cuda selected for int8_linear
2904
+ [DEBUG] Backend cuda selected for int8_linear
2905
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2906
+ [DEBUG] Backend cuda selected for int8_linear
2907
+ worker-comfyui - Websocket receive timed out. Still waiting...
2908
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2909
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2910
+ [DEBUG] Backend cuda selected for int8_linear
2911
+ [DEBUG] Backend cuda selected for int8_linear
2912
+ [DEBUG] Backend cuda selected for int8_linear
2913
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2914
+ [DEBUG] Backend cuda selected for int8_linear
2915
+ [DEBUG] Backend cuda selected for int8_linear
2916
+ [DEBUG] Backend cuda selected for int8_linear
2917
+ [DEBUG] Backend cuda selected for int8_linear
2918
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2919
+ [DEBUG] Backend cuda selected for int8_linear
2920
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2921
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2922
+ [DEBUG] Backend cuda selected for int8_linear
2923
+ [DEBUG] Backend cuda selected for int8_linear
2924
+ [DEBUG] Backend cuda selected for int8_linear
2925
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2926
+ [DEBUG] Backend cuda selected for int8_linear
2927
+ [DEBUG] Backend cuda selected for int8_linear
2928
+ [DEBUG] Backend cuda selected for int8_linear
2929
+ [DEBUG] Backend cuda selected for int8_linear
2930
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2931
+ [DEBUG] Backend cuda selected for int8_linear
2932
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2933
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2934
+ [DEBUG] Backend cuda selected for int8_linear
2935
+ [DEBUG] Backend cuda selected for int8_linear
2936
+ [DEBUG] Backend cuda selected for int8_linear
2937
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2938
+ [DEBUG] Backend cuda selected for int8_linear
2939
+ [DEBUG] Backend cuda selected for int8_linear
2940
+ [DEBUG] Backend cuda selected for int8_linear
2941
+ [DEBUG] Backend cuda selected for int8_linear
2942
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2943
+ [DEBUG] Backend cuda selected for int8_linear
2944
+ worker-comfyui - Websocket receive timed out. Still waiting...
2945
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2946
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2947
+ [DEBUG] Backend cuda selected for int8_linear
2948
+ [DEBUG] Backend cuda selected for int8_linear
2949
+ [DEBUG] Backend cuda selected for int8_linear
2950
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2951
+ [DEBUG] Backend cuda selected for int8_linear
2952
+ [DEBUG] Backend cuda selected for int8_linear
2953
+ [DEBUG] Backend cuda selected for int8_linear
2954
+ [DEBUG] Backend cuda selected for int8_linear
2955
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2956
+ [DEBUG] Backend cuda selected for int8_linear
2957
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2958
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2959
+ [DEBUG] Backend cuda selected for int8_linear
2960
+ [DEBUG] Backend cuda selected for int8_linear
2961
+ [DEBUG] Backend cuda selected for int8_linear
2962
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2963
+ [DEBUG] Backend cuda selected for int8_linear
2964
+ [DEBUG] Backend cuda selected for int8_linear
2965
+ [DEBUG] Backend cuda selected for int8_linear
2966
+ [DEBUG] Backend cuda selected for int8_linear
2967
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2968
+ [DEBUG] Backend cuda selected for int8_linear
2969
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2970
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2971
+ [DEBUG] Backend cuda selected for int8_linear
2972
+ [DEBUG] Backend cuda selected for int8_linear
2973
+ [DEBUG] Backend cuda selected for int8_linear
2974
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2975
+ [DEBUG] Backend cuda selected for int8_linear
2976
+ [DEBUG] Backend cuda selected for int8_linear
2977
+ [DEBUG] Backend cuda selected for int8_linear
2978
+ [DEBUG] Backend cuda selected for int8_linear
2979
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2980
+ [DEBUG] Backend cuda selected for int8_linear
2981
+ worker-comfyui - Websocket receive timed out. Still waiting...
2982
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2983
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2984
+ [DEBUG] Backend cuda selected for int8_linear
2985
+ [DEBUG] Backend cuda selected for int8_linear
2986
+ [DEBUG] Backend cuda selected for int8_linear
2987
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2988
+ [DEBUG] Backend cuda selected for int8_linear
2989
+ [DEBUG] Backend cuda selected for int8_linear
2990
+ [DEBUG] Backend cuda selected for int8_linear
2991
+ [DEBUG] Backend cuda selected for int8_linear
2992
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2993
+ [DEBUG] Backend cuda selected for int8_linear
2994
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
2995
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2996
+ [DEBUG] Backend cuda selected for int8_linear
2997
+ [DEBUG] Backend cuda selected for int8_linear
2998
+ [DEBUG] Backend cuda selected for int8_linear
2999
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3000
+ [DEBUG] Backend cuda selected for int8_linear
3001
+ [DEBUG] Backend cuda selected for int8_linear
3002
+ [DEBUG] Backend cuda selected for int8_linear
3003
+ [DEBUG] Backend cuda selected for int8_linear
3004
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3005
+ [DEBUG] Backend cuda selected for int8_linear
3006
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3007
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3008
+ [DEBUG] Backend cuda selected for int8_linear
3009
+ [DEBUG] Backend cuda selected for int8_linear
3010
+ [DEBUG] Backend cuda selected for int8_linear
3011
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3012
+ [DEBUG] Backend cuda selected for int8_linear
3013
+ [DEBUG] Backend cuda selected for int8_linear
3014
+ [DEBUG] Backend cuda selected for int8_linear
3015
+ [DEBUG] Backend cuda selected for int8_linear
3016
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3017
+ [DEBUG] Backend cuda selected for int8_linear
3018
+ worker-comfyui - Websocket receive timed out. Still waiting...
3019
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3020
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3021
+ [DEBUG] Backend cuda selected for int8_linear
3022
+ [DEBUG] Backend cuda selected for int8_linear
3023
+ [DEBUG] Backend cuda selected for int8_linear
3024
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3025
+ [DEBUG] Backend cuda selected for int8_linear
3026
+ [DEBUG] Backend cuda selected for int8_linear
3027
+ [DEBUG] Backend cuda selected for int8_linear
3028
+ [DEBUG] Backend cuda selected for int8_linear
3029
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3030
+ [DEBUG] Backend cuda selected for int8_linear
3031
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3032
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3033
+ [DEBUG] Backend cuda selected for int8_linear
3034
+ [DEBUG] Backend cuda selected for int8_linear
3035
+ [DEBUG] Backend cuda selected for int8_linear
3036
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3037
+ [DEBUG] Backend cuda selected for int8_linear
3038
+ [DEBUG] Backend cuda selected for int8_linear
3039
+ [DEBUG] Backend cuda selected for int8_linear
3040
+ [DEBUG] Backend cuda selected for int8_linear
3041
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3042
+ [DEBUG] Backend cuda selected for int8_linear
3043
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3044
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3045
+ [DEBUG] Backend cuda selected for int8_linear
3046
+ [DEBUG] Backend cuda selected for int8_linear
3047
+ [DEBUG] Backend cuda selected for int8_linear
3048
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3049
+ [DEBUG] Backend cuda selected for int8_linear
3050
+ [DEBUG] Backend cuda selected for int8_linear
3051
+ [DEBUG] Backend cuda selected for int8_linear
3052
+ [DEBUG] Backend cuda selected for int8_linear
3053
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3054
+ [DEBUG] Backend cuda selected for int8_linear
3055
+ worker-comfyui - Websocket receive timed out. Still waiting...
3056
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3057
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3058
+ [DEBUG] Backend cuda selected for int8_linear
3059
+ [DEBUG] Backend cuda selected for int8_linear
3060
+ [DEBUG] Backend cuda selected for int8_linear
3061
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3062
+ [DEBUG] Backend cuda selected for int8_linear
3063
+ [DEBUG] Backend cuda selected for int8_linear
3064
+ [DEBUG] Backend cuda selected for int8_linear
3065
+ [DEBUG] Backend cuda selected for int8_linear
3066
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3067
+ [DEBUG] Backend cuda selected for int8_linear
3068
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3069
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3070
+ [DEBUG] Backend cuda selected for int8_linear
3071
+ [DEBUG] Backend cuda selected for int8_linear
3072
+ [DEBUG] Backend cuda selected for int8_linear
3073
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3074
+ [DEBUG] Backend cuda selected for int8_linear
3075
+ [DEBUG] Backend cuda selected for int8_linear
3076
+ [DEBUG] Backend cuda selected for int8_linear
3077
+ [DEBUG] Backend cuda selected for int8_linear
3078
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3079
+ [DEBUG] Backend cuda selected for int8_linear
3080
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3081
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3082
+ [DEBUG] Backend cuda selected for int8_linear
3083
+ [DEBUG] Backend cuda selected for int8_linear
3084
+ [DEBUG] Backend cuda selected for int8_linear
3085
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3086
+ [DEBUG] Backend cuda selected for int8_linear
3087
+ [DEBUG] Backend cuda selected for int8_linear
3088
+ [DEBUG] Backend cuda selected for int8_linear
3089
+ [DEBUG] Backend cuda selected for int8_linear
3090
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3091
+ [DEBUG] Backend cuda selected for int8_linear
3092
+ worker-comfyui - Websocket receive timed out. Still waiting...
3093
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3094
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3095
+ [DEBUG] Backend cuda selected for int8_linear
3096
+ [DEBUG] Backend cuda selected for int8_linear
3097
+ [DEBUG] Backend cuda selected for int8_linear
3098
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3099
+ [DEBUG] Backend cuda selected for int8_linear
3100
+ [DEBUG] Backend cuda selected for int8_linear
3101
+ [DEBUG] Backend cuda selected for int8_linear
3102
+ [DEBUG] Backend cuda selected for int8_linear
3103
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3104
+ [DEBUG] Backend cuda selected for int8_linear
3105
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3106
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3107
+ [DEBUG] Backend cuda selected for int8_linear
3108
+ [DEBUG] Backend cuda selected for int8_linear
3109
+ [DEBUG] Backend cuda selected for int8_linear
3110
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3111
+ [DEBUG] Backend cuda selected for int8_linear
3112
+ [DEBUG] Backend cuda selected for int8_linear
3113
+ [DEBUG] Backend cuda selected for int8_linear
3114
+ [DEBUG] Backend cuda selected for int8_linear
3115
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3116
+ [DEBUG] Backend cuda selected for int8_linear
3117
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3118
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3119
+ [DEBUG] Backend cuda selected for int8_linear
3120
+ [DEBUG] Backend cuda selected for int8_linear
3121
+ [DEBUG] Backend cuda selected for int8_linear
3122
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3123
+ [DEBUG] Backend cuda selected for int8_linear
3124
+ [DEBUG] Backend cuda selected for int8_linear
3125
+ [DEBUG] Backend cuda selected for int8_linear
3126
+ [DEBUG] Backend cuda selected for int8_linear
3127
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3128
+ [DEBUG] Backend cuda selected for int8_linear
3129
+ worker-comfyui - Websocket receive timed out. Still waiting...
3130
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3131
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3132
+ [DEBUG] Backend cuda selected for int8_linear
3133
+ [DEBUG] Backend cuda selected for int8_linear
3134
+ [DEBUG] Backend cuda selected for int8_linear
3135
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3136
+ [DEBUG] Backend cuda selected for int8_linear
3137
+ [DEBUG] Backend cuda selected for int8_linear
3138
+ [DEBUG] Backend cuda selected for int8_linear
3139
+ [DEBUG] Backend cuda selected for int8_linear
3140
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3141
+ [DEBUG] Backend cuda selected for int8_linear
3142
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3143
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3144
+ [DEBUG] Backend cuda selected for int8_linear
3145
+ [DEBUG] Backend cuda selected for int8_linear
3146
+ [DEBUG] Backend cuda selected for int8_linear
3147
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3148
+ [DEBUG] Backend cuda selected for int8_linear
3149
+ [DEBUG] Backend cuda selected for int8_linear
3150
+ [DEBUG] Backend cuda selected for int8_linear
3151
+ [DEBUG] Backend cuda selected for int8_linear
3152
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3153
+ [DEBUG] Backend cuda selected for int8_linear
3154
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3155
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3156
+ [DEBUG] Backend cuda selected for int8_linear
3157
+ [DEBUG] Backend cuda selected for int8_linear
3158
+ [DEBUG] Backend cuda selected for int8_linear
3159
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3160
+ [DEBUG] Backend cuda selected for int8_linear
3161
+ [DEBUG] Backend cuda selected for int8_linear
3162
+ [DEBUG] Backend cuda selected for int8_linear
3163
+ [DEBUG] Backend cuda selected for int8_linear
3164
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3165
+ [DEBUG] Backend cuda selected for int8_linear
3166
+ worker-comfyui - Websocket receive timed out. Still waiting...
3167
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3168
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3169
+ [DEBUG] Backend cuda selected for int8_linear
3170
+ [DEBUG] Backend cuda selected for int8_linear
3171
+ [DEBUG] Backend cuda selected for int8_linear
3172
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3173
+ [DEBUG] Backend cuda selected for int8_linear
3174
+ [DEBUG] Backend cuda selected for int8_linear
3175
+ [DEBUG] Backend cuda selected for int8_linear
3176
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=47090 MB
3177
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3178
+
3179
  30%|███ | 6/20 [08:16<19:16, 82.62s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
3180
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x76234b183180: Actual Resident VRAM = 224 MB
3181
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x761e02feb1c0: Actual Resident VRAM = 448 MB
3182
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7614c1d62f50: Actual Resident VRAM = 25888 MB
3183
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x76255e390150: Actual Resident VRAM = 20000 MB
3184
+ [DEBUG] aimdo: /project/src/model-vbar.c:90:DEBUG:Total VRAM for VBARs: 46560 MB
3185
+ [DEBUG] Backend cuda selected for int8_linear
3186
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3187
+ [DEBUG] Backend cuda selected for int8_linear
3188
+ [DEBUG] Backend cuda selected for int8_linear
3189
+ [DEBUG] Backend cuda selected for int8_linear
3190
+ [DEBUG] Backend cuda selected for int8_linear
3191
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3192
+ [DEBUG] Backend cuda selected for int8_linear
3193
+ [DEBUG] Backend cuda selected for int8_linear
3194
+ [DEBUG] Backend cuda selected for int8_linear
3195
+ [DEBUG] Backend cuda selected for int8_linear
3196
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3197
+ [DEBUG] Backend cuda selected for int8_linear
3198
+ [DEBUG] Backend cuda selected for int8_linear
3199
+ [DEBUG] Backend cuda selected for int8_linear
3200
+ [DEBUG] Backend cuda selected for int8_linear
3201
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3202
+ [DEBUG] Backend cuda selected for int8_linear
3203
+ [DEBUG] Backend cuda selected for int8_linear
3204
+ [DEBUG] Backend cuda selected for int8_linear
3205
+ [DEBUG] Backend cuda selected for int8_linear
3206
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3207
+ [DEBUG] Backend cuda selected for int8_linear
3208
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3209
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3210
+ [DEBUG] Backend cuda selected for int8_linear
3211
+ [DEBUG] Backend cuda selected for int8_linear
3212
+ [DEBUG] Backend cuda selected for int8_linear
3213
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3214
+ [DEBUG] Backend cuda selected for int8_linear
3215
+ [DEBUG] Backend cuda selected for int8_linear
3216
+ [DEBUG] Backend cuda selected for int8_linear
3217
+ [DEBUG] Backend cuda selected for int8_linear
3218
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3219
+ [DEBUG] Backend cuda selected for int8_linear
3220
+ worker-comfyui - Websocket receive timed out. Still waiting...
3221
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3222
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3223
+ [DEBUG] Backend cuda selected for int8_linear
3224
+ [DEBUG] Backend cuda selected for int8_linear
3225
+ [DEBUG] Backend cuda selected for int8_linear
3226
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3227
+ [DEBUG] Backend cuda selected for int8_linear
3228
+ [DEBUG] Backend cuda selected for int8_linear
3229
+ [DEBUG] Backend cuda selected for int8_linear
3230
+ [DEBUG] Backend cuda selected for int8_linear
3231
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3232
+ [DEBUG] Backend cuda selected for int8_linear
3233
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3234
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3235
+ [DEBUG] Backend cuda selected for int8_linear
3236
+ [DEBUG] Backend cuda selected for int8_linear
3237
+ [DEBUG] Backend cuda selected for int8_linear
3238
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3239
+ [DEBUG] Backend cuda selected for int8_linear
3240
+ [DEBUG] Backend cuda selected for int8_linear
3241
+ [DEBUG] Backend cuda selected for int8_linear
3242
+ [DEBUG] Backend cuda selected for int8_linear
3243
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3244
+ [DEBUG] Backend cuda selected for int8_linear
3245
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3246
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3247
+ [DEBUG] Backend cuda selected for int8_linear
3248
+ [DEBUG] Backend cuda selected for int8_linear
3249
+ [DEBUG] Backend cuda selected for int8_linear
3250
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3251
+ [DEBUG] Backend cuda selected for int8_linear
3252
+ [DEBUG] Backend cuda selected for int8_linear
3253
+ [DEBUG] Backend cuda selected for int8_linear
3254
+ [DEBUG] Backend cuda selected for int8_linear
3255
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3256
+ [DEBUG] Backend cuda selected for int8_linear
3257
+ worker-comfyui - Websocket receive timed out. Still waiting...
3258
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3259
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3260
+ [DEBUG] Backend cuda selected for int8_linear
3261
+ [DEBUG] Backend cuda selected for int8_linear
3262
+ [DEBUG] Backend cuda selected for int8_linear
3263
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3264
+ [DEBUG] Backend cuda selected for int8_linear
3265
+ [DEBUG] Backend cuda selected for int8_linear
3266
+ [DEBUG] Backend cuda selected for int8_linear
3267
+ [DEBUG] Backend cuda selected for int8_linear
3268
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3269
+ [DEBUG] Backend cuda selected for int8_linear
3270
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3271
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3272
+ [DEBUG] Backend cuda selected for int8_linear
3273
+ [DEBUG] Backend cuda selected for int8_linear
3274
+ [DEBUG] Backend cuda selected for int8_linear
3275
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3276
+ [DEBUG] Backend cuda selected for int8_linear
3277
+ [DEBUG] Backend cuda selected for int8_linear
3278
+ [DEBUG] Backend cuda selected for int8_linear
3279
+ [DEBUG] Backend cuda selected for int8_linear
3280
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3281
+ [DEBUG] Backend cuda selected for int8_linear
3282
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3283
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3284
+ [DEBUG] Backend cuda selected for int8_linear
3285
+ [DEBUG] Backend cuda selected for int8_linear
3286
+ [DEBUG] Backend cuda selected for int8_linear
3287
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3288
+ [DEBUG] Backend cuda selected for int8_linear
3289
+ [DEBUG] Backend cuda selected for int8_linear
3290
+ [DEBUG] Backend cuda selected for int8_linear
3291
+ [DEBUG] Backend cuda selected for int8_linear
3292
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3293
+ [DEBUG] Backend cuda selected for int8_linear
3294
+ worker-comfyui - Websocket receive timed out. Still waiting...
3295
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3296
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3297
+ [DEBUG] Backend cuda selected for int8_linear
3298
+ [DEBUG] Backend cuda selected for int8_linear
3299
+ [DEBUG] Backend cuda selected for int8_linear
3300
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3301
+ [DEBUG] Backend cuda selected for int8_linear
3302
+ [DEBUG] Backend cuda selected for int8_linear
3303
+ [DEBUG] Backend cuda selected for int8_linear
3304
+ [DEBUG] Backend cuda selected for int8_linear
3305
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3306
+ [DEBUG] Backend cuda selected for int8_linear
3307
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3308
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3309
+ [DEBUG] Backend cuda selected for int8_linear
3310
+ [DEBUG] Backend cuda selected for int8_linear
3311
+ [DEBUG] Backend cuda selected for int8_linear
3312
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3313
+ [DEBUG] Backend cuda selected for int8_linear
3314
+ [DEBUG] Backend cuda selected for int8_linear
3315
+ [DEBUG] Backend cuda selected for int8_linear
3316
+ [DEBUG] Backend cuda selected for int8_linear
3317
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3318
+ [DEBUG] Backend cuda selected for int8_linear
3319
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3320
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3321
+ [DEBUG] Backend cuda selected for int8_linear
3322
+ [DEBUG] Backend cuda selected for int8_linear
3323
+ [DEBUG] Backend cuda selected for int8_linear
3324
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3325
+ [DEBUG] Backend cuda selected for int8_linear
3326
+ [DEBUG] Backend cuda selected for int8_linear
3327
+ [DEBUG] Backend cuda selected for int8_linear
3328
+ [DEBUG] Backend cuda selected for int8_linear
3329
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3330
+ [DEBUG] Backend cuda selected for int8_linear
3331
+ worker-comfyui - Websocket receive timed out. Still waiting...
3332
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3333
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3334
+ [DEBUG] Backend cuda selected for int8_linear
3335
+ [DEBUG] Backend cuda selected for int8_linear
3336
+ [DEBUG] Backend cuda selected for int8_linear
3337
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3338
+ [DEBUG] Backend cuda selected for int8_linear
3339
+ [DEBUG] Backend cuda selected for int8_linear
3340
+ [DEBUG] Backend cuda selected for int8_linear
3341
+ [DEBUG] Backend cuda selected for int8_linear
3342
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3343
+ [DEBUG] Backend cuda selected for int8_linear
3344
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3345
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3346
+ [DEBUG] Backend cuda selected for int8_linear
3347
+ [DEBUG] Backend cuda selected for int8_linear
3348
+ [DEBUG] Backend cuda selected for int8_linear
3349
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3350
+ [DEBUG] Backend cuda selected for int8_linear
3351
+ [DEBUG] Backend cuda selected for int8_linear
3352
+ [DEBUG] Backend cuda selected for int8_linear
3353
+ [DEBUG] Backend cuda selected for int8_linear
3354
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3355
+ [DEBUG] Backend cuda selected for int8_linear
3356
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3357
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3358
+ [DEBUG] Backend cuda selected for int8_linear
3359
+ [DEBUG] Backend cuda selected for int8_linear
3360
+ [DEBUG] Backend cuda selected for int8_linear
3361
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3362
+ [DEBUG] Backend cuda selected for int8_linear
3363
+ [DEBUG] Backend cuda selected for int8_linear
3364
+ [DEBUG] Backend cuda selected for int8_linear
3365
+ [DEBUG] Backend cuda selected for int8_linear
3366
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3367
+ [DEBUG] Backend cuda selected for int8_linear
3368
+ worker-comfyui - Websocket receive timed out. Still waiting...
3369
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3370
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3371
+ [DEBUG] Backend cuda selected for int8_linear
3372
+ [DEBUG] Backend cuda selected for int8_linear
3373
+ [DEBUG] Backend cuda selected for int8_linear
3374
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3375
+ [DEBUG] Backend cuda selected for int8_linear
3376
+ [DEBUG] Backend cuda selected for int8_linear
3377
+ [DEBUG] Backend cuda selected for int8_linear
3378
+ [DEBUG] Backend cuda selected for int8_linear
3379
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3380
+ [DEBUG] Backend cuda selected for int8_linear
3381
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=15478 MB total=81079 MB deficit_sync=-15222 MB recorded=53039 MB
3382
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3383
+ [DEBUG] Backend cuda selected for int8_linear
3384
+ [DEBUG] Backend cuda selected for int8_linear
3385
+ [DEBUG] Backend cuda selected for int8_linear
3386
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3387
+ [DEBUG] Backend cuda selected for int8_linear
3388
+ [DEBUG] Backend cuda selected for int8_linear
3389
+ [DEBUG] Backend cuda selected for int8_linear
3390
+ [DEBUG] Backend cuda selected for int8_linear
3391
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3392
+ [DEBUG] Backend cuda selected for int8_linear