nikdevs commited on
Commit
b049aa0
·
verified ·
1 Parent(s): 0577a58

boot log tvfnt4hniqtnma

Browse files
Files changed (1) hide show
  1. logs/boot-tvfnt4hniqtnma.log +1174 -0
logs/boot-tvfnt4hniqtnma.log CHANGED
@@ -1960,3 +1960,1177 @@ worker-comfyui - Websocket receive timed out. Still waiting...
1960
  [DEBUG] Backend cuda selected for int8_linear
1961
  [DEBUG] Backend cuda selected for rms_rope_split_half_
1962
  [DEBUG] Backend cuda selected for int8_linear
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1963
  20%|██ | 4/20 [05:20<21:20, 80.06s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1964
  25%|██▌ | 5/20 [06:40<20:01, 80.11s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1965
  30%|███ | 6/20 [08:00<18:43, 80.24s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1966
  35%|███▌ | 7/20 [09:20<17:22, 80.16s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1960
  [DEBUG] Backend cuda selected for int8_linear
1961
  [DEBUG] Backend cuda selected for rms_rope_split_half_
1962
  [DEBUG] Backend cuda selected for int8_linear
1963
+ [minimax] boot log shipped
1964
+ [DEBUG] Backend cuda selected for int8_linear
1965
+ [DEBUG] Backend cuda selected for int8_linear
1966
+ [DEBUG] Backend cuda selected for int8_linear
1967
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
1968
+ [DEBUG] Backend cuda selected for int8_linear
1969
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
1970
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
1971
+ [DEBUG] Backend cuda selected for int8_linear
1972
+ [DEBUG] Backend cuda selected for int8_linear
1973
+ [DEBUG] Backend cuda selected for int8_linear
1974
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
1975
+ [DEBUG] Backend cuda selected for int8_linear
1976
+ [DEBUG] Backend cuda selected for int8_linear
1977
+ [DEBUG] Backend cuda selected for int8_linear
1978
+ [DEBUG] Backend cuda selected for int8_linear
1979
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
1980
+ [DEBUG] Backend cuda selected for int8_linear
1981
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
1982
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
1983
+ [DEBUG] Backend cuda selected for int8_linear
1984
+ [DEBUG] Backend cuda selected for int8_linear
1985
+ [DEBUG] Backend cuda selected for int8_linear
1986
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
1987
+ [DEBUG] Backend cuda selected for int8_linear
1988
+ [DEBUG] Backend cuda selected for int8_linear
1989
+ [DEBUG] Backend cuda selected for int8_linear
1990
+ [DEBUG] Backend cuda selected for int8_linear
1991
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
1992
+ [DEBUG] Backend cuda selected for int8_linear
1993
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
1994
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
1995
+ [DEBUG] Backend cuda selected for int8_linear
1996
+ [DEBUG] Backend cuda selected for int8_linear
1997
+ [DEBUG] Backend cuda selected for int8_linear
1998
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
1999
+ [DEBUG] Backend cuda selected for int8_linear
2000
+ worker-comfyui - Websocket receive timed out. Still waiting...
2001
+ [DEBUG] Backend cuda selected for int8_linear
2002
+ [DEBUG] Backend cuda selected for int8_linear
2003
+ [DEBUG] Backend cuda selected for int8_linear
2004
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2005
+ [DEBUG] Backend cuda selected for int8_linear
2006
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2007
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2008
+ [DEBUG] Backend cuda selected for int8_linear
2009
+ [DEBUG] Backend cuda selected for int8_linear
2010
+ [DEBUG] Backend cuda selected for int8_linear
2011
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2012
+ [DEBUG] Backend cuda selected for int8_linear
2013
+ [DEBUG] Backend cuda selected for int8_linear
2014
+ [DEBUG] Backend cuda selected for int8_linear
2015
+ [DEBUG] Backend cuda selected for int8_linear
2016
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2017
+ [DEBUG] Backend cuda selected for int8_linear
2018
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2019
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2020
+ [DEBUG] Backend cuda selected for int8_linear
2021
+ [DEBUG] Backend cuda selected for int8_linear
2022
+ [DEBUG] Backend cuda selected for int8_linear
2023
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2024
+ [DEBUG] Backend cuda selected for int8_linear
2025
+ [DEBUG] Backend cuda selected for int8_linear
2026
+ [DEBUG] Backend cuda selected for int8_linear
2027
+ [DEBUG] Backend cuda selected for int8_linear
2028
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2029
+ [DEBUG] Backend cuda selected for int8_linear
2030
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2031
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2032
+ [DEBUG] Backend cuda selected for int8_linear
2033
+ [DEBUG] Backend cuda selected for int8_linear
2034
+ [DEBUG] Backend cuda selected for int8_linear
2035
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2036
+ [DEBUG] Backend cuda selected for int8_linear
2037
+ worker-comfyui - Websocket receive timed out. Still waiting...
2038
+ [DEBUG] Backend cuda selected for int8_linear
2039
+ [DEBUG] Backend cuda selected for int8_linear
2040
+ [DEBUG] Backend cuda selected for int8_linear
2041
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2042
+ [DEBUG] Backend cuda selected for int8_linear
2043
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2044
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2045
+ [DEBUG] Backend cuda selected for int8_linear
2046
+ [DEBUG] Backend cuda selected for int8_linear
2047
+ [DEBUG] Backend cuda selected for int8_linear
2048
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2049
+ [DEBUG] Backend cuda selected for int8_linear
2050
+ [DEBUG] Backend cuda selected for int8_linear
2051
+ [DEBUG] Backend cuda selected for int8_linear
2052
+ [DEBUG] Backend cuda selected for int8_linear
2053
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2054
+ [DEBUG] Backend cuda selected for int8_linear
2055
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2056
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2057
+ [DEBUG] Backend cuda selected for int8_linear
2058
+ [DEBUG] Backend cuda selected for int8_linear
2059
+ [DEBUG] Backend cuda selected for int8_linear
2060
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2061
+ [DEBUG] Backend cuda selected for int8_linear
2062
+ [DEBUG] Backend cuda selected for int8_linear
2063
+ [DEBUG] Backend cuda selected for int8_linear
2064
+ [DEBUG] Backend cuda selected for int8_linear
2065
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2066
+ [DEBUG] Backend cuda selected for int8_linear
2067
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2068
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2069
+ [DEBUG] Backend cuda selected for int8_linear
2070
+ [DEBUG] Backend cuda selected for int8_linear
2071
+ [DEBUG] Backend cuda selected for int8_linear
2072
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2073
+ [DEBUG] Backend cuda selected for int8_linear
2074
+ worker-comfyui - Websocket receive timed out. Still waiting...
2075
+ [DEBUG] Backend cuda selected for int8_linear
2076
+ [DEBUG] Backend cuda selected for int8_linear
2077
+ [DEBUG] Backend cuda selected for int8_linear
2078
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2079
+ [DEBUG] Backend cuda selected for int8_linear
2080
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2081
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2082
+ [DEBUG] Backend cuda selected for int8_linear
2083
+ [DEBUG] Backend cuda selected for int8_linear
2084
+ [DEBUG] Backend cuda selected for int8_linear
2085
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2086
+ [DEBUG] Backend cuda selected for int8_linear
2087
+ [DEBUG] Backend cuda selected for int8_linear
2088
+ [DEBUG] Backend cuda selected for int8_linear
2089
+ [DEBUG] Backend cuda selected for int8_linear
2090
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2091
+ [DEBUG] Backend cuda selected for int8_linear
2092
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2093
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2094
+ [DEBUG] Backend cuda selected for int8_linear
2095
+ [DEBUG] Backend cuda selected for int8_linear
2096
+ [DEBUG] Backend cuda selected for int8_linear
2097
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2098
+ [DEBUG] Backend cuda selected for int8_linear
2099
+ [DEBUG] Backend cuda selected for int8_linear
2100
+ [DEBUG] Backend cuda selected for int8_linear
2101
+ [DEBUG] Backend cuda selected for int8_linear
2102
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2103
+ [DEBUG] Backend cuda selected for int8_linear
2104
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2105
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2106
+ [DEBUG] Backend cuda selected for int8_linear
2107
+ [DEBUG] Backend cuda selected for int8_linear
2108
+ [DEBUG] Backend cuda selected for int8_linear
2109
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2110
+ [DEBUG] Backend cuda selected for int8_linear
2111
+ [DEBUG] Backend cuda selected for int8_linear
2112
+ [DEBUG] Backend cuda selected for int8_linear
2113
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=47294 MB
2114
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2115
+
2116
  20%|██ | 4/20 [05:20<21:20, 80.06s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
2117
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7d02f96a05f0: Actual Resident VRAM = 512 MB
2118
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cf4ab0e12b0: Actual Resident VRAM = 448 MB
2119
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cf476277d50: Actual Resident VRAM = 25888 MB
2120
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cfdee77f880: Actual Resident VRAM = 20000 MB
2121
+ [DEBUG] aimdo: /project/src/model-vbar.c:90:DEBUG:Total VRAM for VBARs: 46848 MB
2122
+ [DEBUG] Backend cuda selected for int8_linear
2123
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2124
+ [DEBUG] Backend cuda selected for int8_linear
2125
+ [DEBUG] Backend cuda selected for int8_linear
2126
+ [DEBUG] Backend cuda selected for int8_linear
2127
+ [DEBUG] Backend cuda selected for int8_linear
2128
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2129
+ [DEBUG] Backend cuda selected for int8_linear
2130
+ [DEBUG] Backend cuda selected for int8_linear
2131
+ [DEBUG] Backend cuda selected for int8_linear
2132
+ [DEBUG] Backend cuda selected for int8_linear
2133
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2134
+ [DEBUG] Backend cuda selected for int8_linear
2135
+ [DEBUG] Backend cuda selected for int8_linear
2136
+ [DEBUG] Backend cuda selected for int8_linear
2137
+ [DEBUG] Backend cuda selected for int8_linear
2138
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2139
+ [DEBUG] Backend cuda selected for int8_linear
2140
+ [DEBUG] Backend cuda selected for int8_linear
2141
+ [DEBUG] Backend cuda selected for int8_linear
2142
+ [DEBUG] Backend cuda selected for int8_linear
2143
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2144
+ [DEBUG] Backend cuda selected for int8_linear
2145
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2146
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2147
+ [DEBUG] Backend cuda selected for int8_linear
2148
+ [DEBUG] Backend cuda selected for int8_linear
2149
+ [DEBUG] Backend cuda selected for int8_linear
2150
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2151
+ [DEBUG] Backend cuda selected for int8_linear
2152
+ [DEBUG] Backend cuda selected for int8_linear
2153
+ [DEBUG] Backend cuda selected for int8_linear
2154
+ [DEBUG] Backend cuda selected for int8_linear
2155
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2156
+ [DEBUG] Backend cuda selected for int8_linear
2157
+ worker-comfyui - Websocket receive timed out. Still waiting...
2158
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2159
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2160
+ [DEBUG] Backend cuda selected for int8_linear
2161
+ [DEBUG] Backend cuda selected for int8_linear
2162
+ [DEBUG] Backend cuda selected for int8_linear
2163
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2164
+ [DEBUG] Backend cuda selected for int8_linear
2165
+ [DEBUG] Backend cuda selected for int8_linear
2166
+ [DEBUG] Backend cuda selected for int8_linear
2167
+ [DEBUG] Backend cuda selected for int8_linear
2168
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2169
+ [DEBUG] Backend cuda selected for int8_linear
2170
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2171
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2172
+ [DEBUG] Backend cuda selected for int8_linear
2173
+ [DEBUG] Backend cuda selected for int8_linear
2174
+ [DEBUG] Backend cuda selected for int8_linear
2175
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2176
+ [DEBUG] Backend cuda selected for int8_linear
2177
+ [DEBUG] Backend cuda selected for int8_linear
2178
+ [DEBUG] Backend cuda selected for int8_linear
2179
+ [DEBUG] Backend cuda selected for int8_linear
2180
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2181
+ [DEBUG] Backend cuda selected for int8_linear
2182
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2183
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2184
+ [DEBUG] Backend cuda selected for int8_linear
2185
+ [DEBUG] Backend cuda selected for int8_linear
2186
+ [DEBUG] Backend cuda selected for int8_linear
2187
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2188
+ [DEBUG] Backend cuda selected for int8_linear
2189
+ [DEBUG] Backend cuda selected for int8_linear
2190
+ [DEBUG] Backend cuda selected for int8_linear
2191
+ [DEBUG] Backend cuda selected for int8_linear
2192
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2193
+ [DEBUG] Backend cuda selected for int8_linear
2194
+ worker-comfyui - Websocket receive timed out. Still waiting...
2195
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2196
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2197
+ [DEBUG] Backend cuda selected for int8_linear
2198
+ [DEBUG] Backend cuda selected for int8_linear
2199
+ [DEBUG] Backend cuda selected for int8_linear
2200
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2201
+ [DEBUG] Backend cuda selected for int8_linear
2202
+ [DEBUG] Backend cuda selected for int8_linear
2203
+ [DEBUG] Backend cuda selected for int8_linear
2204
+ [DEBUG] Backend cuda selected for int8_linear
2205
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2206
+ [DEBUG] Backend cuda selected for int8_linear
2207
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2208
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2209
+ [DEBUG] Backend cuda selected for int8_linear
2210
+ [DEBUG] Backend cuda selected for int8_linear
2211
+ [DEBUG] Backend cuda selected for int8_linear
2212
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2213
+ [DEBUG] Backend cuda selected for int8_linear
2214
+ [DEBUG] Backend cuda selected for int8_linear
2215
+ [DEBUG] Backend cuda selected for int8_linear
2216
+ [DEBUG] Backend cuda selected for int8_linear
2217
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2218
+ [DEBUG] Backend cuda selected for int8_linear
2219
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2220
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2221
+ [DEBUG] Backend cuda selected for int8_linear
2222
+ [DEBUG] Backend cuda selected for int8_linear
2223
+ [DEBUG] Backend cuda selected for int8_linear
2224
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2225
+ [DEBUG] Backend cuda selected for int8_linear
2226
+ [DEBUG] Backend cuda selected for int8_linear
2227
+ [DEBUG] Backend cuda selected for int8_linear
2228
+ [DEBUG] Backend cuda selected for int8_linear
2229
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2230
+ [DEBUG] Backend cuda selected for int8_linear
2231
+ worker-comfyui - Websocket receive timed out. Still waiting...
2232
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2233
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2234
+ [DEBUG] Backend cuda selected for int8_linear
2235
+ [DEBUG] Backend cuda selected for int8_linear
2236
+ [DEBUG] Backend cuda selected for int8_linear
2237
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2238
+ [DEBUG] Backend cuda selected for int8_linear
2239
+ [DEBUG] Backend cuda selected for int8_linear
2240
+ [DEBUG] Backend cuda selected for int8_linear
2241
+ [DEBUG] Backend cuda selected for int8_linear
2242
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2243
+ [DEBUG] Backend cuda selected for int8_linear
2244
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2245
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2246
+ [DEBUG] Backend cuda selected for int8_linear
2247
+ [DEBUG] Backend cuda selected for int8_linear
2248
+ [DEBUG] Backend cuda selected for int8_linear
2249
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2250
+ [DEBUG] Backend cuda selected for int8_linear
2251
+ [DEBUG] Backend cuda selected for int8_linear
2252
+ [DEBUG] Backend cuda selected for int8_linear
2253
+ [DEBUG] Backend cuda selected for int8_linear
2254
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2255
+ [DEBUG] Backend cuda selected for int8_linear
2256
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2257
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2258
+ [DEBUG] Backend cuda selected for int8_linear
2259
+ [DEBUG] Backend cuda selected for int8_linear
2260
+ [DEBUG] Backend cuda selected for int8_linear
2261
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2262
+ [DEBUG] Backend cuda selected for int8_linear
2263
+ [DEBUG] Backend cuda selected for int8_linear
2264
+ [DEBUG] Backend cuda selected for int8_linear
2265
+ [DEBUG] Backend cuda selected for int8_linear
2266
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2267
+ [DEBUG] Backend cuda selected for int8_linear
2268
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2269
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2270
+ [DEBUG] Backend cuda selected for int8_linear
2271
+ [DEBUG] Backend cuda selected for int8_linear
2272
+ worker-comfyui - Websocket receive timed out. Still waiting...
2273
+ [DEBUG] Backend cuda selected for int8_linear
2274
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2275
+ [DEBUG] Backend cuda selected for int8_linear
2276
+ [DEBUG] Backend cuda selected for int8_linear
2277
+ [DEBUG] Backend cuda selected for int8_linear
2278
+ [DEBUG] Backend cuda selected for int8_linear
2279
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2280
+ [DEBUG] Backend cuda selected for int8_linear
2281
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2282
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2283
+ [DEBUG] Backend cuda selected for int8_linear
2284
+ [DEBUG] Backend cuda selected for int8_linear
2285
+ [DEBUG] Backend cuda selected for int8_linear
2286
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2287
+ [DEBUG] Backend cuda selected for int8_linear
2288
+ [DEBUG] Backend cuda selected for int8_linear
2289
+ [DEBUG] Backend cuda selected for int8_linear
2290
+ [DEBUG] Backend cuda selected for int8_linear
2291
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2292
+ [DEBUG] Backend cuda selected for int8_linear
2293
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2294
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2295
+ [DEBUG] Backend cuda selected for int8_linear
2296
+ [DEBUG] Backend cuda selected for int8_linear
2297
+ [DEBUG] Backend cuda selected for int8_linear
2298
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2299
+ [DEBUG] Backend cuda selected for int8_linear
2300
+ [DEBUG] Backend cuda selected for int8_linear
2301
+ [DEBUG] Backend cuda selected for int8_linear
2302
+ [DEBUG] Backend cuda selected for int8_linear
2303
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2304
+ [DEBUG] Backend cuda selected for int8_linear
2305
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2306
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2307
+ [DEBUG] Backend cuda selected for int8_linear
2308
+ [DEBUG] Backend cuda selected for int8_linear
2309
+ [DEBUG] Backend cuda selected for int8_linear
2310
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2311
+ [DEBUG] Backend cuda selected for int8_linear
2312
+ worker-comfyui - Websocket receive timed out. Still waiting...
2313
+ [DEBUG] Backend cuda selected for int8_linear
2314
+ [DEBUG] Backend cuda selected for int8_linear
2315
+ [DEBUG] Backend cuda selected for int8_linear
2316
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2317
+ [DEBUG] Backend cuda selected for int8_linear
2318
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2319
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2320
+ [DEBUG] Backend cuda selected for int8_linear
2321
+ [DEBUG] Backend cuda selected for int8_linear
2322
+ [DEBUG] Backend cuda selected for int8_linear
2323
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2324
+ [DEBUG] Backend cuda selected for int8_linear
2325
+ [DEBUG] Backend cuda selected for int8_linear
2326
+ [DEBUG] Backend cuda selected for int8_linear
2327
+ [DEBUG] Backend cuda selected for int8_linear
2328
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2329
+ [DEBUG] Backend cuda selected for int8_linear
2330
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2331
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2332
+ [DEBUG] Backend cuda selected for int8_linear
2333
+ [DEBUG] Backend cuda selected for int8_linear
2334
+ [DEBUG] Backend cuda selected for int8_linear
2335
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2336
+ [DEBUG] Backend cuda selected for int8_linear
2337
+ [DEBUG] Backend cuda selected for int8_linear
2338
+ [DEBUG] Backend cuda selected for int8_linear
2339
+ [DEBUG] Backend cuda selected for int8_linear
2340
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2341
+ [DEBUG] Backend cuda selected for int8_linear
2342
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2343
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2344
+ [DEBUG] Backend cuda selected for int8_linear
2345
+ [DEBUG] Backend cuda selected for int8_linear
2346
+ [DEBUG] Backend cuda selected for int8_linear
2347
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2348
+ [DEBUG] Backend cuda selected for int8_linear
2349
+ worker-comfyui - Websocket receive timed out. Still waiting...
2350
+ [DEBUG] Backend cuda selected for int8_linear
2351
+ [DEBUG] Backend cuda selected for int8_linear
2352
+ [DEBUG] Backend cuda selected for int8_linear
2353
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2354
+ [DEBUG] Backend cuda selected for int8_linear
2355
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2356
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2357
+ [DEBUG] Backend cuda selected for int8_linear
2358
+ [DEBUG] Backend cuda selected for int8_linear
2359
+ [DEBUG] Backend cuda selected for int8_linear
2360
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2361
+ [DEBUG] Backend cuda selected for int8_linear
2362
+ [DEBUG] Backend cuda selected for int8_linear
2363
+ [DEBUG] Backend cuda selected for int8_linear
2364
+ [DEBUG] Backend cuda selected for int8_linear
2365
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2366
+ [DEBUG] Backend cuda selected for int8_linear
2367
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2368
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2369
+ [DEBUG] Backend cuda selected for int8_linear
2370
+ [DEBUG] Backend cuda selected for int8_linear
2371
+ [DEBUG] Backend cuda selected for int8_linear
2372
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2373
+ [DEBUG] Backend cuda selected for int8_linear
2374
+ [DEBUG] Backend cuda selected for int8_linear
2375
+ [DEBUG] Backend cuda selected for int8_linear
2376
+ [DEBUG] Backend cuda selected for int8_linear
2377
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2378
+ [DEBUG] Backend cuda selected for int8_linear
2379
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2380
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2381
+ [DEBUG] Backend cuda selected for int8_linear
2382
+ [DEBUG] Backend cuda selected for int8_linear
2383
+ [DEBUG] Backend cuda selected for int8_linear
2384
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2385
+ [DEBUG] Backend cuda selected for int8_linear
2386
+ worker-comfyui - Websocket receive timed out. Still waiting...
2387
+ [DEBUG] Backend cuda selected for int8_linear
2388
+ [DEBUG] Backend cuda selected for int8_linear
2389
+ [DEBUG] Backend cuda selected for int8_linear
2390
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2391
+ [DEBUG] Backend cuda selected for int8_linear
2392
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2393
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2394
+ [DEBUG] Backend cuda selected for int8_linear
2395
+ [DEBUG] Backend cuda selected for int8_linear
2396
+ [DEBUG] Backend cuda selected for int8_linear
2397
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2398
+ [DEBUG] Backend cuda selected for int8_linear
2399
+ [DEBUG] Backend cuda selected for int8_linear
2400
+ [DEBUG] Backend cuda selected for int8_linear
2401
+ [DEBUG] Backend cuda selected for int8_linear
2402
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2403
+ [DEBUG] Backend cuda selected for int8_linear
2404
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2405
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2406
+ [DEBUG] Backend cuda selected for int8_linear
2407
+ [DEBUG] Backend cuda selected for int8_linear
2408
+ [DEBUG] Backend cuda selected for int8_linear
2409
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2410
+ [DEBUG] Backend cuda selected for int8_linear
2411
+ [DEBUG] Backend cuda selected for int8_linear
2412
+ [DEBUG] Backend cuda selected for int8_linear
2413
+ [DEBUG] Backend cuda selected for int8_linear
2414
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2415
+ [DEBUG] Backend cuda selected for int8_linear
2416
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2417
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2418
+ [DEBUG] Backend cuda selected for int8_linear
2419
+ [DEBUG] Backend cuda selected for int8_linear
2420
+ [DEBUG] Backend cuda selected for int8_linear
2421
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2422
+ [DEBUG] Backend cuda selected for int8_linear
2423
+ worker-comfyui - Websocket receive timed out. Still waiting...
2424
+ [DEBUG] Backend cuda selected for int8_linear
2425
+ [DEBUG] Backend cuda selected for int8_linear
2426
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=47294 MB
2427
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2428
+
2429
  25%|██▌ | 5/20 [06:40<20:01, 80.11s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
2430
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7d02f96a05f0: Actual Resident VRAM = 512 MB
2431
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cf4ab0e12b0: Actual Resident VRAM = 448 MB
2432
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cf476277d50: Actual Resident VRAM = 25888 MB
2433
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cfdee77f880: Actual Resident VRAM = 20000 MB
2434
+ [DEBUG] aimdo: /project/src/model-vbar.c:90:DEBUG:Total VRAM for VBARs: 46848 MB
2435
+ [DEBUG] Backend cuda selected for int8_linear
2436
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2437
+ [DEBUG] Backend cuda selected for int8_linear
2438
+ [DEBUG] Backend cuda selected for int8_linear
2439
+ [DEBUG] Backend cuda selected for int8_linear
2440
+ [DEBUG] Backend cuda selected for int8_linear
2441
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2442
+ [DEBUG] Backend cuda selected for int8_linear
2443
+ [DEBUG] Backend cuda selected for int8_linear
2444
+ [DEBUG] Backend cuda selected for int8_linear
2445
+ [DEBUG] Backend cuda selected for int8_linear
2446
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2447
+ [DEBUG] Backend cuda selected for int8_linear
2448
+ [DEBUG] Backend cuda selected for int8_linear
2449
+ [DEBUG] Backend cuda selected for int8_linear
2450
+ [DEBUG] Backend cuda selected for int8_linear
2451
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2452
+ [DEBUG] Backend cuda selected for int8_linear
2453
+ [DEBUG] Backend cuda selected for int8_linear
2454
+ [DEBUG] Backend cuda selected for int8_linear
2455
+ [DEBUG] Backend cuda selected for int8_linear
2456
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2457
+ [DEBUG] Backend cuda selected for int8_linear
2458
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2459
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2460
+ [DEBUG] Backend cuda selected for int8_linear
2461
+ [DEBUG] Backend cuda selected for int8_linear
2462
+ [DEBUG] Backend cuda selected for int8_linear
2463
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2464
+ [DEBUG] Backend cuda selected for int8_linear
2465
+ [DEBUG] Backend cuda selected for int8_linear
2466
+ [DEBUG] Backend cuda selected for int8_linear
2467
+ [DEBUG] Backend cuda selected for int8_linear
2468
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2469
+ [DEBUG] Backend cuda selected for int8_linear
2470
+ worker-comfyui - Websocket receive timed out. Still waiting...
2471
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2472
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2473
+ [DEBUG] Backend cuda selected for int8_linear
2474
+ [DEBUG] Backend cuda selected for int8_linear
2475
+ [DEBUG] Backend cuda selected for int8_linear
2476
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2477
+ [DEBUG] Backend cuda selected for int8_linear
2478
+ [DEBUG] Backend cuda selected for int8_linear
2479
+ [DEBUG] Backend cuda selected for int8_linear
2480
+ [DEBUG] Backend cuda selected for int8_linear
2481
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2482
+ [DEBUG] Backend cuda selected for int8_linear
2483
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2484
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2485
+ [DEBUG] Backend cuda selected for int8_linear
2486
+ [DEBUG] Backend cuda selected for int8_linear
2487
+ [DEBUG] Backend cuda selected for int8_linear
2488
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2489
+ [DEBUG] Backend cuda selected for int8_linear
2490
+ [DEBUG] Backend cuda selected for int8_linear
2491
+ [DEBUG] Backend cuda selected for int8_linear
2492
+ [DEBUG] Backend cuda selected for int8_linear
2493
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2494
+ [DEBUG] Backend cuda selected for int8_linear
2495
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2496
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2497
+ [DEBUG] Backend cuda selected for int8_linear
2498
+ [DEBUG] Backend cuda selected for int8_linear
2499
+ [DEBUG] Backend cuda selected for int8_linear
2500
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2501
+ [DEBUG] Backend cuda selected for int8_linear
2502
+ [DEBUG] Backend cuda selected for int8_linear
2503
+ [DEBUG] Backend cuda selected for int8_linear
2504
+ [DEBUG] Backend cuda selected for int8_linear
2505
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2506
+ [DEBUG] Backend cuda selected for int8_linear
2507
+ worker-comfyui - Websocket receive timed out. Still waiting...
2508
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2509
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2510
+ [DEBUG] Backend cuda selected for int8_linear
2511
+ [DEBUG] Backend cuda selected for int8_linear
2512
+ [DEBUG] Backend cuda selected for int8_linear
2513
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2514
+ [DEBUG] Backend cuda selected for int8_linear
2515
+ [DEBUG] Backend cuda selected for int8_linear
2516
+ [DEBUG] Backend cuda selected for int8_linear
2517
+ [DEBUG] Backend cuda selected for int8_linear
2518
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2519
+ [DEBUG] Backend cuda selected for int8_linear
2520
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2521
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2522
+ [DEBUG] Backend cuda selected for int8_linear
2523
+ [DEBUG] Backend cuda selected for int8_linear
2524
+ [DEBUG] Backend cuda selected for int8_linear
2525
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2526
+ [DEBUG] Backend cuda selected for int8_linear
2527
+ [DEBUG] Backend cuda selected for int8_linear
2528
+ [DEBUG] Backend cuda selected for int8_linear
2529
+ [DEBUG] Backend cuda selected for int8_linear
2530
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2531
+ [DEBUG] Backend cuda selected for int8_linear
2532
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2533
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2534
+ [DEBUG] Backend cuda selected for int8_linear
2535
+ [DEBUG] Backend cuda selected for int8_linear
2536
+ [DEBUG] Backend cuda selected for int8_linear
2537
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2538
+ [DEBUG] Backend cuda selected for int8_linear
2539
+ [DEBUG] Backend cuda selected for int8_linear
2540
+ [DEBUG] Backend cuda selected for int8_linear
2541
+ [DEBUG] Backend cuda selected for int8_linear
2542
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2543
+ [DEBUG] Backend cuda selected for int8_linear
2544
+ worker-comfyui - Websocket receive timed out. Still waiting...
2545
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2546
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2547
+ [DEBUG] Backend cuda selected for int8_linear
2548
+ [DEBUG] Backend cuda selected for int8_linear
2549
+ [DEBUG] Backend cuda selected for int8_linear
2550
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2551
+ [DEBUG] Backend cuda selected for int8_linear
2552
+ [DEBUG] Backend cuda selected for int8_linear
2553
+ [DEBUG] Backend cuda selected for int8_linear
2554
+ [DEBUG] Backend cuda selected for int8_linear
2555
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2556
+ [DEBUG] Backend cuda selected for int8_linear
2557
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2558
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2559
+ [DEBUG] Backend cuda selected for int8_linear
2560
+ [DEBUG] Backend cuda selected for int8_linear
2561
+ [DEBUG] Backend cuda selected for int8_linear
2562
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2563
+ [DEBUG] Backend cuda selected for int8_linear
2564
+ [DEBUG] Backend cuda selected for int8_linear
2565
+ [DEBUG] Backend cuda selected for int8_linear
2566
+ [DEBUG] Backend cuda selected for int8_linear
2567
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2568
+ [DEBUG] Backend cuda selected for int8_linear
2569
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2570
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2571
+ [DEBUG] Backend cuda selected for int8_linear
2572
+ [DEBUG] Backend cuda selected for int8_linear
2573
+ [DEBUG] Backend cuda selected for int8_linear
2574
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2575
+ [DEBUG] Backend cuda selected for int8_linear
2576
+ [DEBUG] Backend cuda selected for int8_linear
2577
+ [DEBUG] Backend cuda selected for int8_linear
2578
+ [DEBUG] Backend cuda selected for int8_linear
2579
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2580
+ [DEBUG] Backend cuda selected for int8_linear
2581
+ worker-comfyui - Websocket receive timed out. Still waiting...
2582
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2583
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2584
+ [DEBUG] Backend cuda selected for int8_linear
2585
+ [DEBUG] Backend cuda selected for int8_linear
2586
+ [DEBUG] Backend cuda selected for int8_linear
2587
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2588
+ [DEBUG] Backend cuda selected for int8_linear
2589
+ [DEBUG] Backend cuda selected for int8_linear
2590
+ [DEBUG] Backend cuda selected for int8_linear
2591
+ [DEBUG] Backend cuda selected for int8_linear
2592
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2593
+ [DEBUG] Backend cuda selected for int8_linear
2594
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2595
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2596
+ [DEBUG] Backend cuda selected for int8_linear
2597
+ [DEBUG] Backend cuda selected for int8_linear
2598
+ [DEBUG] Backend cuda selected for int8_linear
2599
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2600
+ [DEBUG] Backend cuda selected for int8_linear
2601
+ [DEBUG] Backend cuda selected for int8_linear
2602
+ [DEBUG] Backend cuda selected for int8_linear
2603
+ [DEBUG] Backend cuda selected for int8_linear
2604
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2605
+ [DEBUG] Backend cuda selected for int8_linear
2606
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2607
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2608
+ [DEBUG] Backend cuda selected for int8_linear
2609
+ [DEBUG] Backend cuda selected for int8_linear
2610
+ [DEBUG] Backend cuda selected for int8_linear
2611
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2612
+ [DEBUG] Backend cuda selected for int8_linear
2613
+ [DEBUG] Backend cuda selected for int8_linear
2614
+ [DEBUG] Backend cuda selected for int8_linear
2615
+ [DEBUG] Backend cuda selected for int8_linear
2616
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2617
+ [DEBUG] Backend cuda selected for int8_linear
2618
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2619
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2620
+ [DEBUG] Backend cuda selected for int8_linear
2621
+ [DEBUG] Backend cuda selected for int8_linear
2622
+ [DEBUG] Backend cuda selected for int8_linear
2623
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2624
+ [DEBUG] Backend cuda selected for int8_linear
2625
+ worker-comfyui - Websocket receive timed out. Still waiting...
2626
+ [DEBUG] Backend cuda selected for int8_linear
2627
+ [DEBUG] Backend cuda selected for int8_linear
2628
+ [DEBUG] Backend cuda selected for int8_linear
2629
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2630
+ [DEBUG] Backend cuda selected for int8_linear
2631
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2632
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2633
+ [DEBUG] Backend cuda selected for int8_linear
2634
+ [DEBUG] Backend cuda selected for int8_linear
2635
+ [DEBUG] Backend cuda selected for int8_linear
2636
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2637
+ [DEBUG] Backend cuda selected for int8_linear
2638
+ [DEBUG] Backend cuda selected for int8_linear
2639
+ [DEBUG] Backend cuda selected for int8_linear
2640
+ [DEBUG] Backend cuda selected for int8_linear
2641
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2642
+ [DEBUG] Backend cuda selected for int8_linear
2643
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2644
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2645
+ [DEBUG] Backend cuda selected for int8_linear
2646
+ [DEBUG] Backend cuda selected for int8_linear
2647
+ [DEBUG] Backend cuda selected for int8_linear
2648
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2649
+ [DEBUG] Backend cuda selected for int8_linear
2650
+ [DEBUG] Backend cuda selected for int8_linear
2651
+ [DEBUG] Backend cuda selected for int8_linear
2652
+ [DEBUG] Backend cuda selected for int8_linear
2653
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2654
+ [DEBUG] Backend cuda selected for int8_linear
2655
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2656
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2657
+ [DEBUG] Backend cuda selected for int8_linear
2658
+ [DEBUG] Backend cuda selected for int8_linear
2659
+ [DEBUG] Backend cuda selected for int8_linear
2660
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2661
+ [DEBUG] Backend cuda selected for int8_linear
2662
+ worker-comfyui - Websocket receive timed out. Still waiting...
2663
+ [DEBUG] Backend cuda selected for int8_linear
2664
+ [DEBUG] Backend cuda selected for int8_linear
2665
+ [DEBUG] Backend cuda selected for int8_linear
2666
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2667
+ [DEBUG] Backend cuda selected for int8_linear
2668
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2669
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2670
+ [DEBUG] Backend cuda selected for int8_linear
2671
+ [DEBUG] Backend cuda selected for int8_linear
2672
+ [DEBUG] Backend cuda selected for int8_linear
2673
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2674
+ [DEBUG] Backend cuda selected for int8_linear
2675
+ [DEBUG] Backend cuda selected for int8_linear
2676
+ [DEBUG] Backend cuda selected for int8_linear
2677
+ [DEBUG] Backend cuda selected for int8_linear
2678
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2679
+ [DEBUG] Backend cuda selected for int8_linear
2680
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2681
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2682
+ [DEBUG] Backend cuda selected for int8_linear
2683
+ [DEBUG] Backend cuda selected for int8_linear
2684
+ [DEBUG] Backend cuda selected for int8_linear
2685
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2686
+ [DEBUG] Backend cuda selected for int8_linear
2687
+ [DEBUG] Backend cuda selected for int8_linear
2688
+ [DEBUG] Backend cuda selected for int8_linear
2689
+ [DEBUG] Backend cuda selected for int8_linear
2690
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2691
+ [DEBUG] Backend cuda selected for int8_linear
2692
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2693
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2694
+ [DEBUG] Backend cuda selected for int8_linear
2695
+ [DEBUG] Backend cuda selected for int8_linear
2696
+ [DEBUG] Backend cuda selected for int8_linear
2697
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2698
+ [DEBUG] Backend cuda selected for int8_linear
2699
+ worker-comfyui - Websocket receive timed out. Still waiting...
2700
+ [DEBUG] Backend cuda selected for int8_linear
2701
+ [DEBUG] Backend cuda selected for int8_linear
2702
+ [DEBUG] Backend cuda selected for int8_linear
2703
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2704
+ [DEBUG] Backend cuda selected for int8_linear
2705
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2706
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2707
+ [DEBUG] Backend cuda selected for int8_linear
2708
+ [DEBUG] Backend cuda selected for int8_linear
2709
+ [DEBUG] Backend cuda selected for int8_linear
2710
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2711
+ [DEBUG] Backend cuda selected for int8_linear
2712
+ [DEBUG] Backend cuda selected for int8_linear
2713
+ [DEBUG] Backend cuda selected for int8_linear
2714
+ [DEBUG] Backend cuda selected for int8_linear
2715
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2716
+ [DEBUG] Backend cuda selected for int8_linear
2717
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2718
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2719
+ [DEBUG] Backend cuda selected for int8_linear
2720
+ [DEBUG] Backend cuda selected for int8_linear
2721
+ [DEBUG] Backend cuda selected for int8_linear
2722
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2723
+ [DEBUG] Backend cuda selected for int8_linear
2724
+ [DEBUG] Backend cuda selected for int8_linear
2725
+ [DEBUG] Backend cuda selected for int8_linear
2726
+ [DEBUG] Backend cuda selected for int8_linear
2727
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2728
+ [DEBUG] Backend cuda selected for int8_linear
2729
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2730
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2731
+ [DEBUG] Backend cuda selected for int8_linear
2732
+ [DEBUG] Backend cuda selected for int8_linear
2733
+ [DEBUG] Backend cuda selected for int8_linear
2734
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2735
+ [DEBUG] Backend cuda selected for int8_linear
2736
+ worker-comfyui - Websocket receive timed out. Still waiting...
2737
+ [DEBUG] Backend cuda selected for int8_linear
2738
+ [DEBUG] Backend cuda selected for int8_linear
2739
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=47294 MB
2740
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2741
+
2742
  30%|███ | 6/20 [08:00<18:43, 80.24s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
2743
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7d02f96a05f0: Actual Resident VRAM = 512 MB
2744
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cf4ab0e12b0: Actual Resident VRAM = 448 MB
2745
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cf476277d50: Actual Resident VRAM = 25888 MB
2746
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cfdee77f880: Actual Resident VRAM = 20000 MB
2747
+ [DEBUG] aimdo: /project/src/model-vbar.c:90:DEBUG:Total VRAM for VBARs: 46848 MB
2748
+ [DEBUG] Backend cuda selected for int8_linear
2749
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2750
+ [DEBUG] Backend cuda selected for int8_linear
2751
+ [DEBUG] Backend cuda selected for int8_linear
2752
+ [DEBUG] Backend cuda selected for int8_linear
2753
+ [DEBUG] Backend cuda selected for int8_linear
2754
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2755
+ [DEBUG] Backend cuda selected for int8_linear
2756
+ [DEBUG] Backend cuda selected for int8_linear
2757
+ [DEBUG] Backend cuda selected for int8_linear
2758
+ [DEBUG] Backend cuda selected for int8_linear
2759
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2760
+ [DEBUG] Backend cuda selected for int8_linear
2761
+ [DEBUG] Backend cuda selected for int8_linear
2762
+ [DEBUG] Backend cuda selected for int8_linear
2763
+ [DEBUG] Backend cuda selected for int8_linear
2764
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2765
+ [DEBUG] Backend cuda selected for int8_linear
2766
+ [DEBUG] Backend cuda selected for int8_linear
2767
+ [DEBUG] Backend cuda selected for int8_linear
2768
+ [DEBUG] Backend cuda selected for int8_linear
2769
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2770
+ [DEBUG] Backend cuda selected for int8_linear
2771
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2772
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2773
+ [DEBUG] Backend cuda selected for int8_linear
2774
+ [DEBUG] Backend cuda selected for int8_linear
2775
+ [DEBUG] Backend cuda selected for int8_linear
2776
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2777
+ [DEBUG] Backend cuda selected for int8_linear
2778
+ [DEBUG] Backend cuda selected for int8_linear
2779
+ [DEBUG] Backend cuda selected for int8_linear
2780
+ [DEBUG] Backend cuda selected for int8_linear
2781
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2782
+ [DEBUG] Backend cuda selected for int8_linear
2783
+ worker-comfyui - Websocket receive timed out. Still waiting...
2784
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2785
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2786
+ [DEBUG] Backend cuda selected for int8_linear
2787
+ [DEBUG] Backend cuda selected for int8_linear
2788
+ [DEBUG] Backend cuda selected for int8_linear
2789
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2790
+ [DEBUG] Backend cuda selected for int8_linear
2791
+ [DEBUG] Backend cuda selected for int8_linear
2792
+ [DEBUG] Backend cuda selected for int8_linear
2793
+ [DEBUG] Backend cuda selected for int8_linear
2794
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2795
+ [DEBUG] Backend cuda selected for int8_linear
2796
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2797
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2798
+ [DEBUG] Backend cuda selected for int8_linear
2799
+ [DEBUG] Backend cuda selected for int8_linear
2800
+ [DEBUG] Backend cuda selected for int8_linear
2801
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2802
+ [DEBUG] Backend cuda selected for int8_linear
2803
+ [DEBUG] Backend cuda selected for int8_linear
2804
+ [DEBUG] Backend cuda selected for int8_linear
2805
+ [DEBUG] Backend cuda selected for int8_linear
2806
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2807
+ [DEBUG] Backend cuda selected for int8_linear
2808
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2809
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2810
+ [DEBUG] Backend cuda selected for int8_linear
2811
+ [DEBUG] Backend cuda selected for int8_linear
2812
+ [DEBUG] Backend cuda selected for int8_linear
2813
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2814
+ [DEBUG] Backend cuda selected for int8_linear
2815
+ [DEBUG] Backend cuda selected for int8_linear
2816
+ [DEBUG] Backend cuda selected for int8_linear
2817
+ [DEBUG] Backend cuda selected for int8_linear
2818
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2819
+ [DEBUG] Backend cuda selected for int8_linear
2820
+ worker-comfyui - Websocket receive timed out. Still waiting...
2821
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2822
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2823
+ [DEBUG] Backend cuda selected for int8_linear
2824
+ [DEBUG] Backend cuda selected for int8_linear
2825
+ [DEBUG] Backend cuda selected for int8_linear
2826
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2827
+ [DEBUG] Backend cuda selected for int8_linear
2828
+ [DEBUG] Backend cuda selected for int8_linear
2829
+ [DEBUG] Backend cuda selected for int8_linear
2830
+ [DEBUG] Backend cuda selected for int8_linear
2831
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2832
+ [DEBUG] Backend cuda selected for int8_linear
2833
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2834
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2835
+ [DEBUG] Backend cuda selected for int8_linear
2836
+ [DEBUG] Backend cuda selected for int8_linear
2837
+ [DEBUG] Backend cuda selected for int8_linear
2838
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2839
+ [DEBUG] Backend cuda selected for int8_linear
2840
+ [DEBUG] Backend cuda selected for int8_linear
2841
+ [DEBUG] Backend cuda selected for int8_linear
2842
+ [DEBUG] Backend cuda selected for int8_linear
2843
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2844
+ [DEBUG] Backend cuda selected for int8_linear
2845
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2846
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2847
+ [DEBUG] Backend cuda selected for int8_linear
2848
+ [DEBUG] Backend cuda selected for int8_linear
2849
+ [DEBUG] Backend cuda selected for int8_linear
2850
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2851
+ [DEBUG] Backend cuda selected for int8_linear
2852
+ [DEBUG] Backend cuda selected for int8_linear
2853
+ [DEBUG] Backend cuda selected for int8_linear
2854
+ [DEBUG] Backend cuda selected for int8_linear
2855
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2856
+ [DEBUG] Backend cuda selected for int8_linear
2857
+ worker-comfyui - Websocket receive timed out. Still waiting...
2858
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2859
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2860
+ [DEBUG] Backend cuda selected for int8_linear
2861
+ [DEBUG] Backend cuda selected for int8_linear
2862
+ [DEBUG] Backend cuda selected for int8_linear
2863
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2864
+ [DEBUG] Backend cuda selected for int8_linear
2865
+ [DEBUG] Backend cuda selected for int8_linear
2866
+ [DEBUG] Backend cuda selected for int8_linear
2867
+ [DEBUG] Backend cuda selected for int8_linear
2868
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2869
+ [DEBUG] Backend cuda selected for int8_linear
2870
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2871
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2872
+ [DEBUG] Backend cuda selected for int8_linear
2873
+ [DEBUG] Backend cuda selected for int8_linear
2874
+ [DEBUG] Backend cuda selected for int8_linear
2875
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2876
+ [DEBUG] Backend cuda selected for int8_linear
2877
+ [DEBUG] Backend cuda selected for int8_linear
2878
+ [DEBUG] Backend cuda selected for int8_linear
2879
+ [DEBUG] Backend cuda selected for int8_linear
2880
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2881
+ [DEBUG] Backend cuda selected for int8_linear
2882
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2883
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2884
+ [DEBUG] Backend cuda selected for int8_linear
2885
+ [DEBUG] Backend cuda selected for int8_linear
2886
+ [DEBUG] Backend cuda selected for int8_linear
2887
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2888
+ [DEBUG] Backend cuda selected for int8_linear
2889
+ [DEBUG] Backend cuda selected for int8_linear
2890
+ [DEBUG] Backend cuda selected for int8_linear
2891
+ [DEBUG] Backend cuda selected for int8_linear
2892
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2893
+ [DEBUG] Backend cuda selected for int8_linear
2894
+ worker-comfyui - Websocket receive timed out. Still waiting...
2895
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2896
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2897
+ [DEBUG] Backend cuda selected for int8_linear
2898
+ [DEBUG] Backend cuda selected for int8_linear
2899
+ [DEBUG] Backend cuda selected for int8_linear
2900
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2901
+ [DEBUG] Backend cuda selected for int8_linear
2902
+ [DEBUG] Backend cuda selected for int8_linear
2903
+ [DEBUG] Backend cuda selected for int8_linear
2904
+ [DEBUG] Backend cuda selected for int8_linear
2905
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2906
+ [DEBUG] Backend cuda selected for int8_linear
2907
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2908
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2909
+ [DEBUG] Backend cuda selected for int8_linear
2910
+ [DEBUG] Backend cuda selected for int8_linear
2911
+ [DEBUG] Backend cuda selected for int8_linear
2912
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2913
+ [DEBUG] Backend cuda selected for int8_linear
2914
+ [DEBUG] Backend cuda selected for int8_linear
2915
+ [DEBUG] Backend cuda selected for int8_linear
2916
+ [DEBUG] Backend cuda selected for int8_linear
2917
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2918
+ [DEBUG] Backend cuda selected for int8_linear
2919
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2920
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2921
+ [DEBUG] Backend cuda selected for int8_linear
2922
+ [DEBUG] Backend cuda selected for int8_linear
2923
+ [DEBUG] Backend cuda selected for int8_linear
2924
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2925
+ [DEBUG] Backend cuda selected for int8_linear
2926
+ [DEBUG] Backend cuda selected for int8_linear
2927
+ [DEBUG] Backend cuda selected for int8_linear
2928
+ [DEBUG] Backend cuda selected for int8_linear
2929
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2930
+ [DEBUG] Backend cuda selected for int8_linear
2931
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2932
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2933
+ [DEBUG] Backend cuda selected for int8_linear
2934
+ [DEBUG] Backend cuda selected for int8_linear
2935
+ [DEBUG] Backend cuda selected for int8_linear
2936
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2937
+ [DEBUG] Backend cuda selected for int8_linear
2938
+ worker-comfyui - Websocket receive timed out. Still waiting...
2939
+ [DEBUG] Backend cuda selected for int8_linear
2940
+ [DEBUG] Backend cuda selected for int8_linear
2941
+ [DEBUG] Backend cuda selected for int8_linear
2942
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2943
+ [DEBUG] Backend cuda selected for int8_linear
2944
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2945
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2946
+ [DEBUG] Backend cuda selected for int8_linear
2947
+ [DEBUG] Backend cuda selected for int8_linear
2948
+ [DEBUG] Backend cuda selected for int8_linear
2949
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2950
+ [DEBUG] Backend cuda selected for int8_linear
2951
+ [DEBUG] Backend cuda selected for int8_linear
2952
+ [DEBUG] Backend cuda selected for int8_linear
2953
+ [DEBUG] Backend cuda selected for int8_linear
2954
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2955
+ [DEBUG] Backend cuda selected for int8_linear
2956
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2957
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2958
+ [DEBUG] Backend cuda selected for int8_linear
2959
+ [DEBUG] Backend cuda selected for int8_linear
2960
+ [DEBUG] Backend cuda selected for int8_linear
2961
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2962
+ [DEBUG] Backend cuda selected for int8_linear
2963
+ [DEBUG] Backend cuda selected for int8_linear
2964
+ [DEBUG] Backend cuda selected for int8_linear
2965
+ [DEBUG] Backend cuda selected for int8_linear
2966
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2967
+ [DEBUG] Backend cuda selected for int8_linear
2968
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2969
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2970
+ [DEBUG] Backend cuda selected for int8_linear
2971
+ [DEBUG] Backend cuda selected for int8_linear
2972
+ [DEBUG] Backend cuda selected for int8_linear
2973
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2974
+ [DEBUG] Backend cuda selected for int8_linear
2975
+ worker-comfyui - Websocket receive timed out. Still waiting...
2976
+ [DEBUG] Backend cuda selected for int8_linear
2977
+ [DEBUG] Backend cuda selected for int8_linear
2978
+ [DEBUG] Backend cuda selected for int8_linear
2979
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2980
+ [DEBUG] Backend cuda selected for int8_linear
2981
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2982
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2983
+ [DEBUG] Backend cuda selected for int8_linear
2984
+ [DEBUG] Backend cuda selected for int8_linear
2985
+ [DEBUG] Backend cuda selected for int8_linear
2986
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2987
+ [DEBUG] Backend cuda selected for int8_linear
2988
+ [DEBUG] Backend cuda selected for int8_linear
2989
+ [DEBUG] Backend cuda selected for int8_linear
2990
+ [DEBUG] Backend cuda selected for int8_linear
2991
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2992
+ [DEBUG] Backend cuda selected for int8_linear
2993
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
2994
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
2995
+ [DEBUG] Backend cuda selected for int8_linear
2996
+ [DEBUG] Backend cuda selected for int8_linear
2997
+ [DEBUG] Backend cuda selected for int8_linear
2998
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
2999
+ [DEBUG] Backend cuda selected for int8_linear
3000
+ [DEBUG] Backend cuda selected for int8_linear
3001
+ [DEBUG] Backend cuda selected for int8_linear
3002
+ [DEBUG] Backend cuda selected for int8_linear
3003
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3004
+ [DEBUG] Backend cuda selected for int8_linear
3005
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3006
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3007
+ [DEBUG] Backend cuda selected for int8_linear
3008
+ [DEBUG] Backend cuda selected for int8_linear
3009
+ [DEBUG] Backend cuda selected for int8_linear
3010
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3011
+ [DEBUG] Backend cuda selected for int8_linear
3012
+ worker-comfyui - Websocket receive timed out. Still waiting...
3013
+ [DEBUG] Backend cuda selected for int8_linear
3014
+ [DEBUG] Backend cuda selected for int8_linear
3015
+ [DEBUG] Backend cuda selected for int8_linear
3016
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3017
+ [DEBUG] Backend cuda selected for int8_linear
3018
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3019
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3020
+ [DEBUG] Backend cuda selected for int8_linear
3021
+ [DEBUG] Backend cuda selected for int8_linear
3022
+ [DEBUG] Backend cuda selected for int8_linear
3023
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3024
+ [DEBUG] Backend cuda selected for int8_linear
3025
+ [DEBUG] Backend cuda selected for int8_linear
3026
+ [DEBUG] Backend cuda selected for int8_linear
3027
+ [DEBUG] Backend cuda selected for int8_linear
3028
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3029
+ [DEBUG] Backend cuda selected for int8_linear
3030
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3031
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3032
+ [DEBUG] Backend cuda selected for int8_linear
3033
+ [DEBUG] Backend cuda selected for int8_linear
3034
+ [DEBUG] Backend cuda selected for int8_linear
3035
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3036
+ [DEBUG] Backend cuda selected for int8_linear
3037
+ [DEBUG] Backend cuda selected for int8_linear
3038
+ [DEBUG] Backend cuda selected for int8_linear
3039
+ [DEBUG] Backend cuda selected for int8_linear
3040
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3041
+ [DEBUG] Backend cuda selected for int8_linear
3042
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3043
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3044
+ [DEBUG] Backend cuda selected for int8_linear
3045
+ [DEBUG] Backend cuda selected for int8_linear
3046
+ [DEBUG] Backend cuda selected for int8_linear
3047
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3048
+ [DEBUG] Backend cuda selected for int8_linear
3049
+ worker-comfyui - Websocket receive timed out. Still waiting...
3050
+ [DEBUG] Backend cuda selected for int8_linear
3051
+ [DEBUG] Backend cuda selected for int8_linear
3052
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=47294 MB
3053
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3054
+
3055
  35%|███▌ | 7/20 [09:20<17:22, 80.16s/it][DEBUG] aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
3056
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7d02f96a05f0: Actual Resident VRAM = 512 MB
3057
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cf4ab0e12b0: Actual Resident VRAM = 448 MB
3058
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cf476277d50: Actual Resident VRAM = 25888 MB
3059
+ [DEBUG] aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x7cfdee77f880: Actual Resident VRAM = 20000 MB
3060
+ [DEBUG] aimdo: /project/src/model-vbar.c:90:DEBUG:Total VRAM for VBARs: 46848 MB
3061
+ [DEBUG] Backend cuda selected for int8_linear
3062
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3063
+ [DEBUG] Backend cuda selected for int8_linear
3064
+ [DEBUG] Backend cuda selected for int8_linear
3065
+ [DEBUG] Backend cuda selected for int8_linear
3066
+ [DEBUG] Backend cuda selected for int8_linear
3067
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3068
+ [DEBUG] Backend cuda selected for int8_linear
3069
+ [DEBUG] Backend cuda selected for int8_linear
3070
+ [DEBUG] Backend cuda selected for int8_linear
3071
+ [DEBUG] Backend cuda selected for int8_linear
3072
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3073
+ [DEBUG] Backend cuda selected for int8_linear
3074
+ [DEBUG] Backend cuda selected for int8_linear
3075
+ [DEBUG] Backend cuda selected for int8_linear
3076
+ [DEBUG] Backend cuda selected for int8_linear
3077
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3078
+ [DEBUG] Backend cuda selected for int8_linear
3079
+ [DEBUG] Backend cuda selected for int8_linear
3080
+ [DEBUG] Backend cuda selected for int8_linear
3081
+ [DEBUG] Backend cuda selected for int8_linear
3082
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3083
+ [DEBUG] Backend cuda selected for int8_linear
3084
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3085
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3086
+ [DEBUG] Backend cuda selected for int8_linear
3087
+ [DEBUG] Backend cuda selected for int8_linear
3088
+ [DEBUG] Backend cuda selected for int8_linear
3089
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3090
+ [DEBUG] Backend cuda selected for int8_linear
3091
+ [DEBUG] Backend cuda selected for int8_linear
3092
+ [DEBUG] Backend cuda selected for int8_linear
3093
+ [DEBUG] Backend cuda selected for int8_linear
3094
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3095
+ [DEBUG] Backend cuda selected for int8_linear
3096
+ worker-comfyui - Websocket receive timed out. Still waiting...
3097
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3098
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3099
+ [DEBUG] Backend cuda selected for int8_linear
3100
+ [DEBUG] Backend cuda selected for int8_linear
3101
+ [DEBUG] Backend cuda selected for int8_linear
3102
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3103
+ [DEBUG] Backend cuda selected for int8_linear
3104
+ [DEBUG] Backend cuda selected for int8_linear
3105
+ [DEBUG] Backend cuda selected for int8_linear
3106
+ [DEBUG] Backend cuda selected for int8_linear
3107
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3108
+ [DEBUG] Backend cuda selected for int8_linear
3109
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3110
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3111
+ [DEBUG] Backend cuda selected for int8_linear
3112
+ [DEBUG] Backend cuda selected for int8_linear
3113
+ [DEBUG] Backend cuda selected for int8_linear
3114
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3115
+ [DEBUG] Backend cuda selected for int8_linear
3116
+ [DEBUG] Backend cuda selected for int8_linear
3117
+ [DEBUG] Backend cuda selected for int8_linear
3118
+ [DEBUG] Backend cuda selected for int8_linear
3119
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3120
+ [DEBUG] Backend cuda selected for int8_linear
3121
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3122
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3123
+ [DEBUG] Backend cuda selected for int8_linear
3124
+ [DEBUG] Backend cuda selected for int8_linear
3125
+ [DEBUG] Backend cuda selected for int8_linear
3126
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3127
+ [DEBUG] Backend cuda selected for int8_linear
3128
+ [DEBUG] Backend cuda selected for int8_linear
3129
+ [DEBUG] Backend cuda selected for int8_linear
3130
+ [DEBUG] Backend cuda selected for int8_linear
3131
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3132
+ [DEBUG] Backend cuda selected for int8_linear
3133
+ worker-comfyui - Websocket receive timed out. Still waiting...
3134
+ [DEBUG] aimdo: /project/src/control.c:170:DEBUG:cuda_budget_deficit: device memory poll free=31436 MB total=97251 MB deficit_sync=-31180 MB recorded=53157 MB
3135
+ [DEBUG] aimdo: /project/src/control.c:174:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
3136
+ [DEBUG] Backend cuda selected for int8_linear
3137
+ [DEBUG] Backend cuda selected for int8_linear
3138
+ [DEBUG] Backend cuda selected for int8_linear
3139
+ [DEBUG] Backend cuda selected for rms_rope_split_half_
3140
+ [DEBUG] Backend cuda selected for int8_linear