Hugging Face's logo

Spaces:
philippendres
/
CellPilot
No application file

App Files Community
CellPilot
File size: 24,252 Bytes
907462b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
2024-06-19 09:14:01,619 INFO    StreamThr :151176 [internal.py:wandb_internal():85] W&B internal server running at pid: 151176, started at: 2024-06-19 09:14:01.617175
2024-06-19 09:14:01,626 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: status
2024-06-19 09:14:01,628 INFO    WriterThread:151176 [datastore.py:open_for_write():87] open: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/run-sbu1f5uu.wandb
2024-06-19 09:14:01,630 DEBUG   SenderThread:151176 [sender.py:send():378] send: header
2024-06-19 09:14:01,630 DEBUG   SenderThread:151176 [sender.py:send():378] send: run
2024-06-19 09:14:02,056 INFO    SenderThread:151176 [dir_watcher.py:__init__():211] watching files in: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files
2024-06-19 09:14:02,056 INFO    SenderThread:151176 [sender.py:_start_run_threads():1123] run started: sbu1f5uu with start time 1718788441.616118
2024-06-19 09:14:02,072 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: check_version
2024-06-19 09:14:02,073 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: check_version
2024-06-19 09:14:02,204 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: run_start
2024-06-19 09:14:02,222 DEBUG   HandlerThread:151176 [system_info.py:__init__():26] System info init
2024-06-19 09:14:02,222 DEBUG   HandlerThread:151176 [system_info.py:__init__():41] System info init done
2024-06-19 09:14:02,222 INFO    HandlerThread:151176 [system_monitor.py:start():194] Starting system monitor
2024-06-19 09:14:02,223 INFO    SystemMonitor:151176 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-19 09:14:02,223 INFO    HandlerThread:151176 [system_monitor.py:probe():214] Collecting system info
2024-06-19 09:14:02,224 INFO    SystemMonitor:151176 [interfaces.py:start():188] Started cpu monitoring
2024-06-19 09:14:02,228 INFO    SystemMonitor:151176 [interfaces.py:start():188] Started disk monitoring
2024-06-19 09:14:02,229 INFO    SystemMonitor:151176 [interfaces.py:start():188] Started gpu monitoring
2024-06-19 09:14:02,231 INFO    SystemMonitor:151176 [interfaces.py:start():188] Started memory monitoring
2024-06-19 09:14:02,232 INFO    SystemMonitor:151176 [interfaces.py:start():188] Started network monitoring
2024-06-19 09:14:02,246 DEBUG   HandlerThread:151176 [system_info.py:probe():150] Probing system
2024-06-19 09:14:02,250 DEBUG   HandlerThread:151176 [gitlib.py:_init_repo():56] git repository is invalid
2024-06-19 09:14:02,250 DEBUG   HandlerThread:151176 [system_info.py:probe():198] Probing system done
2024-06-19 09:14:02,250 DEBUG   HandlerThread:151176 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-112-generic-x86_64-with-glibc2.35', 'python': '3.10.14', 'heartbeatAt': '2024-06-19T09:14:02.246939', 'startedAt': '2024-06-19T09:14:01.608527', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': '/home/ubuntu/thesis/SAMHI/scripts/app.py', 'codePathLocal': 'SAMHI/scripts/app.py', 'codePath': 'SAMHI/scripts/app.py', 'host': 'dilaboct-5193a', 'username': 'ubuntu', 'executable': '/vol/data/miniconda3/envs/histo3.10/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 2800.0, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 48.27519989013672, 'used': 47.303794860839844}}, 'gpu': 'NVIDIA A40', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA A40', 'memory_total': 48305799168}], 'memory': {'total': 62.792388916015625}}
2024-06-19 09:14:02,250 INFO    HandlerThread:151176 [system_monitor.py:probe():224] Finished collecting system info
2024-06-19 09:14:02,250 INFO    HandlerThread:151176 [system_monitor.py:probe():227] Publishing system info
2024-06-19 09:14:02,250 DEBUG   HandlerThread:151176 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-06-19 09:14:03,058 INFO    Thread-12 :151176 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/conda-environment.yaml
2024-06-19 09:14:04,974 DEBUG   HandlerThread:151176 [system_info.py:_save_conda():222] Saving conda packages done
2024-06-19 09:14:04,975 INFO    HandlerThread:151176 [system_monitor.py:probe():229] Finished publishing system info
2024-06-19 09:14:04,980 DEBUG   SenderThread:151176 [sender.py:send():378] send: files
2024-06-19 09:14:04,981 INFO    SenderThread:151176 [sender.py:_save_file():1389] saving file wandb-metadata.json with policy now
2024-06-19 09:14:05,057 INFO    Thread-12 :151176 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/conda-environment.yaml
2024-06-19 09:14:05,057 INFO    Thread-12 :151176 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/wandb-metadata.json
2024-06-19 09:14:05,275 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: python_packages
2024-06-19 09:14:05,276 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: python_packages
2024-06-19 09:14:05,278 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: stop_status
2024-06-19 09:14:05,279 DEBUG   SenderThread:151176 [sender.py:send():378] send: telemetry
2024-06-19 09:14:05,279 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: stop_status
2024-06-19 09:14:05,654 INFO    wandb-upload_0:151176 [upload_job.py:push():130] Uploaded file /tmp/tmpq49dm8yhwandb/6cjqrpqv-wandb-metadata.json
2024-06-19 09:14:06,057 INFO    Thread-12 :151176 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/output.log
2024-06-19 09:14:06,058 INFO    Thread-12 :151176 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/requirements.txt
2024-06-19 09:14:06,652 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: status_report
2024-06-19 09:14:08,058 INFO    Thread-12 :151176 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/output.log
2024-06-19 09:14:11,655 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: status_report
2024-06-19 09:14:16,657 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: status_report
2024-06-19 09:14:20,276 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: stop_status
2024-06-19 09:14:20,277 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: stop_status
2024-06-19 09:14:22,594 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: status_report
2024-06-19 09:14:27,595 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: status_report
2024-06-19 09:14:32,601 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: status_report
2024-06-19 09:14:33,071 INFO    Thread-12 :151176 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/config.yaml
2024-06-19 09:14:35,277 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: stop_status
2024-06-19 09:14:35,278 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: stop_status
2024-06-19 09:14:38,557 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: status_report
2024-06-19 09:14:42,422 DEBUG   SenderThread:151176 [sender.py:send():378] send: exit
2024-06-19 09:14:42,422 INFO    SenderThread:151176 [sender.py:send_exit():585] handling exit code: 0
2024-06-19 09:14:42,423 INFO    SenderThread:151176 [sender.py:send_exit():587] handling runtime: 40
2024-06-19 09:14:42,425 INFO    SenderThread:151176 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-06-19 09:14:42,426 INFO    SenderThread:151176 [sender.py:send_exit():593] send defer
2024-06-19 09:14:42,426 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: defer
2024-06-19 09:14:42,426 INFO    HandlerThread:151176 [handler.py:handle_request_defer():184] handle defer: 0
2024-06-19 09:14:42,427 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: defer
2024-06-19 09:14:42,427 INFO    SenderThread:151176 [sender.py:send_request_defer():609] handle sender defer: 0
2024-06-19 09:14:42,427 INFO    SenderThread:151176 [sender.py:transition_state():613] send defer: 1
2024-06-19 09:14:42,428 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: defer
2024-06-19 09:14:42,428 INFO    HandlerThread:151176 [handler.py:handle_request_defer():184] handle defer: 1
2024-06-19 09:14:42,428 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: defer
2024-06-19 09:14:42,428 INFO    SenderThread:151176 [sender.py:send_request_defer():609] handle sender defer: 1
2024-06-19 09:14:42,428 INFO    SenderThread:151176 [sender.py:transition_state():613] send defer: 2
2024-06-19 09:14:42,429 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: defer
2024-06-19 09:14:42,429 INFO    HandlerThread:151176 [handler.py:handle_request_defer():184] handle defer: 2
2024-06-19 09:14:42,429 INFO    HandlerThread:151176 [system_monitor.py:finish():203] Stopping system monitor
2024-06-19 09:14:42,430 DEBUG   SystemMonitor:151176 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-19 09:14:42,430 INFO    HandlerThread:151176 [interfaces.py:finish():200] Joined cpu monitor
2024-06-19 09:14:42,430 DEBUG   SystemMonitor:151176 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-06-19 09:14:42,431 INFO    HandlerThread:151176 [interfaces.py:finish():200] Joined disk monitor
2024-06-19 09:14:42,431 DEBUG   SystemMonitor:151176 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-06-19 09:14:42,470 INFO    HandlerThread:151176 [interfaces.py:finish():200] Joined gpu monitor
2024-06-19 09:14:42,471 INFO    HandlerThread:151176 [interfaces.py:finish():200] Joined memory monitor
2024-06-19 09:14:42,471 INFO    HandlerThread:151176 [interfaces.py:finish():200] Joined network monitor
2024-06-19 09:14:42,472 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: defer
2024-06-19 09:14:42,472 INFO    SenderThread:151176 [sender.py:send_request_defer():609] handle sender defer: 2
2024-06-19 09:14:42,472 INFO    SenderThread:151176 [sender.py:transition_state():613] send defer: 3
2024-06-19 09:14:42,472 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: defer
2024-06-19 09:14:42,472 INFO    HandlerThread:151176 [handler.py:handle_request_defer():184] handle defer: 3
2024-06-19 09:14:42,473 DEBUG   SenderThread:151176 [sender.py:send():378] send: stats
2024-06-19 09:14:42,473 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: defer
2024-06-19 09:14:42,473 INFO    SenderThread:151176 [sender.py:send_request_defer():609] handle sender defer: 3
2024-06-19 09:14:42,473 INFO    SenderThread:151176 [sender.py:transition_state():613] send defer: 4
2024-06-19 09:14:42,473 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: defer
2024-06-19 09:14:42,474 INFO    HandlerThread:151176 [handler.py:handle_request_defer():184] handle defer: 4
2024-06-19 09:14:42,474 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: defer
2024-06-19 09:14:42,474 INFO    SenderThread:151176 [sender.py:send_request_defer():609] handle sender defer: 4
2024-06-19 09:14:42,474 INFO    SenderThread:151176 [sender.py:transition_state():613] send defer: 5
2024-06-19 09:14:42,474 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: defer
2024-06-19 09:14:42,474 INFO    HandlerThread:151176 [handler.py:handle_request_defer():184] handle defer: 5
2024-06-19 09:14:42,475 DEBUG   SenderThread:151176 [sender.py:send():378] send: summary
2024-06-19 09:14:42,475 INFO    SenderThread:151176 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-06-19 09:14:42,476 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: defer
2024-06-19 09:14:42,476 INFO    SenderThread:151176 [sender.py:send_request_defer():609] handle sender defer: 5
2024-06-19 09:14:42,476 INFO    SenderThread:151176 [sender.py:transition_state():613] send defer: 6
2024-06-19 09:14:42,476 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: defer
2024-06-19 09:14:42,476 INFO    HandlerThread:151176 [handler.py:handle_request_defer():184] handle defer: 6
2024-06-19 09:14:42,477 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: defer
2024-06-19 09:14:42,477 INFO    SenderThread:151176 [sender.py:send_request_defer():609] handle sender defer: 6
2024-06-19 09:14:42,477 INFO    SenderThread:151176 [sender.py:transition_state():613] send defer: 7
2024-06-19 09:14:42,477 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: status_report
2024-06-19 09:14:42,477 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: defer
2024-06-19 09:14:42,477 INFO    HandlerThread:151176 [handler.py:handle_request_defer():184] handle defer: 7
2024-06-19 09:14:42,478 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: defer
2024-06-19 09:14:42,478 INFO    SenderThread:151176 [sender.py:send_request_defer():609] handle sender defer: 7
2024-06-19 09:14:43,077 INFO    Thread-12 :151176 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/wandb-summary.json
2024-06-19 09:14:43,422 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: poll_exit
2024-06-19 09:14:44,288 INFO    SenderThread:151176 [sender.py:transition_state():613] send defer: 8
2024-06-19 09:14:44,288 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: poll_exit
2024-06-19 09:14:44,289 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: defer
2024-06-19 09:14:44,289 INFO    HandlerThread:151176 [handler.py:handle_request_defer():184] handle defer: 8
2024-06-19 09:14:44,289 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: defer
2024-06-19 09:14:44,290 INFO    SenderThread:151176 [sender.py:send_request_defer():609] handle sender defer: 8
2024-06-19 09:14:44,290 INFO    SenderThread:151176 [job_builder.py:build():432] Attempting to build job artifact
2024-06-19 09:14:44,290 INFO    SenderThread:151176 [job_builder.py:_get_source_type():576] no source found
2024-06-19 09:14:44,290 INFO    SenderThread:151176 [sender.py:transition_state():613] send defer: 9
2024-06-19 09:14:44,290 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: defer
2024-06-19 09:14:44,291 INFO    HandlerThread:151176 [handler.py:handle_request_defer():184] handle defer: 9
2024-06-19 09:14:44,291 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: defer
2024-06-19 09:14:44,291 INFO    SenderThread:151176 [sender.py:send_request_defer():609] handle sender defer: 9
2024-06-19 09:14:44,291 INFO    SenderThread:151176 [dir_watcher.py:finish():358] shutting down directory watcher
2024-06-19 09:14:44,424 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: poll_exit
2024-06-19 09:14:45,078 INFO    Thread-12 :151176 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/output.log
2024-06-19 09:14:45,079 INFO    SenderThread:151176 [dir_watcher.py:finish():388] scan: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files
2024-06-19 09:14:45,079 INFO    SenderThread:151176 [dir_watcher.py:finish():402] scan save: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/wandb-metadata.json wandb-metadata.json
2024-06-19 09:14:45,079 INFO    SenderThread:151176 [dir_watcher.py:finish():402] scan save: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/conda-environment.yaml conda-environment.yaml
2024-06-19 09:14:45,080 INFO    SenderThread:151176 [dir_watcher.py:finish():402] scan save: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/requirements.txt requirements.txt
2024-06-19 09:14:45,080 INFO    SenderThread:151176 [dir_watcher.py:finish():402] scan save: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/output.log output.log
2024-06-19 09:14:45,080 INFO    SenderThread:151176 [dir_watcher.py:finish():402] scan save: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/config.yaml config.yaml
2024-06-19 09:14:45,080 INFO    SenderThread:151176 [dir_watcher.py:finish():402] scan save: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/wandb-summary.json wandb-summary.json
2024-06-19 09:14:45,080 INFO    SenderThread:151176 [sender.py:transition_state():613] send defer: 10
2024-06-19 09:14:45,081 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: poll_exit
2024-06-19 09:14:45,081 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: defer
2024-06-19 09:14:45,081 INFO    HandlerThread:151176 [handler.py:handle_request_defer():184] handle defer: 10
2024-06-19 09:14:45,091 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: defer
2024-06-19 09:14:45,092 INFO    SenderThread:151176 [sender.py:send_request_defer():609] handle sender defer: 10
2024-06-19 09:14:45,092 INFO    SenderThread:151176 [file_pusher.py:finish():169] shutting down file pusher
2024-06-19 09:14:45,425 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: poll_exit
2024-06-19 09:14:45,426 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: poll_exit
2024-06-19 09:14:45,624 INFO    wandb-upload_0:151176 [upload_job.py:push():130] Uploaded file /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/requirements.txt
2024-06-19 09:14:45,962 INFO    wandb-upload_4:151176 [upload_job.py:push():130] Uploaded file /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/wandb-summary.json
2024-06-19 09:14:46,004 INFO    wandb-upload_3:151176 [upload_job.py:push():130] Uploaded file /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/config.yaml
2024-06-19 09:14:46,041 INFO    wandb-upload_2:151176 [upload_job.py:push():130] Uploaded file /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/output.log
2024-06-19 09:14:46,050 INFO    wandb-upload_1:151176 [upload_job.py:push():130] Uploaded file /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/files/conda-environment.yaml
2024-06-19 09:14:46,250 INFO    Thread-11 (_thread_body):151176 [sender.py:transition_state():613] send defer: 11
2024-06-19 09:14:46,251 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: defer
2024-06-19 09:14:46,251 INFO    HandlerThread:151176 [handler.py:handle_request_defer():184] handle defer: 11
2024-06-19 09:14:46,252 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: defer
2024-06-19 09:14:46,252 INFO    SenderThread:151176 [sender.py:send_request_defer():609] handle sender defer: 11
2024-06-19 09:14:46,253 INFO    SenderThread:151176 [file_pusher.py:join():175] waiting for file pusher
2024-06-19 09:14:46,253 INFO    SenderThread:151176 [sender.py:transition_state():613] send defer: 12
2024-06-19 09:14:46,253 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: defer
2024-06-19 09:14:46,254 INFO    HandlerThread:151176 [handler.py:handle_request_defer():184] handle defer: 12
2024-06-19 09:14:46,254 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: defer
2024-06-19 09:14:46,254 INFO    SenderThread:151176 [sender.py:send_request_defer():609] handle sender defer: 12
2024-06-19 09:14:46,254 INFO    SenderThread:151176 [file_stream.py:finish():601] file stream finish called
2024-06-19 09:14:46,427 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: poll_exit
2024-06-19 09:14:46,506 INFO    SenderThread:151176 [file_stream.py:finish():605] file stream finish is done
2024-06-19 09:14:46,506 INFO    SenderThread:151176 [sender.py:transition_state():613] send defer: 13
2024-06-19 09:14:46,507 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: poll_exit
2024-06-19 09:14:46,507 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: defer
2024-06-19 09:14:46,507 INFO    HandlerThread:151176 [handler.py:handle_request_defer():184] handle defer: 13
2024-06-19 09:14:46,508 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: defer
2024-06-19 09:14:46,508 INFO    SenderThread:151176 [sender.py:send_request_defer():609] handle sender defer: 13
2024-06-19 09:14:46,508 INFO    SenderThread:151176 [sender.py:transition_state():613] send defer: 14
2024-06-19 09:14:46,508 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: defer
2024-06-19 09:14:46,509 INFO    HandlerThread:151176 [handler.py:handle_request_defer():184] handle defer: 14
2024-06-19 09:14:46,509 DEBUG   SenderThread:151176 [sender.py:send():378] send: final
2024-06-19 09:14:46,510 DEBUG   SenderThread:151176 [sender.py:send():378] send: footer
2024-06-19 09:14:46,510 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: defer
2024-06-19 09:14:46,510 INFO    SenderThread:151176 [sender.py:send_request_defer():609] handle sender defer: 14
2024-06-19 09:14:46,511 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: poll_exit
2024-06-19 09:14:46,511 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: poll_exit
2024-06-19 09:14:46,512 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: server_info
2024-06-19 09:14:46,513 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: poll_exit
2024-06-19 09:14:46,513 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: poll_exit
2024-06-19 09:14:46,513 DEBUG   SenderThread:151176 [sender.py:send_request():405] send_request: server_info
2024-06-19 09:14:46,517 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: get_summary
2024-06-19 09:14:46,518 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: sampled_history
2024-06-19 09:14:46,519 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: internal_messages
2024-06-19 09:14:46,732 INFO    MainThread:151176 [wandb_run.py:_footer_history_summary_info():3994] rendering history
2024-06-19 09:14:46,733 INFO    MainThread:151176 [wandb_run.py:_footer_history_summary_info():4026] rendering summary
2024-06-19 09:14:46,733 INFO    MainThread:151176 [wandb_run.py:_footer_sync_info():3953] logging synced files
2024-06-19 09:14:46,733 DEBUG   HandlerThread:151176 [handler.py:handle_request():158] handle_request: shutdown
2024-06-19 09:14:46,733 INFO    HandlerThread:151176 [handler.py:finish():882] shutting down handler
2024-06-19 09:14:47,513 INFO    WriterThread:151176 [datastore.py:close():296] close: /home/ubuntu/thesis/wandb/run-20240619_091401-sbu1f5uu/run-sbu1f5uu.wandb
2024-06-19 09:14:47,732 INFO    SenderThread:151176 [sender.py:finish():1545] shutting down sender
2024-06-19 09:14:47,732 INFO    SenderThread:151176 [file_pusher.py:finish():169] shutting down file pusher
2024-06-19 09:14:47,733 INFO    SenderThread:151176 [file_pusher.py:join():175] waiting for file pusher