Hugging Face's logo

FINAL-Bench
/
Aether-7B-5Attn

Text Generation
Transformers
Safetensors
Korean
English
aether_v2_7way
foundation-model
sovereign-ai
fully-open
open-source
mixture-of-experts
Mixture of Experts
heterogeneous-attention
latin-square
from-scratch
reproducible
pretrained
korean
vidraft
aether
conversational
custom_code
Model card Files
xet
 Community
SeaWolf-AI commited on
Commit
6d59546
·
verified ·
1 Parent(s): 4c98e6c

sanitize: internal paths/ids/comments

Browse files
Files changed (1) hide show
  1. logs/launcher_v2b_multi.log +24 -24
logs/launcher_v2b_multi.log CHANGED
@@ -4,12 +4,12 @@
4
  [2026-06-15 15:55:17][r2263068] ====================================================================
5
  [2026-06-15 15:55:17][r2263068] AETHER-2B-7Attn Phase 1.6 v2b — Continued Pretraining
6
  [2026-06-15 15:55:17][r2263068] AETHER-2B-7Attn Phase 1.6 v2b — Continued Pretraining
7
- [2026-06-15 15:55:17][r2263068] resume ckpt : /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/runs/v2b_phase16/ckpt/step_0001905
8
- [2026-06-15 15:55:17][r2263068] resume ckpt : /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/runs/v2b_phase16/ckpt/step_0001905
9
- [2026-06-15 15:55:17][r2263068] output : /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/runs/v2b_multi_phase16
10
- [2026-06-15 15:55:17][r2263068] output : /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/runs/v2b_multi_phase16
11
- [2026-06-15 15:55:17][r2263068] log : /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/logs/launcher_v2b_multi.log
12
- [2026-06-15 15:55:17][r2263068] log : /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/logs/launcher_v2b_multi.log
13
  [2026-06-15 15:55:17][r2263068] world_size : 16
14
  [2026-06-15 15:55:17][r2263068] world_size : 16
15
  [2026-06-15 15:55:17][r2263068] LR : 5e-05 → 5e-06 warmup=0
@@ -108,8 +108,8 @@ AETHERV27wayForCausalLM has generative capabilities, as `prepare_inputs_for_gene
108
  [2026-06-15 16:01:00][r2263068] [3/5] Building optimizer (AdamW + LWD)...
109
  [2026-06-15 16:01:00][r2263068] param groups: 2
110
  [2026-06-15 16:01:00][r2263068] param groups: 2
111
- [2026-06-15 16:01:00][r2263068] [4/5] Loading DCP checkpoint: /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/runs/v2b_phase16/ckpt/step_0001905 (model-only, optimizer fresh for 16-GPU)
112
- [2026-06-15 16:01:00][r2263068] [4/5] Loading DCP checkpoint: /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/runs/v2b_phase16/ckpt/step_0001905 (model-only, optimizer fresh for 16-GPU)
113
  /usr/local/lib/python3.12/dist-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:680: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
114
  warnings.warn(
115
  /usr/local/lib/python3.12/dist-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:680: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
@@ -500,8 +500,8 @@ AETHERV27wayForCausalLM has generative capabilities, as `prepare_inputs_for_gene
500
  [2026-06-15 17:12:48][r2263068] [step 2150/324000] loss=20.6418 lr=5.00e-05 tps=31961 total=60.4B tok
501
  [2026-06-15 17:26:28][r2263068] [step 2200/324000] loss=20.6645 lr=5.00e-05 tps=31990 total=60.5B tok
502
  [2026-06-15 17:26:28][r2263068] [step 2200/324000] loss=20.6645 lr=5.00e-05 tps=31990 total=60.5B tok
503
- [2026-06-15 17:33:51][r2263068] [kill] /NHNHOME/STOP_TRAINING_V2B_MULTI → save+exit
504
- [2026-06-15 17:33:51][r2263068] [kill] /NHNHOME/STOP_TRAINING_V2B_MULTI → save+exit
505
  [rank2]:[E615 17:43:51.658291363 ProcessGroupNCCL.cpp:633] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=388567, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=600000) ran for 600000 milliseconds before timing out.
506
  [rank2]:[E615 17:43:51.658477401 ProcessGroupNCCL.cpp:2289] [PG ID 0 PG GUID 0(default_pg) Rank 2] failure detected by watchdog at work sequence id: 388567 PG status: last enqueued work: 388567, last completed work: 388566
507
  [rank2]:[E615 17:43:51.658486935 ProcessGroupNCCL.cpp:671] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
@@ -1040,7 +1040,7 @@ frame #3: <unknown function> + 0xecdb4 (0x734c84ac7db4 in /lib/x86_64-linux-gnu/
1040
  frame #4: <unknown function> + 0x9caa4 (0x734dae600aa4 in /lib/x86_64-linux-gnu/libc.so.6)
1041
  frame #5: <unknown function> + 0x129c6c (0x734dae68dc6c in /lib/x86_64-linux-gnu/libc.so.6)
1042
 
1043
- [W615 17:43:58.175298711 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[DCTN-0527125535-2]:58606, remote=[DCTN-0527125535-1]:29500): Broken pipe
1044
  Exception raised from sendBytes at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/Utils.hpp:646 (most recent call first):
1045
  frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x70fdf276f368 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
1046
  frame #1: <unknown function> + 0x590544e (0x70fe46b1a44e in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
@@ -1061,8 +1061,8 @@ frame #15: /usr/bin/python() [0x6b8fec]
1061
  frame #16: <unknown function> + 0x9caa4 (0x70fe5bc38aa4 in /lib/x86_64-linux-gnu/libc.so.6)
1062
  frame #17: <unknown function> + 0x129c6c (0x70fe5bcc5c6c in /lib/x86_64-linux-gnu/libc.so.6)
1063
 
1064
- W0615 17:43:58.269000 2215298 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1333] The node 'DCTN-0527125535-2_2215298_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
1065
- [W615 17:44:03.178941908 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[DCTN-0527125535-2]:58606, remote=[DCTN-0527125535-1]:29500): Broken pipe
1066
  Exception raised from sendBytes at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/Utils.hpp:646 (most recent call first):
1067
  frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x70fdf276f368 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
1068
  frame #1: <unknown function> + 0x590544e (0x70fe46b1a44e in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
@@ -1083,8 +1083,8 @@ frame #15: /usr/bin/python() [0x6b8fec]
1083
  frame #16: <unknown function> + 0x9caa4 (0x70fe5bc38aa4 in /lib/x86_64-linux-gnu/libc.so.6)
1084
  frame #17: <unknown function> + 0x129c6c (0x70fe5bcc5c6c in /lib/x86_64-linux-gnu/libc.so.6)
1085
 
1086
- W0615 17:44:03.273000 2215298 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1333] The node 'DCTN-0527125535-2_2215298_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
1087
- [W615 17:44:08.182724193 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[DCTN-0527125535-2]:58606, remote=[DCTN-0527125535-1]:29500): Broken pipe
1088
  Exception raised from sendBytes at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/Utils.hpp:646 (most recent call first):
1089
  frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x70fdf276f368 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
1090
  frame #1: <unknown function> + 0x590544e (0x70fe46b1a44e in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
@@ -1105,18 +1105,18 @@ frame #15: /usr/bin/python() [0x6b8fec]
1105
  frame #16: <unknown function> + 0x9caa4 (0x70fe5bc38aa4 in /lib/x86_64-linux-gnu/libc.so.6)
1106
  frame #17: <unknown function> + 0x129c6c (0x70fe5bcc5c6c in /lib/x86_64-linux-gnu/libc.so.6)
1107
 
1108
- W0615 17:44:08.276000 2215298 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1333] The node 'DCTN-0527125535-2_2215298_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
1109
  [rank5]:[W615 17:45:17.956950325 ProcessGroupNCCL.cpp:4782] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
1110
  [2026-06-15 17:45:18][r2273490] ====================================================================
1111
  [2026-06-15 17:45:18][r2273490] ====================================================================
1112
  [2026-06-15 17:45:18][r2273490] AETHER-2B-7Attn Phase 1.6 v2b — Continued Pretraining
1113
  [2026-06-15 17:45:18][r2273490] AETHER-2B-7Attn Phase 1.6 v2b — Continued Pretraining
1114
- [2026-06-15 17:45:18][r2273490] resume ckpt : /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/runs/v2b_phase16/ckpt/step_0001905
1115
- [2026-06-15 17:45:18][r2273490] resume ckpt : /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/runs/v2b_phase16/ckpt/step_0001905
1116
- [2026-06-15 17:45:18][r2273490] output : /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/runs/v2b_multi_phase16
1117
- [2026-06-15 17:45:18][r2273490] output : /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/runs/v2b_multi_phase16
1118
- [2026-06-15 17:45:18][r2273490] log : /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/logs/launcher_v2b_multi.log
1119
- [2026-06-15 17:45:18][r2273490] log : /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/logs/launcher_v2b_multi.log
1120
  [2026-06-15 17:45:18][r2273490] world_size : 16
1121
  [2026-06-15 17:45:18][r2273490] world_size : 16
1122
  [2026-06-15 17:45:18][r2273490] LR : 5e-05 → 5e-06 warmup=0
@@ -1216,8 +1216,8 @@ AETHERV27wayForCausalLM has generative capabilities, as `prepare_inputs_for_gene
1216
  [2026-06-15 17:51:03][r2273490] [3/5] Building optimizer (AdamW + LWD)...
1217
  [2026-06-15 17:51:03][r2273490] param groups: 2
1218
  [2026-06-15 17:51:03][r2273490] param groups: 2
1219
- [2026-06-15 17:51:03][r2273490] [4/5] Loading DCP checkpoint: /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/runs/v2b_phase16/ckpt/step_0001905 (model-only, optimizer fresh for 16-GPU)
1220
- [2026-06-15 17:51:03][r2273490] [4/5] Loading DCP checkpoint: /NHNHOME/WORKSPACE/0426030036_A/aether_1b_7attn_v3/runs/v2b_phase16/ckpt/step_0001905 (model-only, optimizer fresh for 16-GPU)
1221
  /usr/local/lib/python3.12/dist-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:680: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
1222
  warnings.warn(
1223
  /usr/local/lib/python3.12/dist-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:680: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
 
4
  [2026-06-15 15:55:17][r2263068] ====================================================================
5
  [2026-06-15 15:55:17][r2263068] AETHER-2B-7Attn Phase 1.6 v2b — Continued Pretraining
6
  [2026-06-15 15:55:17][r2263068] AETHER-2B-7Attn Phase 1.6 v2b — Continued Pretraining
7
+ [2026-06-15 15:55:17][r2263068] resume ckpt : /workspace/aether/runs/v2b_phase16/ckpt/step_0001905
8
+ [2026-06-15 15:55:17][r2263068] resume ckpt : /workspace/aether/runs/v2b_phase16/ckpt/step_0001905
9
+ [2026-06-15 15:55:17][r2263068] output : /workspace/aether/runs/v2b_multi_phase16
10
+ [2026-06-15 15:55:17][r2263068] output : /workspace/aether/runs/v2b_multi_phase16
11
+ [2026-06-15 15:55:17][r2263068] log : /workspace/aether/logs/launcher_v2b_multi.log
12
+ [2026-06-15 15:55:17][r2263068] log : /workspace/aether/logs/launcher_v2b_multi.log
13
  [2026-06-15 15:55:17][r2263068] world_size : 16
14
  [2026-06-15 15:55:17][r2263068] world_size : 16
15
  [2026-06-15 15:55:17][r2263068] LR : 5e-05 → 5e-06 warmup=0
 
108
  [2026-06-15 16:01:00][r2263068] [3/5] Building optimizer (AdamW + LWD)...
109
  [2026-06-15 16:01:00][r2263068] param groups: 2
110
  [2026-06-15 16:01:00][r2263068] param groups: 2
111
+ [2026-06-15 16:01:00][r2263068] [4/5] Loading DCP checkpoint: /workspace/aether/runs/v2b_phase16/ckpt/step_0001905 (model-only, optimizer fresh for 16-GPU)
112
+ [2026-06-15 16:01:00][r2263068] [4/5] Loading DCP checkpoint: /workspace/aether/runs/v2b_phase16/ckpt/step_0001905 (model-only, optimizer fresh for 16-GPU)
113
  /usr/local/lib/python3.12/dist-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:680: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
114
  warnings.warn(
115
  /usr/local/lib/python3.12/dist-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:680: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
 
500
  [2026-06-15 17:12:48][r2263068] [step 2150/324000] loss=20.6418 lr=5.00e-05 tps=31961 total=60.4B tok
501
  [2026-06-15 17:26:28][r2263068] [step 2200/324000] loss=20.6645 lr=5.00e-05 tps=31990 total=60.5B tok
502
  [2026-06-15 17:26:28][r2263068] [step 2200/324000] loss=20.6645 lr=5.00e-05 tps=31990 total=60.5B tok
503
+ [2026-06-15 17:33:51][r2263068] [kill] /workspace/STOP_TRAINING_V2B_MULTI → save+exit
504
+ [2026-06-15 17:33:51][r2263068] [kill] /workspace/STOP_TRAINING_V2B_MULTI → save+exit
505
  [rank2]:[E615 17:43:51.658291363 ProcessGroupNCCL.cpp:633] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=388567, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=600000) ran for 600000 milliseconds before timing out.
506
  [rank2]:[E615 17:43:51.658477401 ProcessGroupNCCL.cpp:2289] [PG ID 0 PG GUID 0(default_pg) Rank 2] failure detected by watchdog at work sequence id: 388567 PG status: last enqueued work: 388567, last completed work: 388566
507
  [rank2]:[E615 17:43:51.658486935 ProcessGroupNCCL.cpp:671] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
 
1040
  frame #4: <unknown function> + 0x9caa4 (0x734dae600aa4 in /lib/x86_64-linux-gnu/libc.so.6)
1041
  frame #5: <unknown function> + 0x129c6c (0x734dae68dc6c in /lib/x86_64-linux-gnu/libc.so.6)
1042
 
1043
+ [W615 17:43:58.175298711 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[node]:58606, remote=[node]:29500): Broken pipe
1044
  Exception raised from sendBytes at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/Utils.hpp:646 (most recent call first):
1045
  frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x70fdf276f368 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
1046
  frame #1: <unknown function> + 0x590544e (0x70fe46b1a44e in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
 
1061
  frame #16: <unknown function> + 0x9caa4 (0x70fe5bc38aa4 in /lib/x86_64-linux-gnu/libc.so.6)
1062
  frame #17: <unknown function> + 0x129c6c (0x70fe5bcc5c6c in /lib/x86_64-linux-gnu/libc.so.6)
1063
 
1064
+ W0615 17:43:58.269000 2215298 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1333] The node 'node_2215298_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
1065
+ [W615 17:44:03.178941908 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[node]:58606, remote=[node]:29500): Broken pipe
1066
  Exception raised from sendBytes at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/Utils.hpp:646 (most recent call first):
1067
  frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x70fdf276f368 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
1068
  frame #1: <unknown function> + 0x590544e (0x70fe46b1a44e in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
 
1083
  frame #16: <unknown function> + 0x9caa4 (0x70fe5bc38aa4 in /lib/x86_64-linux-gnu/libc.so.6)
1084
  frame #17: <unknown function> + 0x129c6c (0x70fe5bcc5c6c in /lib/x86_64-linux-gnu/libc.so.6)
1085
 
1086
+ W0615 17:44:03.273000 2215298 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1333] The node 'node_2215298_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
1087
+ [W615 17:44:08.182724193 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[node]:58606, remote=[node]:29500): Broken pipe
1088
  Exception raised from sendBytes at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/Utils.hpp:646 (most recent call first):
1089
  frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x70fdf276f368 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
1090
  frame #1: <unknown function> + 0x590544e (0x70fe46b1a44e in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
 
1105
  frame #16: <unknown function> + 0x9caa4 (0x70fe5bc38aa4 in /lib/x86_64-linux-gnu/libc.so.6)
1106
  frame #17: <unknown function> + 0x129c6c (0x70fe5bcc5c6c in /lib/x86_64-linux-gnu/libc.so.6)
1107
 
1108
+ W0615 17:44:08.276000 2215298 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1333] The node 'node_2215298_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
1109
  [rank5]:[W615 17:45:17.956950325 ProcessGroupNCCL.cpp:4782] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
1110
  [2026-06-15 17:45:18][r2273490] ====================================================================
1111
  [2026-06-15 17:45:18][r2273490] ====================================================================
1112
  [2026-06-15 17:45:18][r2273490] AETHER-2B-7Attn Phase 1.6 v2b — Continued Pretraining
1113
  [2026-06-15 17:45:18][r2273490] AETHER-2B-7Attn Phase 1.6 v2b — Continued Pretraining
1114
+ [2026-06-15 17:45:18][r2273490] resume ckpt : /workspace/aether/runs/v2b_phase16/ckpt/step_0001905
1115
+ [2026-06-15 17:45:18][r2273490] resume ckpt : /workspace/aether/runs/v2b_phase16/ckpt/step_0001905
1116
+ [2026-06-15 17:45:18][r2273490] output : /workspace/aether/runs/v2b_multi_phase16
1117
+ [2026-06-15 17:45:18][r2273490] output : /workspace/aether/runs/v2b_multi_phase16
1118
+ [2026-06-15 17:45:18][r2273490] log : /workspace/aether/logs/launcher_v2b_multi.log
1119
+ [2026-06-15 17:45:18][r2273490] log : /workspace/aether/logs/launcher_v2b_multi.log
1120
  [2026-06-15 17:45:18][r2273490] world_size : 16
1121
  [2026-06-15 17:45:18][r2273490] world_size : 16
1122
  [2026-06-15 17:45:18][r2273490] LR : 5e-05 → 5e-06 warmup=0
 
1216
  [2026-06-15 17:51:03][r2273490] [3/5] Building optimizer (AdamW + LWD)...
1217
  [2026-06-15 17:51:03][r2273490] param groups: 2
1218
  [2026-06-15 17:51:03][r2273490] param groups: 2
1219
+ [2026-06-15 17:51:03][r2273490] [4/5] Loading DCP checkpoint: /workspace/aether/runs/v2b_phase16/ckpt/step_0001905 (model-only, optimizer fresh for 16-GPU)
1220
+ [2026-06-15 17:51:03][r2273490] [4/5] Loading DCP checkpoint: /workspace/aether/runs/v2b_phase16/ckpt/step_0001905 (model-only, optimizer fresh for 16-GPU)
1221
  /usr/local/lib/python3.12/dist-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:680: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
1222
  warnings.warn(
1223
  /usr/local/lib/python3.12/dist-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:680: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .