darkstudio009 commited on
Commit
f6d0bbf
·
verified ·
1 Parent(s): b75effd

Upload training_log.txt with huggingface_hub

Browse files
Files changed (1) hide show
  1. training_log.txt +5 -23
training_log.txt CHANGED
@@ -1,30 +1,12 @@
1
- Model parameters: 1,103,194,112 (1.103B) | world_size=2 GPUs (FSDP) | ram 19.9 free/30.0 GiB | rss 4.8 GiB
2
  [resume] rank 0 will load vanessa_ckpt_rank0of2.pt from /kaggle/tmp/hub_rank0
3
- [mem-guard pre-resume] need ~11.2 GiB, 19.9 GiB available -> ok
4
  [resume[rank0]] torch.load vanessa_ckpt_rank0of2.pt (6.16 GiB on disk) with {'weights_only': True, 'mmap': True}
5
  [fsdp] fp16 scaler: torch ShardedGradScaler
6
  [resume] restored 201 local parameter shards on this rank.
7
  [resume] optimizer + scaler state restored.
8
  [resume] removed the local copy of vanessa_ckpt_rank0of2.pt after loading.
9
- [resume] resumed from a per-rank sharded checkpoint | ram 27.6 free/30.0 GiB | rss 1.0 GiB | vram 6.2/14.6 GiB
10
  [disk] 1050.2 free/8656.9 GB (reported) | train.bin=50.0GB
11
- Starting loop at step 3,060/95,367 (802,160,640/25,000,000,000 tokens seen so far) | ram 27.6 free/30.0 GiB | rss 1.0 GiB | vram 6.2/14.6 GiB | 0 steps scheduled this session
12
- step 3,080 | loss 3.2877 (avg 3.3304, overall 4.1709) | tokens 807.4M/25B (3.230%) | lr 3.00e-04 | bos_seen 387,933 eos_seen 387,935 | 91.0s/step | ram 26.4 free/30.0 GiB | rss 1.8 GiB | vram 13.4/14.6 GiB peak_vram 13.4 GiB | disk 1050.2 free/8656.9 GB (reported) | elapsed 36.7m, 650m left
13
- step 3,100 | loss 3.3429 (avg 3.3255, overall 4.1654) | tokens 812.6M/25B (3.251%) | lr 3.00e-04 | bos_seen 390,443 eos_seen 390,445 | 87.4s/step | ram 26.4 free/30.0 GiB | rss 1.8 GiB | vram 13.4/14.6 GiB peak_vram 13.4 GiB | disk 1050.2 free/8656.9 GB (reported) | elapsed 65.7m, 621m left
14
- step 3,120 | loss 3.2633 (avg 3.3116, overall 4.1598) | tokens 817.9M/25B (3.272%) | lr 3.00e-04 | bos_seen 393,069 eos_seen 393,071 | 87.0s/step | ram 26.4 free/30.0 GiB | rss 1.8 GiB | vram 13.4/14.6 GiB peak_vram 13.4 GiB | disk 1050.2 free/8656.9 GB (reported) | elapsed 94.7m, 592m left
15
- [checkpoint saved scheduled at step 3,123 | sharded | 6.62 GB/rank | 150s including push | ram 23.6 free/30.0 GiB | rss 4.2 GiB]
16
- [disk] 1036.9 free/8656.9 GB (reported) | train.bin=50.0GB | vanessa_ckpt_rank0of2.pt=6.6GB | vanessa_ckpt_rank1of2.pt=6.6GB
17
- step 3,140 | loss 3.2704 (avg 3.3114, overall 4.1544) | tokens 823.1M/25B (3.293%) | lr 3.00e-04 | bos_seen 395,569 eos_seen 395,571 | 86.8s/step | ram 26.2 free/30.0 GiB | rss 1.7 GiB | vram 13.4/14.6 GiB peak_vram 13.4 GiB | disk 1036.9 free/8656.9 GB (reported) | elapsed 126.2m, 560m left
18
- [checkpoint saved scheduled at step 3,150 | sharded | 6.62 GB/rank | 146s including push | ram 26.0 free/30.0 GiB | rss 1.5 GiB]
19
- [disk] 1023.6 free/8656.9 GB (reported) | train.bin=50.0GB | vanessa_ckpt_rank0of2.pt=6.6GB | vanessa_ckpt_rank0of2.pt.prev=6.6GB | vanessa_ckpt_rank1of2.pt.prev=6.6GB | vanessa_ckpt_rank1of2.pt=6.6GB | hf_cache=0.1GB
20
- step 3,160 | loss 3.3380 (avg 3.3032, overall 4.1489) | tokens 828.4M/25B (3.314%) | lr 3.00e-04 | bos_seen 398,161 eos_seen 398,163 | 87.0s/step | ram 26.0 free/30.0 GiB | rss 1.5 GiB | vram 13.4/14.6 GiB peak_vram 13.4 GiB | disk 1023.6 free/8656.9 GB (reported) | elapsed 157.7m, 529m left
21
- step 3,180 | loss 3.2923 (avg 3.2994, overall 4.1435) | tokens 833.6M/25B (3.334%) | lr 3.00e-04 | bos_seen 400,631 eos_seen 400,633 | 86.9s/step | ram 26.0 free/30.0 GiB | rss 1.5 GiB | vram 13.4/14.6 GiB peak_vram 13.4 GiB | disk 1023.6 free/8656.9 GB (reported) | elapsed 186.7m, 500m left
22
- step 3,200 | loss 3.3295 (avg 3.2933, overall 4.1381) | tokens 838.9M/25B (3.355%) | lr 3.00e-04 | bos_seen 403,305 eos_seen 403,307 | 86.9s/step | ram 26.0 free/30.0 GiB | rss 1.5 GiB | vram 13.4/14.6 GiB peak_vram 13.4 GiB | disk 1023.6 free/8656.9 GB (reported) | elapsed 215.7m, 471m left
23
- [checkpoint saved scheduled at step 3,212 | sharded | 6.62 GB/rank | 173s including push | ram 24.8 free/30.0 GiB | rss 1.5 GiB]
24
- [disk] 1023.6 free/8656.9 GB (reported) | train.bin=50.0GB | vanessa_ckpt_rank0of2.pt=6.6GB | vanessa_ckpt_rank0of2.pt.prev=6.6GB | vanessa_ckpt_rank1of2.pt.prev=6.6GB | vanessa_ckpt_rank1of2.pt=6.6GB | hf_cache=0.1GB
25
- step 3,220 | loss 3.2980 (avg 3.2972, overall 4.1329) | tokens 844.1M/25B (3.376%) | lr 3.00e-04 | bos_seen 405,754 eos_seen 405,756 | 86.4s/step | ram 26.1 free/30.0 GiB | rss 1.5 GiB | vram 13.4/14.6 GiB peak_vram 13.4 GiB | disk 1023.6 free/8656.9 GB (reported) | elapsed 247.5m, 439m left
26
- step 3,240 | loss 3.3301 (avg 3.2949, overall 4.1277) | tokens 849.3M/25B (3.397%) | lr 3.00e-04 | bos_seen 408,310 eos_seen 408,312 | 86.8s/step | ram 26.1 free/30.0 GiB | rss 1.5 GiB | vram 13.4/14.6 GiB peak_vram 13.4 GiB | disk 1023.6 free/8656.9 GB (reported) | elapsed 276.5m, 410m left
27
- step 3,260 | loss 3.2853 (avg 3.2926, overall 4.1225) | tokens 854.6M/25B (3.418%) | lr 3.00e-04 | bos_seen 410,699 eos_seen 410,701 | 86.8s/step | ram 26.1 free/30.0 GiB | rss 1.5 GiB | vram 13.4/14.6 GiB peak_vram 13.4 GiB | disk 1023.6 free/8656.9 GB (reported) | elapsed 305.5m, 381m left
28
- [checkpoint saved scheduled at step 3,275 | sharded | 6.62 GB/rank | 162s including push | ram 25.9 free/30.0 GiB | rss 1.5 GiB]
29
- [disk] 1023.6 free/8656.9 GB (reported) | train.bin=50.0GB | vanessa_ckpt_rank0of2.pt=6.6GB | vanessa_ckpt_rank0of2.pt.prev=6.6GB | vanessa_ckpt_rank1of2.pt.prev=6.6GB | vanessa_ckpt_rank1of2.pt=6.6GB | hf_cache=0.2GB
30
- step 3,280 | loss 3.2913 (avg 3.2838, overall 4.1173) | tokens 859.8M/25B (3.439%) | lr 3.00e-04 | bos_seen 413,257 eos_seen 413,258 | 86.5s/step | ram 26.1 free/30.0 GiB | rss 1.6 GiB | vram 13.4/14.6 GiB peak_vram 13.4 GiB | disk 1023.6 free/8656.9 GB (reported) | elapsed 337.1m, 349m left
 
1
+ Model parameters: 1,103,194,112 (1.103B) | world_size=2 GPUs (FSDP) | ram 20.0 free/30.0 GiB | rss 4.8 GiB
2
  [resume] rank 0 will load vanessa_ckpt_rank0of2.pt from /kaggle/tmp/hub_rank0
3
+ [mem-guard pre-resume] need ~11.2 GiB, 20.0 GiB available -> ok
4
  [resume[rank0]] torch.load vanessa_ckpt_rank0of2.pt (6.16 GiB on disk) with {'weights_only': True, 'mmap': True}
5
  [fsdp] fp16 scaler: torch ShardedGradScaler
6
  [resume] restored 201 local parameter shards on this rank.
7
  [resume] optimizer + scaler state restored.
8
  [resume] removed the local copy of vanessa_ckpt_rank0of2.pt after loading.
9
+ [resume] resumed from a per-rank sharded checkpoint | ram 27.7 free/30.0 GiB | rss 0.9 GiB | vram 6.2/14.6 GiB
10
  [disk] 1050.2 free/8656.9 GB (reported) | train.bin=50.0GB
11
+ Starting loop at step 3,275/95,367 (858,521,600/25,000,000,000 tokens seen so far) | ram 27.7 free/30.0 GiB | rss 0.9 GiB | vram 6.2/14.6 GiB | 0 steps scheduled this session
12
+ step 3,280 | loss 3.2913 (avg 3.2349, overall 4.1173) | tokens 859.8M/25B (3.439%) | lr 3.00e-04 | bos_seen 413,257 eos_seen 413,258 | 108.8s/step | ram 26.6 free/30.0 GiB | rss 1.6 GiB | vram 13.4/14.6 GiB peak_vram 13.4 GiB | disk 1050.2 free/8656.9 GB (reported) | elapsed 15.6m, 671m left