algomaster99 commited on
Commit
b319e33
·
verified ·
1 Parent(s): c5360f5

Upload checkpoints from pretrain_2.2e18_bf16_qknorm

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
README.md ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # algomaster99/pretrain-2.2e18
2
+
3
+ Pretrain run `pretrain_2.2e18_bf16_qknorm` (see `src/model/train.py`).
4
+
5
+ ## Model config
6
+
7
+ ```
8
+ GPTConfig(vocab_size=32000, block_size=1024, n_layer=11, n_head=16, n_embd=1024, dropout=0.0, bias=True, qk_norm=True)
9
+ ```
10
+
11
+ ## Final training state
12
+
13
+ - iter: 259653
14
+ - final_train_loss: 3.1813
15
+ - final_val_loss: 2.6673
16
+
17
+ ## Files
18
+
19
+ - `ckpt_final.pt` -- final trained weights (`model_state_dict`, `model_cfg`, `iter_num`, `final_train_loss`, `final_val_loss`)
20
+ - `ckpt.pt` -- best-val-loss checkpoint seen during training
21
+ - `ckpt_iter*.pt` -- periodic snapshots (no optimizer state)
22
+
23
+ Load with `torch.load(path, weights_only=False)` and `src.model.gpt.GPTConfig`/`GPT`.
ckpt.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:00b69366e97f357c40fb811723ede3bd64d7de7cf7487bb89d4d2c99f702a95b
3
+ size 689574313
ckpt_final.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cd98c094ef2fb0c41efcd4d98d70be12832d20bb5dd26ccc848bb3d1cf4dacd5
3
+ size 689587013
ckpt_iter0003246.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:067c969c2684c7e87833a01084128f8fbc51bf32dda338da5a293cc677542c2a
3
+ size 689588065
ckpt_iter0006492.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:50baa42935589aea1420a36ea953274958c5d8d8feebea6c57e747081c9efe82
3
+ size 689588065
ckpt_iter0009738.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:22e5776ae2d51b5f6c637842ef75c0e1b5179c57929edb52a52022fd2a167732
3
+ size 689588065
ckpt_iter0012984.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8561a2bdaabb7024f69fbdafc60eab56c42817c30809d9114625705fb71f0080
3
+ size 689588065
ckpt_iter0016230.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c45270034713f4959969d34a329fdfe0afc2e76fed5368c6811d5933e7d3c255
3
+ size 689588065
ckpt_iter0019476.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a91ed563d4e1274d2c2054bc213d9bdd835f6e869f93dc692b486c0cd87e9cb2
3
+ size 689588065
ckpt_iter0022722.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4692775068ec5ad4bd3c8346b62a66b619a3e4d0c243d14447387f5e11771efc
3
+ size 689588065
ckpt_iter0025968.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f6696d6cab52762bdff47d9da418d78832f41f614f1e9ca9001b9abc399a70f7
3
+ size 689588065
ckpt_iter0029214.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8955c279e688f3c97ab3e10612bd256985f5404eede08546f6ad0f982132215e
3
+ size 689588065
ckpt_iter0032460.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4165e6eb9f679770c9597225f455caac8e25baceb471a4fd67e43e27f2ce1da0
3
+ size 689588065
ckpt_iter0035706.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6c2771dafadea41976bbe8b749cdfe7bb5e63a9b0d4cbf5035d88c47bf8ee31b
3
+ size 689588065
ckpt_iter0038952.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f202091ba38599e828949695f2cb1df7f5eae5992d339b8751e603a24dcb42f1
3
+ size 689588065
ckpt_iter0042198.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b8f631bb2b885d3b6f75bcda0ae9c6d1c76c48541be3a346e00febe000aa2bfd
3
+ size 689588065
ckpt_iter0045444.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6dbe20ab9396bbd209f7cf5fb35196713d14d40a96c6f6d6228390341bc3d915
3
+ size 689588065
ckpt_iter0048690.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fb6ca357207b7f50432003d37d1d0ab2b92baefd3e73a162c28b0d87ee757f7b
3
+ size 689588065
ckpt_iter0051936.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f5b160c5662c8bcb3b70d3dbbbc6c09f0fe94081ff2e58af65f127b5a540d1ca
3
+ size 689588065
ckpt_iter0055182.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:983a8a60bd6fdc10d085bf9cff62e2c69bde857336d62c1768b44b999bcd7e3e
3
+ size 689588065
ckpt_iter0058428.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:19a597987cbed154e7ab56078e15e1de4daccaed2380ed8c3a75ef06923bc66c
3
+ size 689588065
ckpt_iter0061674.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1616e643c84a8912eb69f28cb3e4ca4b367f346656d03cf70fb18fa0b768cbe8
3
+ size 689588065
ckpt_iter0064920.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1c00de502e1fdebcb040c0428cb7aba059260f014d1ed67e9c3d7da628529739
3
+ size 689588065
ckpt_iter0068166.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cc8d4ee21081061fad768573fad5ac012aa7ea712d7c642d3d677c1fdc7f2bba
3
+ size 689588065
ckpt_iter0071412.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4c49c0d764e53c9191815dac2197531bd00abc8bd8fcac1138f22598a1fd87eb
3
+ size 689588065
ckpt_iter0074658.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9c199c302d04d91762b72df28f4634049ab9f108352198ad766d9fdbea7159f2
3
+ size 689588065
ckpt_iter0077904.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fe1c8139ea85d5bc605541089140d1944dacaee24e7276050ef089a0f4dabc95
3
+ size 689588065
ckpt_iter0081150.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5c9e57e42093e1502faed96cffce93d9a0548bb1f59e6616566af51fcd4011af
3
+ size 689588065
ckpt_iter0084396.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:32efcd3e8e28d53fea5a96936622c8f958ff636d9e52282ed9fbdb2ab3ce8d8e
3
+ size 689588065
ckpt_iter0087642.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:225f9bb488d5cd7d9627b6c8fedafa36b5fb3c880f0fdc48946bba49cf75114a
3
+ size 689588065
ckpt_iter0090888.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6c66f9e6edf5b6865d13cb6111c62f58514730f9200accd786e0fc8a4eb519e3
3
+ size 689588065
ckpt_iter0094134.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8cbf42c2b9a72ede2444c8f568fe45b78ce2ae45385ae41e5c1f706e1b647a13
3
+ size 689588065
ckpt_iter0097380.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:33df769767ccdb755364b92a0302757571826d29594e250938473ab0c82b60ef
3
+ size 689588065
ckpt_iter0100626.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ebbc80debac7da89def4fd448ec60e12a3f03c5e93ef970afdb43c27cb0adb21
3
+ size 689588065
ckpt_iter0103872.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bfa99c308f58288aa4cce0172d3ca6effbaeecd8d3a88bcbd7f61dc212d37e3e
3
+ size 689588065
ckpt_iter0107118.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e9ce854b1931710504f320486c9aba996ed7fdac39ad45a8485ccc276e65e234
3
+ size 689588065
ckpt_iter0110364.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4047e2e954c5f541e4b0d79a3af430558c3e19c31881809b4dbb189779d11ff8
3
+ size 689588065
ckpt_iter0113610.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2ea0b28da5b9993661231baf2dd05fc2768fa27577274405dd68ae21c43cd384
3
+ size 689588065
ckpt_iter0116856.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:deb5ce96b27961db8c5462a0e1f2c332dc27cec09dc3471b7a82b117133fa359
3
+ size 689588065
ckpt_iter0120102.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:98ab49311864383967159512e3b1ac65a8dee7c1664f0513ee207e3c30223a74
3
+ size 689588065
ckpt_iter0123348.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ac0223812f37472d380422dad6b27b5311ce1e3bc114404e7eea8d095d0eafa1
3
+ size 689588065
ckpt_iter0126594.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fb5c53d243c774aae7fe79ccb6fa0257c816dffaa4316632e675a48331aac952
3
+ size 689588065
ckpt_iter0129840.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:995a0106dadfb0b5fb69389d6e82951984a71041abb4dc4491cd90cabbe4247f
3
+ size 689588065
ckpt_iter0133086.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f1892965065f18e3f2df44813dc6dfc07c3d8fb1b14b5838833b8e6099c1b1c0
3
+ size 689588065
ckpt_iter0136332.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2e3bf7aa646fe9512fdc67982be9b70ab00a6a01fafc0d574ca5dc0049a429e3
3
+ size 689588065
ckpt_iter0139578.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:95767e3e67ca62c7d9cc488abe7594d9579facea6801af5a1171710ec1f69d9a
3
+ size 689588065
ckpt_iter0142824.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:917c1ace0d00de5bffb4ea789e95062aae9693ea28df1982dd547ca72c68830f
3
+ size 689588065
ckpt_iter0146070.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:413010c07e113c8d8d67449e60066121312d60bf6f7a4950399cbb47503c89d0
3
+ size 689588065
ckpt_iter0149316.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2dbea4764dc22ed1455fd9df143e0fa2ab895ee5f0251ae6ca0399df8b8d19f1
3
+ size 689588065
ckpt_iter0152562.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:699a849519a25fc40afc959cb2fc3461e1a3f926ee02205ff92b36568aa1f80e
3
+ size 689588065