aryan5555/5555LM_20M

Model architecture(5555LM model family): Decoder-only Transformer(Causal autoregressive language model) Pre-RMSNorm architecture Grouped-Query Attention (GQA) Rotary Positional Embeddings (RoPE) Causal self-attention SwiGLU feed-forward network with SiLU Residual connections Attention & hidden-state dropout

Model/train parameters: { "model_name": "model5555", "dataset_dir": "dataset", "train_bin": "dataset/train.bin", "val_bin": "dataset/val.bin", "dataset_dtype": "uint16", "tokenizer_path": "tokenizer/tokenizer.json", "vocab_size": 8192, "hidden_size": 256, "num_hidden_layers": 16, "intermediate_size": 1280, "num_attention_heads": 8, "num_key_value_heads": 2, "attention_dropout": 0.0, "hidden_dropout": 0.0, "rms_norm_eps": 0.00001, "rope_theta": 10000.0, "max_seq_len": 256, "tie_word_embeddings": true, "initializer_range": 0.02, "batch_size": 128, "gradient_accumulation_steps": 16, "max_iters": 1000, "eval_interval": 100, "eval_iters": 50, "log_interval": 10, "eval_only": false, "learning_rate": 0.001, "min_lr": 0.0001, "weight_decay": 0.1, "beta1": 0.9, "beta2": 0.95, "grad_clip": 1.0, "lr_decay": true, "warmup_iters": 100, "lr_decay_iters": 1000, "device": "cuda", "dtype": "float16", "compile_model": true, "backend": "nccl", "out_dir": "out_model5555", "checkpoint_name": "ckpt.pt", "save_checkpoint": true, "always_save_checkpoint": true, "init_from": "scratch", "seed": 1337, "print_model_info": true, "peak_flops": null, "sample_count": 5, "sample_prompt_tokens": 16, "sample_new_tokens": 80, "sample_temperature": 0.8, "sample_top_k": 50, "sample_top_p": 0.95, "eos_token_id": null } (note: this file is the original config.json used in training)

5555LM_20M is the first model of the 5555LM Ai LM family. This model has been trained on google collab on 1x Tesla T4. VRAM usage : 10-13 GB / 15GB (unlogged?)

This model uses the 5555LM_tokenizer8192_BBPE as a JSON file.

Train metrics:

/content/nGPT/test5 device=cuda, world_size=1, dtype=float16 tokens per optimizer step=524,288 parameters=20,455,680, fp16 size=39.016113 MB W0926 00:28:59.301000 3324 torch/_inductor/utils.py:1731] [2/0] Not enough SMs to use max_autotune_gemm mode step=000000 train_loss=9.020222663879395 val_loss=9.02020263671875 lr=9.900990099009901e-06 samples: [1] play with it tomorrow."

Tim and Sam agreed to share the mar custom Sometimes captch necessarilyitalital Spe max X� breast."orp strongorp strong Sum��oughtDon coinatorsators lay inchesâ rele inches 35 Democr�� electricking� Spe thrig XReg watched flebersbers� electricutions medicIS James James�rip analyst hear hear fabric fabriculty phil did phil pointoman regional regional Que Bat�ke wood woodzzM�ib rejectiod [2] next weekend. The sun was shining. Little Jenny was playing in polic derwich pickLetduublic diwo club August Augustoring helpingitureody stay afternoon sumists licensomin routeagesureau skynessness sum estim Word destination markets developing clinical significant GMness stay noise Energy� Pass CAsel chromatographyFree clinical L order orderarrant -- poll helpingages teniturerit thoughtoringreen continued ideas remind sum regularly helping destinationitureselselowl sum B consc mile attachedneri [3] the tiger is attractive. She wants to try the new cereal.most Childension glizesanks Jacksonupsiger modeain hasn� Per Sus enforcegot Radxtelxtel ;ute efficient Jackson plays plays plays plays those Jackson Jackson�aught dynam dynam mini dynamgot Training beat customer Rad� scream Rad2008 Communityaf dynam� cou Ira dynamiments entertainment miniimentsrought beat isn Diego minigotingildizes native private hatartedarted JacksonimentseHe�got dynam conce mini [4] "Yes, you can ride it too. But you'll have to wait until literLastyn pull feature outarn tomorrow visitors several appearanceED himself himselfulation taken Sun Tre prepare whenware AM when whenEx date flatWithistics Friday buyulation alreadyings Agency regulation regulation payment 07inet E their existingSerom scream Turkey Turkey Turkey Agencyvers His printed Ir Maybe determ supported helping court Up keepsanny bornchan visitors prepare bike mixinetinetinet cities cities visitors visitors Josephising Plan Ge pun [5] mommy and daddy were so proud. They thanked the sweetwichened white manufact manufactsequri regular on;; manufacthent� assist ab incredible incrediblesequ onribleurningrit academ realize realize opened gall gall bus coverretereteri holds holds assist} regularly impro impro gall Sh Ireland academ 3 iron bus beginsoid Russ gall gall gall holds enableMost conver merely regularlyorders fro merely buy gall Su leave leaveiveness detild ge dead responsibilitywich Peterwood27FindFind iter=000000 loss=9.019601821899414 time=96.61564s tokens/s=5426.5334 FLOP/token=42991616 FLOP/s=2.3329544e+11 MFU=n/a iter=000010 loss=8.17706298828125 time=6.7839785s tokens/s=77283.264 FLOP/token=42991616 FLOP/s=3.3225324e+12 MFU=n/a iter=000020 loss=7.367835998535156 time=6.7437387s tokens/s=77744.412 FLOP/token=42991616 FLOP/s=3.3423579e+12 MFU=n/a iter=000030 loss=6.542900085449219 time=6.7881376s tokens/s=77235.912 FLOP/token=42991616 FLOP/s=3.3204967e+12 MFU=n/a iter=000040 loss=5.694801330566406 time=6.7979796s tokens/s=77124.092 FLOP/token=42991616 FLOP/s=3.3156893e+12 MFU=n/a iter=000050 loss=4.924409866333008 time=6.7726127s tokens/s=77412.961 FLOP/token=42991616 FLOP/s=3.3281083e+12 MFU=n/a iter=000060 loss=4.343236923217773 time=6.7779894s tokens/s=77351.552 FLOP/token=42991616 FLOP/s=3.3254682e+12 MFU=n/a iter=000070 loss=3.945407629013062 time=6.7983165s tokens/s=77120.269 FLOP/token=42991616 FLOP/s=3.315525e+12 MFU=n/a iter=000080 loss=3.661749362945557 time=6.8210636s tokens/s=76863.087 FLOP/token=42991616 FLOP/s=3.3044683e+12 MFU=n/a iter=000090 loss=3.445833683013916 time=6.799036s tokens/s=77112.108 FLOP/token=42991616 FLOP/s=3.3151741e+12 MFU=n/a step=000100 train_loss=3.227960824966431 val_loss=3.227255821228027 lr=0.001 samples: [1] mbrella and told her to be careful. Lily opened the ug and thanked him. She saw many things in the box and wanted to play with her.

"No, Lily! It's help me! It's our family and angry!"

Her mom was so happy. She said, "We canys, I love you, it is that."

Jill saw Lily and said, "We are something [2] up the hill. That way, it could conserve its energy and trophant!"

"Wow, he said, "Hey, the man. Thank you forgive me." The man smiled and said, "You can't have many things. But you should be bow? You can't you. You have to make the jar. You can make some of you. I will the laughed the ground and get my friend [3] was a snake who was very sad. He was so sad he couldn't feel. I'll help you for a new doll. He hugged his mouth and said he could look for a nice man. I have to be a big mouse. I'm a smile and the loyal bear. Maybe you can have to the lion."

"You're sorry, we do you. The box is the toys."

"Hello, [4] at nods and bleats happily. It licks Tom's face and saw it. They had many pink eyes.

She thought about the car is. They are happy. They will have a lot of fun. They go down on their room and play with them. They will not find a dog. They are a big tree with a big dog. They are not want to be sad.

They are a long tail. They say, " [5] toy. Her mom said, "We can go to the shop, but it's a special, it is too."

They started to the table and put the bird in the park. They started to the tree and wore the bird. They had fun. They had fun. They would not find a big box and a moment. They learned that they do not like the big box. They were sad and scared. They played with their eyes and hopped saved checkpoint: out_model5555/model5555_ckpt_100.pt iter=000100 loss=3.233522415161133 time=31.857571s tokens/s=16457.25 FLOP/token=42991616 FLOP/s=7.0752377e+11 MFU=n/a iter=000110 loss=3.068276405334473 time=6.7625488s tokens/s=77528.166 FLOP/token=42991616 FLOP/s=3.3330611e+12 MFU=n/a iter=000120 loss=2.917527675628662 time=6.8015097s tokens/s=77084.062 FLOP/token=42991616 FLOP/s=3.3139684e+12 MFU=n/a iter=000130 loss=2.785095930099487 time=6.8051933s tokens/s=77042.338 FLOP/token=42991616 FLOP/s=3.3121746e+12 MFU=n/a iter=000140 loss=2.664446592330933 time=6.8171648s tokens/s=76907.045 FLOP/token=42991616 FLOP/s=3.3063582e+12 MFU=n/a iter=000150 loss=2.556045532226562 time=6.7957497s tokens/s=77149.399 FLOP/token=42991616 FLOP/s=3.3167773e+12 MFU=n/a iter=000160 loss=2.460245370864868 time=6.7994673s tokens/s=77107.217 FLOP/token=42991616 FLOP/s=3.3149638e+12 MFU=n/a iter=000170 loss=2.416778326034546 time=6.835985s tokens/s=76695.312 FLOP/token=42991616 FLOP/s=3.2972554e+12 MFU=n/a iter=000180 loss=2.330549716949463 time=6.8349584s tokens/s=76706.831 FLOP/token=42991616 FLOP/s=3.2977506e+12 MFU=n/a iter=000190 loss=2.262391090393066 time=6.7984825s tokens/s=77118.386 FLOP/token=42991616 FLOP/s=3.315444e+12 MFU=n/a step=000200 train_loss=2.212054491043091 val_loss=2.209667444229126 lr=0.0009728616793536588 samples: [1] giant and saving the woods. Once upon a time there was a man who was very hungry. He decided to go to the park and explore the world. He was so happy when he got to explore and see a big, green field. So, he jumped up and down and began to look around.

The old man walked around and looked around, even in the grass. The old man was so excited and shiny and shiny, [2] station's mascot. Everywhere the pup went it glowed, it was a special place to make it work.

The dictionary was so beautiful! Everywhere the ducks went on, it was so glad and full of joy.

The tap was not so good because it was its new and the ducks were the best of friends. Once upon a time there was a little girl. She was a very happy girl and loved [3] treat. Blue felt good and calm. He looked at the sky and smiled.

The end. One day, a little girl named Lisa was playing in her garden. She was so happy, she was having a fun time, looking for her friends.

After a while, Lisa's mom came in. She said, "That's okay, Larry. I will do it without you."

Luna was so [4] ike was very happy, and all the dogs in the park learned to share and have fun. They all learned that they can still be friends and be happy. Lily and Ben were friends who liked to play in the park. One day, they found a big, old lake. The jellyfly was very small and fluffy. They wanted to see the jungle.

"Can we use the swamp?" Ben asked.

[5] bear and the fox both cheered. As they sat down to have dinner, the fox felt so happy and happy.

The next day, the fox decided to give the fox some of his adventure. It said goodbye and thanked the fox. From that day on, the fox and the fox were friends and would often look for the other animals in the woods. Once upon a time there was a little girl saved checkpoint: out_model5555/model5555_ckpt_200.pt iter=000200 loss=2.199012756347656 time=31.073566s tokens/s=16872.476 FLOP/token=42991616 FLOP/s=7.2537502e+11 MFU=n/a iter=000210 loss=2.162654638290405 time=6.7966915s tokens/s=77138.708 FLOP/token=42991616 FLOP/s=3.3163177e+12 MFU=n/a iter=000220 loss=2.107671976089478 time=6.8564388s tokens/s=76466.518 FLOP/token=42991616 FLOP/s=3.2874192e+12 MFU=n/a iter=000230 loss=2.055487394332886 time=6.7996157s tokens/s=77105.534 FLOP/token=42991616 FLOP/s=3.3148915e+12 MFU=n/a iter=000240 loss=2.037257671356201 time=6.8273432s tokens/s=76792.39 FLOP/token=42991616 FLOP/s=3.3014289e+12 MFU=n/a iter=000250 loss=2.008619546890259 time=6.8159746s tokens/s=76920.475 FLOP/token=42991616 FLOP/s=3.3069355e+12 MFU=n/a iter=000260 loss=1.959845185279846 time=6.8314355s tokens/s=76746.388 FLOP/token=42991616 FLOP/s=3.2994512e+12 MFU=n/a iter=000270 loss=1.924401164054871 time=6.8234028s tokens/s=76836.736 FLOP/token=42991616 FLOP/s=3.3033354e+12 MFU=n/a iter=000280 loss=1.908273458480835 time=6.8295409s tokens/s=76767.679 FLOP/token=42991616 FLOP/s=3.3003666e+12 MFU=n/a iter=000290 loss=1.882551789283752 time=6.8348166s tokens/s=76708.423 FLOP/token=42991616 FLOP/s=3.297819e+12 MFU=n/a step=000300 train_loss=1.851970911026001 val_loss=1.85030722618103 lr=0.0008947199994035401 samples: [1] and it always kept him warm! Once upon a time, there was a little boy named Timmy. Timmy liked to play with his toys, especially his toy cars. One day, Timmy's mom told him they were going to a new park. Timmy was very excited and he couldn't wait to play.

Timmy's mom said, "I want to play on the swings, please." Timmy was happy that his mom [2] , please! My nose is melting!" he cried.

Sam looked at his nose. He felt sad. He loved his nose.

He looked around. He saw his nose. He remembered that his nose was not too tight. He looked at his nose and his nose. He looked at his nose.

He said, "I'm sorry, Sam. I was silly. I was [3] know if it would be safe.

But then he heard a voice say "Welcome to my house. It's time to go home." Tom was happy to hear that and said "Please, I want to be your friend again".

He took a deep breath and slowly stepped into the house. He was very excited and ran to the kitchen. Suddenly, he saw a small, shiny red ball. He was so excited and shouted [4] ommy thanked Lily for letting him borrow the truck and they both went home together. Once upon a time, there was a little girl named Lily. She loved to eat peaches and take care of them. One day, she saw a big, scary bear. She wanted to run away, but she didn't know how.

Lily started to worry. She said, "I don't want to eat that. I want to eat [5] back to playing with her friends. From that day on, she learned that sometimes it's okay to do something that it's important to always be careful with things in need. Once upon a time, there was a little girl named Lily. She loved to play outside in the grass. One day, she saw a pretty butterfly and wanted to catch it. She tried to catch it, but it was too fast.

Lily's mom told saved checkpoint: out_model5555/model5555_ckpt_300.pt iter=000300 loss=1.864209175109863 time=36.783381s tokens/s=14253.394 FLOP/token=42991616 FLOP/s=6.1277642e+11 MFU=n/a iter=000310 loss=1.837927222251892 time=6.6861898s tokens/s=78413.568 FLOP/token=42991616 FLOP/s=3.371126e+12 MFU=n/a iter=000320 loss=1.822239279747009 time=6.8949636s tokens/s=76039.27 FLOP/token=42991616 FLOP/s=3.2690511e+12 MFU=n/a iter=000330 loss=1.793344378471375 time=6.7938492s tokens/s=77170.98 FLOP/token=42991616 FLOP/s=3.3177051e+12 MFU=n/a iter=000340 loss=1.785236597061157 time=6.8074982s tokens/s=77016.253 FLOP/token=42991616 FLOP/s=3.3110532e+12 MFU=n/a iter=000350 loss=1.759562730789185 time=6.7761337s tokens/s=77372.736 FLOP/token=42991616 FLOP/s=3.3263789e+12 MFU=n/a iter=000360 loss=1.763575792312622 time=6.8100964s tokens/s=76986.869 FLOP/token=42991616 FLOP/s=3.3097899e+12 MFU=n/a iter=000370 loss=1.736714243888855 time=6.8399082s tokens/s=76651.321 FLOP/token=42991616 FLOP/s=3.2953641e+12 MFU=n/a iter=000380 loss=1.715463399887085 time=6.8710985s tokens/s=76303.375 FLOP/token=42991616 FLOP/s=3.2804054e+12 MFU=n/a iter=000390 loss=1.714152097702026 time=6.8027788s tokens/s=77069.682 FLOP/token=42991616 FLOP/s=3.3133502e+12 MFU=n/a step=000400 train_loss=1.70381236076355 val_loss=1.696491837501526 lr=0.0007750000000000001 samples: [1] , that's so cool!" Lily said.

They played together for a while and then they decided to go home. Lily was sad to leave the park, but she said goodbye to the old man and went home.

The moral of the story is that it's important to be honest and kind to others. Sometimes it's better to ask for help and not be selfish with anything. Once upon a time, there was a little girl [2] a and Sam were playing with their toy cars in the garden. They liked to build tall towers and houses with their blocks. They also liked to pretend they were driving on a tour of the park. They had a lot of fun, but they also had to be careful not to break anything else.

One day, they saw a big dog in the park. The dog was barking and running towards them. Lila and Sam were [3] very big and had many leaves. One day, the gardener came to the forest and saw a little cave. The cave had a shiny cobweb in its nose.

The cobweb wanted to be friends with the cobweb, but it was too scared to move. The cobweb was surprised to see a little squirrel nearby. The cobweb was scared and didn't [4] the room.

"Maybe I can climb into the bin and see if you need me," she said.

Max ran over to the bin and he grabbed the box. Suddenly, the box opened. Inside, it was filled with toys. Max was so excited and shouted, "I found this box!"

Max ran to the box and started to grab the box. He shout [5] Sam and Lily were friends. They liked to play outside in the park. One day, they found a big hole in the ground. The hole was deep and dark. It was very deep.

"Let's go in," Sam said. "Maybe there are toys or candy there."

"OK," said Sam. "But we have to be careful. The hole is deep and dark."

They climbed up the saved checkpoint: out_model5555/model5555_ckpt_400.pt iter=000400 loss=1.701935768127441 time=35.666589s tokens/s=14699.695 FLOP/token=42991616 FLOP/s=6.3196366e+11 MFU=n/a iter=000410 loss=1.691343188285828 time=6.684616s tokens/s=78432.03 FLOP/token=42991616 FLOP/s=3.3719197e+12 MFU=n/a iter=000420 loss=1.679758191108704 time=6.8100812s tokens/s=76987.041 FLOP/token=42991616 FLOP/s=3.3097973e+12 MFU=n/a iter=000430 loss=1.671901822090149 time=6.7852095s tokens/s=77269.243 FLOP/token=42991616 FLOP/s=3.3219296e+12 MFU=n/a iter=000440 loss=1.667677640914917 time=6.7927763s tokens/s=77183.169 FLOP/token=42991616 FLOP/s=3.3182292e+12 MFU=n/a iter=000450 loss=1.654768109321594 time=6.8336208s tokens/s=76721.845 FLOP/token=42991616 FLOP/s=3.2983961e+12 MFU=n/a iter=000460 loss=1.62972354888916 time=6.8422009s tokens/s=76625.636 FLOP/token=42991616 FLOP/s=3.2942599e+12 MFU=n/a iter=000470 loss=1.64052677154541 time=6.8658637s tokens/s=76361.551 FLOP/token=42991616 FLOP/s=3.2829065e+12 MFU=n/a iter=000480 loss=1.628792643547058 time=6.8440753s tokens/s=76604.651 FLOP/token=42991616 FLOP/s=3.2933577e+12 MFU=n/a iter=000490 loss=1.622517943382263 time=6.801838s tokens/s=77080.342 FLOP/token=42991616 FLOP/s=3.3138085e+12 MFU=n/a step=000500 train_loss=1.628132104873657 val_loss=1.614776611328125 lr=0.0006281416799501188 samples: [1] lady. It's a good thing, not a bad thing".

John smiled and said, "It was so cool! I love playing with it!"

The lady smiled and said, "Yes, it's very important to have such a fun game. I'm so glad you had a great time playing the game and I can't wait to come back and play!"

John was thrilled and ran off to play. He was so [2] is too heavy and she loses her balance. She drops the jar and falls into the hole. She cannot get out. She drops the jar and cries. She does not want to play with the jar anymore.

Tim and Mia go home. They are sad and sorry. They are not cheerful and sad anymore. They wish they did not go to the jar. They wish they had not played with the jar [3] They feel sorry. They say, "Sorry, Mom. Sorry, sir. We did not mean to break it. We did not mean to break the box."

Mom smiles. She says, "It's OK, Tom and Lily. I forgive you. But you have to listen to me. I know the key is dangerous. It is for cleaning and working. Do you understand?"

Tom and L [4] na's face. She said, "Thank you, Anna. You are very kind. I don't want to play with the ball. It is fun. I will kick it and throw it. I will kick it. It is a good game."

Anna felt happy. She said, "Thank you, Tom. You are a good friend. You are a good friend too. You can play with the ball too. I will call you Snowy [5] . She was careful to clean up all her toys and make her room clean. So, she did. She was very careful not to lose her toys.

But when she came home, she saw that her toys were gone. She looked everywhere for them, but they were gone. She felt sad. She realized that she had made a mistake. She had been careless. She had ruined her toys. She had made saved checkpoint: out_model5555/model5555_ckpt_500.pt iter=000500 loss=1.619901537895203 time=31.538496s tokens/s=16623.748 FLOP/token=42991616 FLOP/s=7.1468178e+11 MFU=n/a iter=000510 loss=1.607285022735596 time=6.7668696s tokens/s=77478.661 FLOP/token=42991616 FLOP/s=3.3309329e+12 MFU=n/a iter=000520 loss=1.593489646911621 time=6.8558913s tokens/s=76472.624 FLOP/token=42991616 FLOP/s=3.2876817e+12 MFU=n/a iter=000530 loss=1.599012851715088 time=6.814813s tokens/s=76933.585 FLOP/token=42991616 FLOP/s=3.3074992e+12 MFU=n/a iter=000540 loss=1.584777593612671 time=6.801086s tokens/s=77088.865 FLOP/token=42991616 FLOP/s=3.3141749e+12 MFU=n/a iter=000550 loss=1.566122770309448 time=6.796376s tokens/s=77142.289 FLOP/token=42991616 FLOP/s=3.3164717e+12 MFU=n/a iter=000560 loss=1.555447340011597 time=6.7952216s tokens/s=77155.394 FLOP/token=42991616 FLOP/s=3.3170351e+12 MFU=n/a iter=000570 loss=1.576649188995361 time=6.7939986s tokens/s=77169.283 FLOP/token=42991616 FLOP/s=3.3176322e+12 MFU=n/a iter=000580 loss=1.562054395675659 time=6.7849144s tokens/s=77272.603 FLOP/token=42991616 FLOP/s=3.3220741e+12 MFU=n/a iter=000590 loss=1.556951522827148 time=6.8315651s tokens/s=76744.932 FLOP/token=42991616 FLOP/s=3.2993886e+12 MFU=n/a step=000600 train_loss=1.554375767707825 val_loss=1.554735660552979 lr=0.0004718583200498814 samples: [1] ucks!"

Jill thought for a minute and then smiled and said, "I think it's time to go inside."

So the two of them went inside and had a nice nice morning together. The end. Once upon a time there was a little girl named Kayla. She loved to look at the beautiful landscape around her. One day, she saw a ranch that looked very different from the other flowers. It looked so pretty, [2] 't worry, Lily. We just need to tighten it up a bit," said her mom.

Lily didn't want to wait, but she wanted to keep playing. So, they started to play together. They raced around the field, pretending to be a bird, and even pretended they were flying. But then, Lily accidentally bumped into her mom and she fell down.

"Ouch! My knee [3] 't work out the way we want them to, and that's okay. Once upon a time, there was a little girl named Lily. She loved to play outside with her friends. One day, she saw a little ant crawling on a leaf. She wanted to touch it, but her friend, Timmy, said no.

Timmy got upset and said, "I'm going to steal it. I want to make it [4] ," Tom says. "I was just trying to have fun. I didn't know the flower would be gone."

"Me too," Sue says. "But we have to be careful. The flower is very special. We have to be careful with it. And we have to follow the rules. The book is not for us. It has our words. We have to keep it safe and strong."

Tom and Sue n [5] sun and were happy. Once upon a time, in a small village, there was a little boy named Tim. He loved to play with his toy cars and trucks. One day, Tim saw a big red ball in his yard. He wanted to play with it, but his mom said no.

Tim felt anxious because he wanted to play with the ball outside. So, he went to his room and started to play with his car saved checkpoint: out_model5555/model5555_ckpt_600.pt iter=000600 loss=1.553627967834473 time=41.899542s tokens/s=12512.977 FLOP/token=42991616 FLOP/s=5.3795309e+11 MFU=n/a iter=000610 loss=1.548794627189636 time=6.6631301s tokens/s=78684.941 FLOP/token=42991616 FLOP/s=3.3827928e+12 MFU=n/a iter=000620 loss=1.550044059753418 time=6.8395552s tokens/s=76655.277 FLOP/token=42991616 FLOP/s=3.2955342e+12 MFU=n/a iter=000630 loss=1.56082546710968 time=6.8054785s tokens/s=77039.109 FLOP/token=42991616 FLOP/s=3.3120358e+12 MFU=n/a iter=000640 loss=1.536441564559937 time=6.7962337s tokens/s=77143.905 FLOP/token=42991616 FLOP/s=3.3165411e+12 MFU=n/a iter=000650 loss=1.533891081809998 time=6.8063478s tokens/s=77029.27 FLOP/token=42991616 FLOP/s=3.3116128e+12 MFU=n/a iter=000660 loss=1.532094955444336 time=6.8088719s tokens/s=77000.714 FLOP/token=42991616 FLOP/s=3.3103851e+12 MFU=n/a iter=000670 loss=1.526175498962402 time=6.8035271s tokens/s=77061.205 FLOP/token=42991616 FLOP/s=3.3129857e+12 MFU=n/a iter=000680 loss=1.52797794342041 time=6.8134134s tokens/s=76949.389 FLOP/token=42991616 FLOP/s=3.3081786e+12 MFU=n/a iter=000690 loss=1.525654792785645 time=6.805953s tokens/s=77033.738 FLOP/token=42991616 FLOP/s=3.3118049e+12 MFU=n/a step=000700 train_loss=1.512338638305664 val_loss=1.512636661529541 lr=0.0003250000000000001 samples: [1] the boat began to slow down and a friendly voice said, “Hello, little boy! Why do you have to stop?â€

The little boy looked up and saw the friendly old man holding a big balloon. The man smiled and said, “I’m a little girl, who’s sitting there. He has been waiting for a long time and now [2] ."

So, Tom and his friends played with the cheap log all day. They made it look like the biggest log in the forest. They had a great time together. Once upon a time, there was a little girl named Sue. Sue was a very brave girl. She loved to play with her toys. One day, she found a big box in her room. She opened the box and saw many toys. Sue was very happy [3] !" Ben says. He fills his hat with water and pours it on his head. He feels happy.

"See, Ben, I'm a princess!" Lily says. She laughs too.

They have fun with the pink prints. They make more stories and different sounds. They share their ideas and laugh.

Mom and Dad watch them. They smile. They are proud. They are proud of their children. [4] . Bun was free. He was happy. He hugged Lily and Ben.

"Thank you, Lily and Ben. You are my friends. I love you. Let's play again," the farmer said.

Lily and Ben ran back to the house. They told Mom about the dog. Mom looked at the toys. She smiled. She was proud of her children.

"That's a good idea [5] top and smiled. They hugged each other and said, "We are friends forever."

Their moms smiled and hugged them back. They were proud of them and happy. They all had a wonderful day together. Lily and Ben are twins. They like to play with their toys in their room. They have many toys, but their favorite is a big dragon. The dragon is red and shiny and saved checkpoint: out_model5555/model5555_ckpt_700.pt iter=000700 loss=1.522422075271606 time=41.406762s tokens/s=12661.893 FLOP/token=42991616 FLOP/s=5.4435526e+11 MFU=n/a iter=000710 loss=1.511780619621277 time=6.6348468s tokens/s=79020.363 FLOP/token=42991616 FLOP/s=3.3972131e+12 MFU=n/a iter=000720 loss=1.507643103599548 time=6.7911959s tokens/s=77201.13 FLOP/token=42991616 FLOP/s=3.3190014e+12 MFU=n/a iter=000730 loss=1.501144170761108 time=6.7937604s tokens/s=77171.988 FLOP/token=42991616 FLOP/s=3.3177485e+12 MFU=n/a iter=000740 loss=1.501596331596375 time=6.8067426s tokens/s=77024.802 FLOP/token=42991616 FLOP/s=3.3114207e+12 MFU=n/a iter=000750 loss=1.495803475379944 time=6.7894624s tokens/s=77220.841 FLOP/token=42991616 FLOP/s=3.3198487e+12 MFU=n/a iter=000760 loss=1.494728326797485 time=6.8025415s tokens/s=77072.371 FLOP/token=42991616 FLOP/s=3.3134658e+12 MFU=n/a iter=000770 loss=1.477253079414368 time=6.8101317s tokens/s=76986.47 FLOP/token=42991616 FLOP/s=3.3097728e+12 MFU=n/a iter=000780 loss=1.483962893486023 time=6.8468542s tokens/s=76573.56 FLOP/token=42991616 FLOP/s=3.2920211e+12 MFU=n/a iter=000790 loss=1.488822221755981 time=6.8462672s tokens/s=76580.126 FLOP/token=42991616 FLOP/s=3.2923034e+12 MFU=n/a step=000800 train_loss=1.49064576625824 val_loss=1.48790442943573 lr=0.00020528000059646 samples: [1] to stay.

The bear explained that he was cold and that he needed to find a way to get warm. So, the bear got a big coat and some soft blanket. He wrapped the bear in the coat, and with a warm, warm breath, the bear began to warm up.

The bear felt much better, and the two of them played together in the sun. They played tag and then went home, and the bear [2] the lady and the dog. They said, "You are very kind. You can play with us if you want. But you have to be nice and share. That is the best gift of all."

The lady said, "I will, and I will be nice. And I will play with you. You are my friend. And you are my friend too."

The lady said, "Thank you, Lily and Ben. I am [3] are not afraid of me and you make me laugh. I want to be friends with you. Can we play together?" Anna says.

"Of course we can. Come on, let's play. We can have fun. We can be friends." Ben says.

They hug and laugh. They play with the ball and the doll. They have fun. They are happy. They are friends. Lily was sick. She had a fe [4] quito is fast and dodges her hand.

The mule does not like Lila's toy. It bit Lila's finger and makes a lot of noise. Lila is scared and runs away. The mule runs back to the cage and sits on a chair. Lila is relieved and happy to have her toy. She hugs Lila and thanks her for saving [5] She wondered what to write. She thought of her family, her friends, and her toys. She said, "Let's write down."

She got some paper and crayons and started to write. She wrote "Twinkle, yellow, crayons, and more. She wrote "Mail" and "Look, Mommy, I wrote!" She wrote "Mail is green!"

She was very proud of her saved checkpoint: out_model5555/model5555_ckpt_800.pt iter=000800 loss=1.472851634025574 time=31.218138s tokens/s=16794.339 FLOP/token=42991616 FLOP/s=7.2201578e+11 MFU=n/a iter=000810 loss=1.481930494308472 time=6.7713942s tokens/s=77426.891 FLOP/token=42991616 FLOP/s=3.3287072e+12 MFU=n/a iter=000820 loss=1.479142785072327 time=6.8532052s tokens/s=76502.598 FLOP/token=42991616 FLOP/s=3.2889703e+12 MFU=n/a iter=000830 loss=1.474742531776428 time=6.8246315s tokens/s=76822.902 FLOP/token=42991616 FLOP/s=3.3027407e+12 MFU=n/a iter=000840 loss=1.481850624084473 time=6.8030088s tokens/s=77067.077 FLOP/token=42991616 FLOP/s=3.3132382e+12 MFU=n/a iter=000850 loss=1.478882551193237 time=6.8376861s tokens/s=76676.232 FLOP/token=42991616 FLOP/s=3.2964351e+12 MFU=n/a iter=000860 loss=1.471592545509338 time=6.8537632s tokens/s=76496.369 FLOP/token=42991616 FLOP/s=3.2887025e+12 MFU=n/a iter=000870 loss=1.473869562149048 time=6.8128382s tokens/s=76955.886 FLOP/token=42991616 FLOP/s=3.3084579e+12 MFU=n/a iter=000880 loss=1.458152174949646 time=6.8048526s tokens/s=77046.195 FLOP/token=42991616 FLOP/s=3.3123405e+12 MFU=n/a iter=000890 loss=1.46212911605835 time=6.8041429s tokens/s=77054.231 FLOP/token=42991616 FLOP/s=3.3126859e+12 MFU=n/a step=000900 train_loss=1.458112478256226 val_loss=1.46475076675415 lr=0.0001271383206463413 samples: [1] . She was very surprised, but quickly realized the bear was friendly.

Mama knew the bear was friendly, so she got down on her hands and rubbed the bear's nose. The bear was so happy and started to purr.

Mama was so proud of her little baby. She wanted to show it to her best friend. She told the bear that it was a special kind of fur, just like the bear.

The [2] .â€

Robbie got up and walked over to the door. He pulled out his jacket and opened it. He took out his favorite toy, a bright yellow ball. He started to play and make all sorts of things.

When the toy was finished, Robbie looked around. He noticed the door was locked. He asked his mom, “What is in the door, momm [3] and laughed as the water sprayed her. She realized it was fun to spray, but also to be careful. She said, "Lily, do you want to play with the boat?"

Lily smiled and said, "Yes, I do. I like it here. It is a good boat. Thank you for being my friend."

They hugged and played with the boat for a while. Then Lily said [4] She saw the cake mix. She saw Lily and Ben. She was not happy. She said, "What have you done? You are a naughty girl! You made a mess! You have to clean up right now! You have to clean up the mess and say sorry!"

Lily and Ben felt sorry. They said, "We are sorry, Mom. We were trying to bake a cake. We wanted to make a cake. We [5] your babies?" The hen thought about it and said, "Okay, but only one for me."

The hen was very happy and said, "Thank you, hen. You are a great helper." The hen felt proud that she could help the hen. From that day on, the hen would visit the hen every day to visit her. Once upon a time, there was a little girl named Lily. saved checkpoint: out_model5555/model5555_ckpt_900.pt iter=000900 loss=1.472051501274109 time=31.659761s tokens/s=16560.075 FLOP/token=42991616 FLOP/s=7.1194437e+11 MFU=n/a iter=000910 loss=1.48569667339325 time=6.7730167s tokens/s=77408.343 FLOP/token=42991616 FLOP/s=3.3279098e+12 MFU=n/a iter=000920 loss=1.458486795425415 time=6.8576479s tokens/s=76453.036 FLOP/token=42991616 FLOP/s=3.2868396e+12 MFU=n/a iter=000930 loss=1.460655093193054 time=6.8325625s tokens/s=76733.729 FLOP/token=42991616 FLOP/s=3.298907e+12 MFU=n/a iter=000940 loss=1.462050199508667 time=6.7959758s tokens/s=77146.832 FLOP/token=42991616 FLOP/s=3.316667e+12 MFU=n/a iter=000950 loss=1.454442262649536 time=6.816064s tokens/s=76919.466 FLOP/token=42991616 FLOP/s=3.3068921e+12 MFU=n/a iter=000960 loss=1.436673402786255 time=6.7937202s tokens/s=77172.445 FLOP/token=42991616 FLOP/s=3.3177681e+12 MFU=n/a iter=000970 loss=1.454544305801392 time=6.7981259s tokens/s=77122.431 FLOP/token=42991616 FLOP/s=3.315618e+12 MFU=n/a iter=000980 loss=1.452312469482422 time=6.8037699s tokens/s=77058.456 FLOP/token=42991616 FLOP/s=3.3128675e+12 MFU=n/a iter=000990 loss=1.456324696540833 time=6.8014133s tokens/s=77085.155 FLOP/token=42991616 FLOP/s=3.3140154e+12 MFU=n/a saved checkpoint: out_model5555/model5555_ckpt_1000.pt

train.py end

dataset tokenization and information

====================================================================== tinystories TEXT → BIN DATASET

Input : ts_sample.txt Size : 1.77 GB Chunk : 32.0 MB Train : 90% Validation : 10%

Loading tokenizer... Vocabulary : 8,192 Load time : 0.06s

---

CHUNK DONE SPEED ETA TOKENS RAM

  1    1.76%  779.6 KB/s     39:04       9,318,569    2.82 GB
  2    3.52%  784.5 KB/s     38:08      18,617,846    2.86 GB
  3    5.28%  783.8 KB/s     37:28      27,917,342    2.87 GB
  4    7.05%  784.0 KB/s     36:45      37,268,992    2.88 GB
  5    8.81%  783.1 KB/s     36:06      46,647,300    2.89 GB
  6   10.57%  783.5 KB/s     35:23      55,997,405    2.89 GB
  7   12.33%  782.5 KB/s     34:44      65,385,975    2.90 GB
  8   14.09%  787.3 KB/s     33:50      74,710,413    2.91 GB
  9   15.85%  789.9 KB/s     33:01      84,041,233    2.91 GB
 10   17.61%  789.4 KB/s     32:21      93,358,882    2.92 GB
 11   19.37%  790.5 KB/s     31:37     102,695,789    2.93 GB
 12   21.14%  783.4 KB/s     31:12     112,013,754    2.93 GB
 13   22.90%  780.2 KB/s     30:38     121,357,175    2.94 GB
 14   24.66%  781.0 KB/s     29:54     130,681,862    2.94 GB
 15   26.42%  781.5 KB/s     29:11     140,054,828    2.95 GB
 16   28.18%  782.3 KB/s     28:27     149,402,863    2.96 GB
 17   29.94%  782.9 KB/s     27:44     158,732,403    2.96 GB
 18   31.70%  782.4 KB/s     27:04     168,072,632    2.97 GB
 19   33.47%  782.9 KB/s     26:21     177,441,189    2.97 GB
 20   35.23%  783.2 KB/s     25:38     186,815,779    2.98 GB
 21   36.99%  785.0 KB/s     24:53     196,145,991    2.98 GB
 22   38.75%  786.8 KB/s     24:08     205,469,283    2.99 GB
 23   40.51%  787.0 KB/s     23:26     214,771,604    2.99 GB
 24   42.27%  786.4 KB/s     22:45     224,091,914    2.99 GB
 25   44.03%  786.4 KB/s     22:03     233,454,178    3.00 GB
 26   45.79%  786.8 KB/s     21:21     242,754,917    3.00 GB
 27   47.56%  777.8 KB/s     20:54     252,117,525    3.01 GB
 28   49.32%  777.7 KB/s     20:12     261,458,023    3.01 GB
 29   51.08%  777.1 KB/s     19:31     270,771,155    3.01 GB
 30   52.84%  778.4 KB/s     18:47     280,057,891    3.01 GB
 31   54.60%  778.9 KB/s     18:04     289,395,682    3.02 GB
 32   56.36%  779.4 KB/s     17:21     298,684,232    3.02 GB
 33   58.12%  779.2 KB/s     16:39     308,028,126    3.02 GB
 34   59.89%  779.6 KB/s     15:57     317,376,523    3.02 GB
 35   61.65%  779.9 KB/s     15:14     326,756,602    3.02 GB
 36   63.41%  780.2 KB/s     14:32     336,064,047    3.03 GB
 37   65.17%  780.6 KB/s     13:50     345,407,240    3.03 GB
 38   66.93%  780.6 KB/s     13:08     354,734,467    3.03 GB
 39   68.69%  781.3 KB/s     12:25     364,042,689    3.03 GB
 40   70.45%  782.9 KB/s     11:42     373,340,740    3.03 GB
 41   72.21%  780.5 KB/s     11:02     382,670,935    3.04 GB
 42   73.98%  780.2 KB/s     10:20     391,990,763    3.04 GB
 43   75.74%  780.3 KB/s     09:38     401,359,905    3.04 GB
 44   77.50%  780.2 KB/s     08:56     410,699,639    3.04 GB
 45   79.26%  780.3 KB/s     08:14     420,028,000    3.05 GB
 46   81.02%  780.1 KB/s     07:32     429,383,494    3.05 GB
 47   82.78%  780.4 KB/s     06:50     438,696,634    3.05 GB
 48   84.54%  780.8 KB/s     06:08     448,006,945    3.05 GB
 49   86.31%  780.9 KB/s     05:26     457,265,432    3.06 GB
 50   88.07%  780.3 KB/s     04:44     466,604,292    3.06 GB
 51   89.83%  780.4 KB/s     04:02     475,944,476    3.06 GB
 52   91.59%  780.7 KB/s     03:20     485,293,862    3.06 GB
 53   93.35%  780.6 KB/s     02:38     494,619,031    3.07 GB
 54   95.11%  781.8 KB/s     01:56     503,951,863    3.07 GB
 55   96.87%  782.2 KB/s     01:14     513,227,320    3.07 GB
 56   98.63%  778.3 KB/s     00:32     522,573,156    3.07 GB
 57  100.00%  779.0 KB/s     00:00     529,850,782    3.08 GB

====================================================================== DONE

Time : 39:48 Average speed : 778.9 KB/s

Input : 1.77 GB Train tokens : 485,293,862 Validation tokens: 44,556,920 Total tokens : 529,850,782

train.bin : 925.6 MB val.bin : 85.0 MB Final RAM : 3.08 GB

Created: dataset/train.bin dataset/val.bin dataset/dataset_config.json

============================================================ BIN TOKEN COUNT CHECK

train.bin size : 925.62 MB train.bin tokens : 485,293,862

val.bin size : 84.99 MB val.bin tokens : 44,556,920

TOTAL tokens : 529,850,782

PASS: Both files contain valid uint16 token counts.

(note: I know the code is AI slop, you can not do anything about it)