Delete logs/output_run_20260202_placater_lyricism.log

logs/output_run_20260202_placater_lyricism.log

@@ -1,84 +0,0 @@
-2026-02-02 10:01:46,576 - root - INFO - Run: run_20260202_placater_lyricism
-2026-02-02 10:01:46,576 - root - INFO - Log directory: /root/tiny_moe/training_runs/Tiny_MoE/logs
-2026-02-02 10:01:46,576 - root - INFO - Output dir: /root/tiny_moe/training_runs
-2026-02-02 10:01:49,085 - jax._src.xla_bridge - INFO - Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
-2026-02-02 10:01:52,423 - root - INFO - Flax version: 0.11.1
-2026-02-02 10:01:52,423 - root - INFO - Optax version: 0.2.6
-2026-02-02 10:01:52,423 - root - INFO - Platform: gpu
-2026-02-02 10:01:52,423 - root - INFO - Num Devices: 8
-2026-02-02 10:01:52,423 - root - INFO - Devices: [CudaDevice(id=0), CudaDevice(id=1), CudaDevice(id=2), CudaDevice(id=3), CudaDevice(id=4), CudaDevice(id=5), CudaDevice(id=6), CudaDevice(id=7)]
-2026-02-02 10:01:54,659 - root - INFO - Model config:
-Config(name='Tiny_MoE',
-       dtype=<class 'jax.numpy.bfloat16'>,
-       vocab_size=50304,
-       block_size=2048,
-       n_layer=30,
-       n_embed=672,
-       n_glu_hidden=2048,
-       n_head=12,
-       n_kv_head=4,
-       n_experts=8,
-       init_stddev=0.02,
-       expert_load_factor=1.25,
-       aux_loss_coeff=0.01,
-       moe_bias=True,
-       mlp_bias=False,
-       attention_bias=False,
-       load_balance_loss_coeff=0.01,
-       z_loss_coeff=0.0005,
-       expert_top_k=2,
-       ln_epsilon=1e-05,
-       rope_theta=0.0001,
-       expert_partition_spec=PartitionSpec('devices',),
-       sdpa_implementation='flash_attn_jax',
-       window_size=(512, 512),
-       value_residual_init=0.5,
-       logit_softcap=30.0)
-2026-02-02 10:03:43,007 - root - INFO - Parameter Count: 1,062,185,550
-2026-02-02 10:03:43,007 - root - INFO - Sharded / MoE Parameter Count: 992,210,160
-2026-02-02 10:03:43,007 - root - INFO - Replicated Parameter Count: 69,975,390
-2026-02-02 10:03:44,528 - root - INFO - Weight decay param count: 1,062,140,928
-2026-02-02 10:03:44,529 - root - INFO - Training config:
-TrainerConfig(num_tokens=100000000000,
-              num_tokens_per_batch=262144,
-              mB=128,
-              T=2048,
-              max_steps=381469,
-              max_lr=0.008,
-              min_lr=0.0008,
-              max_grad_norm=1.0,
-              weight_decay=0.1,
-              adam_b1=0.9,
-              adam_b2=0.95,
-              warmup_steps=3814,
-              print_interval=100,
-              val=True,
-              val_interval=5000,
-              val_batches=50,
-              checkpoint_model=False,
-              checkpoint_optimizer=False,
-              checkpoint_interval=10000)
-2026-02-02 10:03:44,529 - root - INFO - Effective batch size per device: 16
-2026-02-02 10:03:48,910 - root - INFO - HuggingfaceDataLoader: 1030 shards (train)
-2026-02-02 10:03:48,994 - root - INFO - HuggingfaceDataLoader initialized:
-------------------------
-label:          train
-shards:         1,030
-shard size:     100,000,000
-batch size:     128
-block size:     2048
-device rank:    1
-start shard:    0
-start pos:      0
-------------------------
-2026-02-02 10:03:48,994 - root - INFO - HuggingfaceDataLoader: 1 shards (val)
-2026-02-02 10:03:49,088 - root - INFO - Starting from step: 0
-2026-02-02 10:08:37,436 - root - INFO - 100 | lr: 0.0002 | loss: 6.9875 | logits loss: 6.6875 | load balance loss: 30.2952 | z loss: 21.1250 | avg iter time: 0.00ms | avg tok/sec: 0.00 | tokens processed: 26,214,400 | elapsed: 0h 4m 48s | ETA: calculating...
-2026-02-02 10:09:58,109 - root - INFO - 200 | lr: 0.0004 | loss: 5.9627 | logits loss: 5.6562 | load balance loss: 30.2515 | z loss: 13.3125 | avg iter time: 806.64ms | avg tok/sec: 324,984.59 | tokens processed: 52,428,800 | elapsed: 0h 6m 9s | ETA: 85h 25m
-2026-02-02 10:11:18,870 - root - INFO - 300 | lr: 0.0006 | loss: 5.3933 | logits loss: 5.0938 | load balance loss: 30.1811 | z loss: 5.0000 | avg iter time: 807.54ms | avg tok/sec: 324,622.05 | tokens processed: 78,643,200 | elapsed: 0h 7m 29s | ETA: 85h 30m
-2026-02-02 10:12:22,553 - root - INFO - Downloading fineweb_train_000002.bin from kjj0/fineweb100B-gpt2...
-2026-02-02 10:12:22,767 - httpx - INFO - HTTP Request: HEAD https://huggingface.co/datasets/kjj0/fineweb100B-gpt2/resolve/main/fineweb_train_000002.bin "HTTP/1.1 302 Found"
-2026-02-02 10:12:22,792 - httpx - INFO - HTTP Request: GET https://huggingface.co/api/datasets/kjj0/fineweb100B-gpt2/xet-read-token/50d1422b27e1a928440c26a8829f3f827f44ac56 "HTTP/1.1 200 OK"
-2026-02-02 10:12:39,635 - root - INFO - 400 | lr: 0.0008 | loss: 5.2321 | logits loss: 4.9375 | load balance loss: 30.2736 | z loss: 4.7812 | avg iter time: 807.59ms | avg tok/sec: 324,600.30 | tokens processed: 104,857,600 | elapsed: 0h 8m 50s | ETA: 85h 29m
-2026-02-02 10:12:56,482 - root - WARNING - Received KeyboardInterrupt. Exiting...
-2026-02-02 10:12:56,758 - root - INFO - Training completed.