AbhishekGhosh1998 commited on
Commit
157e056
·
verified ·
1 Parent(s): 9283f6a

granularity results tierb-fixed-moonep-full @ 2026-08-30T00:30:27Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0

Browse files
.gitattributes CHANGED
@@ -57,3 +57,4 @@ moonep/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8_
57
  moonep/tierbtrace/tierbtrace_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone/nsys/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone.nsys-rep filter=lfs diff=lfs merge=lfs -text
58
  tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375/nsys/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375.nsys-rep filter=lfs diff=lfs merge=lfs -text
59
  tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125/nsys/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125.nsys-rep filter=lfs diff=lfs merge=lfs -text
 
 
57
  moonep/tierbtrace/tierbtrace_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone/nsys/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone.nsys-rep filter=lfs diff=lfs merge=lfs -text
58
  tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375/nsys/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375.nsys-rep filter=lfs diff=lfs merge=lfs -text
59
  tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125/nsys/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125.nsys-rep filter=lfs diff=lfs merge=lfs -text
60
+ tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333.nsys-rep filter=lfs diff=lfs merge=lfs -text
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333.nsys-rep ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d34c00d2677ccce5088f6ec814c1f4967fd78717b8c83c2fbeca0059a1d153da
3
+ size 18115411
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/train.log CHANGED
@@ -1388,3 +1388,184 @@ Overwriting rerun_state_machine.current_iteration from -1 to 0...
1388
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1389
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1390
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1388
  warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1389
  /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1390
  warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1391
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1392
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1393
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1394
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1395
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1396
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1397
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1398
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1399
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1400
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1401
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1402
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1403
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1404
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1405
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1406
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1407
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1408
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1409
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1410
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1411
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1412
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1413
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1414
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1415
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1416
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1417
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1418
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1419
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1420
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1421
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1422
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1423
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1424
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1425
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1426
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1427
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1428
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1429
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1430
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1431
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1432
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1433
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1434
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1435
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1436
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1437
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1438
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1439
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1440
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1441
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1442
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1443
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1444
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1445
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1446
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1447
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1448
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1449
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1450
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1451
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1452
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1453
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1454
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1455
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1456
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1457
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1458
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1459
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1460
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1461
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1462
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1463
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1464
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1465
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1466
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1467
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1468
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1469
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1470
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1471
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1472
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1473
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1474
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1475
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1476
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1477
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1478
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1479
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1480
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1481
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1482
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1483
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1484
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1485
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1486
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1487
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1488
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1489
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1490
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1491
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1492
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1493
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1494
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1495
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1496
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1497
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1498
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1499
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1500
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1501
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1502
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1503
+ [2026-08-30 00:27:56] iteration 1/ 10 | consumed samples: 8 | elapsed time per iteration (ms): 46530.8 | throughput per GPU (TFLOP/s/GPU): 2.7 | learning rate: 2.000000E-05 | global batch size: 8 | lm loss: 1.092139E+01 | loss scale: 1.0 | grad norm: 30.619 | number of skipped iterations: 0 | number of nan iterations: 0 |
1504
+ Number of parameters in transformer block in billions: 13.53
1505
+ Number of parameters in embedding layers in billions: 0.21
1506
+ Total number of parameters in billions: 13.74
1507
+ Number of parameters in most loaded shard in billions: 13.7356
1508
+ Theoretical memory footprints: weight and optimizer=98244.96 MB
1509
+ [Rank 0] (after 1 iterations) memory (MB) | allocated: 40504.66845703125 | max allocated: 106236.974609375 | reserved: 109990.0 | max reserved: 109990.0
1510
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1511
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1512
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1513
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1514
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1515
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1516
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1517
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1518
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1519
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1520
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1521
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1522
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1523
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1524
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1525
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1526
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1527
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1528
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1529
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1530
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1531
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1532
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1533
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1534
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1535
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1536
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1537
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1538
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1539
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1540
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
1541
+ warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
1542
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1543
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1544
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1545
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1546
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1547
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1548
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1549
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1550
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1551
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1552
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1553
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1554
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1555
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1556
+ /workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
1557
+ warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
1558
+ [2026-08-30 00:27:56] iteration 2/ 10 | consumed samples: 16 | elapsed time per iteration (ms): 646.1 | throughput per GPU (TFLOP/s/GPU): 197.5 | learning rate: 4.000000E-05 | global batch size: 8 | lm loss: 1.092003E+01 | loss scale: 1.0 | grad norm: 30.571 | number of skipped iterations: 0 | number of nan iterations: 0 |
1559
+ [2026-08-30 00:27:57] iteration 3/ 10 | consumed samples: 24 | elapsed time per iteration (ms): 639.7 | throughput per GPU (TFLOP/s/GPU): 199.5 | learning rate: 6.000000E-05 | global batch size: 8 | lm loss: 9.952750E+00 | loss scale: 1.0 | grad norm: 42.271 | number of skipped iterations: 0 | number of nan iterations: 0 |
1560
+ Capture CUDA graph for training!!!
1561
+ CUDA graph capture done!!!
1562
+ [2026-08-30 00:27:59] iteration 4/ 10 | consumed samples: 32 | elapsed time per iteration (ms): 1959.3 | throughput per GPU (TFLOP/s/GPU): 65.1 | learning rate: 8.000000E-05 | global batch size: 8 | lm loss: 9.477107E+00 | loss scale: 1.0 | grad norm: 7.154 | number of skipped iterations: 0 | number of nan iterations: 0 |
1563
+ [2026-08-30 00:28:00] iteration 5/ 10 | consumed samples: 40 | elapsed time per iteration (ms): 588.8 | throughput per GPU (TFLOP/s/GPU): 216.7 | learning rate: 1.000000E-04 | global batch size: 8 | lm loss: 1.112597E+01 | loss scale: 1.0 | grad norm: 413.140 | number of skipped iterations: 0 | number of nan iterations: 0 |
1564
+ [2026-08-30 00:28:00] iteration 6/ 10 | consumed samples: 48 | elapsed time per iteration (ms): 593.3 | throughput per GPU (TFLOP/s/GPU): 215.0 | learning rate: 7.750000E-05 | global batch size: 8 | lm loss: 9.369524E+00 | loss scale: 1.0 | grad norm: 5.038 | number of skipped iterations: 0 | number of nan iterations: 0 |
1565
+ [2026-08-30 00:28:04] iteration 7/ 10 | consumed samples: 56 | elapsed time per iteration (ms): 3393.2 | throughput per GPU (TFLOP/s/GPU): 37.6 | learning rate: 3.250000E-05 | global batch size: 8 | lm loss: 8.974425E+00 | loss scale: 1.0 | grad norm: 3.708 | number of skipped iterations: 0 | number of nan iterations: 0 |
1566
+ [2026-08-30 00:28:04] iteration 8/ 10 | consumed samples: 64 | elapsed time per iteration (ms): 608.2 | throughput per GPU (TFLOP/s/GPU): 209.8 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 8.785715E+00 | loss scale: 1.0 | grad norm: 5.467 | number of skipped iterations: 0 | number of nan iterations: 0 |
1567
+ Capture range started in the application.
1568
+ Capture range ended in the application.
1569
+ Generating '/tmp/nsys-report-9615.qdstrm'
1570
+
1571
+
tierb-fixed-moonep-full/hf_watcher.log CHANGED
@@ -30,3 +30,4 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
30
  [2026-08-30T00:20:20+00:00] backup pass
31
  uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full
32
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
 
 
30
  [2026-08-30T00:20:20+00:00] backup pass
31
  uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full
32
  https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
33
+ [2026-08-30T00:30:27+00:00] backup pass