granularity results tierb-fixed-moonep-full @ 2026-08-30T00:30:27Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0
Browse files- .gitattributes +1 -0
- tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333.nsys-rep +3 -0
- tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/train.log +181 -0
- tierb-fixed-moonep-full/hf_watcher.log +1 -0
.gitattributes
CHANGED
|
@@ -57,3 +57,4 @@ moonep/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__moonep__matched__seq8192__gbs8_
|
|
| 57 |
moonep/tierbtrace/tierbtrace_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone/nsys/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
| 58 |
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375/nsys/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
| 59 |
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125/nsys/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 57 |
moonep/tierbtrace/tierbtrace_gbs8/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone/nsys/QWEN2_57B__moonep__matched__seq4096__gbs8__capnone.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
| 58 |
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375/nsys/QWEN3_235B__echo__matched__seq8192__gbs8__cap1p083984375.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
| 59 |
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125/nsys/QWEN3_30B__echo__matched__seq4096__gbs8__cap1p11328125.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
| 60 |
+
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333.nsys-rep
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d34c00d2677ccce5088f6ec814c1f4967fd78717b8c83c2fbeca0059a1d153da
|
| 3 |
+
size 18115411
|
tierb-fixed-moonep-full/echo/tierbtrace/tierbtrace_gbs8/DSV2_LITE_L24__echo__matched__seq8192__gbs8__cap1p0755208333333333/nsys/train.log
CHANGED
|
@@ -1388,3 +1388,184 @@ Overwriting rerun_state_machine.current_iteration from -1 to 0...
|
|
| 1388 |
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1389 |
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1390 |
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1388 |
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1389 |
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1390 |
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1391 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1392 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1393 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1394 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1395 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1396 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1397 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1398 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1399 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1400 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1401 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1402 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1403 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1404 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1405 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1406 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1407 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1408 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1409 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1410 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1411 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1412 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1413 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1414 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1415 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1416 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1417 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1418 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1419 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1420 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1421 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1422 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1423 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1424 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1425 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1426 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1427 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1428 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1429 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1430 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1431 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1432 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1433 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1434 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1435 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1436 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1437 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1438 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1439 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1440 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1441 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1442 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1443 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1444 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1445 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1446 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1447 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1448 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1449 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1450 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1451 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1452 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1453 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1454 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1455 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1456 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1457 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1458 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1459 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1460 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1461 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1462 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1463 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1464 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1465 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1466 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1467 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1468 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1469 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1470 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1471 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1472 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1473 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1474 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1475 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1476 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1477 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1478 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1479 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1480 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1481 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1482 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1483 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1484 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1485 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1486 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1487 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1488 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1489 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1490 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1491 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1492 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1493 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1494 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1495 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1496 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1497 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1498 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1499 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1500 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1501 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1502 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1503 |
+
[2026-08-30 00:27:56] iteration 1/ 10 | consumed samples: 8 | elapsed time per iteration (ms): 46530.8 | throughput per GPU (TFLOP/s/GPU): 2.7 | learning rate: 2.000000E-05 | global batch size: 8 | lm loss: 1.092139E+01 | loss scale: 1.0 | grad norm: 30.619 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1504 |
+
Number of parameters in transformer block in billions: 13.53
|
| 1505 |
+
Number of parameters in embedding layers in billions: 0.21
|
| 1506 |
+
Total number of parameters in billions: 13.74
|
| 1507 |
+
Number of parameters in most loaded shard in billions: 13.7356
|
| 1508 |
+
Theoretical memory footprints: weight and optimizer=98244.96 MB
|
| 1509 |
+
[Rank 0] (after 1 iterations) memory (MB) | allocated: 40504.66845703125 | max allocated: 106236.974609375 | reserved: 109990.0 | max reserved: 109990.0
|
| 1510 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1511 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1512 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1513 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1514 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1515 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1516 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1517 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1518 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1519 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1520 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1521 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1522 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1523 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1524 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1525 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1526 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1527 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1528 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1529 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1530 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1531 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1532 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1533 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1534 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1535 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1536 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1537 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1538 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:334: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1539 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1540 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:336: UserWarning: The use_host_meta is deprecated, it will be removed in the future.
|
| 1541 |
+
warnings.warn("The use_host_meta is deprecated, it will be removed in the future.")
|
| 1542 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1543 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1544 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1545 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1546 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1547 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1548 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1549 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1550 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1551 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1552 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1553 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1554 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1555 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1556 |
+
/workspace/DeepEP/deep_ep/hybrid_ep_buffer.py:482: UserWarning: The num_dispatched_tokens is deprecated, it will be removed in the future.
|
| 1557 |
+
warnings.warn("The num_dispatched_tokens is deprecated, it will be removed in the future.")
|
| 1558 |
+
[2026-08-30 00:27:56] iteration 2/ 10 | consumed samples: 16 | elapsed time per iteration (ms): 646.1 | throughput per GPU (TFLOP/s/GPU): 197.5 | learning rate: 4.000000E-05 | global batch size: 8 | lm loss: 1.092003E+01 | loss scale: 1.0 | grad norm: 30.571 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1559 |
+
[2026-08-30 00:27:57] iteration 3/ 10 | consumed samples: 24 | elapsed time per iteration (ms): 639.7 | throughput per GPU (TFLOP/s/GPU): 199.5 | learning rate: 6.000000E-05 | global batch size: 8 | lm loss: 9.952750E+00 | loss scale: 1.0 | grad norm: 42.271 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1560 |
+
Capture CUDA graph for training!!!
|
| 1561 |
+
CUDA graph capture done!!!
|
| 1562 |
+
[2026-08-30 00:27:59] iteration 4/ 10 | consumed samples: 32 | elapsed time per iteration (ms): 1959.3 | throughput per GPU (TFLOP/s/GPU): 65.1 | learning rate: 8.000000E-05 | global batch size: 8 | lm loss: 9.477107E+00 | loss scale: 1.0 | grad norm: 7.154 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1563 |
+
[2026-08-30 00:28:00] iteration 5/ 10 | consumed samples: 40 | elapsed time per iteration (ms): 588.8 | throughput per GPU (TFLOP/s/GPU): 216.7 | learning rate: 1.000000E-04 | global batch size: 8 | lm loss: 1.112597E+01 | loss scale: 1.0 | grad norm: 413.140 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1564 |
+
[2026-08-30 00:28:00] iteration 6/ 10 | consumed samples: 48 | elapsed time per iteration (ms): 593.3 | throughput per GPU (TFLOP/s/GPU): 215.0 | learning rate: 7.750000E-05 | global batch size: 8 | lm loss: 9.369524E+00 | loss scale: 1.0 | grad norm: 5.038 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1565 |
+
[2026-08-30 00:28:04] iteration 7/ 10 | consumed samples: 56 | elapsed time per iteration (ms): 3393.2 | throughput per GPU (TFLOP/s/GPU): 37.6 | learning rate: 3.250000E-05 | global batch size: 8 | lm loss: 8.974425E+00 | loss scale: 1.0 | grad norm: 3.708 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1566 |
+
[2026-08-30 00:28:04] iteration 8/ 10 | consumed samples: 64 | elapsed time per iteration (ms): 608.2 | throughput per GPU (TFLOP/s/GPU): 209.8 | learning rate: 1.000000E-05 | global batch size: 8 | lm loss: 8.785715E+00 | loss scale: 1.0 | grad norm: 5.467 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
| 1567 |
+
Capture range started in the application.
|
| 1568 |
+
Capture range ended in the application.
|
| 1569 |
+
Generating '/tmp/nsys-report-9615.qdstrm'
|
| 1570 |
+
|
| 1571 |
+
|
tierb-fixed-moonep-full/hf_watcher.log
CHANGED
|
@@ -30,3 +30,4 @@ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
|
| 30 |
[2026-08-30T00:20:20+00:00] backup pass
|
| 31 |
uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full
|
| 32 |
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
|
|
|
|
|
| 30 |
[2026-08-30T00:20:20+00:00] backup pass
|
| 31 |
uploaded: /workspace/tierb-fixed-moonep-full -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tierb-fixed-moonep-full
|
| 32 |
https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
|
| 33 |
+
[2026-08-30T00:30:27+00:00] backup pass
|