granularity results granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace @ 2026-09-21T05:53:09Z from sp-aiinfra-interactive-aghosh-8-mb99k-8xb200-moe-cg-0
Browse files- granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125/nsys/train.log +43 -0
- granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125/nsys/verify_trace_ranks.log +6 -0
- granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/trace_results.tsv +1 -0
granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125/nsys/train.log
CHANGED
|
@@ -1526,3 +1526,46 @@ Generating '/tmp/nsys-report-026f.qdstrm'
|
|
| 1526 |
|
| 1527 |
|
| 1528 |
Collection stop is not allowed in this state.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1526 |
|
| 1527 |
|
| 1528 |
Collection stop is not allowed in this state.
|
| 1529 |
+
W0921 05:52:51.804000 116018 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
|
| 1530 |
+
W0921 05:52:51.804000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116129 closing signal SIGTERM
|
| 1531 |
+
Generated:
|
| 1532 |
+
/workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125/nsys/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125.nsys-rep
|
| 1533 |
+
W0921 05:52:51.806000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116130 closing signal SIGTERM
|
| 1534 |
+
W0921 05:52:51.809000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116131 closing signal SIGTERM
|
| 1535 |
+
W0921 05:52:51.811000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116132 closing signal SIGTERM
|
| 1536 |
+
W0921 05:52:51.815000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116133 closing signal SIGTERM
|
| 1537 |
+
W0921 05:52:51.816000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116134 closing signal SIGTERM
|
| 1538 |
+
W0921 05:52:51.821000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116135 closing signal SIGTERM
|
| 1539 |
+
W0921 05:52:51.823000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116136 closing signal SIGTERM
|
| 1540 |
+
Traceback (most recent call last):
|
| 1541 |
+
File "/usr/local/bin/torchrun", line 8, in <module>
|
| 1542 |
+
sys.exit(main())
|
| 1543 |
+
^^^^^^
|
| 1544 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
|
| 1545 |
+
return f(*args, **kwargs)
|
| 1546 |
+
^^^^^^^^^^^^^^^^^^
|
| 1547 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
|
| 1548 |
+
run(args)
|
| 1549 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
|
| 1550 |
+
elastic_launch(
|
| 1551 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
|
| 1552 |
+
return launch_agent(self._config, self._entrypoint, list(args))
|
| 1553 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 1554 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
|
| 1555 |
+
result = agent.run()
|
| 1556 |
+
^^^^^^^^^^^
|
| 1557 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
|
| 1558 |
+
result = f(*args, **kwargs)
|
| 1559 |
+
^^^^^^^^^^^^^^^^^^
|
| 1560 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
|
| 1561 |
+
result = self._invoke_run(role)
|
| 1562 |
+
^^^^^^^^^^^^^^^^^^^^^^
|
| 1563 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 870, in _invoke_run
|
| 1564 |
+
time.sleep(monitor_interval)
|
| 1565 |
+
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
|
| 1566 |
+
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
|
| 1567 |
+
torch.distributed.elastic.multiprocessing.api.SignalException: Process 116018 got signal: 15
|
| 1568 |
+
The target application terminated. One or more process it created re-parented.
|
| 1569 |
+
Waiting for termination of re-parented processes.
|
| 1570 |
+
Use the `--wait` option to modify this behavior.
|
| 1571 |
+
There are no active sessions.
|
granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125/nsys/verify_trace_ranks.log
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
report : /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125/nsys/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125.nsys-rep
|
| 2 |
+
kernels : 186566
|
| 3 |
+
devices : 8 of 8 -> [0, 1, 2, 3, 4, 5, 6, 7]
|
| 4 |
+
processes: 1
|
| 5 |
+
markers : 26624 ultraep_trace__ kernels on devices [0, 1, 2, 3, 4, 5, 6, 7]
|
| 6 |
+
PASS: all ranks present and backend markers emitted
|
granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/trace_results.tsv
CHANGED
|
@@ -7,3 +7,4 @@ G4_K4 ultraep 1.171875 OK /workspace/granularity_next_b200/tiera_topk_trace/ultr
|
|
| 7 |
G4_K6 ultraep 1.1354166666666667 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K6__ultraep__matched__seq4096__gbs16__cap1p1354166666666667/nsys/G4_K6__ultraep__matched__seq4096__gbs16__cap1p1354166666666667.nsys-rep
|
| 8 |
G4_K8 ultraep 1.109375 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K8__ultraep__matched__seq4096__gbs16__cap1p109375/nsys/G4_K8__ultraep__matched__seq4096__gbs16__cap1p109375.nsys-rep
|
| 9 |
G4_K10 ultraep 1.1 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K10__ultraep__matched__seq4096__gbs16__cap1p1/nsys/G4_K10__ultraep__matched__seq4096__gbs16__cap1p1.nsys-rep
|
|
|
|
|
|
| 7 |
G4_K6 ultraep 1.1354166666666667 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K6__ultraep__matched__seq4096__gbs16__cap1p1354166666666667/nsys/G4_K6__ultraep__matched__seq4096__gbs16__cap1p1354166666666667.nsys-rep
|
| 8 |
G4_K8 ultraep 1.109375 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K8__ultraep__matched__seq4096__gbs16__cap1p109375/nsys/G4_K8__ultraep__matched__seq4096__gbs16__cap1p109375.nsys-rep
|
| 9 |
G4_K10 ultraep 1.1 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K10__ultraep__matched__seq4096__gbs16__cap1p1/nsys/G4_K10__ultraep__matched__seq4096__gbs16__cap1p1.nsys-rep
|
| 10 |
+
G4_K16 ultraep 1.08203125 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125/nsys/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125.nsys-rep
|