AbhishekGhosh1998 commited on
Commit
67fc4d2
·
verified ·
1 Parent(s): 5e7fd80

granularity results granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace @ 2026-09-21T05:53:09Z from sp-aiinfra-interactive-aghosh-8-mb99k-8xb200-moe-cg-0

Browse files
granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125/nsys/train.log CHANGED
@@ -1526,3 +1526,46 @@ Generating '/tmp/nsys-report-026f.qdstrm'
1526
 
1527
 
1528
  Collection stop is not allowed in this state.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1526
 
1527
 
1528
  Collection stop is not allowed in this state.
1529
+ W0921 05:52:51.804000 116018 torch/distributed/elastic/agent/server/api.py:719] Received 15 death signal, shutting down workers
1530
+ W0921 05:52:51.804000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116129 closing signal SIGTERM
1531
+ Generated:
1532
+ /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125/nsys/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125.nsys-rep
1533
+ W0921 05:52:51.806000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116130 closing signal SIGTERM
1534
+ W0921 05:52:51.809000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116131 closing signal SIGTERM
1535
+ W0921 05:52:51.811000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116132 closing signal SIGTERM
1536
+ W0921 05:52:51.815000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116133 closing signal SIGTERM
1537
+ W0921 05:52:51.816000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116134 closing signal SIGTERM
1538
+ W0921 05:52:51.821000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116135 closing signal SIGTERM
1539
+ W0921 05:52:51.823000 116018 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 116136 closing signal SIGTERM
1540
+ Traceback (most recent call last):
1541
+ File "/usr/local/bin/torchrun", line 8, in <module>
1542
+ sys.exit(main())
1543
+ ^^^^^^
1544
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
1545
+ return f(*args, **kwargs)
1546
+ ^^^^^^^^^^^^^^^^^^
1547
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
1548
+ run(args)
1549
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
1550
+ elastic_launch(
1551
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
1552
+ return launch_agent(self._config, self._entrypoint, list(args))
1553
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1554
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
1555
+ result = agent.run()
1556
+ ^^^^^^^^^^^
1557
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
1558
+ result = f(*args, **kwargs)
1559
+ ^^^^^^^^^^^^^^^^^^
1560
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
1561
+ result = self._invoke_run(role)
1562
+ ^^^^^^^^^^^^^^^^^^^^^^
1563
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/agent/server/api.py", line 870, in _invoke_run
1564
+ time.sleep(monitor_interval)
1565
+ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/api.py", line 84, in _terminate_process_handler
1566
+ raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
1567
+ torch.distributed.elastic.multiprocessing.api.SignalException: Process 116018 got signal: 15
1568
+ The target application terminated. One or more process it created re-parented.
1569
+ Waiting for termination of re-parented processes.
1570
+ Use the `--wait` option to modify this behavior.
1571
+ There are no active sessions.
granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125/nsys/verify_trace_ranks.log ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ report : /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125/nsys/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125.nsys-rep
2
+ kernels : 186566
3
+ devices : 8 of 8 -> [0, 1, 2, 3, 4, 5, 6, 7]
4
+ processes: 1
5
+ markers : 26624 ultraep_trace__ kernels on devices [0, 1, 2, 3, 4, 5, 6, 7]
6
+ PASS: all ranks present and backend markers emitted
granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/trace_results.tsv CHANGED
@@ -7,3 +7,4 @@ G4_K4 ultraep 1.171875 OK /workspace/granularity_next_b200/tiera_topk_trace/ultr
7
  G4_K6 ultraep 1.1354166666666667 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K6__ultraep__matched__seq4096__gbs16__cap1p1354166666666667/nsys/G4_K6__ultraep__matched__seq4096__gbs16__cap1p1354166666666667.nsys-rep
8
  G4_K8 ultraep 1.109375 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K8__ultraep__matched__seq4096__gbs16__cap1p109375/nsys/G4_K8__ultraep__matched__seq4096__gbs16__cap1p109375.nsys-rep
9
  G4_K10 ultraep 1.1 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K10__ultraep__matched__seq4096__gbs16__cap1p1/nsys/G4_K10__ultraep__matched__seq4096__gbs16__cap1p1.nsys-rep
 
 
7
  G4_K6 ultraep 1.1354166666666667 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K6__ultraep__matched__seq4096__gbs16__cap1p1354166666666667/nsys/G4_K6__ultraep__matched__seq4096__gbs16__cap1p1354166666666667.nsys-rep
8
  G4_K8 ultraep 1.109375 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K8__ultraep__matched__seq4096__gbs16__cap1p109375/nsys/G4_K8__ultraep__matched__seq4096__gbs16__cap1p109375.nsys-rep
9
  G4_K10 ultraep 1.1 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K10__ultraep__matched__seq4096__gbs16__cap1p1/nsys/G4_K10__ultraep__matched__seq4096__gbs16__cap1p1.nsys-rep
10
+ G4_K16 ultraep 1.08203125 OK /workspace/granularity_next_b200/tiera_topk_trace/ultraep/tierbtrace/tierbtrace_gbs16/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125/nsys/G4_K16__ultraep__matched__seq4096__gbs16__cap1p08203125.nsys-rep