AbhishekGhosh1998 commited on
Commit
0662f0e
·
verified ·
1 Parent(s): b53279c

granularity results tiera-rerun-fixed-moonep @ 2026-08-30T10:26:09Z from sp-aiinfra-interactive-aghosh-7jtjl-8xb200-moe-cg-0 (part 3)

Browse files
tiera-rerun-fixed-moonep/steps.tsv CHANGED
@@ -22,3 +22,6 @@ OK moonep tierbmem 16 2026-08-30T10:05:11+00:00 2026-08-30T10:07:42+00:00 /works
22
  OK echo tierbmem 16 2026-08-30T10:08:07+00:00 2026-08-30T10:11:03+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
23
  OK ultraep tierbmem 16 2026-08-30T10:11:14+00:00 2026-08-30T10:13:56+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
24
  OK moonep tierbmem 16 2026-08-30T10:14:05+00:00 2026-08-30T10:16:46+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
 
 
 
 
22
  OK echo tierbmem 16 2026-08-30T10:08:07+00:00 2026-08-30T10:11:03+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
23
  OK ultraep tierbmem 16 2026-08-30T10:11:14+00:00 2026-08-30T10:13:56+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
24
  OK moonep tierbmem 16 2026-08-30T10:14:05+00:00 2026-08-30T10:16:46+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
25
+ OK echo tierbmem 16 2026-08-30T10:17:11+00:00 2026-08-30T10:20:07+00:00 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
26
+ OK ultraep tierbmem 16 2026-08-30T10:20:18+00:00 2026-08-30T10:23:00+00:00 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
27
+ OK moonep tierbmem 16 2026-08-30T10:23:07+00:00 2026-08-30T10:25:51+00:00 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
tiera-rerun-fixed-moonep/study.log CHANGED
@@ -3178,3 +3178,372 @@ driver_repo: /workspace/blackwell_run_scripts @ 48db41e
3178
  === STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
3179
 
3180
  === STEP echo__tierbmem__gbs16__G4
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3178
  === STAGE 8 -- Tier B allocator dumps (static CG capacities + dynamic no-CG)
3179
 
3180
  === STEP echo__tierbmem__gbs16__G4
3181
+ ==============================================================
3182
+ CAMPAIGN: echo backend=echo phase=tierbmem
3183
+ Megatron: /workspace/Megatron-MoE-Echo @ better-memory-dump-support
3184
+ GBS=16 seq=4096 repeats=3 nsys=0
3185
+ results -> /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
3186
+ ==============================================================
3187
+ --------------------------------------------------------------------------
3188
+ [10:17:26] CELL G4__echo__matched__seq4096__gbs16__cap1p095703125__memiteration__it6 (slots=16)
3189
+ [guard] iter 4/6 (70s)
3190
+ [guard] iter 6/6 (80s)
3191
+ --------------------------------------------------------------------------
3192
+ [10:18:46] CELL G4__echo__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5 (slots=16)
3193
+ [guard] iter 1/5 (70s)
3194
+ [guard] iter 5/5 (80s)
3195
+ [guard] tearing down process group (outcome=DONE)
3196
+ --------------------------------------------------------------------------
3197
+
3198
+ ==============================================================
3199
+ SWEEP COMPLETE ok=2 failed=0
3200
+ ==============================================================
3201
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
3202
+ G4 echo matched 16 4096 16 1.095703125 none 1 OK 501.20 501.20 168261 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/G4__echo__matched__seq4096__gbs16__cap1p095703125__memiteration__it6/rep1
3203
+ G4 echo matched 16 4096 16 dynamic none 1 OK NA NA 125233 /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/G4__echo__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5/rep1
3204
+
3205
+ Results: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/results.tsv
3206
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16
3207
+ ==============================================================
3208
+ 8 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/echo/tierbmem/tierbmem_gbs16/memory_summary.tsv
3209
+ model backend cg allocation capacity status peak GiB dumps
3210
+ ----------------------------------------------------------------------------------
3211
+ G1 echo 1 static 2.40625 OK 157.9 48/48
3212
+ G1 echo 0 dynamic dynamic OK 112.6 40/40
3213
+ G2 echo 1 static1.08203125 OK 142.5 48/48
3214
+ G2 echo 0 dynamic dynamic OK 115.1 40/40
3215
+ G3 echo 1 static1.072265625 OK 144.5 48/48
3216
+ G3 echo 0 dynamic dynamic OK 116.1 40/40
3217
+ G4 echo 1 static1.095703125 OK 164.3 48/48
3218
+ G4 echo 0 dynamic dynamic OK 122.3 40/40
3219
+
3220
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
3221
+
3222
+ TIER B MEMORY DUMPS -- how to read this
3223
+ ---------------------------------------
3224
+ ECHO / UltraEP:
3225
+ cap1p00__memiteration__it6 graph on, static C=1.00
3226
+ cap1p125__memiteration__it6 graph on, static C=1.125
3227
+ cap1p25__memiteration__it6 graph on, static C=1.25
3228
+ capdynamic__nocg__memiteration__it5
3229
+ graph off, no capacity flag, dynamic allocation
3230
+
3231
+ MoonEP:
3232
+ capnone__memiteration__it6 graph on, no capacity factor
3233
+ capnone__nocg__memiteration__it5 graph off, no capacity factor
3234
+
3235
+ Every successful cell contains:
3236
+ train.log Megatron and OOM diagnostics
3237
+ mem_samples.csv driver-level nvidia-smi samples every 2 seconds
3238
+ run.env exact branch/config/capacity provenance
3239
+ memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
3240
+ memory_snapshots/*.json matching summaries and before/after deltas
3241
+
3242
+ The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
3243
+ "graph cost": capacity allocation policy and device-initiated grouped GEMM also
3244
+ change. Compare the three graph-on rows to measure static-capacity sensitivity,
3245
+ and use the dynamic row as the natural eager-allocation reference.
3246
+
3247
+ Campaign results: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem
3248
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/echo/tierbmem --compare
3249
+ Already logged in to Hugging Face as: AbhishekGhosh1998
3250
+ uploaded: /workspace/tiera-rerun-fixed-moonep/echo/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/echo/tierbmem
3251
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
3252
+
3253
+ === STEP ultraep__tierbmem__gbs16__G4
3254
+ ==============================================================
3255
+ CAMPAIGN: ultraep backend=ultraep phase=tierbmem
3256
+ Megatron: /workspace/Megatron-UltraEP @ ultraep-better-memory-dump-support
3257
+ GBS=16 seq=4096 repeats=3 nsys=0
3258
+ results -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
3259
+ ==============================================================
3260
+ --------------------------------------------------------------------------
3261
+ [10:20:33] CELL G4__ultraep__matched__seq4096__gbs16__cap1p095703125__memiteration__it6 (slots=16)
3262
+ [guard] iter 3/6 (60s)
3263
+ [guard] iter 6/6 (70s)
3264
+ [guard] tearing down process group (outcome=DONE)
3265
+ --------------------------------------------------------------------------
3266
+ [10:21:46] CELL G4__ultraep__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5 (slots=16)
3267
+ [guard] iter 1/5 (60s)
3268
+ [guard] iter 5/5 (70s)
3269
+ [guard] tearing down process group (outcome=DONE)
3270
+ --------------------------------------------------------------------------
3271
+
3272
+ ==============================================================
3273
+ SWEEP COMPLETE ok=2 failed=0
3274
+ ==============================================================
3275
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
3276
+ G4 ultraep matched 16 4096 16 1.095703125 none 1 OK 498.70 498.70 132437 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/G4__ultraep__matched__seq4096__gbs16__cap1p095703125__memiteration__it6/rep1
3277
+ G4 ultraep matched 16 4096 16 dynamic none 1 OK NA NA 120957 /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/G4__ultraep__matched__seq4096__gbs16__capdynamic__nocg__memiteration__it5/rep1
3278
+
3279
+ Results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/results.tsv
3280
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16
3281
+ ==============================================================
3282
+ 8 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem/tierbmem_gbs16/memory_summary.tsv
3283
+ model backend cg allocation capacity status peak GiB dumps
3284
+ ----------------------------------------------------------------------------------
3285
+ G1 ultraep 1 static 2.40625 OK 126.4 48/48
3286
+ G1 ultraep 0 dynamic dynamic OK 106.7 40/40
3287
+ G2 ultraep 1 static1.08203125 OK 109.7 48/48
3288
+ G2 ultraep 0 dynamic dynamic OK 105.7 40/40
3289
+ G3 ultraep 1 static1.072265625 OK 111.3 48/48
3290
+ G3 ultraep 0 dynamic dynamic OK 109.1 40/40
3291
+ G4 ultraep 1 static1.095703125 OK 129.3 48/48
3292
+ G4 ultraep 0 dynamic dynamic OK 118.1 40/40
3293
+
3294
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
3295
+
3296
+ TIER B MEMORY DUMPS -- how to read this
3297
+ ---------------------------------------
3298
+ ECHO / UltraEP:
3299
+ cap1p00__memiteration__it6 graph on, static C=1.00
3300
+ cap1p125__memiteration__it6 graph on, static C=1.125
3301
+ cap1p25__memiteration__it6 graph on, static C=1.25
3302
+ capdynamic__nocg__memiteration__it5
3303
+ graph off, no capacity flag, dynamic allocation
3304
+
3305
+ MoonEP:
3306
+ capnone__memiteration__it6 graph on, no capacity factor
3307
+ capnone__nocg__memiteration__it5 graph off, no capacity factor
3308
+
3309
+ Every successful cell contains:
3310
+ train.log Megatron and OOM diagnostics
3311
+ mem_samples.csv driver-level nvidia-smi samples every 2 seconds
3312
+ run.env exact branch/config/capacity provenance
3313
+ memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
3314
+ memory_snapshots/*.json matching summaries and before/after deltas
3315
+
3316
+ The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
3317
+ "graph cost": capacity allocation policy and device-initiated grouped GEMM also
3318
+ change. Compare the three graph-on rows to measure static-capacity sensitivity,
3319
+ and use the dynamic row as the natural eager-allocation reference.
3320
+
3321
+ Campaign results: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem
3322
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem --compare
3323
+ Already logged in to Hugging Face as: AbhishekGhosh1998
3324
+ uploaded: /workspace/tiera-rerun-fixed-moonep/ultraep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/ultraep/tierbmem
3325
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
3326
+
3327
+ === STEP moonep__tierbmem__gbs16__G4
3328
+ ==============================================================
3329
+ CAMPAIGN: moonep backend=moonep phase=tierbmem
3330
+ Megatron: /workspace/Megatron-MoE-Echo @ moonep_backend_async_grad_reduce_better_memory_dump_support
3331
+ GBS=16 seq=4096 repeats=3 nsys=0
3332
+ results -> /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
3333
+ ==============================================================
3334
+ --------------------------------------------------------------------------
3335
+ [10:23:22] CELL G4__moonep__matched__seq4096__gbs16__capnone__async__memiteration__it6 (slots=16)
3336
+ [guard] iter 2/6 (60s)
3337
+ [guard] iter 6/6 (70s)
3338
+ [guard] tearing down process group (outcome=DONE)
3339
+ --------------------------------------------------------------------------
3340
+ [10:24:36] CELL G4__moonep__matched__seq4096__gbs16__capnone__nocg__async__memiteration__it5 (slots=16)
3341
+ [guard] iter 5/5 (70s)
3342
+ [guard] tearing down process group (outcome=DONE)
3343
+ --------------------------------------------------------------------------
3344
+
3345
+ ==============================================================
3346
+ SWEEP COMPLETE ok=2 failed=0
3347
+ ==============================================================
3348
+ config backend policy slots seq gbs cap recompute rep status iter_ms_mean iter_ms_min peak_mib run_dir
3349
+ G4 moonep matched 16 4096 16 none none 1 OK 810.50 810.50 106029 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/G4__moonep__matched__seq4096__gbs16__capnone__async__memiteration__it6/rep1
3350
+ G4 moonep matched 16 4096 16 none none 1 OK NA NA 105101 /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/G4__moonep__matched__seq4096__gbs16__capnone__nocg__async__memiteration__it5/rep1
3351
+
3352
+ Results: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/results.tsv
3353
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16
3354
+ ==============================================================
3355
+ 8 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem/tierbmem_gbs16/memory_summary.tsv
3356
+ model backend cg allocation capacity status peak GiB dumps
3357
+ ----------------------------------------------------------------------------------
3358
+ G1 moonep 1 none none OK 94.6 48/48
3359
+ G1 moonep 0 none none OK 90.4 40/40
3360
+ G2 moonep 1 none none OK 95.4 48/48
3361
+ G2 moonep 0 none none OK 93.2 40/40
3362
+ G3 moonep 1 none none OK 97.1 48/48
3363
+ G3 moonep 0 none none OK 96.3 40/40
3364
+ G4 moonep 1 none none OK 103.5 48/48
3365
+ G4 moonep 0 none none OK 102.6 40/40
3366
+
3367
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
3368
+
3369
+ TIER B MEMORY DUMPS -- how to read this
3370
+ ---------------------------------------
3371
+ ECHO / UltraEP:
3372
+ cap1p00__memiteration__it6 graph on, static C=1.00
3373
+ cap1p125__memiteration__it6 graph on, static C=1.125
3374
+ cap1p25__memiteration__it6 graph on, static C=1.25
3375
+ capdynamic__nocg__memiteration__it5
3376
+ graph off, no capacity flag, dynamic allocation
3377
+
3378
+ MoonEP:
3379
+ capnone__memiteration__it6 graph on, no capacity factor
3380
+ capnone__nocg__memiteration__it5 graph off, no capacity factor
3381
+
3382
+ Every successful cell contains:
3383
+ train.log Megatron and OOM diagnostics
3384
+ mem_samples.csv driver-level nvidia-smi samples every 2 seconds
3385
+ run.env exact branch/config/capacity provenance
3386
+ memory_snapshots/*.pickle 48 captured / 40 eager allocator histories
3387
+ memory_snapshots/*.json matching summaries and before/after deltas
3388
+
3389
+ The graph-on/static vs graph-off/dynamic difference is intentionally NOT called
3390
+ "graph cost": capacity allocation policy and device-initiated grouped GEMM also
3391
+ change. Compare the three graph-on rows to measure static-capacity sensitivity,
3392
+ and use the dynamic row as the natural eager-allocation reference.
3393
+
3394
+ Campaign results: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem
3395
+ Roll up: python3 /workspace/blackwell_run_scripts/granularity/tools/collect.py /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem --compare
3396
+ Already logged in to Hugging Face as: AbhishekGhosh1998
3397
+ uploaded: /workspace/tiera-rerun-fixed-moonep/moonep/tierbmem -> MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tiera-rerun-fixed-moonep/moonep/tierbmem
3398
+ https://huggingface.co/MoE-CUDA-Graph-Analysis/tierb-b200-gbs8/tree/main
3399
+
3400
+ === ROLL-UP
3401
+ 140 runs ok=128 oom=0 -> /workspace/tiera-rerun-fixed-moonep/collected_all.tsv
3402
+
3403
+ WARNING: 11 run(s) mention capacity saturation. A run that drops assignments does less expert work and must not be quoted as a win:
3404
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=283999 total_dropped_tok
3405
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__matched__seq4096__gbs16__cap1p8125/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=284540 total_dropped_tok
3406
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p0625/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
3407
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=0 iteration=0 records=32 total_expected_recv=1048800 total_dropped_to
3408
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__matched__seq4096__gbs16__cap1p25__forcebal/rep1: WARNING:megatron.training.drop_debug_utils:[MOE_DROP_DEBUG][iteration_summary] global_rank=1 iteration=0 records=32 total_expected_recv=1047984 total_dropped_to
3409
+
3410
+ (excluded 14 nsys-traced run(s) from the timing comparison)
3411
+
3412
+ phase config seq gbs policy rc cg asy backend iter_ms vs_best
3413
+ -------------------------------------------------------------------------------------
3414
+ a1 G1 4096 16 one none 1 0 echo 466.68 1.00x
3415
+ ultraep 891.00 1.91x
3416
+
3417
+ a1 G1 4096 16 one none 1 1 moonep 480.30 1.00x
3418
+
3419
+ a1 G2 4096 16 one none 1 0 echo 394.28 1.00x
3420
+ ultraep 466.65 1.18x
3421
+
3422
+ a1 G3 4096 16 one none 1 0 echo 412.83 1.00x
3423
+ ultraep 447.84 1.08x
3424
+
3425
+ a1 G4 4096 16 one none 1 0 echo 461.03 1.00x
3426
+ ultraep 467.57 1.01x
3427
+
3428
+ a2 G1 4096 16 matched none 1 0 echo 466.91 1.00x
3429
+ ultraep 891.92 1.91x
3430
+
3431
+ a2 G1 4096 16 matched none 1 1 moonep 481.37 1.00x
3432
+
3433
+ a2 G2 4096 16 matched none 1 0 echo 415.21 1.00x
3434
+ ultraep 492.54 1.19x
3435
+
3436
+ a2 G2 4096 16 matched none 1 1 moonep 432.02 1.00x
3437
+
3438
+ a2 G3 4096 16 matched none 1 0 echo 437.35 1.00x
3439
+ ultraep 459.60 1.05x
3440
+
3441
+ a2 G3 4096 16 matched none 1 1 moonep 559.88 1.00x
3442
+
3443
+ a2 G4 4096 16 matched none 1 0 ultraep 482.64 1.00x
3444
+ echo 486.54 1.01x
3445
+
3446
+ a2 G4 4096 16 matched none 1 1 moonep 783.34 1.00x
3447
+
3448
+ ponly PONLY 4096 16 one none 1 0 echo 174.81 1.00x
3449
+ ultraep 176.76 1.01x
3450
+
3451
+ ponly PONLY 4096 16 one none 1 1 moonep 165.22 1.00x
3452
+
3453
+ ponly PONLY 8192 16 one none 1 0 ultraep 292.62 1.00x
3454
+ echo 316.70 1.08x
3455
+
3456
+ ponly PONLY 8192 16 one none 1 1 moonep 300.61 1.00x
3457
+
3458
+ tierbmem G1 4096 16 matched none 1 0 echo 474.40 1.00x
3459
+ ultraep 909.90 1.92x
3460
+
3461
+ tierbmem G1 4096 16 matched none 1 1 moonep 497.40 1.00x
3462
+
3463
+ tierbmem G2 4096 16 matched none 1 0 echo 430.10 1.00x
3464
+ ultraep 475.40 1.11x
3465
+
3466
+ tierbmem G2 4096 16 matched none 1 1 moonep 439.70 1.00x
3467
+
3468
+ tierbmem G3 4096 16 matched none 1 0 echo 452.40 1.00x
3469
+ ultraep 477.80 1.06x
3470
+
3471
+ tierbmem G3 4096 16 matched none 1 1 moonep 578.30 1.00x
3472
+
3473
+ tierbmem G4 4096 16 matched none 1 0 ultraep 498.70 1.00x
3474
+ echo 501.20 1.01x
3475
+
3476
+ tierbmem G4 4096 16 matched none 1 1 moonep 810.50 1.00x
3477
+
3478
+ tierbprobeG1 4096 16 matched none 0 0 echo 671.53 1.00x
3479
+ ultraep 904.16 1.35x
3480
+
3481
+ tierbprobeG2 4096 16 matched none 0 0 ultraep 512.37 1.00x
3482
+ echo 654.76 1.28x
3483
+
3484
+ tierbprobeG3 4096 16 matched none 0 0 ultraep 512.31 1.00x
3485
+ echo 689.13 1.35x
3486
+
3487
+ tierbprobeG4 4096 16 matched none 0 0 ultraep 600.19 1.00x
3488
+ echo 790.97 1.32x
3489
+
3490
+ 21 probe runs -> /workspace/tiera-rerun-fixed-moonep/probe_summary_all.tsv
3491
+ run arm cap rawExpI preRankI postRankI drop% reqCap
3492
+ --------------------------------------------------------------------------------------------------------------------
3493
+ echo/tierbprobe/tierbprobe_gbs16/G1__echo__matched__ natural 2.00 3.6198* 3.6198 1.0827 1.9062
3494
+ echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 1.08203125 3.8720* 2.1703 1.0000 1.0273
3495
+ echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ forced 1.08203125 1.0201 1.0069 1.0000 1.0234
3496
+ echo/tierbprobe/tierbprobe_gbs16/G2__echo__matched__ natural 2.00 3.8724* 2.1875 1.0000 1.0273
3497
+ echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 1.072265625 3.9096* 1.7838 1.0000 1.0195
3498
+ echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ forced 1.072265625 1.0231 1.0048 1.0000 1.0176
3499
+ echo/tierbprobe/tierbprobe_gbs16/G3__echo__matched__ natural 2.00 3.9096* 1.7916 1.0000 1.0195
3500
+ echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 1.068359375 3.9303* 1.5507 1.0000 1.0166
3501
+ echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ forced 1.068359375 1.0249 1.0034 1.0000 1.0137
3502
+ echo/tierbprobe/tierbprobe_gbs16/G4__echo__matched__ natural 2.00 3.9301* 1.5543 1.0000 1.0166
3503
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.25 3.6085 1.0640 2.8918 1.7246
3504
+ ultraep/tierbprobe/tierbprobe_gbs16/G1__ultraep__mat natural 1.8125 3.6152 1.0747 1.5460 2.3091
3505
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.0625 2.2032 1.0100 0.0000 1.0101
3506
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat natural 1.25 2.2136 1.0100 0.0000 1.0101
3507
+ ultraep/tierbprobe/tierbprobe_gbs16/G2__ultraep__mat forced 1.25 1.0069 1.0067 0.0000 1.0100
3508
+ ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.0625 1.7842 1.0100 0.0000 1.0101
3509
+ ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat natural 1.25 1.7752 1.0100 0.0000 1.0101
3510
+ ultraep/tierbprobe/tierbprobe_gbs16/G3__ultraep__mat forced 1.25 1.0047 1.0047 0.0000 1.0100
3511
+ ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.095703125 1.5612 1.0100 0.0000 1.0429
3512
+ ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat natural 1.25 1.5504 1.0100 0.0000 1.0429
3513
+ ultraep/tierbprobe/tierbprobe_gbs16/G4__ultraep__mat forced 1.25 1.0033 1.0033 0.0000 1.0071
3514
+
3515
+ * At least one globally aggregated snapshot reached the E/K expert-Imax ceiling.
3516
+ Inspect raw_expert_imax_mean and hottest_expert_token_fraction_mean before generalising.
3517
+
3518
+ ECHO note: echo_logged_pre_rank_metric_valid=0 and echo_logged_load_reduction_valid=0 are intentional.
3519
+ The branch's built-in fields mix source-rank and destination-rank axes; this tool reconstructs both globally.
3520
+ 24 memory-dump runs -> /workspace/tiera-rerun-fixed-moonep/memory_summary_all.tsv
3521
+ model backend cg allocation capacity status peak GiB dumps
3522
+ ----------------------------------------------------------------------------------
3523
+ G1 echo 1 static 2.40625 OK 157.9 48/48
3524
+ G1 echo 0 dynamic dynamic OK 112.6 40/40
3525
+ G1 moonep 1 none none OK 94.6 48/48
3526
+ G1 moonep 0 none none OK 90.4 40/40
3527
+ G1 ultraep 1 static 2.40625 OK 126.4 48/48
3528
+ G1 ultraep 0 dynamic dynamic OK 106.7 40/40
3529
+ G2 echo 1 static1.08203125 OK 142.5 48/48
3530
+ G2 echo 0 dynamic dynamic OK 115.1 40/40
3531
+ G2 moonep 1 none none OK 95.4 48/48
3532
+ G2 moonep 0 none none OK 93.2 40/40
3533
+ G2 ultraep 1 static1.08203125 OK 109.7 48/48
3534
+ G2 ultraep 0 dynamic dynamic OK 105.7 40/40
3535
+ G3 echo 1 static1.072265625 OK 144.5 48/48
3536
+ G3 echo 0 dynamic dynamic OK 116.1 40/40
3537
+ G3 moonep 1 none none OK 97.1 48/48
3538
+ G3 moonep 0 none none OK 96.3 40/40
3539
+ G3 ultraep 1 static1.072265625 OK 111.3 48/48
3540
+ G3 ultraep 0 dynamic dynamic OK 109.1 40/40
3541
+ G4 echo 1 static1.095703125 OK 164.3 48/48
3542
+ G4 echo 0 dynamic dynamic OK 122.3 40/40
3543
+ G4 moonep 1 none none OK 103.5 48/48
3544
+ G4 moonep 0 none none OK 102.6 40/40
3545
+ G4 ultraep 1 static1.095703125 OK 129.3 48/48
3546
+ G4 ultraep 0 dynamic dynamic OK 118.1 40/40
3547
+
3548
+ ECHO/UltraEP note: graph-on uses static capacity + device-initiated GEMM while graph-off uses dynamic allocation + host-derived sizes. Their difference is not labelled as pure graph cost.
3549
+ Already logged in to Hugging Face as: AbhishekGhosh1998