GH200, H100 PCIe and one MI250X GCD
Attention and HEPTv2 batch-size sweeps
Runs from 29–30 October 2026
120 steps per case, batches 8–128
GH200, H100 PCIe and one MI250X GCD
Attention and HEPTv2 batch-size sweeps
Runs from 29–30 October 2026
120 steps per case, batches 8–128
| Platform | Model | Batch | Events/s | kHits/s |
|---|---|---|---|---|
| GH200 | Attention | 32 | 14.55 | 140.6 |
| GH200 | HEPTv2 | 32 | 13.83 | 133.7 |
| H100 PCIe | Attention | 16 | 12.81 | 123.6 |
| H100 PCIe | HEPTv2 | 16 | 11.21 | 108.1 |
| MI250X (1 GCD) | Attention | 16 | 7.35 | 70.9 |
| MI250X (1 GCD) | HEPTv2 | 16 | 6.18 | 59.6 |
GH200 leads these platform runs. All six configurations slow down at batch 64 relative to their observed peak.
| Device | Visible GiB | CPU arch. | TFDS | Runtime | Workers |
|---|---|---|---|---|---|
| GH200 | 95.0 | aarch64 | 3.2.2 | CUDA 12.8 | 4 |
| H100 PCIe | 79.2 | x86_64 | 3.2.1 | CUDA 12.8 | 4 |
| MI250X (1 GCD) | 64.0 | x86_64 | 3.2.2 | ROCm 7.2 | 2 |
All runs use PyTorch 2.11, BF16 and one visible device. NVIDIA uses native attention. LUMI uses external FlashAttention.
LUMI allocates one 64 GiB GCD, half of an MI250X module. H100 is the PCIe variant. The GH200 device name includes “480GB”, but PyTorch exposes about 95 GiB of device memory.
Two full training recipes: six backbone layers, width 256 and 16 heads, with a shared four-layer, 256-slot set decoder. HEPTv2 enables activation checkpointing.
120 training steps: discard the first 20 for throughput. Seed 12345, LAMB, learning rate 0.001, cosine schedule and 20-step loss calibration.
Training rate: sum of events or hits divided by summed host step durations over steps 21–120. Includes data wait and existing synchronizations. Excludes validation and checkpoint writes.
End-of-case evaluation: step 120, 32 validation events and 8 test events. Event exposure grows with batch size, so final losses do not measure matched training quality.
Batch 128 fails with OOM for both recipes on all three platforms. Missing throughput at 128 represents failure, not zero performance.
All 120 per-step sequence-length vectors match across platforms for every completed model/batch pair. This checks event-size exposure, not dataset payload identity.
Raw losses for all 120 steps, without smoothing. Dotted vertical line: end of 20-step calibration. Compare platforms within each batch. TFDS versions and attention backends differ.
Raw losses for all 120 steps, without smoothing. Dotted vertical line: end of 20-step calibration. Compare platforms within each batch. TFDS versions and attention backends differ.
The largest within-batch platform spread is 0.169 for Attention at batch 32. Only 32 validation events and one run per case support these snapshots. No convergence or significance claim.
At matched batches, GH200 is 1.06–1.35× H100 and 1.79–2.49× one MI250X GCD. Attention is faster in all 12 within-platform comparisons.
At batch 64, Attention allocates about 43.1 GiB and HEPTv2 about 54.3 GiB. HEPTv2 uses about 26% more allocated memory despite activation checkpointing.
| Platform | Model | Result | Events before failure | Allocated GiB | Reserved GiB |
|---|---|---|---|---|---|
| GH200 | Attention | OOM | 256 | 80.7 | 94.0 |
| GH200 | HEPTv2 | OOM | 0 | 87.3 | 91.3 |
| H100 PCIe | Attention | OOM | 0 | 77.1 | 78.6 |
| H100 PCIe | HEPTv2 | OOM | 0 | 73.0 | 78.3 |
| MI250X (1 GCD) | Attention | OOM | 0 | 61.6 | 63.4 |
| MI250X (1 GCD) | HEPTv2 | OOM | 0 | 57.5 | 62.8 |
H100 HEPTv2 reports a fused bucket-attention RuntimeError. Its chained exception and sweep log establish OOM as the cause.
Batch 64 is the largest successful tested batch. The exact limit lies somewhere between the tested points. Failed-run peaks do not include the allocation that failed.
The p99/p50 ratio approaches 2× for some batch-8 runs. Each percentile comes from just 100 measured steps in one run. These are descriptive tails, not confidence intervals.
Steady rate uses the last 100 steps. Whole-worker and whole-case rates use all 120. The case includes launch/import time, warmup, validation and checkpointing.
| Batch 16 | Model | GPU busy | Busy coverage | Mean sampled W |
|---|---|---|---|---|
| GH200 | Attention | 53.7% | 99.2% | 256 |
| GH200 | HEPTv2 | 54.0% | 99.2% | 289 |
| H100 PCIe | Attention | 56.6% | 99.4% | 189 |
| H100 PCIe | HEPTv2 | 65.3% | 99.5% | 206 |
| MI250X (1 GCD) | Attention | 86.8% | 99.3% | Unavailable |
| MI250X (1 GCD) | HEPTv2 | 88.5% | 99.8% | Unavailable |
MI250X shows higher device busy while delivering fewer events/s. Busy is a vendor device metric, not SM occupancy or FLOP efficiency.
LUMI has no usable power measurements. GH200 Attention batch 32 has no usable phase-attributed busy or power interval. Energy-efficiency ranking is unsupported.
| Batch | Mean real hits/event | Padded / real hits | Events used for steady rate |
|---|---|---|---|
| 8 | 9,722 | 1.274× | 800 |
| 16 | 9,647 | 1.328× | 1600 |
| 32 | 9,665 | 1.365× | 3200 |
| 64 | 9,613 | 1.396× | 6400 |
Across completed cases, measured data wait accounts for 0.04–0.52% of steady step time. This does not identify a loader bottleneck in the timed window.
Padding grows with batch size. It may contribute to scaling loss, but these records cannot isolate kernel, memory or CPU submission costs. A profiler run is needed to assign cause.
| Platform | Recorded Git base | Dirty tree | Job ID |
|---|---|---|---|
| GH200 | d2d991ce | Yes | 153260 |
| H100 PCIe | 9e3d96bf | Yes | 7363958 |
| MI250X (1 GCD) | 20ac400a | Yes | 23308832 |
H100 uses TFDS 3.2.1 rather than 3.2.2. Matching sequence lengths strengthens size comparability but does not prove matching features or targets.
Different Git bases and dirty trees prevent claiming an identical source snapshot. Metadata retains tracked-diff and untracked-file hashes, not complete historical patches.
One seed and one timed run per case provide no estimate of run-to-run variation. Model quality and full-module MI250X performance remain outside this study.
Shared batch-sweep driver: isolated processes, frozen cases, explicit batch sizes, timeout/OOM classification and CSV/JSON reports.
Instrumentation: raw event and hit counts, step timings, nested phase accounting, device telemetry and framework memory peaks. Validation history now records cycle timings.
Platform support: GH200 environment, H100 and single-GCD LUMI profiles. The set decoder selects native NVIDIA or external ROCm variable-length attention.
LUMI retry changes: two loader workers, file-descriptor sharing and shared-memory/cgroup diagnostics address the earlier loader failure mode.
Candidate operating points: batch 32 on GH200 and batch 16 on H100 / one MI250X GCD for both recipes. These maximize throughput in this sweep, not proven production optima.
Matched rerun: use one clean commit and an identical dataset copy. Repeat batches 8, 16 and 32 in varied order, with enough duration to measure run-to-run spread.
Targeted profiling: compare Attention and HEPTv2 at batch 16. Inspect kernel and host timelines before changing loader settings or attributing the gap to hardware.
Allocation evidence: observed ten-case spans are 42.0, 52.8 and 85.9 minutes respectively. Scheduler records and valid power scope are needed for cost or energy rankings.