CLD training on three GPU platforms

GH200, H100 PCIe and one MI250X GCD

Attention and HEPTv2 batch-size sweeps

Runs from 29–30 October 2026
120 steps per case, batches 8–128

Best observed batches are 16 or 32

Measured training throughput after 20 warmup steps
PlatformModelBatchEvents/skHits/s
GH200Attention3214.55140.6
GH200HEPTv23213.83133.7
H100 PCIeAttention1612.81123.6
H100 PCIeHEPTv21611.21108.1
MI250X (1 GCD)Attention167.3570.9
MI250X (1 GCD)HEPTv2166.1859.6

GH200 leads these platform runs. All six configurations slow down at batch 64 relative to their observed peak.

Platform and software configuration

These are platform comparisons with different software and host environments
DeviceVisible GiBCPU arch.TFDSRuntimeWorkers
GH20095.0aarch643.2.2CUDA 12.84
H100 PCIe79.2x86_643.2.1CUDA 12.84
MI250X (1 GCD)64.0x86_643.2.2ROCm 7.22

All runs use PyTorch 2.11, BF16 and one visible device. NVIDIA uses native attention. LUMI uses external FlashAttention.

LUMI allocates one 64 GiB GCD, half of an MI250X module. H100 is the PCIe variant. The GH200 device name includes “480GB”, but PyTorch exposes about 95 GiB of device memory.

Measurement protocol and timing scope

Two full training recipes: six backbone layers, width 256 and 16 heads, with a shared four-layer, 256-slot set decoder. HEPTv2 enables activation checkpointing.

120 training steps: discard the first 20 for throughput. Seed 12345, LAMB, learning rate 0.001, cosine schedule and 20-step loss calibration.

Training rate: sum of events or hits divided by summed host step durations over steps 21–120. Includes data wait and existing synchronizations. Excludes validation and checkpoint writes.

End-of-case evaluation: step 120, 32 validation events and 8 test events. Event exposure grows with batch size, so final losses do not measure matched training quality.

Training throughput peaks before capacity

events_per_second

Batch 128 fails with OOM for both recipes on all three platforms. Missing throughput at 128 represents failure, not zero performance.

Hit throughput confirms the same ordering

hits_per_second

All 120 per-step sequence-length vectors match across platforms for every completed model/batch pair. This checks event-size exposure, not dataset payload identity.

Attention: unweighted training loss by platform

attention_unweighted_training_loss

Raw losses for all 120 steps, without smoothing. Dotted vertical line: end of 20-step calibration. Compare platforms within each batch. TFDS versions and attention backends differ.

HEPTv2: unweighted training loss by platform

heptv2_unweighted_training_loss

Raw losses for all 120 steps, without smoothing. Dotted vertical line: end of 20-step calibration. Compare platforms within each batch. TFDS versions and attention backends differ.

Step-120 validation losses remain close across platforms

validation_unweighted_loss

The largest within-batch platform spread is 0.169 for Attention at batch 32. Only 32 validation events and one run per case support these snapshots. No convergence or significance claim.

GH200’s relative lead grows with batch size

matched_batch_ratios

At matched batches, GH200 is 1.06–1.35× H100 and 1.79–2.49× one MI250X GCD. Attention is faster in all 12 within-platform comparisons.

Memory demand is nearly identical across platforms

allocated_memory

At batch 64, Attention allocates about 43.1 GiB and HEPTv2 about 54.3 GiB. HEPTv2 uses about 26% more allocated memory despite activation checkpointing.

Batch 128 exceeds the observed memory envelope

PlatformModelResultEvents before failureAllocated GiBReserved GiB
GH200AttentionOOM25680.794.0
GH200HEPTv2OOM087.391.3
H100 PCIeAttentionOOM077.178.6
H100 PCIeHEPTv2OOM073.078.3
MI250X (1 GCD)AttentionOOM061.663.4
MI250X (1 GCD)HEPTv2OOM057.562.8

H100 HEPTv2 reports a fused bucket-attention RuntimeError. Its chained exception and sweep log establish OOM as the cause.

Batch 64 is the largest successful tested batch. The exact limit lies somewhere between the tested points. Failed-run peaks do not include the allocation that failed.

Small batches show larger relative latency tails

latency_tails

The p99/p50 ratio approaches 2× for some batch-8 runs. Each percentile comes from just 100 measured steps in one run. These are descriptive tails, not confidence intervals.

Launch overhead matters for short benchmarks

timing_scopes

Steady rate uses the last 100 steps. Whole-worker and whole-case rates use all 120. The case includes launch/import time, warmup, validation and checkpointing.

Higher device busy does not mean higher throughput

Batch 16ModelGPU busyBusy coverageMean sampled W
GH200Attention53.7%99.2%256
GH200HEPTv254.0%99.2%289
H100 PCIeAttention56.6%99.4%189
H100 PCIeHEPTv265.3%99.5%206
MI250X (1 GCD)Attention86.8%99.3%Unavailable
MI250X (1 GCD)HEPTv288.5%99.8%Unavailable

MI250X shows higher device busy while delivering fewer events/s. Busy is a vendor device metric, not SM occupancy or FLOP efficiency.

LUMI has no usable power measurements. GH200 Attention batch 32 has no usable phase-attributed busy or power interval. Energy-efficiency ranking is unsupported.

Event-size and loader diagnostics narrow the hypotheses

BatchMean real hits/eventPadded / real hitsEvents used for steady rate
89,7221.274×800
169,6471.328×1600
329,6651.365×3200
649,6131.396×6400

Across completed cases, measured data wait accounts for 0.04–0.52% of steady step time. This does not identify a loader bottleneck in the timed window.

Padding grows with batch size. It may contribute to scaling loss, but these records cannot isolate kernel, memory or CPU submission costs. A profiler run is needed to assign cause.

Run provenance limits hardware-only conclusions

PlatformRecorded Git baseDirty treeJob ID
GH200d2d991ceYes153260
H100 PCIe9e3d96bfYes7363958
MI250X (1 GCD)20ac400aYes23308832

H100 uses TFDS 3.2.1 rather than 3.2.2. Matching sequence lengths strengthens size comparability but does not prove matching features or targets.

Different Git bases and dirty trees prevent claiming an identical source snapshot. Metadata retains tracked-diff and untracked-file hashes, not complete historical patches.

One seed and one timed run per case provide no estimate of run-to-run variation. Model quality and full-module MI250X performance remain outside this study.

Branch changes support a portable benchmark workflow

Shared batch-sweep driver: isolated processes, frozen cases, explicit batch sizes, timeout/OOM classification and CSV/JSON reports.

Instrumentation: raw event and hit counts, step timings, nested phase accounting, device telemetry and framework memory peaks. Validation history now records cycle timings.

Platform support: GH200 environment, H100 and single-GCD LUMI profiles. The set decoder selects native NVIDIA or external ROCm variable-length attention.

LUMI retry changes: two loader workers, file-descriptor sharing and shared-memory/cgroup diagnostics address the earlier loader failure mode.

Observed choices and the next controlled comparison

Candidate operating points: batch 32 on GH200 and batch 16 on H100 / one MI250X GCD for both recipes. These maximize throughput in this sweep, not proven production optima.

Matched rerun: use one clean commit and an identical dataset copy. Repeat batches 8, 16 and 32 in varied order, with enough duration to measure run-to-run spread.

Targeted profiling: compare Attention and HEPTv2 at batch 16. Inspect kernel and host timelines before changing loader settings or attributing the gap to hardware.

Allocation evidence: observed ten-case spans are 42.0, 52.8 and 85.9 minutes respectively. Scheduler records and valid power scope are needed for cost or energy rankings.