Energy: calibrated whole-node factor 1.26 (range 1.23-1.29), replacing the provisional 1.17
Browse files
README.md
CHANGED
|
@@ -259,44 +259,45 @@ Every GPU job was wrapped in [perun](https://github.com/Helmholtz-AI-Energy/peru
|
|
| 259 |
of 1.3; CO₂e assumes 30 g/kWh (Swedish SE3 grid). Neither factor is confirmed by NAISS yet; seconds perun did not
|
| 260 |
cover count at idle power, so the totals are a floor.
|
| 261 |
|
| 262 |
-
| Run | Jobs | GH200-h | kWh | kg CO₂e | Whole-node est. (×1.
|
| 263 |
|---|---|---|---|---|---|
|
| 264 |
-
| Latent cache (SD-VAE encode of the dataset) | 11 | 4.2 | 2.6 | 0.08 | 3.
|
| 265 |
-
| FCDM v1, epochs 1-10 | 2 | 13.7 | 8.7 | 0.26 |
|
| 266 |
-
| FCDM v1, epochs 11-20 + Flan-T5 fine-tune | 1 | 20.6 | 13.5 | 0.41 |
|
| 267 |
-
| DiT reproduction, epochs 1-10 | 1 | 13.1 | 8.8 | 0.26 |
|
| 268 |
-
| fcdm2, all phases incl. restarts | 14 | 68.0 | 33.8 | 1.01 |
|
| 269 |
-
| Run 1 thinker, epochs 1-10 | 1 | 19.0 | 12.6 | 0.38 |
|
| 270 |
-
| T40r first try (diverged at step ~1,800) | 1 | 7.8 | 3.0 | 0.09 | 3.
|
| 271 |
-
| **T40r phase 1 (epochs 1-10)** | 1 | 47.3 | 27.7 | 0.83 |
|
| 272 |
-
| **T40r phase 2 (joint Ettin training, to epoch 16)** | 1 | 37.1 | 18.8 | 0.56 |
|
| 273 |
-
| **T40r caption phase (2 nodes, 8 h, cosine anneal)** | 1 | 60.3 | 35.2 | 1.06 |
|
| 274 |
-
| T40r caption phase, two failed starts | 2 | 3.5 | 0.5 | 0.02 | 0.
|
| 275 |
-
| **Agate 002: cap2 continuation, 82.8k to 138k** | 1 | 61.0 | 35.9 | 1.08 |
|
| 276 |
-
| 001 release benchmark and GenEval | 3 | 5.1 | 2.7 | 0.08 | 3.
|
| 277 |
-
| A/B wave 1 (six 256 px arms) + noise-scale study | 9 | 17.5 | 8.7 | 0.26 |
|
| 278 |
-
| A/B wave 2 (does 512 px work?) | 2 | 12.1 | 6.7 | 0.20 |
|
| 279 |
-
| A/B wave 2, failed attempts | 4 | 3.0 | 1.4 | 0.04 | 1.
|
| 280 |
-
| 512 px latents + saliency maps (data prep) | 21 | 22.5 | 14.7 | 0.44 |
|
| 281 |
-
| Perceptual-loss profiling | 2 | 1.3 | 0.5 | 0.02 | 0.
|
| 282 |
| Perceptual-loss profiling, OOM | 2 | 0.6 | 0.1 | 0.00 | 0.1 kWh / 0.00 kg |
|
| 283 |
-
| Research after 003: logo LoRA, router, MoE pilots | 34 | 68.5 | 32.7 | 0.98 |
|
| 284 |
-
| 512 px runs that became 003 (long1, long2, perc1) | 3 | 113.5 | 68.2 | 2.05 |
|
| 285 |
-
| Evaluations, smoke tests, benchmarks | 47 | 30.9 | 11.7 | 0.35 |
|
| 286 |
-
| **Whole Agate project to 2026-09-28** | 164 | **630.4** | **348.6** | **10.46** | **
|
| 287 |
|
| 288 |
Preview 002's own lineage, counted from scratch with the caption-phase job pro-rated to the 82,800 steps 002
|
| 289 |
-
inherits, cost **187.7 GH200-hours, 107.1 kWh and 3.21 kg CO₂e** (whole-node est.
|
| 290 |
date, including Preview 003 and every experiment, cost 630.4 GPU-hours.
|
| 291 |
|
| 292 |
-
**About these energy figures.** perun measured GPU power only, through NVML: on the GH200 superchip it
|
| 293 |
see the Grace CPU and its memory (Grace has no RAPL counters) or the node's Slingshot network cards, and Arrhenius has
|
| 294 |
-
no Slurm energy accounting to check against.
|
| 295 |
-
|
| 296 |
-
|
| 297 |
-
|
| 298 |
-
|
| 299 |
-
|
|
|
|
| 300 |
|
| 301 |
**Training compute.** Estimated at **~6e19 FLOP** (5.8e19–7.2e19): measured forward FLOPs per image (94.4 GFLOP per 256 px image for the generator, 10.8 GFLOP for the text encoder at 128 tokens; `torch.utils.flop_counter`) × the 186.6M images seen in the whole lineage × 3 for the backward pass. Counting the pretraining of the Ettin-68M text encoder we started from (~8e20 FLOP, per its authors) gives ~9e20 FLOP. Both are far below the 1e23 FLOP indicative threshold for general-purpose AI models in the European Commission's guidelines. The compute that third parties used to generate the synthetic training images (FLUX-Reason-6M) is not included.
|
| 302 |
|
|
|
|
| 259 |
of 1.3; CO₂e assumes 30 g/kWh (Swedish SE3 grid). Neither factor is confirmed by NAISS yet; seconds perun did not
|
| 260 |
cover count at idle power, so the totals are a floor.
|
| 261 |
|
| 262 |
+
| Run | Jobs | GH200-h | kWh | kg CO₂e | Whole-node est. (×1.26) |
|
| 263 |
|---|---|---|---|---|---|
|
| 264 |
+
| Latent cache (SD-VAE encode of the dataset) | 11 | 4.2 | 2.6 | 0.08 | 3.3 kWh / 0.10 kg |
|
| 265 |
+
| FCDM v1, epochs 1-10 | 2 | 13.7 | 8.7 | 0.26 | 11.0 kWh / 0.33 kg |
|
| 266 |
+
| FCDM v1, epochs 11-20 + Flan-T5 fine-tune | 1 | 20.6 | 13.5 | 0.41 | 17.1 kWh / 0.51 kg |
|
| 267 |
+
| DiT reproduction, epochs 1-10 | 1 | 13.1 | 8.8 | 0.26 | 11.1 kWh / 0.33 kg |
|
| 268 |
+
| fcdm2, all phases incl. restarts | 14 | 68.0 | 33.8 | 1.01 | 42.6 kWh / 1.28 kg |
|
| 269 |
+
| Run 1 thinker, epochs 1-10 | 1 | 19.0 | 12.6 | 0.38 | 15.9 kWh / 0.48 kg |
|
| 270 |
+
| T40r first try (diverged at step ~1,800) | 1 | 7.8 | 3.0 | 0.09 | 3.7 kWh / 0.11 kg |
|
| 271 |
+
| **T40r phase 1 (epochs 1-10)** | 1 | 47.3 | 27.7 | 0.83 | 34.9 kWh / 1.05 kg |
|
| 272 |
+
| **T40r phase 2 (joint Ettin training, to epoch 16)** | 1 | 37.1 | 18.8 | 0.56 | 23.7 kWh / 0.71 kg |
|
| 273 |
+
| **T40r caption phase (2 nodes, 8 h, cosine anneal)** | 1 | 60.3 | 35.2 | 1.06 | 44.4 kWh / 1.33 kg |
|
| 274 |
+
| T40r caption phase, two failed starts | 2 | 3.5 | 0.5 | 0.02 | 0.7 kWh / 0.02 kg |
|
| 275 |
+
| **Agate 002: cap2 continuation, 82.8k to 138k** | 1 | 61.0 | 35.9 | 1.08 | 45.2 kWh / 1.36 kg |
|
| 276 |
+
| 001 release benchmark and GenEval | 3 | 5.1 | 2.7 | 0.08 | 3.4 kWh / 0.10 kg |
|
| 277 |
+
| A/B wave 1 (six 256 px arms) + noise-scale study | 9 | 17.5 | 8.7 | 0.26 | 11.0 kWh / 0.33 kg |
|
| 278 |
+
| A/B wave 2 (does 512 px work?) | 2 | 12.1 | 6.7 | 0.20 | 8.5 kWh / 0.25 kg |
|
| 279 |
+
| A/B wave 2, failed attempts | 4 | 3.0 | 1.4 | 0.04 | 1.8 kWh / 0.05 kg |
|
| 280 |
+
| 512 px latents + saliency maps (data prep) | 21 | 22.5 | 14.7 | 0.44 | 18.5 kWh / 0.55 kg |
|
| 281 |
+
| Perceptual-loss profiling | 2 | 1.3 | 0.5 | 0.02 | 0.7 kWh / 0.02 kg |
|
| 282 |
| Perceptual-loss profiling, OOM | 2 | 0.6 | 0.1 | 0.00 | 0.1 kWh / 0.00 kg |
|
| 283 |
+
| Research after 003: logo LoRA, router, MoE pilots | 34 | 68.5 | 32.7 | 0.98 | 41.2 kWh / 1.24 kg |
|
| 284 |
+
| 512 px runs that became 003 (long1, long2, perc1) | 3 | 113.5 | 68.2 | 2.05 | 86.0 kWh / 2.58 kg |
|
| 285 |
+
| Evaluations, smoke tests, benchmarks | 47 | 30.9 | 11.7 | 0.35 | 14.7 kWh / 0.44 kg |
|
| 286 |
+
| **Whole Agate project to 2026-09-28** | 164 | **630.4** | **348.6** | **10.46** | **439.2 kWh / 13.18 kg** |
|
| 287 |
|
| 288 |
Preview 002's own lineage, counted from scratch with the caption-phase job pro-rated to the 82,800 steps 002
|
| 289 |
+
inherits, cost **187.7 GH200-hours, 107.1 kWh and 3.21 kg CO₂e** (whole-node est. 134.9 kWh / 4.05 kg). The whole project to
|
| 290 |
date, including Preview 003 and every experiment, cost 630.4 GPU-hours.
|
| 291 |
|
| 292 |
+
**About these energy figures.** perun measured GPU power only, through NVML: on the GH200 superchip it did not
|
| 293 |
see the Grace CPU and its memory (Grace has no RAPL counters) or the node's Slingshot network cards, and Arrhenius has
|
| 294 |
+
no Slurm energy accounting to check against. We calibrated the gap on 2026-09-30 (one node, 4 GH200, 12 load phases,
|
| 295 |
+
whole-module power cross-checked against NVML's module energy counter to 0.03 %): the Grace CPU and its memory add a
|
| 296 |
+
near-constant 83-100 W per superchip on top of the GPU reading, and the network cards about 17 W per GPU. At our average
|
| 297 |
+
GPU power that makes whole-node energy about 1.26x the figures above (range 1.23-1.29; about 1.2 at full GPU load,
|
| 298 |
+
more for idle-heavy jobs such as evaluation), shown in the whole-node column. The PUE of 1.3 and 30 g/kWh are already
|
| 299 |
+
in these figures and should not be applied again. About 93.5 % of the project's GPU-hours were measured directly by
|
| 300 |
+
perun and the rest estimated; our jobs now record whole-superchip power directly.
|
| 301 |
|
| 302 |
**Training compute.** Estimated at **~6e19 FLOP** (5.8e19–7.2e19): measured forward FLOPs per image (94.4 GFLOP per 256 px image for the generator, 10.8 GFLOP for the text encoder at 128 tokens; `torch.utils.flop_counter`) × the 186.6M images seen in the whole lineage × 3 for the backward pass. Counting the pretraining of the Ettin-68M text encoder we started from (~8e20 FLOP, per its authors) gives ~9e20 FLOP. Both are far below the 1e23 FLOP indicative threshold for general-purpose AI models in the European Commission's guidelines. The compute that third parties used to generate the synthetic training images (FLUX-Reason-6M) is not included.
|
| 303 |
|