Energy: calibrated whole-node factor 1.26 (range 1.23-1.29), replacing the provisional 1.17

README.md

@@ -281,46 +281,47 @@ Every GPU job was wrapped in [perun](https://github.com/Helmholtz-AI-Energy/peru
 of 1.3; CO₂e assumes 30 g/kWh (Swedish SE3 grid). Neither factor is confirmed by NAISS yet; seconds perun did not
 cover count at idle power, so the totals are a floor.
 
-| Run | Jobs | GH200-h | kWh | kg CO₂e | Whole-node est. (×1.17) |
+| Run | Jobs | GH200-h | kWh | kg CO₂e | Whole-node est. (×1.26) |
 |---|---|---|---|---|---|
-| Latent cache (SD-VAE encode of the dataset) | 11 | 4.2 | 2.6 | 0.08 | 3.1 kWh / 0.09 kg |
-| FCDM v1, epochs 1-10 | 2 | 13.7 | 8.7 | 0.26 | 10.2 kWh / 0.31 kg |
-| FCDM v1, epochs 11-20 + Flan-T5 fine-tune | 1 | 20.6 | 13.5 | 0.41 | 15.8 kWh / 0.48 kg |
-| DiT reproduction, epochs 1-10 | 1 | 13.1 | 8.8 | 0.26 | 10.3 kWh / 0.31 kg |
-| fcdm2, all phases incl. restarts | 14 | 68.0 | 33.8 | 1.01 | 39.6 kWh / 1.19 kg |
-| Run 1 thinker, epochs 1-10 | 1 | 19.0 | 12.6 | 0.38 | 14.7 kWh / 0.44 kg |
-| T40r first try (diverged at step ~1,800) | 1 | 7.8 | 3.0 | 0.09 | 3.5 kWh / 0.10 kg |
-| **T40r phase 1 (epochs 1-10)** | 1 | 47.3 | 27.7 | 0.83 | 32.4 kWh / 0.97 kg |
-| **T40r phase 2 (joint Ettin training, to epoch 16)** | 1 | 37.1 | 18.8 | 0.56 | 22.0 kWh / 0.66 kg |
-| **T40r caption phase (2 nodes, 8 h, cosine anneal)** | 1 | 60.3 | 35.2 | 1.06 | 41.2 kWh / 1.24 kg |
-| T40r caption phase, two failed starts | 2 | 3.5 | 0.5 | 0.02 | 0.6 kWh / 0.02 kg |
-| **cap2 continuation (003 branches at step 121,440)** | 1 | 61.0 | 35.9 | 1.08 | 41.9 kWh / 1.26 kg |
-| **512 px: mr512_long1 (crashed; step 12,000 used)** | 1 | 19.6 | 12.5 | 0.37 | 14.6 kWh / 0.44 kg |
-| **512 px: mr512_long2 (everything trainable)** | 1 | 21.5 | 12.0 | 0.36 | 14.0 kWh / 0.42 kg |
-| **512 px: mr512_perc1 (perceptual + saliency)** | 1 | 72.4 | 43.8 | 1.31 | 51.3 kWh / 1.54 kg |
-| 001 release benchmark and GenEval | 3 | 5.1 | 2.7 | 0.08 | 3.1 kWh / 0.09 kg |
-| A/B wave 1 (six 256 px arms) + noise-scale study | 9 | 17.5 | 8.7 | 0.26 | 10.2 kWh / 0.31 kg |
-| A/B wave 2 (does 512 px work?) | 2 | 12.1 | 6.7 | 0.20 | 7.9 kWh / 0.24 kg |
-| A/B wave 2, failed attempts | 4 | 3.0 | 1.4 | 0.04 | 1.7 kWh / 0.05 kg |
-| 512 px latents + saliency maps (data prep) | 21 | 22.5 | 14.7 | 0.44 | 17.1 kWh / 0.51 kg |
-| Perceptual-loss profiling | 2 | 1.3 | 0.5 | 0.02 | 0.6 kWh / 0.02 kg |
+| Latent cache (SD-VAE encode of the dataset) | 11 | 4.2 | 2.6 | 0.08 | 3.3 kWh / 0.10 kg |
+| FCDM v1, epochs 1-10 | 2 | 13.7 | 8.7 | 0.26 | 11.0 kWh / 0.33 kg |
+| FCDM v1, epochs 11-20 + Flan-T5 fine-tune | 1 | 20.6 | 13.5 | 0.41 | 17.1 kWh / 0.51 kg |
+| DiT reproduction, epochs 1-10 | 1 | 13.1 | 8.8 | 0.26 | 11.1 kWh / 0.33 kg |
+| fcdm2, all phases incl. restarts | 14 | 68.0 | 33.8 | 1.01 | 42.6 kWh / 1.28 kg |
+| Run 1 thinker, epochs 1-10 | 1 | 19.0 | 12.6 | 0.38 | 15.9 kWh / 0.48 kg |
+| T40r first try (diverged at step ~1,800) | 1 | 7.8 | 3.0 | 0.09 | 3.7 kWh / 0.11 kg |
+| **T40r phase 1 (epochs 1-10)** | 1 | 47.3 | 27.7 | 0.83 | 34.9 kWh / 1.05 kg |
+| **T40r phase 2 (joint Ettin training, to epoch 16)** | 1 | 37.1 | 18.8 | 0.56 | 23.7 kWh / 0.71 kg |
+| **T40r caption phase (2 nodes, 8 h, cosine anneal)** | 1 | 60.3 | 35.2 | 1.06 | 44.4 kWh / 1.33 kg |
+| T40r caption phase, two failed starts | 2 | 3.5 | 0.5 | 0.02 | 0.7 kWh / 0.02 kg |
+| **cap2 continuation (003 branches at step 121,440)** | 1 | 61.0 | 35.9 | 1.08 | 45.2 kWh / 1.36 kg |
+| **512 px: mr512_long1 (crashed; step 12,000 used)** | 1 | 19.6 | 12.5 | 0.37 | 15.7 kWh / 0.47 kg |
+| **512 px: mr512_long2 (everything trainable)** | 1 | 21.5 | 12.0 | 0.36 | 15.1 kWh / 0.45 kg |
+| **512 px: mr512_perc1 (perceptual + saliency)** | 1 | 72.4 | 43.8 | 1.31 | 55.2 kWh / 1.66 kg |
+| 001 release benchmark and GenEval | 3 | 5.1 | 2.7 | 0.08 | 3.4 kWh / 0.10 kg |
+| A/B wave 1 (six 256 px arms) + noise-scale study | 9 | 17.5 | 8.7 | 0.26 | 11.0 kWh / 0.33 kg |
+| A/B wave 2 (does 512 px work?) | 2 | 12.1 | 6.7 | 0.20 | 8.5 kWh / 0.25 kg |
+| A/B wave 2, failed attempts | 4 | 3.0 | 1.4 | 0.04 | 1.8 kWh / 0.05 kg |
+| 512 px latents + saliency maps (data prep) | 21 | 22.5 | 14.7 | 0.44 | 18.5 kWh / 0.55 kg |
+| Perceptual-loss profiling | 2 | 1.3 | 0.5 | 0.02 | 0.7 kWh / 0.02 kg |
 | Perceptual-loss profiling, OOM | 2 | 0.6 | 0.1 | 0.00 | 0.1 kWh / 0.00 kg |
-| Research after 003: logo LoRA, router, MoE pilots | 34 | 68.5 | 32.7 | 0.98 | 38.2 kWh / 1.15 kg |
-| Evaluations, smoke tests, benchmarks | 47 | 30.9 | 11.7 | 0.35 | 13.6 kWh / 0.41 kg |
-| **Whole Agate project to 2026-09-28** | 164 | **630.4** | **348.6** | **10.46** | **407.9 kWh / 12.24 kg** |
+| Research after 003: logo LoRA, router, MoE pilots | 34 | 68.5 | 32.7 | 0.98 | 41.2 kWh / 1.24 kg |
+| Evaluations, smoke tests, benchmarks | 47 | 30.9 | 11.7 | 0.35 | 14.7 kWh / 0.44 kg |
+| **Whole Agate project to 2026-09-28** | 164 | **630.4** | **348.6** | **10.46** | **439.2 kWh / 13.18 kg** |
 
 Preview 003's own lineage, counted from scratch with jobs pro-rated at the steps where it branched off, cost
-**282.8 GH200-hours, 164.5 kWh and 4.94 kg CO₂e** (whole-node est. 192.5 kWh / 5.78 kg), plus 22.5
+**282.8 GH200-hours, 164.5 kWh and 4.94 kg CO₂e** (whole-node est. 207.3 kWh / 6.22 kg), plus 22.5
 GH200-hours to encode the 512 px latents and saliency maps. The whole project to date cost 630.4 GPU-hours.
 
-**About these energy figures.** perun measured GPU power only, through NVML: on the GH200 superchip it does not
+**About these energy figures.** perun measured GPU power only, through NVML: on the GH200 superchip it did not
 see the Grace CPU and its memory (Grace has no RAPL counters) or the node's Slingshot network cards, and Arrhenius has
-no Slurm energy accounting to check against. A probe on one GH200 (2026-09-30) read 94 W for the GPU against 172 W for
-the whole module at idle, 578 W against 621 W under full bf16 load, and about 17 W of network cards per GPU. As a
-provisional estimate of whole-node energy, multiply the kWh and CO₂e above by about 1.17 (roughly 1.1 for GPU-heavy
-training, more for idle-heavy jobs such as evaluation); the PUE of 1.3 and 30 g/kWh are already in these figures and
-should not be applied again. About 93.5 % of the project's GPU-hours were measured directly by perun and the rest
-estimated; from now on our jobs record whole-superchip power directly.
+no Slurm energy accounting to check against. We calibrated the gap on 2026-09-30 (one node, 4 GH200, 12 load phases,
+whole-module power cross-checked against NVML's module energy counter to 0.03 %): the Grace CPU and its memory add a
+near-constant 83-100 W per superchip on top of the GPU reading, and the network cards about 17 W per GPU. At our average
+GPU power that makes whole-node energy about 1.26x the figures above (range 1.23-1.29; about 1.2 at full GPU load,
+more for idle-heavy jobs such as evaluation), shown in the whole-node column. The PUE of 1.3 and 30 g/kWh are already
+in these figures and should not be applied again. About 93.5 % of the project's GPU-hours were measured directly by
+perun and the rest estimated; our jobs now record whole-superchip power directly.
 
 **Training compute.** Estimated at **~8e19 FLOP** (7.1e19–8.5e19): measured forward FLOPs per image (97.5 GFLOP per 256 px and 255.9 GFLOP per 512 px image for the generator, 10.8 GFLOP for the text encoder at 128 tokens; `torch.utils.flop_counter`) × the 196.6M images seen in the whole lineage × 3 for the backward pass. Counting the pretraining of the Ettin-68M text encoder we started from (~8e20 FLOP, per its authors) gives ~9e20 FLOP. Both are far below the 1e23 FLOP indicative threshold for general-purpose AI models in the European Commission's guidelines. The compute that third parties used to generate the synthetic training images (FLUX-Reason-6M) is not included.