Spaces:
Running on Zero
Running on Zero
| (.venv) macbook@austen:~/Desktop/miradi/gnn/PoC_v1 python -m scripts.train_spatiotemporal_gnn --data-root datasets | |
| eure: 2883 nodes, 3358 edges | |
| risle: 1642 nodes, 1873 edges | |
| Combined graph: 4525 nodes, 5231 edges | |
| Real discharge coverage: 8923 / 23136325 | |
| Standardized 'discharge': train-period mean=2809.26, std=3286.51 (fit on real training-period values only) | |
| Standardized 'groundwater_level': train-period mean=102.62, std=39.37 (fit on real training-period values only) | |
| Physics indices: 28 confluences, 696 braid pairs, 5231 routing edges | |
| Training windows: X=(4903, 30, 4525, 4), Y=(4903, 4525, 10) | |
| Train examples: 3257, val: 730, test: 916 | |
| Using device: cpu | |
| Physics loss weights: {'confluence': 1.0, 'split_rejoin': 1.0, 'routing': 2.0} | |
| epoch 1/100: train loss = 0.8412, val supervised loss = 1.1929 (best) | |
| epoch 2/100: train loss = 0.7969, val supervised loss = 1.1486 (best) | |
| epoch 3/100: train loss = 0.7908, val supervised loss = 1.1795 | |
| epoch 4/100: train loss = 0.7955, val supervised loss = 1.1162 (best) | |
| epoch 5/100: train loss = 0.7808, val supervised loss = 1.1066 (best) | |
| epoch 6/100: train loss = 0.7651, val supervised loss = 1.1549 | |
| epoch 7/100: train loss = 0.7411, val supervised loss = 1.0764 (best) | |
| epoch 8/100: train loss = 0.7027, val supervised loss = 0.9924 (best) | |
| epoch 9/100: train loss = 0.6924, val supervised loss = 1.3795 | |
| epoch 10/100: train loss = 0.6889, val supervised loss = 1.0943 | |
| epoch 11/100: train loss = 0.6836, val supervised loss = 1.0572 | |
| epoch 12/100: train loss = 0.6827, val supervised loss = 0.9846 (best) | |
| epoch 13/100: train loss = 0.6763, val supervised loss = 0.9921 | |
| epoch 14/100: train loss = 0.6743, val supervised loss = 1.0505 | |
| epoch 15/100: train loss = 0.6750, val supervised loss = 1.0760 | |
| epoch 16/100: train loss = 0.6751, val supervised loss = 1.0425 | |
| epoch 17/100: train loss = 0.6696, val supervised loss = 0.9860 | |
| epoch 18/100: train loss = 0.6670, val supervised loss = 1.0494 | |
| epoch 19/100: train loss = 0.6643, val supervised loss = 1.0208 | |
| epoch 20/100: train loss = 0.6631, val supervised loss = 1.0441 | |
| epoch 21/100: train loss = 0.6619, val supervised loss = 1.0921 | |
| epoch 22/100: train loss = 0.6603, val supervised loss = 1.0351 | |
| stopped early at epoch 22 | |
| Best validation loss: 0.9846 | |
| Saved model to datasets/spatiotemporal_gnn/model.pt | |
| eure: 2883 real nodes, 3358 real edges (full reach graph) | |
| eure: reduced to 50 node(s) (real gauges + confluences/braids only, no virtual infill nodes), 67 collapsed edge(s), real connectivity preserved | |
| risle: 1642 real nodes, 1873 real edges (full reach graph) | |
| risle: reduced to 50 node(s) (real gauges + confluences/braids only, no virtual infill nodes), 86 collapsed edge(s), real connectivity preserved | |
| Combined graph: 100 nodes, 153 edges | |
| Real discharge coverage: 8923 / 511300 | |
| Standardized 'discharge': train-period mean=2809.26, std=3286.51 (fit on real training-period values only) | |
| Standardized 'groundwater_level': train-period mean=106.01, std=39.00 (fit on real training-period values only) | |
| Physics indices: 11 confluences, 1 braid pairs, 153 routing edges | |
| Training windows: X=(4903, 30, 100, 4), Y=(4903, 100, 10) | |
| Train examples: 3257, val: 730, test: 916 | |
| Using device: cpu | |
| Physics loss weights: {'confluence': 1.0, 'split_rejoin': 1.0, 'routing': 2.0} | |
| epoch 1/100: train loss = 1.1519, val supervised loss = 1.4610 (best) | |
| epoch 2/100: train loss = 0.7972, val supervised loss = 1.3024 (best) | |
| epoch 3/100: train loss = 0.7899, val supervised loss = 1.2201 (best) | |
| epoch 4/100: train loss = 0.7846, val supervised loss = 1.2223 | |
| epoch 5/100: train loss = 0.7906, val supervised loss = 1.2654 | |
| epoch 6/100: train loss = 0.8115, val supervised loss = 1.1744 (best) | |
| epoch 7/100: train loss = 0.7939, val supervised loss = 1.2215 | |
| epoch 8/100: train loss = 0.7976, val supervised loss = 1.2869 | |
| epoch 9/100: train loss = 0.8045, val supervised loss = 1.2263 | |
| epoch 10/100: train loss = 0.7929, val supervised loss = 1.3447 | |
| epoch 11/100: train loss = 0.7906, val supervised loss = 1.2693 | |
| epoch 12/100: train loss = 0.7902, val supervised loss = 1.2461 | |
| epoch 13/100: train loss = 0.7915, val supervised loss = 1.1545 (best) | |
| epoch 14/100: train loss = 0.7886, val supervised loss = 1.2151 | |
| epoch 15/100: train loss = 0.8031, val supervised loss = 1.1819 | |
| epoch 16/100: train loss = 0.7914, val supervised loss = 1.1730 | |
| epoch 17/100: train loss = 0.7864, val supervised loss = 1.1729 | |
| epoch 18/100: train loss = 0.7863, val supervised loss = 1.1727 | |
| epoch 19/100: train loss = 0.7858, val supervised loss = 1.3309 | |
| epoch 20/100: train loss = 0.7856, val supervised loss = 1.2335 | |
| epoch 21/100: train loss = 0.7862, val supervised loss = 1.1961 | |
| epoch 22/100: train loss = 0.7830, val supervised loss = 1.1632 | |
| epoch 23/100: train loss = 0.7835, val supervised loss = 1.2377 | |
| stopped early at epoch 23 | |
| Best validation loss: 1.1545 | |
| Saved model to datasets/spatiotemporal_gnn/model.pt | |
| Naive baseline (predict the training-period mean, i.e. 0 in standardized units): 2.5756 | |
| Model beats the naive baseline by 55.2% -- real, not just baseline-matching skill | |
| (.venv) macbook@austen:~/Desktop/miradi/gnn/PoC_v1 python -m scripts.train_spatiotemporal_gnn --data-root datasets --train-end 2025-06-30 --val-end 2025-12-31 | |
| eure: joined real channel width for 2805/3358 edge(s) from datasets/edge_widths.csv | |
| eure: 2883 real nodes, 3358 real edges (full reach graph) | |
| eure: reduced to 50 node(s) (real gauges + confluences/braids only, no virtual infill nodes), 67 collapsed edge(s), real connectivity preserved | |
| risle: joined real channel width for 1488/1873 edge(s) from datasets/edge_widths.csv | |
| risle: 1642 real nodes, 1873 real edges (full reach graph) | |
| risle: reduced to 50 node(s) (real gauges + confluences/braids only, no virtual infill nodes), 86 collapsed edge(s), real connectivity preserved | |
| Combined graph: 100 nodes, 153 edges | |
| Real discharge coverage: 8923 / 511300 | |
| Real water-level coverage: 21947 / 511300 | |
| Real catchment area available for 16/100 node(s) | |
| Fitted specific-discharge coefficient k=0.0766876 on 2603 real (Q, precip, area) triple(s) from gauged stations, training period only | |
| Real precipitation-based discharge estimate available for 16496 node-date value(s) (input feature, never a target) | |
| Standardized 'discharge': train-period mean=4168.71, std=5311.31 (fit on real training-period values only) | |
| Standardized 'groundwater_level': train-period mean=105.86, std=39.03 (fit on real training-period values only) | |
| Standardized 'precipitation': train-period mean=1.78, std=4.00 (fit on real training-period values only) | |
| Standardized 'estimated_discharge': train-period mean=3659.35, std=7011.12 (fit on real training-period values only) | |
| Standardized 'waterlevel': train-period mean=402.00, std=590.37 (fit on real training-period values only) | |
| Physics indices: 11 confluences, 1 braid pairs, 153 routing edges | |
| Training windows: X=(4903, 30, 100, 8), Y=(4903, 100, 10) | |
| Water-level targets at the same 4903 anchor dates: Y_H=(4903, 100, 10), 215936 real value(s) | |
| Real forecast precipitation available for 61830 (example, node, horizon) value(s) -- real coverage is partial by design (only horizons [1, 2, 3, 5, 7], only from 2024-03-01 onward) | |
| Train examples: 4534, val: 184, test: 185 | |
| Using device: cpu | |
| Physics loss weights: {'confluence': 1.0, 'split_rejoin': 1.0, 'routing': 2.0, 'manning': 1.0, 'water_balance': 1.0} | |
| Real precipitation available -- water balance physics enabled (channels 4/5) | |
| [debug] example (epoch 1, remaining debug budget 24): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.06501, max=14.41 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.16, max=5.462 | |
| [debug] Y_i real value count: 10/1000 | |
| [debug] Y_H_i real value count: 30/1000 | |
| [debug] confluence term: 81.1335, finite=True | |
| [debug] split_rejoin term: 0.000716772, finite=True | |
| [debug] manning term: 24.3307, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0283629, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=4314, max=7186, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=10, max=25, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.449464, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.53244, finite=True | |
| [debug] supervised_H: 0.179025, finite=True | |
| [debug] physics total: 105.943, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 23): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.06501, max=14.41 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.16, max=5.462 | |
| [debug] Y_i real value count: 26/1000 | |
| [debug] Y_H_i real value count: 50/1000 | |
| [debug] confluence term: 81.1334, finite=True | |
| [debug] split_rejoin term: 0.000716772, finite=True | |
| [debug] manning term: 24.3307, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0283612, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=4314, max=7186, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=4, max=13, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.0549348, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.367937, finite=True | |
| [debug] supervised_H: 0.193409, finite=True | |
| [debug] physics total: 105.548, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 22): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.06501, max=14.41 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.16, max=5.462 | |
| [debug] Y_i real value count: 5/1000 | |
| [debug] Y_H_i real value count: 51/1000 | |
| [debug] confluence term: 81.1334, finite=True | |
| [debug] split_rejoin term: 0.000716772, finite=True | |
| [debug] manning term: 24.3307, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0283658, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=4314, max=7186, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=6.6, max=20.8, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.191883, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 1.01059, finite=True | |
| [debug] supervised_H: 0.253062, finite=True | |
| [debug] physics total: 105.685, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 21): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.06501, max=14.41 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.16, max=5.462 | |
| [debug] Y_i real value count: 10/1000 | |
| [debug] Y_H_i real value count: 37/1000 | |
| [debug] confluence term: 81.1335, finite=True | |
| [debug] split_rejoin term: 0.000716772, finite=True | |
| [debug] manning term: 24.3307, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.028364, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=4314, max=7186, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=8.6, max=19.7, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.204664, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.52279, finite=True | |
| [debug] supervised_H: 0.156358, finite=True | |
| [debug] physics total: 105.698, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 20): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.065, max=14.41 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.16, max=5.462 | |
| [debug] Y_i real value count: 20/1000 | |
| [debug] Y_H_i real value count: 50/1000 | |
| [debug] confluence term: 81.1334, finite=True | |
| [debug] split_rejoin term: 0.000716772, finite=True | |
| [debug] manning term: 24.3307, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0283685, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=4314, max=7186, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=-9.537e-07, max=6.2, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.360339, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.562039, finite=True | |
| [debug] supervised_H: 0.346614, finite=True | |
| [debug] physics total: 105.853, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 19): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.06501, max=14.41 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.16, max=5.462 | |
| [debug] Y_i real value count: 10/1000 | |
| [debug] Y_H_i real value count: 30/1000 | |
| [debug] confluence term: 81.1334, finite=True | |
| [debug] split_rejoin term: 0.000716772, finite=True | |
| [debug] manning term: 24.3307, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0283629, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=4314, max=7186, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=-9.537e-07, max=0.4, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.914814, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.589836, finite=True | |
| [debug] supervised_H: 0.211875, finite=True | |
| [debug] physics total: 106.408, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 18): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.065, max=14.41 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.16, max=5.462 | |
| [debug] Y_i real value count: 20/1000 | |
| [debug] Y_H_i real value count: 50/1000 | |
| [debug] confluence term: 81.1337, finite=True | |
| [debug] split_rejoin term: 0.000716772, finite=True | |
| [debug] manning term: 24.3307, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0283742, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=4314, max=7186, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=10.6, max=21, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.357288, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.548801, finite=True | |
| [debug] supervised_H: 0.431054, finite=True | |
| [debug] physics total: 105.851, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 17): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.06501, max=14.41 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.16, max=5.462 | |
| [debug] Y_i real value count: 10/1000 | |
| [debug] Y_H_i real value count: 22/1000 | |
| [debug] confluence term: 81.1335, finite=True | |
| [debug] split_rejoin term: 0.000716772, finite=True | |
| [debug] manning term: 24.3307, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0283632, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=4314, max=7186, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=9, max=16.5, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.206747, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.586394, finite=True | |
| [debug] supervised_H: 0.193314, finite=True | |
| [debug] physics total: 105.7, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 16): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.1888, max=12.57 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1493, max=4.844 | |
| [debug] Y_i real value count: 22/1000 | |
| [debug] Y_H_i real value count: 43/1000 | |
| [debug] confluence term: 58.8818, finite=True | |
| [debug] split_rejoin term: 0.000370772, finite=True | |
| [debug] manning term: 17.935, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.032642, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=3637, max=6725, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=0.6, max=2, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.362139, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.540033, finite=True | |
| [debug] supervised_H: 0.188887, finite=True | |
| [debug] physics total: 77.2119, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 15): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.189, max=12.57 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1493, max=4.844 | |
| [debug] Y_i real value count: 9/1000 | |
| [debug] Y_H_i real value count: 59/1000 | |
| [debug] confluence term: 58.882, finite=True | |
| [debug] split_rejoin term: 0.000370772, finite=True | |
| [debug] manning term: 17.935, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0326606, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=3637, max=6725, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=5.4, max=21.4, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.464889, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.478132, finite=True | |
| [debug] supervised_H: 0.300983, finite=True | |
| [debug] physics total: 77.3149, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 14): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.2166, max=12.57 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1493, max=4.844 | |
| [debug] Y_i real value count: 30/1000 | |
| [debug] Y_H_i real value count: 40/1000 | |
| [debug] confluence term: 58.8827, finite=True | |
| [debug] split_rejoin term: 0.000370772, finite=True | |
| [debug] manning term: 17.9346, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0341154, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=3549, max=6725, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=-9.537e-07, max=1, n_real=20 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.622464, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.525364, finite=True | |
| [debug] supervised_H: 0.151529, finite=True | |
| [debug] physics total: 77.4742, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 13): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.1888, max=12.57 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1493, max=4.844 | |
| [debug] Y_i real value count: 9/1000 | |
| [debug] Y_H_i real value count: 59/1000 | |
| [debug] confluence term: 58.8818, finite=True | |
| [debug] split_rejoin term: 0.000370772, finite=True | |
| [debug] manning term: 17.935, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0326413, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=3637, max=6725, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=2, max=16.7, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.58037, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.521639, finite=True | |
| [debug] supervised_H: 0.299622, finite=True | |
| [debug] physics total: 77.4302, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 12): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.1888, max=12.57 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1493, max=4.844 | |
| [debug] Y_i real value count: 25/1000 | |
| [debug] Y_H_i real value count: 49/1000 | |
| [debug] confluence term: 58.8818, finite=True | |
| [debug] split_rejoin term: 0.000370772, finite=True | |
| [debug] manning term: 17.935, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0326409, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=3637, max=6725, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=4.4, max=12.3, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.240601, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.300028, finite=True | |
| [debug] supervised_H: 0.206646, finite=True | |
| [debug] physics total: 77.0904, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 11): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.189, max=12.57 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1493, max=4.844 | |
| [debug] Y_i real value count: 14/1000 | |
| [debug] Y_H_i real value count: 47/1000 | |
| [debug] confluence term: 58.882, finite=True | |
| [debug] split_rejoin term: 0.000370772, finite=True | |
| [debug] manning term: 17.9349, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0326483, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=3637, max=6725, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=5.6, max=13.5, n_real=19 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.471311, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.766813, finite=True | |
| [debug] supervised_H: 0.201794, finite=True | |
| [debug] physics total: 77.3212, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 10): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.2775, max=12.57 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1493, max=4.844 | |
| [debug] Y_i real value count: 29/1000 | |
| [debug] Y_H_i real value count: 46/1000 | |
| [debug] confluence term: 58.8864, finite=True | |
| [debug] split_rejoin term: 0.000370772, finite=True | |
| [debug] manning term: 17.935, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0402923, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=3402, max=6725, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=0.2, max=6.3, n_real=20 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.522789, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.438725, finite=True | |
| [debug] supervised_H: 0.157988, finite=True | |
| [debug] physics total: 77.3848, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 9): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.1889, max=12.57 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1493, max=4.844 | |
| [debug] Y_i real value count: 20/1000 | |
| [debug] Y_H_i real value count: 50/1000 | |
| [debug] confluence term: 58.8818, finite=True | |
| [debug] split_rejoin term: 0.000370772, finite=True | |
| [debug] manning term: 17.935, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.032641, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=3637, max=6725, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=0.4, max=4.6, n_real=13 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.431347, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.478395, finite=True | |
| [debug] supervised_H: 0.352821, finite=True | |
| [debug] physics total: 77.2811, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 8): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.2806, max=10.76 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1475, max=4.197 | |
| [debug] Y_i real value count: 13/1000 | |
| [debug] Y_H_i real value count: 45/1000 | |
| [debug] confluence term: 40.6323, finite=True | |
| [debug] split_rejoin term: 0.000253812, finite=True | |
| [debug] manning term: 12.8064, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0413045, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=2999, max=5719, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=2.4, max=5.2, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.174067, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.376802, finite=True | |
| [debug] supervised_H: 0.258603, finite=True | |
| [debug] physics total: 53.6543, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 7): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.2806, max=10.76 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1475, max=4.197 | |
| [debug] Y_i real value count: 29/1000 | |
| [debug] Y_H_i real value count: 50/1000 | |
| [debug] confluence term: 40.6323, finite=True | |
| [debug] split_rejoin term: 0.000253812, finite=True | |
| [debug] manning term: 12.8064, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0413093, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=2999, max=5719, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=-9.537e-07, max=0.6, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 1, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.350273, finite=True | |
| [debug] supervised_H: 0.168079, finite=True | |
| [debug] physics total: 54.4802, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 6): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.2807, max=10.76 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1475, max=4.197 | |
| [debug] Y_i real value count: 10/1000 | |
| [debug] Y_H_i real value count: 30/1000 | |
| [debug] confluence term: 40.6323, finite=True | |
| [debug] split_rejoin term: 0.000253812, finite=True | |
| [debug] manning term: 12.8064, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0413074, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=2999, max=5719, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=19.7, max=92.5, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.810393, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.380917, finite=True | |
| [debug] supervised_H: 0.157874, finite=True | |
| [debug] physics total: 54.2906, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 5): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.2806, max=10.76 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1475, max=4.197 | |
| [debug] Y_i real value count: 29/1000 | |
| [debug] Y_H_i real value count: 41/1000 | |
| [debug] confluence term: 40.6323, finite=True | |
| [debug] split_rejoin term: 0.000253812, finite=True | |
| [debug] manning term: 12.8063, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0413193, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=2999, max=5719, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=-9.537e-07, max=0.6, n_real=20 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.601746, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.353307, finite=True | |
| [debug] supervised_H: 0.20659, finite=True | |
| [debug] physics total: 54.0818, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 4): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.2806, max=10.76 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1475, max=4.197 | |
| [debug] Y_i real value count: 10/1000 | |
| [debug] Y_H_i real value count: 59/1000 | |
| [debug] confluence term: 40.6322, finite=True | |
| [debug] split_rejoin term: 0.000253812, finite=True | |
| [debug] manning term: 12.8064, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0413075, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=2999, max=5719, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=0.6, max=24.7, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.546189, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.43953, finite=True | |
| [debug] supervised_H: 0.320977, finite=True | |
| [debug] physics total: 54.0264, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 3): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.2806, max=10.76 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1475, max=4.197 | |
| [debug] Y_i real value count: 27/1000 | |
| [debug] Y_H_i real value count: 49/1000 | |
| [debug] confluence term: 40.6323, finite=True | |
| [debug] split_rejoin term: 0.000253812, finite=True | |
| [debug] manning term: 12.8064, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0413097, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=2999, max=5719, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=3.6, max=26.3, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.131423, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.359537, finite=True | |
| [debug] supervised_H: 0.187993, finite=True | |
| [debug] physics total: 53.6116, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 2): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.2806, max=10.76 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1475, max=4.197 | |
| [debug] Y_i real value count: 20/1000 | |
| [debug] Y_H_i real value count: 50/1000 | |
| [debug] confluence term: 40.6323, finite=True | |
| [debug] split_rejoin term: 0.000253812, finite=True | |
| [debug] manning term: 12.8064, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0413066, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=2999, max=5719, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=0.6, max=2.4, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.471807, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.415129, finite=True | |
| [debug] supervised_H: 0.417917, finite=True | |
| [debug] physics total: 53.952, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 1): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.3152, max=10.76 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1475, max=4.197 | |
| [debug] Y_i real value count: 30/1000 | |
| [debug] Y_H_i real value count: 40/1000 | |
| [debug] confluence term: 40.6364, finite=True | |
| [debug] split_rejoin term: 0.000253812, finite=True | |
| [debug] manning term: 12.8064, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.044186, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=2847, max=5719, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=-9.537e-07, max=0.4, n_real=20 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.972517, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.357991, finite=True | |
| [debug] supervised_H: 0.150745, finite=True | |
| [debug] physics total: 54.4598, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] example (epoch 1, remaining debug budget 0): | |
| [debug] Q_pred_quantiles: finite_all=True, min=-0.3686, max=9.093 | |
| [debug] H_pred_quantiles: finite_all=True, min=-0.1423, max=4.156 | |
| [debug] Y_i real value count: 10/1000 | |
| [debug] Y_H_i real value count: 30/1000 | |
| [debug] confluence term: 26.4635, finite=True | |
| [debug] split_rejoin term: 0.000407118, finite=True | |
| [debug] manning term: 8.60421, finite=True | |
| [debug] GRADIENT of confluence term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of split_rejoin term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] GRADIENT of manning term: wrt Q_median finite=True, wrt H_median finite=True | |
| [debug] routing term: 0.0488792, finite=True | |
| [debug] GRADIENT of routing term: wrt Q_median finite=True | |
| [debug] Q_short_block_avg (real L/s): min=2472, max=5445, finite_all=True | |
| [debug] recent_precip_sum_mm (real mm): min=0.2, max=1.2, n_real=14 | |
| [debug] catchment_area_km2: n_real=16, min=149, max=5935 | |
| [debug] water_balance term (pre-weight): 0.633285, finite=True | |
| [debug] GRADIENT of water_balance term: wrt Q_median finite=True | |
| [debug] supervised_Q: 0.290417, finite=True | |
| [debug] supervised_H: 0.157283, finite=True | |
| [debug] physics total: 35.7503, finite=True | |
| [debug] GRADIENT of supervised_Q: wrt Q_pred_quantiles finite=True | |
| [debug] GRADIENT of supervised_H: wrt H_pred_quantiles finite=True | |
| [debug] raw accounting: n_batches=567, total_loss=794.492, len(train_loader)=567 | |
| epoch 1/100: train loss = 1.4012, val Q loss = 0.1503, val H loss = 0.0783, val combined = 0.2286 (best) | |
| [debug] raw accounting: n_batches=567, total_loss=420.515, len(train_loader)=567 | |
| epoch 2/100: train loss = 0.7416, val Q loss = 0.1521, val H loss = 0.0698, val combined = 0.2219 (best) | |
| [debug] raw accounting: n_batches=567, total_loss=418.261, len(train_loader)=567 | |
| epoch 3/100: train loss = 0.7377, val Q loss = 0.1614, val H loss = 0.0744, val combined = 0.2358 | |
| [debug] raw accounting: n_batches=567, total_loss=416.014, len(train_loader)=567 | |
| epoch 4/100: train loss = 0.7337, val Q loss = 0.1463, val H loss = 0.0757, val combined = 0.2221 | |
| [debug] raw accounting: n_batches=567, total_loss=418.748, len(train_loader)=567 | |
| epoch 5/100: train loss = 0.7385, val Q loss = 0.1590, val H loss = 0.0767, val combined = 0.2357 | |
| [debug] raw accounting: n_batches=567, total_loss=414.083, len(train_loader)=567 | |
| epoch 6/100: train loss = 0.7303, val Q loss = 0.1702, val H loss = 0.0769, val combined = 0.2471 | |
| [debug] raw accounting: n_batches=567, total_loss=413.883, len(train_loader)=567 | |
| epoch 7/100: train loss = 0.7300, val Q loss = 0.1551, val H loss = 0.0756, val combined = 0.2307 | |
| [debug] raw accounting: n_batches=567, total_loss=412.084, len(train_loader)=567 | |
| epoch 8/100: train loss = 0.7268, val Q loss = 0.1655, val H loss = 0.0748, val combined = 0.2404 | |
| [debug] raw accounting: n_batches=567, total_loss=411.92, len(train_loader)=567 | |
| epoch 9/100: train loss = 0.7265, val Q loss = 0.1620, val H loss = 0.0783, val combined = 0.2403 | |
| [debug] raw accounting: n_batches=567, total_loss=411.669, len(train_loader)=567 | |
| epoch 10/100: train loss = 0.7260, val Q loss = 0.1568, val H loss = 0.0724, val combined = 0.2293 | |
| [debug] raw accounting: n_batches=567, total_loss=412.194, len(train_loader)=567 | |
| epoch 11/100: train loss = 0.7270, val Q loss = 0.1463, val H loss = 0.0748, val combined = 0.2211 (best) | |
| [debug] raw accounting: n_batches=567, total_loss=405.851, len(train_loader)=567 | |
| epoch 12/100: train loss = 0.7158, val Q loss = 0.1439, val H loss = 0.0649, val combined = 0.2088 (best) | |
| [debug] raw accounting: n_batches=567, total_loss=415.095, len(train_loader)=567 | |
| epoch 13/100: train loss = 0.7321, val Q loss = 0.1703, val H loss = 0.0836, val combined = 0.2540 | |
| [debug] raw accounting: n_batches=567, total_loss=416.673, len(train_loader)=567 | |
| epoch 14/100: train loss = 0.7349, val Q loss = 0.1684, val H loss = 0.0829, val combined = 0.2513 | |
| [debug] raw accounting: n_batches=567, total_loss=417.424, len(train_loader)=567 | |
| epoch 15/100: train loss = 0.7362, val Q loss = 0.1841, val H loss = 0.0858, val combined = 0.2699 | |
| [debug] raw accounting: n_batches=567, total_loss=414.205, len(train_loader)=567 | |
| epoch 16/100: train loss = 0.7305, val Q loss = 0.1647, val H loss = 0.0795, val combined = 0.2441 | |
| [debug] raw accounting: n_batches=567, total_loss=411.842, len(train_loader)=567 | |
| epoch 17/100: train loss = 0.7264, val Q loss = 0.1664, val H loss = 0.0793, val combined = 0.2457 | |
| [debug] raw accounting: n_batches=567, total_loss=411.079, len(train_loader)=567 | |
| epoch 18/100: train loss = 0.7250, val Q loss = 0.1611, val H loss = 0.0809, val combined = 0.2420 | |
| [debug] raw accounting: n_batches=567, total_loss=411.693, len(train_loader)=567 | |
| epoch 19/100: train loss = 0.7261, val Q loss = 0.1516, val H loss = 0.0746, val combined = 0.2261 | |
| [debug] raw accounting: n_batches=567, total_loss=411.414, len(train_loader)=567 | |
| epoch 20/100: train loss = 0.7256, val Q loss = 0.1677, val H loss = 0.0796, val combined = 0.2473 | |
| [debug] raw accounting: n_batches=567, total_loss=411.121, len(train_loader)=567 | |
| epoch 21/100: train loss = 0.7251, val Q loss = 0.1586, val H loss = 0.0812, val combined = 0.2398 | |
| [debug] raw accounting: n_batches=567, total_loss=410.599, len(train_loader)=567 | |
| epoch 22/100: train loss = 0.7242, val Q loss = 0.1534, val H loss = 0.0750, val combined = 0.2283 | |
| stopped early at epoch 22 | |
| Best validation loss: 0.2088 | |
| Learned Manning's roughness n = 0.9932 (0.03 was the initialization; real natural streams typically fall in ~0.025-0.1 -- a value drifting far outside that range is worth investigating, not just noting) | |
| Learned bank height = 0.10m (2.5m was the initialization, an assumption with no real measured data behind it -- compare this against real rating-curve data for the gauges with one, if the model's learned value looks implausible for these specific rivers) | |
| Saved model to datasets/spatiotemporal_gnn/model.pt | |
| Naive baseline, discharge (real training-period empirical quantiles [0.5, 0.9, 0.95, 0.99]): 0.1728 | |
| Naive baseline, water level (real training-period empirical quantiles [0.5, 0.9, 0.95, 0.99]): 0.1276 | |
| Discharge: model beats the naive baseline by 16.7% -- real, not just baseline-matching skill | |
| Water level: model beats the naive baseline by 49.1% -- real, not just baseline-matching skill | |
| (.venv) macbook@austen:~/Desktop/miradi/gnn/PoC_v1 python -m scripts.evaluation.evaluate_flood_detection --data-root datasets --train-end 2025-06-30 --val-end 2025-12-31 | |
| Loaded 125 real event(s) across 6 station(s) | |
| eure: joined real channel width for 2805/3358 edge(s) from datasets/edge_widths.csv | |
| risle: joined real channel width for 1488/1873 edge(s) from datasets/edge_widths.csv | |
| Fitted specific-discharge coefficient k=0.0766876 on 2603 real (Q, precip, area) triple(s) from gauged stations, training period only | |
| Evaluating on 185 real held-out test example(s) | |
| Real forecast precipitation available for 13320 test (example, node, horizon) value(s) | |
| Loaded trained model from datasets/spatiotemporal_gnn/model.pt | |
| Flagging a real event when the model's predicted 0.95 quantile exceeds that station's real threshold -- not the median, since flood risk is a tail question: "is there a real 5% chance this exceeds the threshold", not "what's the single most likely value." | |
| ====================================================================== | |
| Flood-detection confusion matrix by lead time (real held-out test data) | |
| ====================================================================== | |
| horizon_days tp fp fn tn precision recall | |
| 1 15 351 23 721 0.040984 0.394737 | |
| 2 17 381 21 691 0.042714 0.447368 | |
| 3 24 472 14 600 0.048387 0.631579 | |
| 4 0 0 38 1072 NaN 0.000000 | |
| 5 21 457 17 615 0.043933 0.552632 | |
| 10 0 0 38 1072 NaN 0.000000 | |
| 30 0 0 38 1072 NaN 0.000000 | |
| 60 0 0 0 1110 NaN NaN | |
| 90 0 0 0 1110 NaN NaN | |
| 180 0 0 0 1110 NaN NaN | |
| Saved to datasets/flood_detection_evaluation.csv |