Konuktor92 commited on
Commit
cb29b3c
·
verified ·
1 Parent(s): aa1a18c

expected-greenness v1

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ model_a.ubj filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ library_name: xgboost
4
+ tags: [nasa, modis, ndvi, imerg, gldas, pastures, kyrgyzstan, tabular-regression]
5
+ datasets: [TrendeX90/naryn-cube]
6
+ ---
7
+ # TrendeX expected-greenness model (v1)
8
+
9
+ Predicts the **summer greenness (NDVI) anomaly** of every 1 km pasture cell in Naryn oblast, Kyrgyzstan
10
+ (May-September, 2001-2025) **from weather and terrain alone**. The gap between observed and expected NDVI is
11
+ change **not explained by precipitation, temperature, snow and soil moisture**. It is not a cause and says
12
+ nothing about forage quality. Built for NASA Space Apps 2026 ("Be An Earth System Trend Detective!").
13
+
14
+ ## Inputs
15
+ - Weather anomalies vs each cell's 2001-2025 mean for the same month: GPM IMERG precipitation (month, 3-month,
16
+ Oct-Apr), MODIS LST day/night (month, previous month), MODIS snow (Mar-May, month), GLDAS soil moisture
17
+ (month) and April snow water.
18
+ - Terrain from NASADEM (elevation, spread, slope, steep share, aspect, TPI) and land cover (MCD12Q1 2020).
19
+ - The cell's own NDVI climatology for that month. **Not used on purpose:** year, coordinates, past NDVI.
20
+
21
+ ## How it was tested (protocol locked in git before the first fit: `model_v1.yaml`, commit 92ed159)
22
+ - Tuning only on 2001-2015 -> 2016-2019. The 2020-2025 years were used once, for the final test.
23
+ - Spatial test: 5-fold GroupKFold over ~30 x 30 km super-blocks (all years; easier than the temporal test
24
+ because the same years' weather appears in training).
25
+ - Baselines: climatology (anomaly = 0) and ridge regression on the same features.
26
+ - Pre-registered gate: ship XGBoost only if the 95% CI lower bound of its skill over ridge is > 0 on both tests.
27
+ **Gate passed: True. Shipped: xgb.**
28
+
29
+ | Test | RMSE xgb | RMSE ridge | RMSE climatology | anomaly corr xgb | anomaly corr ridge | tercile bal. acc. xgb |
30
+ |---|---|---|---|---|---|---|
31
+ | Temporal (2020-2025) | 0.0554 | 0.0588 | 0.0616 | 0.424 | 0.288 | 0.455 |
32
+ | Spatial (unseen areas) | 0.0523 | 0.0578 | 0.0619 | 0.540 | 0.358 | 0.524 |
33
+
34
+ - Skill vs ridge, temporal: 0.113 (95% CI 0.089 to 0.133, 6 units). Five seeds: 0.110 +/- 0.003.
35
+ - Skill vs ridge, spatial: 0.180 (95% CI 0.169 to 0.192, 76 units).
36
+ - Skill vs climatology, temporal: 0.191 (95% CI 0.160 to 0.215, 6 units).
37
+ - What each NASA dataset adds (skill vs climatology, temporal test): precip: 0.055 -> +lst: 0.167 -> +snow: 0.181 -> +gldas (full): 0.188.
38
+ - Most used inputs: lst_day_m, lst_day_prev, precip_3m, lst_night_m, soil_m, month.
39
+
40
+ ## Limits (read before using)
41
+ - The model explains about a fifth of the year-to-year NDVI anomaly on unseen years; most variation is not weather-explained.
42
+ - P10-P90 band: on 2020-2025 it covered **74% instead of 80%**, so treat it as too narrow.
43
+ - Weather comes from coarse grids (IMERG 0.1 deg, GLDAS 0.25 deg) interpolated to 1 km.
44
+ - NDVI is greenness, not forage: shrub encroachment can look like improvement.
45
+ - Land cover is a fixed 2020 map; MODIS Terra drifted in recent years (see Aqua-Terra check in the dataset products).
46
+ - What-if runs: drier and hotter-than-normal conditions lower expected July NDVI, but wetter-than-normal
47
+ precipitation does not raise it (slightly lowers it on average). Treat what-if outputs as model
48
+ sensitivities, not causal effects.
49
+ - No field validation. Relative to Naryn pastures only.
50
+
51
+ ## Use
52
+ ```python
53
+ from predict import load
54
+ m = load(".") # needs only numpy, pandas, xgboost (CPU)
55
+ out = m.predict(features_df) # columns as in feature_schema.json
56
+ scen = m.what_if(cells, cell_ids, month=7, precip_pct=-30) # cells = naryn-cube cells.parquet
57
+ ```
58
+ `pytest test_golden.py` reproduces the 100 golden rows.
59
+
60
+ ## Data and tools
61
+ MOD13A3.061, MYD13A3.061, MOD11A2.061, MOD10A2.061, MCD12Q1.061, NASADEM_HGT.001 (LP DAAC / NSIDC via earthaccess),
62
+ GPM_3IMERGM.07 and GLDAS_NOAH025_M.2.1 (GES DISC). XGBoost, Optuna, scikit-learn, pymannkendall.
63
+ AI use: implementation was assisted by Claude (Anthropic); design decisions and the product idea are the team's.
climatology.parquet ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ccf6e4786bbbbb94dc8fc63d0a789d572a8181da43ff3a16e538aadf215881ec
3
+ size 2032126
conformal.json ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "5": [
3
+ -0.08275514096021652,
4
+ 0.09044573456048965
5
+ ],
6
+ "6": [
7
+ -0.0724983811378479,
8
+ 0.06799467653036118
9
+ ],
10
+ "7": [
11
+ -0.05535222217440605,
12
+ 0.0496993213891983
13
+ ],
14
+ "8": [
15
+ -0.05151964724063873,
16
+ 0.048559289425611496
17
+ ],
18
+ "9": [
19
+ -0.05011269450187683,
20
+ 0.04957366734743118
21
+ ]
22
+ }
feature_schema.json ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "features": [
3
+ "elev_mean",
4
+ "elev_std",
5
+ "slope_mean",
6
+ "steep_share",
7
+ "aspect_sin",
8
+ "aspect_cos",
9
+ "tpi",
10
+ "grass_frac",
11
+ "barren_frac",
12
+ "lc_type1",
13
+ "month",
14
+ "clim_ndvi",
15
+ "precip_m",
16
+ "precip_3m",
17
+ "precip_winter",
18
+ "lst_day_m",
19
+ "lst_night_m",
20
+ "lst_day_prev",
21
+ "snow_spring",
22
+ "snow_m",
23
+ "soil_m",
24
+ "swe_apr"
25
+ ],
26
+ "target": "ndvi_anomaly = ndvi - clim_ndvi",
27
+ "months": [
28
+ 5,
29
+ 6,
30
+ 7,
31
+ 8,
32
+ 9
33
+ ],
34
+ "weather_anomaly_baseline": "cell-month mean 2001-2025 (see climatology.parquet / weather_climatology.parquet)",
35
+ "shipped_model": "xgb"
36
+ }
golden/expected.parquet ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cce46e2d1390c95a9ddee29f254923debb754b6debc5d15679d0801497bdbfdd
3
+ size 5263
golden/inputs.parquet ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8c7176e3a378efa5f8c46261289941d490dceb7f146b0793d8d71dfab9bfd80e
3
+ size 23213
metrics.json ADDED
@@ -0,0 +1,211 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "protocol": "configs/model_v1.yaml",
3
+ "git_commit": "92ed159",
4
+ "features": [
5
+ "elev_mean",
6
+ "elev_std",
7
+ "slope_mean",
8
+ "steep_share",
9
+ "aspect_sin",
10
+ "aspect_cos",
11
+ "tpi",
12
+ "grass_frac",
13
+ "barren_frac",
14
+ "lc_type1",
15
+ "month",
16
+ "clim_ndvi",
17
+ "precip_m",
18
+ "precip_3m",
19
+ "precip_winter",
20
+ "lst_day_m",
21
+ "lst_night_m",
22
+ "lst_day_prev",
23
+ "snow_spring",
24
+ "snow_m",
25
+ "soil_m",
26
+ "swe_apr"
27
+ ],
28
+ "data": {
29
+ "cells": 50168,
30
+ "years": [
31
+ 2001,
32
+ 2025
33
+ ],
34
+ "months": [
35
+ 5,
36
+ 6,
37
+ 7,
38
+ 8,
39
+ 9
40
+ ]
41
+ },
42
+ "tuning": {
43
+ "trials": 40,
44
+ "best_params": {
45
+ "max_depth": 4,
46
+ "learning_rate": 0.05621263498077372,
47
+ "min_child_weight": 20.75429877129615,
48
+ "subsample": 0.9749316758605804,
49
+ "colsample_bytree": 0.5763769882944643,
50
+ "reg_lambda": 21.09479080136915,
51
+ "reg_alpha": 0.015374612569742166
52
+ },
53
+ "n_estimators": 145,
54
+ "inner_rmse_xgb": 0.05335228517651558,
55
+ "ridge_alpha": 1000,
56
+ "inner_rmse_ridge": {
57
+ "0.1": 0.055551912635564804,
58
+ "1": 0.055551912635564804,
59
+ "10": 0.05555190145969391,
60
+ "100": 0.05555180460214615,
61
+ "1000": 0.05555080249905586
62
+ }
63
+ },
64
+ "temporal_test": {
65
+ "n_rows": 1484947,
66
+ "xgb": {
67
+ "rmse": 0.05543375015258789,
68
+ "mae": 0.03968954086303711,
69
+ "anomaly_corr": 0.4242786380666133,
70
+ "tercile_bacc": 0.45527536336862856
71
+ },
72
+ "ridge": {
73
+ "rmse": 0.05884538218379021,
74
+ "mae": 0.04250144585967064,
75
+ "anomaly_corr": 0.28780787848363554,
76
+ "tercile_bacc": 0.41035457854199836
77
+ },
78
+ "clim": {
79
+ "rmse": 0.061612908672877956,
80
+ "mae": 0.04333119232217224,
81
+ "anomaly_corr": null,
82
+ "tercile_bacc": 0.3333333333333333
83
+ },
84
+ "skill_xgb_vs_ridge": {
85
+ "point": 0.11259126663208008,
86
+ "ci_lo": 0.0890129804611206,
87
+ "ci_hi": 0.13281476497650146,
88
+ "units": 6
89
+ },
90
+ "skill_xgb_vs_clim": {
91
+ "point": 0.1905218060292877,
92
+ "ci_lo": 0.15966808101130808,
93
+ "ci_hi": 0.21508542508221518,
94
+ "units": 6
95
+ },
96
+ "skill_ridge_vs_clim": {
97
+ "point": 0.08781810471926532,
98
+ "ci_lo": 0.047335278856051044,
99
+ "ci_hi": 0.12555581705053517,
100
+ "units": 6
101
+ },
102
+ "seed_skill_vs_ridge": {
103
+ "values": [
104
+ 0.10996484756469727,
105
+ 0.11416596174240112,
106
+ 0.10624825954437256,
107
+ 0.11326617002487183,
108
+ 0.10692000389099121
109
+ ],
110
+ "mean": 0.11011304855346679,
111
+ "sd": 0.0032100583223650296
112
+ },
113
+ "conformal_coverage_p10_p90": 0.7434184519716865
114
+ },
115
+ "ablation_skill_vs_clim_temporal": {
116
+ "precip": 0.054882168769836426,
117
+ "+lst": 0.16703283786773682,
118
+ "+snow": 0.1810346245765686,
119
+ "+gldas (full)": 0.18812614679336548
120
+ },
121
+ "spatial_test": {
122
+ "n_rows": 6154869,
123
+ "xgb": {
124
+ "rmse": 0.052341818058410906,
125
+ "mae": 0.035987544774873564,
126
+ "anomaly_corr": 0.5401110764463342,
127
+ "tercile_bacc": 0.5242309440180889
128
+ },
129
+ "ridge": {
130
+ "rmse": 0.05780769190076838,
131
+ "mae": 0.040356925267091316,
132
+ "anomaly_corr": 0.3584913306633923,
133
+ "tercile_bacc": 0.4518557704096291
134
+ },
135
+ "clim": {
136
+ "rmse": 0.06192350673620555,
137
+ "mae": 0.0441928267668356,
138
+ "anomaly_corr": null,
139
+ "tercile_bacc": 0.3333333333333333
140
+ },
141
+ "skill_xgb_vs_ridge": {
142
+ "point": 0.18016520234683842,
143
+ "ci_lo": 0.16931476346897326,
144
+ "ci_hi": 0.1921305862591789,
145
+ "units": 76
146
+ },
147
+ "skill_xgb_vs_clim": {
148
+ "point": 0.28552584752017784,
149
+ "ci_lo": 0.25587941077953436,
150
+ "ci_hi": 0.31732067499447747,
151
+ "units": 76
152
+ },
153
+ "skill_ridge_vs_clim": {
154
+ "point": 0.12851448300918933,
155
+ "ci_lo": 0.09990488385165495,
156
+ "ci_hi": 0.1578925273258722,
157
+ "units": 76
158
+ }
159
+ },
160
+ "gate": {
161
+ "rule": "CI lower bound of skill_xgb_vs_ridge > 0 on temporal AND spatial tests",
162
+ "passed": true,
163
+ "shipped_model": "xgb"
164
+ },
165
+ "conformal_p10_p90_by_month": {
166
+ "5": [
167
+ -0.08275514096021652,
168
+ 0.09044573456048965
169
+ ],
170
+ "6": [
171
+ -0.0724983811378479,
172
+ 0.06799467653036118
173
+ ],
174
+ "7": [
175
+ -0.05535222217440605,
176
+ 0.0496993213891983
177
+ ],
178
+ "8": [
179
+ -0.05151964724063873,
180
+ 0.048559289425611496
181
+ ],
182
+ "9": [
183
+ -0.05011269450187683,
184
+ 0.04957366734743118
185
+ ]
186
+ },
187
+ "feature_importance_mean_abs": {
188
+ "lst_day_m": 0.007868999615311623,
189
+ "lst_day_prev": 0.006517000030726194,
190
+ "precip_3m": 0.005698999855667353,
191
+ "lst_night_m": 0.0051369997672736645,
192
+ "soil_m": 0.0031030001118779182,
193
+ "month": 0.0029990000184625387,
194
+ "elev_mean": 0.002767000114545226,
195
+ "snow_spring": 0.002471999963745475,
196
+ "precip_winter": 0.002409999957308173,
197
+ "snow_m": 0.002343999920412898,
198
+ "precip_m": 0.0018370000179857016,
199
+ "clim_ndvi": 0.0016789999790489674,
200
+ "swe_apr": 0.001019999966956675,
201
+ "slope_mean": 0.00029200001154094934,
202
+ "barren_frac": 0.00014699999883305281,
203
+ "steep_share": 0.0001230000052601099,
204
+ "elev_std": 5.700000110664405e-05,
205
+ "grass_frac": 5.400000009103678e-05,
206
+ "tpi": 7.999999979801942e-06,
207
+ "lc_type1": 1.9999999949504854e-06,
208
+ "aspect_cos": 9.999999974752427e-07,
209
+ "aspect_sin": 0.0
210
+ }
211
+ }
model_a.ubj ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:99c776c812434d712ac96c4cbde733b9ffd3f73a57c161182f171026bee4cd73
3
+ size 249918
model_v1.yaml ADDED
@@ -0,0 +1,87 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # TrendeX "expected greenness" model, protocol v1.
2
+ # Locked BEFORE the first model fit. Every output records the git commit of this file.
3
+ # Changing anything below after results are seen = a new version (model_v2.yaml), never an edit.
4
+
5
+ name: expected-greenness
6
+ version: v1
7
+ question: >
8
+ How much of each cell's summer greenness anomaly is explained by weather (precipitation, temperature,
9
+ snow, soil moisture) and terrain alone? Residuals = change not explained by weather (never called a cause).
10
+
11
+ data:
12
+ cube: TrendeX90/naryn-cube # cells.parquet + monthly/
13
+ cells: in_oblast and mask_reason is null # pasture candidates in the 5 districts
14
+ months: [5, 6, 7, 8, 9]
15
+ years: [2001, 2025] # 2000 dropped: IMERG starts 2000-06, no Oct-Apr precipitation
16
+ target_rows: ndvi_ok == true # pixel reliability 0-1 only; others are not imputed
17
+
18
+ target:
19
+ name: ndvi_anomaly
20
+ definition: ndvi - clim(cell, month)
21
+ climatology: >
22
+ mean NDVI of the same cell and month over the TRAINING years of the current split, excluding the row's own
23
+ year (leave-one-year-out inside training years). Never uses test years.
24
+
25
+ features:
26
+ static: [elev_mean, elev_std, slope_mean, steep_share, aspect_sin, aspect_cos, tpi, grass_frac, barren_frac, lc_type1]
27
+ calendar: [month]
28
+ climatology: [clim_ndvi]
29
+ weather_anomalies: # anomaly = value - full-period (2001-2025) mean for the same cell and month
30
+ - precip_m # this month, mm
31
+ - precip_3m # this + 2 previous months
32
+ - precip_winter # Oct (previous year) .. Apr sum
33
+ - lst_day_m
34
+ - lst_night_m
35
+ - lst_day_prev # previous month
36
+ - snow_spring # Mar-May mean snow fraction
37
+ - snow_m
38
+ - soil_m # GLDAS 0-10 cm soil moisture, this month
39
+ - swe_apr # GLDAS snow water equivalent, April
40
+ weather_climatology_note: >
41
+ Weather anomalies use a full-period climatology. These are model inputs, not the target, so this leaks no
42
+ target information; the target climatology is fold-aware (see above).
43
+ excluded_on_purpose: [year, lat, lon, x, y, any NDVI lag]
44
+
45
+ splits:
46
+ temporal_test: {train: [2001, 2019], test: [2020, 2025]}
47
+ spatial_test: {kind: GroupKFold, folds: 5, group: superblock_33x33_cells} # all years, unseen areas
48
+ cross_fit_for_residuals: {kind: contiguous_year_blocks, blocks: [[2001, 2005], [2006, 2010], [2011, 2015], [2016, 2020], [2021, 2025]]}
49
+ inner_validation_for_tuning: {train: [2001, 2015], valid: [2016, 2019]} # never touches 2020-2025
50
+
51
+ baselines:
52
+ B0_climatology: anomaly = 0
53
+ B1_ridge: sklearn Ridge on the same standardized features, alpha chosen on the inner validation split
54
+
55
+ model:
56
+ kind: xgboost
57
+ params_fixed: {tree_method: hist, device: cuda, objective: "reg:squarederror"}
58
+ tuning: {sampler: optuna_tpe, max_minutes: 25, max_trials: 40, metric: rmse_inner_valid}
59
+ seeds: [0, 1, 2, 3, 4]
60
+
61
+ metrics:
62
+ primary: skill_vs_ridge = 1 - MSE_model / MSE_ridge
63
+ others: [rmse, mae, skill_vs_climatology, anomaly_correlation, tercile_balanced_accuracy]
64
+ uncertainty:
65
+ temporal: bootstrap over test years (2000 resamples)
66
+ spatial: bootstrap over superblocks (2000 resamples)
67
+ report_per: [cell_month, block_summer_mean, district_summer_mean]
68
+
69
+ gate:
70
+ rule: ship xgboost only if the 95% CI lower bound of skill_vs_ridge > 0 on BOTH temporal and spatial tests
71
+ otherwise: ship ridge as the expected-greenness model and say so on the Methods page
72
+
73
+ intervals:
74
+ method: split conformal on cross-fitted residuals of training years, per month
75
+ levels: [0.1, 0.9]
76
+ check: empirical coverage on the temporal test years
77
+
78
+ residual_trends:
79
+ series: summer (Jun-Aug) mean of out-of-fold residuals per cell and year
80
+ test: trend.py trend_1d (Sen slope + Mann-Kendall, Hamed-Rao) - copied unchanged from the backend
81
+ multiple_testing: Benjamini-Hochberg across cells, q < 0.05
82
+ min_years: 18
83
+ placebo: shuffle years within each cell, 200 permutations, mk_vectorized
84
+ wording: "change not explained by precipitation, temperature, snow and soil moisture"
85
+
86
+ cross_check:
87
+ aqua_vs_terra: per-block summer NDVI trend 2003-2025 from MYD13A3 vs MOD13A3; report sign agreement
predict.py ADDED
@@ -0,0 +1,105 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """TrendeX expected-greenness model: CPU inference. Imports only numpy, pandas and xgboost.
2
+
3
+ Files expected next to this module (the HF model repo TrendeX90/expected-greenness):
4
+ model_a.ubj XGBoost model (predicts the NDVI anomaly)
5
+ feature_schema.json feature order
6
+ climatology.parquet cell_id, month, clim_ndvi (2001-2025 mean NDVI of that cell and month)
7
+ weather_climatology.parquet cell_id, month, <feature>_mean (raw weather value minus this = model anomaly)
8
+ conformal.json month -> [q10, q90] of residuals (P10/P90 band)
9
+
10
+ from predict import load
11
+ m = load("path/to/repo")
12
+ out = m.predict(features_df) # features in feature_schema order
13
+ out = m.what_if(cells_df, cell_ids, month=7, precip_pct=-30, lst_day_c=+1.5)
14
+
15
+ Output: pred_anom, expected (= clim_ndvi + pred_anom), p10, p90. NDVI units.
16
+ Meaning: greenness expected from weather and terrain alone. The gap to observed NDVI is change NOT
17
+ explained by precipitation, temperature, snow and soil moisture. It is not a cause and not forage quality.
18
+ """
19
+ from __future__ import annotations
20
+
21
+ import json
22
+ from pathlib import Path
23
+
24
+ import numpy as np
25
+ import pandas as pd
26
+ import xgboost as xgb
27
+
28
+ STATIC = ["elev_mean", "elev_std", "slope_mean", "steep_share", "aspect_sin", "aspect_cos", "tpi",
29
+ "grass_frac", "barren_frac", "lc_type1"]
30
+ WEATHER = ["precip_m", "precip_3m", "precip_winter", "lst_day_m", "lst_night_m", "lst_day_prev",
31
+ "snow_spring", "snow_m", "soil_m", "swe_apr"]
32
+
33
+
34
+ class ExpectedGreenness:
35
+ def __init__(self, root: str | Path):
36
+ root = Path(root)
37
+ self.schema = json.loads((root / "feature_schema.json").read_text())
38
+ self.features = self.schema["features"]
39
+ self.booster = xgb.Booster()
40
+ self.booster.load_model(str(root / "model_a.ubj"))
41
+ self.booster.set_param({"device": "cpu"})
42
+ self.clim = pd.read_parquet(root / "climatology.parquet").set_index(["cell_id", "month"])
43
+ self.wclim = pd.read_parquet(root / "weather_climatology.parquet").set_index(["cell_id", "month"])
44
+ self.conf = {int(k): v for k, v in json.loads((root / "conformal.json").read_text()).items()}
45
+
46
+ # ------------------------------------------------------------------ core
47
+ def predict(self, df: pd.DataFrame) -> pd.DataFrame:
48
+ """df must contain every feature in feature_schema.json (NaN allowed for weather)."""
49
+ missing = [c for c in self.features if c not in df.columns]
50
+ if missing:
51
+ raise ValueError(f"missing features: {missing}")
52
+ X = df[self.features].astype("float32")
53
+ pred = self.booster.predict(xgb.DMatrix(X, feature_names=self.features))
54
+ months = df["month"].astype(int).values
55
+ lo = np.array([self.conf[m][0] for m in months])
56
+ hi = np.array([self.conf[m][1] for m in months])
57
+ exp = df["clim_ndvi"].values + pred
58
+ return pd.DataFrame({"pred_anom": pred, "expected": exp, "p10": exp + lo, "p90": exp + hi}, index=df.index)
59
+
60
+ # ------------------------------------------------------------------ helpers
61
+ def features_from_raw(self, cells: pd.DataFrame, raw: pd.DataFrame) -> pd.DataFrame:
62
+ """raw: cell_id, month and raw weather values named like WEATHER (mm, deg C, fractions, kg m-2).
63
+
64
+ precip_3m = this + 2 previous months; precip_winter = Oct..Apr sum; snow_spring = Mar-May mean;
65
+ swe_apr = April SWE. cells: rows of the cube's cells.parquet (for static features).
66
+ """
67
+ key = pd.MultiIndex.from_arrays([raw["cell_id"].values, raw["month"].astype(int).values])
68
+ out = raw[["cell_id", "month"]].copy()
69
+ for w in WEATHER:
70
+ out[w] = raw[w].values - self.wclim[f"{w}_mean"].reindex(key).values if w in raw else np.nan
71
+ out["clim_ndvi"] = self.clim["clim_ndvi"].reindex(key).values
72
+ out = out.merge(cells[["cell_id"] + STATIC], on="cell_id", how="left")
73
+ return out
74
+
75
+ def what_if(self, cells: pd.DataFrame, cell_ids, month: int, precip_pct: float = 0.0,
76
+ lst_day_c: float = 0.0, lst_night_c: float = 0.0, snow_spring_delta: float = 0.0,
77
+ soil_pct: float = 0.0) -> pd.DataFrame:
78
+ """Scenario vs typical weather: e.g. precip_pct=-30 means every precipitation input 30% below normal."""
79
+ ids = np.asarray(cell_ids)
80
+ key = pd.MultiIndex.from_arrays([ids, np.full(len(ids), int(month))])
81
+ w = self.wclim.reindex(key)
82
+
83
+ def frame(pp, ld, ln, ss, sp):
84
+ f = pd.DataFrame({"cell_id": ids, "month": float(month)})
85
+ for k in ("precip_m", "precip_3m", "precip_winter"):
86
+ f[k] = w[f"{k}_mean"].values * pp / 100.0
87
+ f["lst_day_m"] = f["lst_day_prev"] = ld
88
+ f["lst_night_m"] = ln
89
+ f["snow_spring"] = ss
90
+ f["snow_m"] = 0.0
91
+ f["soil_m"] = w["soil_m_mean"].values * sp / 100.0
92
+ f["swe_apr"] = 0.0
93
+ f["clim_ndvi"] = self.clim["clim_ndvi"].reindex(key).values
94
+ return f.merge(cells[["cell_id"] + STATIC], on="cell_id", how="left")
95
+
96
+ res = self.predict(frame(precip_pct, lst_day_c, lst_night_c, snow_spring_delta, soil_pct))
97
+ base = self.predict(frame(0.0, 0.0, 0.0, 0.0, 0.0))
98
+ res.insert(0, "cell_id", ids)
99
+ res["expected_typical_weather"] = base["expected"].values
100
+ res["change_vs_typical"] = res["expected"].values - base["expected"].values # the scenario effect
101
+ return res
102
+
103
+
104
+ def load(root: str | Path = Path(__file__).parent) -> ExpectedGreenness:
105
+ return ExpectedGreenness(root)
test_golden.py ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """pytest test_golden.py - the serving code must reproduce these outputs exactly."""
2
+ from pathlib import Path
3
+
4
+ import numpy as np
5
+ import pandas as pd
6
+
7
+ from predict import load
8
+
9
+ HERE = Path(__file__).parent
10
+
11
+
12
+ def test_golden():
13
+ m = load(HERE)
14
+ got = m.predict(pd.read_parquet(HERE / "golden" / "inputs.parquet"))
15
+ want = pd.read_parquet(HERE / "golden" / "expected.parquet")
16
+ assert np.allclose(got.values, want.values, atol=1e-6)
weather_climatology.parquet ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:77ad08468a0a514916d581cdafad1c3b921e28431e15e5a0a99ff50db7cb8281
3
+ size 11915708