expected-greenness v1
Browse files- .gitattributes +1 -0
- README.md +63 -0
- climatology.parquet +3 -0
- conformal.json +22 -0
- feature_schema.json +36 -0
- golden/expected.parquet +3 -0
- golden/inputs.parquet +3 -0
- metrics.json +211 -0
- model_a.ubj +3 -0
- model_v1.yaml +87 -0
- predict.py +105 -0
- test_golden.py +16 -0
- weather_climatology.parquet +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
model_a.ubj filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,63 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: other
|
| 3 |
+
library_name: xgboost
|
| 4 |
+
tags: [nasa, modis, ndvi, imerg, gldas, pastures, kyrgyzstan, tabular-regression]
|
| 5 |
+
datasets: [TrendeX90/naryn-cube]
|
| 6 |
+
---
|
| 7 |
+
# TrendeX expected-greenness model (v1)
|
| 8 |
+
|
| 9 |
+
Predicts the **summer greenness (NDVI) anomaly** of every 1 km pasture cell in Naryn oblast, Kyrgyzstan
|
| 10 |
+
(May-September, 2001-2025) **from weather and terrain alone**. The gap between observed and expected NDVI is
|
| 11 |
+
change **not explained by precipitation, temperature, snow and soil moisture**. It is not a cause and says
|
| 12 |
+
nothing about forage quality. Built for NASA Space Apps 2026 ("Be An Earth System Trend Detective!").
|
| 13 |
+
|
| 14 |
+
## Inputs
|
| 15 |
+
- Weather anomalies vs each cell's 2001-2025 mean for the same month: GPM IMERG precipitation (month, 3-month,
|
| 16 |
+
Oct-Apr), MODIS LST day/night (month, previous month), MODIS snow (Mar-May, month), GLDAS soil moisture
|
| 17 |
+
(month) and April snow water.
|
| 18 |
+
- Terrain from NASADEM (elevation, spread, slope, steep share, aspect, TPI) and land cover (MCD12Q1 2020).
|
| 19 |
+
- The cell's own NDVI climatology for that month. **Not used on purpose:** year, coordinates, past NDVI.
|
| 20 |
+
|
| 21 |
+
## How it was tested (protocol locked in git before the first fit: `model_v1.yaml`, commit 92ed159)
|
| 22 |
+
- Tuning only on 2001-2015 -> 2016-2019. The 2020-2025 years were used once, for the final test.
|
| 23 |
+
- Spatial test: 5-fold GroupKFold over ~30 x 30 km super-blocks (all years; easier than the temporal test
|
| 24 |
+
because the same years' weather appears in training).
|
| 25 |
+
- Baselines: climatology (anomaly = 0) and ridge regression on the same features.
|
| 26 |
+
- Pre-registered gate: ship XGBoost only if the 95% CI lower bound of its skill over ridge is > 0 on both tests.
|
| 27 |
+
**Gate passed: True. Shipped: xgb.**
|
| 28 |
+
|
| 29 |
+
| Test | RMSE xgb | RMSE ridge | RMSE climatology | anomaly corr xgb | anomaly corr ridge | tercile bal. acc. xgb |
|
| 30 |
+
|---|---|---|---|---|---|---|
|
| 31 |
+
| Temporal (2020-2025) | 0.0554 | 0.0588 | 0.0616 | 0.424 | 0.288 | 0.455 |
|
| 32 |
+
| Spatial (unseen areas) | 0.0523 | 0.0578 | 0.0619 | 0.540 | 0.358 | 0.524 |
|
| 33 |
+
|
| 34 |
+
- Skill vs ridge, temporal: 0.113 (95% CI 0.089 to 0.133, 6 units). Five seeds: 0.110 +/- 0.003.
|
| 35 |
+
- Skill vs ridge, spatial: 0.180 (95% CI 0.169 to 0.192, 76 units).
|
| 36 |
+
- Skill vs climatology, temporal: 0.191 (95% CI 0.160 to 0.215, 6 units).
|
| 37 |
+
- What each NASA dataset adds (skill vs climatology, temporal test): precip: 0.055 -> +lst: 0.167 -> +snow: 0.181 -> +gldas (full): 0.188.
|
| 38 |
+
- Most used inputs: lst_day_m, lst_day_prev, precip_3m, lst_night_m, soil_m, month.
|
| 39 |
+
|
| 40 |
+
## Limits (read before using)
|
| 41 |
+
- The model explains about a fifth of the year-to-year NDVI anomaly on unseen years; most variation is not weather-explained.
|
| 42 |
+
- P10-P90 band: on 2020-2025 it covered **74% instead of 80%**, so treat it as too narrow.
|
| 43 |
+
- Weather comes from coarse grids (IMERG 0.1 deg, GLDAS 0.25 deg) interpolated to 1 km.
|
| 44 |
+
- NDVI is greenness, not forage: shrub encroachment can look like improvement.
|
| 45 |
+
- Land cover is a fixed 2020 map; MODIS Terra drifted in recent years (see Aqua-Terra check in the dataset products).
|
| 46 |
+
- What-if runs: drier and hotter-than-normal conditions lower expected July NDVI, but wetter-than-normal
|
| 47 |
+
precipitation does not raise it (slightly lowers it on average). Treat what-if outputs as model
|
| 48 |
+
sensitivities, not causal effects.
|
| 49 |
+
- No field validation. Relative to Naryn pastures only.
|
| 50 |
+
|
| 51 |
+
## Use
|
| 52 |
+
```python
|
| 53 |
+
from predict import load
|
| 54 |
+
m = load(".") # needs only numpy, pandas, xgboost (CPU)
|
| 55 |
+
out = m.predict(features_df) # columns as in feature_schema.json
|
| 56 |
+
scen = m.what_if(cells, cell_ids, month=7, precip_pct=-30) # cells = naryn-cube cells.parquet
|
| 57 |
+
```
|
| 58 |
+
`pytest test_golden.py` reproduces the 100 golden rows.
|
| 59 |
+
|
| 60 |
+
## Data and tools
|
| 61 |
+
MOD13A3.061, MYD13A3.061, MOD11A2.061, MOD10A2.061, MCD12Q1.061, NASADEM_HGT.001 (LP DAAC / NSIDC via earthaccess),
|
| 62 |
+
GPM_3IMERGM.07 and GLDAS_NOAH025_M.2.1 (GES DISC). XGBoost, Optuna, scikit-learn, pymannkendall.
|
| 63 |
+
AI use: implementation was assisted by Claude (Anthropic); design decisions and the product idea are the team's.
|
climatology.parquet
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ccf6e4786bbbbb94dc8fc63d0a789d572a8181da43ff3a16e538aadf215881ec
|
| 3 |
+
size 2032126
|
conformal.json
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"5": [
|
| 3 |
+
-0.08275514096021652,
|
| 4 |
+
0.09044573456048965
|
| 5 |
+
],
|
| 6 |
+
"6": [
|
| 7 |
+
-0.0724983811378479,
|
| 8 |
+
0.06799467653036118
|
| 9 |
+
],
|
| 10 |
+
"7": [
|
| 11 |
+
-0.05535222217440605,
|
| 12 |
+
0.0496993213891983
|
| 13 |
+
],
|
| 14 |
+
"8": [
|
| 15 |
+
-0.05151964724063873,
|
| 16 |
+
0.048559289425611496
|
| 17 |
+
],
|
| 18 |
+
"9": [
|
| 19 |
+
-0.05011269450187683,
|
| 20 |
+
0.04957366734743118
|
| 21 |
+
]
|
| 22 |
+
}
|
feature_schema.json
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"features": [
|
| 3 |
+
"elev_mean",
|
| 4 |
+
"elev_std",
|
| 5 |
+
"slope_mean",
|
| 6 |
+
"steep_share",
|
| 7 |
+
"aspect_sin",
|
| 8 |
+
"aspect_cos",
|
| 9 |
+
"tpi",
|
| 10 |
+
"grass_frac",
|
| 11 |
+
"barren_frac",
|
| 12 |
+
"lc_type1",
|
| 13 |
+
"month",
|
| 14 |
+
"clim_ndvi",
|
| 15 |
+
"precip_m",
|
| 16 |
+
"precip_3m",
|
| 17 |
+
"precip_winter",
|
| 18 |
+
"lst_day_m",
|
| 19 |
+
"lst_night_m",
|
| 20 |
+
"lst_day_prev",
|
| 21 |
+
"snow_spring",
|
| 22 |
+
"snow_m",
|
| 23 |
+
"soil_m",
|
| 24 |
+
"swe_apr"
|
| 25 |
+
],
|
| 26 |
+
"target": "ndvi_anomaly = ndvi - clim_ndvi",
|
| 27 |
+
"months": [
|
| 28 |
+
5,
|
| 29 |
+
6,
|
| 30 |
+
7,
|
| 31 |
+
8,
|
| 32 |
+
9
|
| 33 |
+
],
|
| 34 |
+
"weather_anomaly_baseline": "cell-month mean 2001-2025 (see climatology.parquet / weather_climatology.parquet)",
|
| 35 |
+
"shipped_model": "xgb"
|
| 36 |
+
}
|
golden/expected.parquet
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cce46e2d1390c95a9ddee29f254923debb754b6debc5d15679d0801497bdbfdd
|
| 3 |
+
size 5263
|
golden/inputs.parquet
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8c7176e3a378efa5f8c46261289941d490dceb7f146b0793d8d71dfab9bfd80e
|
| 3 |
+
size 23213
|
metrics.json
ADDED
|
@@ -0,0 +1,211 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"protocol": "configs/model_v1.yaml",
|
| 3 |
+
"git_commit": "92ed159",
|
| 4 |
+
"features": [
|
| 5 |
+
"elev_mean",
|
| 6 |
+
"elev_std",
|
| 7 |
+
"slope_mean",
|
| 8 |
+
"steep_share",
|
| 9 |
+
"aspect_sin",
|
| 10 |
+
"aspect_cos",
|
| 11 |
+
"tpi",
|
| 12 |
+
"grass_frac",
|
| 13 |
+
"barren_frac",
|
| 14 |
+
"lc_type1",
|
| 15 |
+
"month",
|
| 16 |
+
"clim_ndvi",
|
| 17 |
+
"precip_m",
|
| 18 |
+
"precip_3m",
|
| 19 |
+
"precip_winter",
|
| 20 |
+
"lst_day_m",
|
| 21 |
+
"lst_night_m",
|
| 22 |
+
"lst_day_prev",
|
| 23 |
+
"snow_spring",
|
| 24 |
+
"snow_m",
|
| 25 |
+
"soil_m",
|
| 26 |
+
"swe_apr"
|
| 27 |
+
],
|
| 28 |
+
"data": {
|
| 29 |
+
"cells": 50168,
|
| 30 |
+
"years": [
|
| 31 |
+
2001,
|
| 32 |
+
2025
|
| 33 |
+
],
|
| 34 |
+
"months": [
|
| 35 |
+
5,
|
| 36 |
+
6,
|
| 37 |
+
7,
|
| 38 |
+
8,
|
| 39 |
+
9
|
| 40 |
+
]
|
| 41 |
+
},
|
| 42 |
+
"tuning": {
|
| 43 |
+
"trials": 40,
|
| 44 |
+
"best_params": {
|
| 45 |
+
"max_depth": 4,
|
| 46 |
+
"learning_rate": 0.05621263498077372,
|
| 47 |
+
"min_child_weight": 20.75429877129615,
|
| 48 |
+
"subsample": 0.9749316758605804,
|
| 49 |
+
"colsample_bytree": 0.5763769882944643,
|
| 50 |
+
"reg_lambda": 21.09479080136915,
|
| 51 |
+
"reg_alpha": 0.015374612569742166
|
| 52 |
+
},
|
| 53 |
+
"n_estimators": 145,
|
| 54 |
+
"inner_rmse_xgb": 0.05335228517651558,
|
| 55 |
+
"ridge_alpha": 1000,
|
| 56 |
+
"inner_rmse_ridge": {
|
| 57 |
+
"0.1": 0.055551912635564804,
|
| 58 |
+
"1": 0.055551912635564804,
|
| 59 |
+
"10": 0.05555190145969391,
|
| 60 |
+
"100": 0.05555180460214615,
|
| 61 |
+
"1000": 0.05555080249905586
|
| 62 |
+
}
|
| 63 |
+
},
|
| 64 |
+
"temporal_test": {
|
| 65 |
+
"n_rows": 1484947,
|
| 66 |
+
"xgb": {
|
| 67 |
+
"rmse": 0.05543375015258789,
|
| 68 |
+
"mae": 0.03968954086303711,
|
| 69 |
+
"anomaly_corr": 0.4242786380666133,
|
| 70 |
+
"tercile_bacc": 0.45527536336862856
|
| 71 |
+
},
|
| 72 |
+
"ridge": {
|
| 73 |
+
"rmse": 0.05884538218379021,
|
| 74 |
+
"mae": 0.04250144585967064,
|
| 75 |
+
"anomaly_corr": 0.28780787848363554,
|
| 76 |
+
"tercile_bacc": 0.41035457854199836
|
| 77 |
+
},
|
| 78 |
+
"clim": {
|
| 79 |
+
"rmse": 0.061612908672877956,
|
| 80 |
+
"mae": 0.04333119232217224,
|
| 81 |
+
"anomaly_corr": null,
|
| 82 |
+
"tercile_bacc": 0.3333333333333333
|
| 83 |
+
},
|
| 84 |
+
"skill_xgb_vs_ridge": {
|
| 85 |
+
"point": 0.11259126663208008,
|
| 86 |
+
"ci_lo": 0.0890129804611206,
|
| 87 |
+
"ci_hi": 0.13281476497650146,
|
| 88 |
+
"units": 6
|
| 89 |
+
},
|
| 90 |
+
"skill_xgb_vs_clim": {
|
| 91 |
+
"point": 0.1905218060292877,
|
| 92 |
+
"ci_lo": 0.15966808101130808,
|
| 93 |
+
"ci_hi": 0.21508542508221518,
|
| 94 |
+
"units": 6
|
| 95 |
+
},
|
| 96 |
+
"skill_ridge_vs_clim": {
|
| 97 |
+
"point": 0.08781810471926532,
|
| 98 |
+
"ci_lo": 0.047335278856051044,
|
| 99 |
+
"ci_hi": 0.12555581705053517,
|
| 100 |
+
"units": 6
|
| 101 |
+
},
|
| 102 |
+
"seed_skill_vs_ridge": {
|
| 103 |
+
"values": [
|
| 104 |
+
0.10996484756469727,
|
| 105 |
+
0.11416596174240112,
|
| 106 |
+
0.10624825954437256,
|
| 107 |
+
0.11326617002487183,
|
| 108 |
+
0.10692000389099121
|
| 109 |
+
],
|
| 110 |
+
"mean": 0.11011304855346679,
|
| 111 |
+
"sd": 0.0032100583223650296
|
| 112 |
+
},
|
| 113 |
+
"conformal_coverage_p10_p90": 0.7434184519716865
|
| 114 |
+
},
|
| 115 |
+
"ablation_skill_vs_clim_temporal": {
|
| 116 |
+
"precip": 0.054882168769836426,
|
| 117 |
+
"+lst": 0.16703283786773682,
|
| 118 |
+
"+snow": 0.1810346245765686,
|
| 119 |
+
"+gldas (full)": 0.18812614679336548
|
| 120 |
+
},
|
| 121 |
+
"spatial_test": {
|
| 122 |
+
"n_rows": 6154869,
|
| 123 |
+
"xgb": {
|
| 124 |
+
"rmse": 0.052341818058410906,
|
| 125 |
+
"mae": 0.035987544774873564,
|
| 126 |
+
"anomaly_corr": 0.5401110764463342,
|
| 127 |
+
"tercile_bacc": 0.5242309440180889
|
| 128 |
+
},
|
| 129 |
+
"ridge": {
|
| 130 |
+
"rmse": 0.05780769190076838,
|
| 131 |
+
"mae": 0.040356925267091316,
|
| 132 |
+
"anomaly_corr": 0.3584913306633923,
|
| 133 |
+
"tercile_bacc": 0.4518557704096291
|
| 134 |
+
},
|
| 135 |
+
"clim": {
|
| 136 |
+
"rmse": 0.06192350673620555,
|
| 137 |
+
"mae": 0.0441928267668356,
|
| 138 |
+
"anomaly_corr": null,
|
| 139 |
+
"tercile_bacc": 0.3333333333333333
|
| 140 |
+
},
|
| 141 |
+
"skill_xgb_vs_ridge": {
|
| 142 |
+
"point": 0.18016520234683842,
|
| 143 |
+
"ci_lo": 0.16931476346897326,
|
| 144 |
+
"ci_hi": 0.1921305862591789,
|
| 145 |
+
"units": 76
|
| 146 |
+
},
|
| 147 |
+
"skill_xgb_vs_clim": {
|
| 148 |
+
"point": 0.28552584752017784,
|
| 149 |
+
"ci_lo": 0.25587941077953436,
|
| 150 |
+
"ci_hi": 0.31732067499447747,
|
| 151 |
+
"units": 76
|
| 152 |
+
},
|
| 153 |
+
"skill_ridge_vs_clim": {
|
| 154 |
+
"point": 0.12851448300918933,
|
| 155 |
+
"ci_lo": 0.09990488385165495,
|
| 156 |
+
"ci_hi": 0.1578925273258722,
|
| 157 |
+
"units": 76
|
| 158 |
+
}
|
| 159 |
+
},
|
| 160 |
+
"gate": {
|
| 161 |
+
"rule": "CI lower bound of skill_xgb_vs_ridge > 0 on temporal AND spatial tests",
|
| 162 |
+
"passed": true,
|
| 163 |
+
"shipped_model": "xgb"
|
| 164 |
+
},
|
| 165 |
+
"conformal_p10_p90_by_month": {
|
| 166 |
+
"5": [
|
| 167 |
+
-0.08275514096021652,
|
| 168 |
+
0.09044573456048965
|
| 169 |
+
],
|
| 170 |
+
"6": [
|
| 171 |
+
-0.0724983811378479,
|
| 172 |
+
0.06799467653036118
|
| 173 |
+
],
|
| 174 |
+
"7": [
|
| 175 |
+
-0.05535222217440605,
|
| 176 |
+
0.0496993213891983
|
| 177 |
+
],
|
| 178 |
+
"8": [
|
| 179 |
+
-0.05151964724063873,
|
| 180 |
+
0.048559289425611496
|
| 181 |
+
],
|
| 182 |
+
"9": [
|
| 183 |
+
-0.05011269450187683,
|
| 184 |
+
0.04957366734743118
|
| 185 |
+
]
|
| 186 |
+
},
|
| 187 |
+
"feature_importance_mean_abs": {
|
| 188 |
+
"lst_day_m": 0.007868999615311623,
|
| 189 |
+
"lst_day_prev": 0.006517000030726194,
|
| 190 |
+
"precip_3m": 0.005698999855667353,
|
| 191 |
+
"lst_night_m": 0.0051369997672736645,
|
| 192 |
+
"soil_m": 0.0031030001118779182,
|
| 193 |
+
"month": 0.0029990000184625387,
|
| 194 |
+
"elev_mean": 0.002767000114545226,
|
| 195 |
+
"snow_spring": 0.002471999963745475,
|
| 196 |
+
"precip_winter": 0.002409999957308173,
|
| 197 |
+
"snow_m": 0.002343999920412898,
|
| 198 |
+
"precip_m": 0.0018370000179857016,
|
| 199 |
+
"clim_ndvi": 0.0016789999790489674,
|
| 200 |
+
"swe_apr": 0.001019999966956675,
|
| 201 |
+
"slope_mean": 0.00029200001154094934,
|
| 202 |
+
"barren_frac": 0.00014699999883305281,
|
| 203 |
+
"steep_share": 0.0001230000052601099,
|
| 204 |
+
"elev_std": 5.700000110664405e-05,
|
| 205 |
+
"grass_frac": 5.400000009103678e-05,
|
| 206 |
+
"tpi": 7.999999979801942e-06,
|
| 207 |
+
"lc_type1": 1.9999999949504854e-06,
|
| 208 |
+
"aspect_cos": 9.999999974752427e-07,
|
| 209 |
+
"aspect_sin": 0.0
|
| 210 |
+
}
|
| 211 |
+
}
|
model_a.ubj
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:99c776c812434d712ac96c4cbde733b9ffd3f73a57c161182f171026bee4cd73
|
| 3 |
+
size 249918
|
model_v1.yaml
ADDED
|
@@ -0,0 +1,87 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# TrendeX "expected greenness" model, protocol v1.
|
| 2 |
+
# Locked BEFORE the first model fit. Every output records the git commit of this file.
|
| 3 |
+
# Changing anything below after results are seen = a new version (model_v2.yaml), never an edit.
|
| 4 |
+
|
| 5 |
+
name: expected-greenness
|
| 6 |
+
version: v1
|
| 7 |
+
question: >
|
| 8 |
+
How much of each cell's summer greenness anomaly is explained by weather (precipitation, temperature,
|
| 9 |
+
snow, soil moisture) and terrain alone? Residuals = change not explained by weather (never called a cause).
|
| 10 |
+
|
| 11 |
+
data:
|
| 12 |
+
cube: TrendeX90/naryn-cube # cells.parquet + monthly/
|
| 13 |
+
cells: in_oblast and mask_reason is null # pasture candidates in the 5 districts
|
| 14 |
+
months: [5, 6, 7, 8, 9]
|
| 15 |
+
years: [2001, 2025] # 2000 dropped: IMERG starts 2000-06, no Oct-Apr precipitation
|
| 16 |
+
target_rows: ndvi_ok == true # pixel reliability 0-1 only; others are not imputed
|
| 17 |
+
|
| 18 |
+
target:
|
| 19 |
+
name: ndvi_anomaly
|
| 20 |
+
definition: ndvi - clim(cell, month)
|
| 21 |
+
climatology: >
|
| 22 |
+
mean NDVI of the same cell and month over the TRAINING years of the current split, excluding the row's own
|
| 23 |
+
year (leave-one-year-out inside training years). Never uses test years.
|
| 24 |
+
|
| 25 |
+
features:
|
| 26 |
+
static: [elev_mean, elev_std, slope_mean, steep_share, aspect_sin, aspect_cos, tpi, grass_frac, barren_frac, lc_type1]
|
| 27 |
+
calendar: [month]
|
| 28 |
+
climatology: [clim_ndvi]
|
| 29 |
+
weather_anomalies: # anomaly = value - full-period (2001-2025) mean for the same cell and month
|
| 30 |
+
- precip_m # this month, mm
|
| 31 |
+
- precip_3m # this + 2 previous months
|
| 32 |
+
- precip_winter # Oct (previous year) .. Apr sum
|
| 33 |
+
- lst_day_m
|
| 34 |
+
- lst_night_m
|
| 35 |
+
- lst_day_prev # previous month
|
| 36 |
+
- snow_spring # Mar-May mean snow fraction
|
| 37 |
+
- snow_m
|
| 38 |
+
- soil_m # GLDAS 0-10 cm soil moisture, this month
|
| 39 |
+
- swe_apr # GLDAS snow water equivalent, April
|
| 40 |
+
weather_climatology_note: >
|
| 41 |
+
Weather anomalies use a full-period climatology. These are model inputs, not the target, so this leaks no
|
| 42 |
+
target information; the target climatology is fold-aware (see above).
|
| 43 |
+
excluded_on_purpose: [year, lat, lon, x, y, any NDVI lag]
|
| 44 |
+
|
| 45 |
+
splits:
|
| 46 |
+
temporal_test: {train: [2001, 2019], test: [2020, 2025]}
|
| 47 |
+
spatial_test: {kind: GroupKFold, folds: 5, group: superblock_33x33_cells} # all years, unseen areas
|
| 48 |
+
cross_fit_for_residuals: {kind: contiguous_year_blocks, blocks: [[2001, 2005], [2006, 2010], [2011, 2015], [2016, 2020], [2021, 2025]]}
|
| 49 |
+
inner_validation_for_tuning: {train: [2001, 2015], valid: [2016, 2019]} # never touches 2020-2025
|
| 50 |
+
|
| 51 |
+
baselines:
|
| 52 |
+
B0_climatology: anomaly = 0
|
| 53 |
+
B1_ridge: sklearn Ridge on the same standardized features, alpha chosen on the inner validation split
|
| 54 |
+
|
| 55 |
+
model:
|
| 56 |
+
kind: xgboost
|
| 57 |
+
params_fixed: {tree_method: hist, device: cuda, objective: "reg:squarederror"}
|
| 58 |
+
tuning: {sampler: optuna_tpe, max_minutes: 25, max_trials: 40, metric: rmse_inner_valid}
|
| 59 |
+
seeds: [0, 1, 2, 3, 4]
|
| 60 |
+
|
| 61 |
+
metrics:
|
| 62 |
+
primary: skill_vs_ridge = 1 - MSE_model / MSE_ridge
|
| 63 |
+
others: [rmse, mae, skill_vs_climatology, anomaly_correlation, tercile_balanced_accuracy]
|
| 64 |
+
uncertainty:
|
| 65 |
+
temporal: bootstrap over test years (2000 resamples)
|
| 66 |
+
spatial: bootstrap over superblocks (2000 resamples)
|
| 67 |
+
report_per: [cell_month, block_summer_mean, district_summer_mean]
|
| 68 |
+
|
| 69 |
+
gate:
|
| 70 |
+
rule: ship xgboost only if the 95% CI lower bound of skill_vs_ridge > 0 on BOTH temporal and spatial tests
|
| 71 |
+
otherwise: ship ridge as the expected-greenness model and say so on the Methods page
|
| 72 |
+
|
| 73 |
+
intervals:
|
| 74 |
+
method: split conformal on cross-fitted residuals of training years, per month
|
| 75 |
+
levels: [0.1, 0.9]
|
| 76 |
+
check: empirical coverage on the temporal test years
|
| 77 |
+
|
| 78 |
+
residual_trends:
|
| 79 |
+
series: summer (Jun-Aug) mean of out-of-fold residuals per cell and year
|
| 80 |
+
test: trend.py trend_1d (Sen slope + Mann-Kendall, Hamed-Rao) - copied unchanged from the backend
|
| 81 |
+
multiple_testing: Benjamini-Hochberg across cells, q < 0.05
|
| 82 |
+
min_years: 18
|
| 83 |
+
placebo: shuffle years within each cell, 200 permutations, mk_vectorized
|
| 84 |
+
wording: "change not explained by precipitation, temperature, snow and soil moisture"
|
| 85 |
+
|
| 86 |
+
cross_check:
|
| 87 |
+
aqua_vs_terra: per-block summer NDVI trend 2003-2025 from MYD13A3 vs MOD13A3; report sign agreement
|
predict.py
ADDED
|
@@ -0,0 +1,105 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""TrendeX expected-greenness model: CPU inference. Imports only numpy, pandas and xgboost.
|
| 2 |
+
|
| 3 |
+
Files expected next to this module (the HF model repo TrendeX90/expected-greenness):
|
| 4 |
+
model_a.ubj XGBoost model (predicts the NDVI anomaly)
|
| 5 |
+
feature_schema.json feature order
|
| 6 |
+
climatology.parquet cell_id, month, clim_ndvi (2001-2025 mean NDVI of that cell and month)
|
| 7 |
+
weather_climatology.parquet cell_id, month, <feature>_mean (raw weather value minus this = model anomaly)
|
| 8 |
+
conformal.json month -> [q10, q90] of residuals (P10/P90 band)
|
| 9 |
+
|
| 10 |
+
from predict import load
|
| 11 |
+
m = load("path/to/repo")
|
| 12 |
+
out = m.predict(features_df) # features in feature_schema order
|
| 13 |
+
out = m.what_if(cells_df, cell_ids, month=7, precip_pct=-30, lst_day_c=+1.5)
|
| 14 |
+
|
| 15 |
+
Output: pred_anom, expected (= clim_ndvi + pred_anom), p10, p90. NDVI units.
|
| 16 |
+
Meaning: greenness expected from weather and terrain alone. The gap to observed NDVI is change NOT
|
| 17 |
+
explained by precipitation, temperature, snow and soil moisture. It is not a cause and not forage quality.
|
| 18 |
+
"""
|
| 19 |
+
from __future__ import annotations
|
| 20 |
+
|
| 21 |
+
import json
|
| 22 |
+
from pathlib import Path
|
| 23 |
+
|
| 24 |
+
import numpy as np
|
| 25 |
+
import pandas as pd
|
| 26 |
+
import xgboost as xgb
|
| 27 |
+
|
| 28 |
+
STATIC = ["elev_mean", "elev_std", "slope_mean", "steep_share", "aspect_sin", "aspect_cos", "tpi",
|
| 29 |
+
"grass_frac", "barren_frac", "lc_type1"]
|
| 30 |
+
WEATHER = ["precip_m", "precip_3m", "precip_winter", "lst_day_m", "lst_night_m", "lst_day_prev",
|
| 31 |
+
"snow_spring", "snow_m", "soil_m", "swe_apr"]
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
class ExpectedGreenness:
|
| 35 |
+
def __init__(self, root: str | Path):
|
| 36 |
+
root = Path(root)
|
| 37 |
+
self.schema = json.loads((root / "feature_schema.json").read_text())
|
| 38 |
+
self.features = self.schema["features"]
|
| 39 |
+
self.booster = xgb.Booster()
|
| 40 |
+
self.booster.load_model(str(root / "model_a.ubj"))
|
| 41 |
+
self.booster.set_param({"device": "cpu"})
|
| 42 |
+
self.clim = pd.read_parquet(root / "climatology.parquet").set_index(["cell_id", "month"])
|
| 43 |
+
self.wclim = pd.read_parquet(root / "weather_climatology.parquet").set_index(["cell_id", "month"])
|
| 44 |
+
self.conf = {int(k): v for k, v in json.loads((root / "conformal.json").read_text()).items()}
|
| 45 |
+
|
| 46 |
+
# ------------------------------------------------------------------ core
|
| 47 |
+
def predict(self, df: pd.DataFrame) -> pd.DataFrame:
|
| 48 |
+
"""df must contain every feature in feature_schema.json (NaN allowed for weather)."""
|
| 49 |
+
missing = [c for c in self.features if c not in df.columns]
|
| 50 |
+
if missing:
|
| 51 |
+
raise ValueError(f"missing features: {missing}")
|
| 52 |
+
X = df[self.features].astype("float32")
|
| 53 |
+
pred = self.booster.predict(xgb.DMatrix(X, feature_names=self.features))
|
| 54 |
+
months = df["month"].astype(int).values
|
| 55 |
+
lo = np.array([self.conf[m][0] for m in months])
|
| 56 |
+
hi = np.array([self.conf[m][1] for m in months])
|
| 57 |
+
exp = df["clim_ndvi"].values + pred
|
| 58 |
+
return pd.DataFrame({"pred_anom": pred, "expected": exp, "p10": exp + lo, "p90": exp + hi}, index=df.index)
|
| 59 |
+
|
| 60 |
+
# ------------------------------------------------------------------ helpers
|
| 61 |
+
def features_from_raw(self, cells: pd.DataFrame, raw: pd.DataFrame) -> pd.DataFrame:
|
| 62 |
+
"""raw: cell_id, month and raw weather values named like WEATHER (mm, deg C, fractions, kg m-2).
|
| 63 |
+
|
| 64 |
+
precip_3m = this + 2 previous months; precip_winter = Oct..Apr sum; snow_spring = Mar-May mean;
|
| 65 |
+
swe_apr = April SWE. cells: rows of the cube's cells.parquet (for static features).
|
| 66 |
+
"""
|
| 67 |
+
key = pd.MultiIndex.from_arrays([raw["cell_id"].values, raw["month"].astype(int).values])
|
| 68 |
+
out = raw[["cell_id", "month"]].copy()
|
| 69 |
+
for w in WEATHER:
|
| 70 |
+
out[w] = raw[w].values - self.wclim[f"{w}_mean"].reindex(key).values if w in raw else np.nan
|
| 71 |
+
out["clim_ndvi"] = self.clim["clim_ndvi"].reindex(key).values
|
| 72 |
+
out = out.merge(cells[["cell_id"] + STATIC], on="cell_id", how="left")
|
| 73 |
+
return out
|
| 74 |
+
|
| 75 |
+
def what_if(self, cells: pd.DataFrame, cell_ids, month: int, precip_pct: float = 0.0,
|
| 76 |
+
lst_day_c: float = 0.0, lst_night_c: float = 0.0, snow_spring_delta: float = 0.0,
|
| 77 |
+
soil_pct: float = 0.0) -> pd.DataFrame:
|
| 78 |
+
"""Scenario vs typical weather: e.g. precip_pct=-30 means every precipitation input 30% below normal."""
|
| 79 |
+
ids = np.asarray(cell_ids)
|
| 80 |
+
key = pd.MultiIndex.from_arrays([ids, np.full(len(ids), int(month))])
|
| 81 |
+
w = self.wclim.reindex(key)
|
| 82 |
+
|
| 83 |
+
def frame(pp, ld, ln, ss, sp):
|
| 84 |
+
f = pd.DataFrame({"cell_id": ids, "month": float(month)})
|
| 85 |
+
for k in ("precip_m", "precip_3m", "precip_winter"):
|
| 86 |
+
f[k] = w[f"{k}_mean"].values * pp / 100.0
|
| 87 |
+
f["lst_day_m"] = f["lst_day_prev"] = ld
|
| 88 |
+
f["lst_night_m"] = ln
|
| 89 |
+
f["snow_spring"] = ss
|
| 90 |
+
f["snow_m"] = 0.0
|
| 91 |
+
f["soil_m"] = w["soil_m_mean"].values * sp / 100.0
|
| 92 |
+
f["swe_apr"] = 0.0
|
| 93 |
+
f["clim_ndvi"] = self.clim["clim_ndvi"].reindex(key).values
|
| 94 |
+
return f.merge(cells[["cell_id"] + STATIC], on="cell_id", how="left")
|
| 95 |
+
|
| 96 |
+
res = self.predict(frame(precip_pct, lst_day_c, lst_night_c, snow_spring_delta, soil_pct))
|
| 97 |
+
base = self.predict(frame(0.0, 0.0, 0.0, 0.0, 0.0))
|
| 98 |
+
res.insert(0, "cell_id", ids)
|
| 99 |
+
res["expected_typical_weather"] = base["expected"].values
|
| 100 |
+
res["change_vs_typical"] = res["expected"].values - base["expected"].values # the scenario effect
|
| 101 |
+
return res
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
def load(root: str | Path = Path(__file__).parent) -> ExpectedGreenness:
|
| 105 |
+
return ExpectedGreenness(root)
|
test_golden.py
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""pytest test_golden.py - the serving code must reproduce these outputs exactly."""
|
| 2 |
+
from pathlib import Path
|
| 3 |
+
|
| 4 |
+
import numpy as np
|
| 5 |
+
import pandas as pd
|
| 6 |
+
|
| 7 |
+
from predict import load
|
| 8 |
+
|
| 9 |
+
HERE = Path(__file__).parent
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
def test_golden():
|
| 13 |
+
m = load(HERE)
|
| 14 |
+
got = m.predict(pd.read_parquet(HERE / "golden" / "inputs.parquet"))
|
| 15 |
+
want = pd.read_parquet(HERE / "golden" / "expected.parquet")
|
| 16 |
+
assert np.allclose(got.values, want.values, atol=1e-6)
|
weather_climatology.parquet
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:77ad08468a0a514916d581cdafad1c3b921e28431e15e5a0a99ff50db7cb8281
|
| 3 |
+
size 11915708
|