Tabular Regression
Transformers
Safetensors
yield-text-image
feature-extraction
crop-yield-estimation
digital-agriculture
multimodal
multi-temporal
regression
custom_code
Instructions to use ICICLE-AI/multimodal-yield-estimation with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ICICLE-AI/multimodal-yield-estimation with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("ICICLE-AI/multimodal-yield-estimation", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from ICICLE-AI/multimodal-yield-estimation: direct link, hf CLI and curl.
- Browser
- Download file 17.3 kB
-
https://huggingface.co/ICICLE-AI/multimodal-yield-estimation/resolve/main/README.md
- Command line
-
hf download hf://ICICLE-AI/multimodal-yield-estimation/README.md
-
curl -L -o README.md https://huggingface.co/ICICLE-AI/multimodal-yield-estimation/resolve/main/README.md
17.3 kB
| license: mit | |
| library_name: transformers | |
| pipeline_tag: tabular-regression | |
| base_model: | |
| - Qwen/Qwen3-Embedding-0.6B | |
| - facebook/dinov3-vitb16-pretrain-lvd1689m | |
| tags: | |
| - crop-yield-estimation | |
| - digital-agriculture | |
| - multimodal | |
| - multi-temporal | |
| - regression | |
| - custom_code | |
| # Multimodal Yield Estimation | |
| A transformer-based model that estimates plot-level corn yield (bu/acre) from **weekly text captions** of field conditions (weather, soil chemistry, topography and treatment) and **UAS RGB images** of the plot, at any point in the growing season. This repository contains the trained checkpoint, inference scripts that compute the Qwen3 text and DINOv3 image embeddings and predict yield, and the training code. | |
| **Tags:** Foundation-AI | |
| ### License | |
| [](https://huggingface.co/ICICLE-AI/multimodal-yield-estimation/blob/main/LICENSE) | |
| ## References | |
| - [Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B): text embedding model used for the weekly captions (1024-d). | |
| - [facebook/dinov3-vitb16-pretrain-lvd1689m](https://huggingface.co/facebook/dinov3-vitb16-pretrain-lvd1689m): image backbone used for the UAS plot tiles (768-d). | |
| - [Daymet](https://daymet.ornl.gov/): source of the daily weather variables summarized in the captions. | |
| - [Hugging Face custom models](https://huggingface.co/docs/transformers/custom_models): how `trust_remote_code=True` loading works. | |
| - **Cutoff**: the number of weeks after planting the model is allowed to see. Week 12 means the prediction uses captions and images from the first 12 weeks only. | |
| - **UAS**: Uncrewed Aircraft System (drone) imagery. | |
| ## Acknowledgements | |
| *National Science Foundation (NSF) funded AI institute for Intelligent Cyberinfrastructure with Computational Learning in the Environment (ICICLE) (OAC 2112606)* | |
| ## Issue reporting | |
| For questions, issues, or support, please contact: | |
| Sarikaa Sridhar | |
| sridhar.86@osu.edu | |
| --- | |
| # Tutorials | |
| ## Overview | |
| | | | | |
| |---|---| | |
| | Task | Plot-level corn yield regression (bu/acre) | | |
| | Text input | One caption per week after planting, embedded with [`Qwen/Qwen3-Embedding-0.6B`](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) (1024-d) | | |
| | Image input | UAS RGB plot tiles (`.tif`) with capture dates, embedded with [`facebook/dinov3-vitb16-pretrain-lvd1689m`](https://huggingface.co/facebook/dinov3-vitb16-pretrain-lvd1689m) (768-d) | | |
| | Season length | 22 weeks (150 days) after planting | | |
| | Cutoffs | Trained and evaluated at weeks `4, 8, 12, 16, 20, 22`; any week 1–22 can be estimated | | |
| ## Prerequisites | |
| - Python 3.11 or newer | |
| - A GPU is recommended. On CPU, the inference scripts take several minutes because they run Qwen3 and DINOv3. | |
| ## Getting Started: your first prediction | |
| 1. Clone the repository and install the dependencies: | |
| ```bash | |
| git clone https://huggingface.co/ICICLE-AI/multimodal-yield-estimation | |
| cd multimodal-yield-estimation | |
| pip install -r requirements.txt | |
| ``` | |
| 2. Point the example at your images. `examples/sample_input.json` contains real weekly captions for one plot, but the UAS `.tif` tiles are not included. Each image entry has a placeholder path such as `/path/to/images/northwest_bftb_om_corn_rgb_101_20240530.tif`; replace `/path/to/images/` with the folder holding your tiles (or remove the `images` list to predict from text only). | |
| 3. Run single-sample inference at week 12: | |
| ```bash | |
| python scripts/inference_single_hf.py \ | |
| --hf_model_dir ICICLE-AI/multimodal-yield-estimation \ | |
| --sample_json examples/sample_input.json \ | |
| --week 12 \ | |
| --output_json prediction.json | |
| ``` | |
| 4. Read the result, printed and saved to `prediction.json`: | |
| ```json | |
| { | |
| "sample_id": "northwest_bftb_2024_plot101", | |
| "week": 12, | |
| "has_image_by_week": true, | |
| "image_age_weeks": 0, | |
| "yield_pred_bu_per_acre": 175.66 | |
| } | |
| ``` | |
| --- | |
| # How-To Guides | |
| ## Installation | |
| ```bash | |
| pip install -r requirements.txt | |
| ``` | |
| Tested versions: | |
| ```text | |
| torch==2.12.1 | |
| torchvision==0.27.1 | |
| transformers==5.13.0 | |
| sentence-transformers==5.7.0 | |
| safetensors==0.8.0 | |
| numpy==2.4.4 | |
| pandas==3.0.3 | |
| scikit-learn==1.9.0 | |
| pillow==12.2.0 | |
| ``` | |
| ## Prepare Input Data | |
| ### Input format | |
| Both inference scripts use the same per-plot sample format: | |
| ```json | |
| { | |
| "sample_id": "plot_001", | |
| "planting_date": "2024-05-16", | |
| "week": 12, | |
| "texts": [ | |
| "<caption for week 0>", | |
| "<caption for week 1>", | |
| "..." | |
| ], | |
| "images": [ | |
| {"path": "/path/to/images/plot_001_20240530.tif", "date": "2024-05-30"}, | |
| {"path": "/path/to/images/plot_001_20240606.tif", "date": "2024-06-06"} | |
| ] | |
| } | |
| ``` | |
| | Field | Required | Description | | |
| |---|---|---| | |
| | `planting_date` | yes | `YYYY-MM-DD`. Weeks are counted from this date. | | |
| | `week` | yes, unless `--week` is passed | Cutoff week, 1–22. | | |
| | `texts` | yes | Weekly captions starting at week 0 (planting week). At least `week` captions are needed; at most 22 are used. | | |
| | `images` | no | List of `{path, date}`. Order does not matter. An empty or missing list means no images. | | |
| | `sample_id` | no | Copied to the output. | | |
| - **Single inference**: one JSON object (see `examples/sample_input.json`). | |
| - **Batch inference**: a JSONL file with one object per line (see `examples/sample_batch.jsonl`). | |
| ### Write the weekly captions | |
| Week `w` covers days `7w` to `7w+6` after planting; the last week (week 21) covers days 147–149. Each caption describes: | |
| - **11 weekly weather variables** (Daymet): day length, precipitation total, solar radiation, maximum, minimum and mean temperature, vapor pressure, GDD accumulation, and 3-, 7- and 14-day precipitation averages. Values are weekly means, except precipitation and GDD, which are weekly sums. | |
| - **23 soil variables**: pH, buffer pH, organic matter, CEC, P, K, Mg, Ca, K/Ca/Mg TE, S, Zn, Fe, Mn, Cu, B, Na, clay, sand, silt, bulk density and available water capacity. | |
| - **Topography**: elevation and slope. | |
| - **Pre-plant soil moisture proxy**: mean precipitation over the 14 days before planting. | |
| - **Hybrid/treatment** identifier. | |
| Each numeric value is converted to `low`, `medium` or `high` using tertile thresholds computed on the full corn caption dataset (`not available` when missing). The hybrid/treatment value is written as is. | |
| Example caption (one week of one plot): | |
| ```text | |
| Weekly mean day length: medium. Weekly precipitation total: low. Weekly mean solar radiation: high. Weekly mean maximum temperature: high. Weekly mean minimum temperature: medium. Weekly mean vapor pressure: medium. Weekly mean temperature: high. Weekly GDD accumulation: high. Weekly mean 3-day precipitation average: medium. Weekly mean 7-day precipitation average: medium. Weekly mean 14-day precipitation average: high. Soil pH: low. Buffer pH: not available. Soil organic matter: low. Soil cation exchange capacity: high. Soil phosphorus: medium. Soil potassium: high. Soil magnesium: high. Soil calcium: high. Soil potassium TE: low. Soil calcium TE: high. Soil magnesium TE: low. Soil sulfur: high. Soil zinc: medium. Soil iron: high. Soil manganese: low. Soil copper: high. Soil boron: medium. Soil sodium: medium. Soil clay: high. Soil sand: medium. Soil silt: low. Soil bulk density: medium. Soil available water capacity: low. Elevation: low. Slope: low. Pre-plant soil moisture proxy: high. Hybrid/treatment: 100.0. | |
| ``` | |
| The model only learned this template. Captions should use the same wording, order and categories; free-form text produces embeddings the model has not seen. The caption generator and the tertile thresholds are not included in this repository. | |
| ### Prepare the UAS plot tiles | |
| - RGB plot tiles (`.tif`) cropped from UAS orthomosaics, one tile per plot per flight date. | |
| - Any size works: the DINOv3 image processor resizes and normalizes each image. | |
| - Each image needs its capture `date`. | |
| ## Run Inference | |
| Both scripts take raw captions and image files, compute the Qwen3 and DINOv3 embeddings, place the images on the weekly grid and predict yield. `--hf_model_dir` accepts a local checkpoint folder (use `.` from the repository root) or a Hub repo ID. The model is trained on weather/soil text caption and UAS plot images with 29.8 bu/acre RMSE yield error at evaluation. | |
| ### Single sample | |
| ```bash | |
| python scripts/inference_single_hf.py \ | |
| --hf_model_dir ICICLE-AI/multimodal-yield-estimation \ | |
| --sample_json examples/sample_input.json \ | |
| --week 12 \ | |
| --output_json prediction.json | |
| ``` | |
| ### Batch | |
| ```bash | |
| python scripts/inference_batch_hf.py \ | |
| --hf_model_dir ICICLE-AI/multimodal-yield-estimation \ | |
| --input_jsonl examples/sample_batch.jsonl \ | |
| --output_json predictions.json \ | |
| --output_csv predictions.csv | |
| ``` | |
| - The JSON output is a list with one object per sample, using the same fields as the single-sample output. `--output_csv` is optional. | |
| - Each sample uses its own `week`; pass `--week` to use the same week for all samples. | |
| | Option | Description | | |
| |---|---| | |
| | `--hf_model_dir` | Local checkpoint folder or Hub repo ID | | |
| | `--sample_json` / `--input_jsonl` | Input file | | |
| | `--week` | Cutoff week (1–22); overrides the sample's `week` | | |
| | `--output_json` | Output JSON path (output folder must exist) | | |
| | `--output_csv` | Batch only: optional CSV output | | |
| | `--batch_size` | Batch only: model batch size (default 64) | | |
| ### On a SLURM cluster | |
| `multimodal_yield_estimation.slurm` runs both inference scripts on one GPU. Fill in `--account`, the conda paths and the repository path before submitting: | |
| ```bash | |
| sbatch multimodal_yield_estimation.slurm | |
| ``` | |
| ## Load the Model from Python | |
| The model uses custom code, so load it with `trust_remote_code=True`. The model takes embeddings, not raw text or images; use `data/embeddings.py` (as the inference scripts do) to build them. Run this from the repository root so `data/` can be imported. | |
| ```python | |
| import torch | |
| from transformers import AutoModel | |
| from data.embeddings import ImageEmbedder, TextEmbedder, embed_samples | |
| model = AutoModel.from_pretrained( | |
| "ICICLE-AI/multimodal-yield-estimation", trust_remote_code=True | |
| ).eval() | |
| x = embed_samples( | |
| [sample], # dict in the input format above | |
| TextEmbedder(model.config.text_model), | |
| ImageEmbedder(model.config.image_model), | |
| ) | |
| with torch.no_grad(): | |
| out = model( | |
| text=torch.from_numpy(x["text"]), | |
| image=torch.from_numpy(x["image"]), | |
| img_mask=torch.from_numpy(x["img_mask"]), | |
| img_age=torch.from_numpy(x["img_age"]), | |
| cutoff=int(x["week"][0]), | |
| ) | |
| print(out.predictions) # bu/acre | |
| ``` | |
| Model inputs: | |
| | Input | Shape | Description | | |
| |---|---|---| | |
| | `text` | `[B, T, 1024]` | Raw Qwen3 caption embeddings | | |
| | `image` | `[B, T, 768]` | Raw DINOv3 features (any value where `img_mask` is 0) | | |
| | `img_mask` | `[B, T]` | 1 if an image is available at or before that week | | |
| | `img_age` | `[B, T]` | Weeks since that image was taken | | |
| | `cutoff` | int | Number of weeks to use (defaults to `T`) | | |
| Normalization statistics are stored in `config.json`, so inputs are passed un-normalized. The output has `predictions` (bu/acre) and `logits` (standardized yield). | |
| ## Train on Your Own Data | |
| 1. Build an `.npz` file of embeddings with these arrays: | |
| | Array | Shape | Description | | |
| |---|---|---| | |
| | `text` | `[N, 22, 1024]` float32 | Weekly Qwen3 caption embeddings | | |
| | `image` | `[N, 22, 768]` float32 | DINOv3 feature of the latest image at or before each week (zeros where none) | | |
| | `img_mask` | `[N, 22]` float32 | 1 if an image is available at or before that week | | |
| | `img_age` | `[N, 22]` float32 | Weeks since that image was taken | | |
| | `yield_` | `[N]` float32 | Yield (bu/acre) | | |
| | `farm_field`, `plot`, `planting_date`, `row_key` | `[N]` | Plot metadata | | |
| The image arrays can be built with `images_to_weeks()` in `data/multimodal_dataset.py`, which is the same function the inference scripts use. | |
| 2. Train. Your data needs at least two `farm_field` values; pass `--test_field <name>` to choose which one is held out. | |
| ```bash | |
| python scripts/train_multimodal_hf.py \ | |
| --data /path/to/embeddings.npz \ | |
| --out_dir checkpoints | |
| ``` | |
| Other options: `--cutoffs`, `--epochs`, `--patience`, `--lr`, `--batch_size`, `--d_model`, `--num_layers`, `--dropout`, `--input_dropout`, `--seed`. | |
| 3. To train on SLURM, replace the inference commands in `multimodal_yield_estimation.slurm` with: | |
| ```bash | |
| python scripts/train_multimodal_hf.py --data /path/to/your.npz --out_dir checkpoints | |
| ``` | |
| 4. Use the new checkpoint by passing `--hf_model_dir checkpoints` to the inference scripts. | |
| --- | |
| # Explanation | |
| ## Problem Description | |
| Given what is known about a corn plot up to a certain week of the season, estimate its final yield. | |
| What is known each week is described in two ways: | |
| - **Text**: a caption summarizing that week's weather and the plot's static soil, topography and treatment information. | |
| - **Image**: UAS RGB images of the plot, taken on flight dates that do not line up with weeks (every 1–2 weeks during the season). | |
| The model gives one yield estimate per plot for the requested cutoff week. | |
| ## Placing Images on the Weekly Grid | |
| The inference scripts do this automatically: | |
| - Image week = `(date − planting_date).days // 7`. Images before planting or after week 21 are ignored. | |
| - Each week uses the latest image taken at or before that week (forward-fill). If two images fall in the same week, the later one is used. | |
| - Weeks before the first image have no image (`has_image = 0`). | |
| - The scripts also compute the image age: the number of weeks since the image in use was taken (0 = new image that week). | |
| Example (flights in weeks 2, 3, 4, 6, 7, 9, 11, 14 and 15): | |
| ```text | |
| week: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | |
| has_image: 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 | |
| image_age: 0 0 0 0 0 1 0 0 1 0 1 0 1 2 0 0 1 2 3 4 5 6 | |
| ``` | |
| ## Multi-Temporal Yield Estimation | |
| - Each week after planting is represented by a Qwen3 embedding of that week's caption and a DINOv3 embedding of the most recent UAS image, concatenated with a has-image flag and the image age. | |
| - The model reads tokens for weeks `0 … cutoff−1`. A projection layer maps each token to 64 dimensions, then a week signal, a learned position embedding and a cutoff (horizon) embedding are added. | |
| - A 2-layer causal transformer encoder (4 heads) processes the sequence. The causal mask and forward-fill mean that no week sees later captions or images. | |
| - The last token, together with the horizon embedding, goes to a small regression head. | |
| - During training, the loss is averaged over all cutoffs (4, 8, 12, 16, 20, 22) for each batch, so one checkpoint serves every cutoff. | |
| ## Features | |
| - Text and image embeddings are concatenated per week. | |
| - Inference starts from raw captions and UAS images: the scripts run Qwen3 and DINOv3, so no precomputed embeddings are needed. | |
| - Missing weeks are handled with image forward-fill, a has-image flag and image age. | |
| - One checkpoint predicts at any cutoff week from 1 to 22. | |
| - The model is a standard Hugging Face `PreTrainedModel` and loads with `AutoModel.from_pretrained(..., trust_remote_code=True)`. | |
| - Single-sample and batch inference scripts write JSON (and optionally CSV) outputs. | |
| ## Repository Structure | |
| ```text | |
| config.json model config, including normalization statistics | |
| model.safetensors trained weights | |
| configuration_multimodal.py HF config class (loaded by trust_remote_code) | |
| modeling_multimodal.py HF model class (loaded by trust_remote_code) | |
| multimodal_concat_tf.py temporal transformer module (loaded by trust_remote_code) | |
| train_args.json arguments used to train the released checkpoint | |
| hf/ same model code, imported by the local scripts | |
| data/embeddings.py Qwen3 / DINOv3 embedding and input building | |
| data/multimodal_dataset.py weekly image placement, dataset, split, normalization | |
| scripts/inference_single_hf.py single-sample inference | |
| scripts/inference_batch_hf.py batch inference (JSONL) | |
| scripts/train_multimodal_hf.py training | |
| examples/ sample_input.json, sample_batch.jsonl | |
| multimodal_yield_estimation.slurm SLURM example | |
| component-info.yaml ICICLE component catalog metadata | |
| requirements.txt | |
| LICENSE | |
| ``` | |
| ## Training Details | |
| - **Split**: by field. One `farm_field` is held out (`--test_field`, default: the last field in alphabetical order) and its plots are split 50/50 into validation and test; all other fields are used for training. | |
| - **Loss**: MSE on standardized yield, averaged over all training cutoffs for each batch. | |
| - **Defaults**: AdamW, lr 3e-4, weight decay 1e-2, batch size 16, up to 300 epochs, early stopping on mean validation RMSE with patience 30. | |
| - **Checkpoint**: the epoch with the best validation RMSE is saved in Hugging Face format to `--out_dir`. | |
| ## Evaluation | |
| The training script evaluates the best checkpoint on the validation and test plots at every cutoff, and writes: | |
| - `metrics.json`: validation metrics at the best epoch | |
| - `test_metrics.json`: validation and test RMSE, MAE, bias and R² per cutoff | |
| - `test_predictions.csv`: per-plot predictions for the test plots | |