Instructions to use ICICLE-AI/multimodal-yield-estimation with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ICICLE-AI/multimodal-yield-estimation with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("ICICLE-AI/multimodal-yield-estimation", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Download README.md from ICICLE-AI/multimodal-yield-estimation: direct link, hf CLI and curl.
- Browser
- Download file 17.3 kB
-
https://huggingface.co/ICICLE-AI/multimodal-yield-estimation/resolve/main/README.md
- Command line
-
hf download hf://ICICLE-AI/multimodal-yield-estimation/README.md
-
curl -L -o README.md https://huggingface.co/ICICLE-AI/multimodal-yield-estimation/resolve/main/README.md
license: mit
library_name: transformers
pipeline_tag: tabular-regression
base_model:
- Qwen/Qwen3-Embedding-0.6B
- facebook/dinov3-vitb16-pretrain-lvd1689m
tags:
- crop-yield-estimation
- digital-agriculture
- multimodal
- multi-temporal
- regression
- custom_code
Multimodal Yield Estimation
A transformer-based model that estimates plot-level corn yield (bu/acre) from weekly text captions of field conditions (weather, soil chemistry, topography and treatment) and UAS RGB images of the plot, at any point in the growing season. This repository contains the trained checkpoint, inference scripts that compute the Qwen3 text and DINOv3 image embeddings and predict yield, and the training code.
Tags: Foundation-AI
License
References
- Qwen/Qwen3-Embedding-0.6B: text embedding model used for the weekly captions (1024-d).
- facebook/dinov3-vitb16-pretrain-lvd1689m: image backbone used for the UAS plot tiles (768-d).
- Daymet: source of the daily weather variables summarized in the captions.
- Hugging Face custom models: how
trust_remote_code=Trueloading works. - Cutoff: the number of weeks after planting the model is allowed to see. Week 12 means the prediction uses captions and images from the first 12 weeks only.
- UAS: Uncrewed Aircraft System (drone) imagery.
Acknowledgements
National Science Foundation (NSF) funded AI institute for Intelligent Cyberinfrastructure with Computational Learning in the Environment (ICICLE) (OAC 2112606)
Issue reporting
For questions, issues, or support, please contact:
Sarikaa Sridhar sridhar.86@osu.edu
Tutorials
Overview
| Task | Plot-level corn yield regression (bu/acre) |
| Text input | One caption per week after planting, embedded with Qwen/Qwen3-Embedding-0.6B (1024-d) |
| Image input | UAS RGB plot tiles (.tif) with capture dates, embedded with facebook/dinov3-vitb16-pretrain-lvd1689m (768-d) |
| Season length | 22 weeks (150 days) after planting |
| Cutoffs | Trained and evaluated at weeks 4, 8, 12, 16, 20, 22; any week 1–22 can be estimated |
Prerequisites
- Python 3.11 or newer
- A GPU is recommended. On CPU, the inference scripts take several minutes because they run Qwen3 and DINOv3.
Getting Started: your first prediction
Clone the repository and install the dependencies:
git clone https://huggingface.co/ICICLE-AI/multimodal-yield-estimation cd multimodal-yield-estimation pip install -r requirements.txtPoint the example at your images.
examples/sample_input.jsoncontains real weekly captions for one plot, but the UAS.tiftiles are not included. Each image entry has a placeholder path such as/path/to/images/northwest_bftb_om_corn_rgb_101_20240530.tif; replace/path/to/images/with the folder holding your tiles (or remove theimageslist to predict from text only).Run single-sample inference at week 12:
python scripts/inference_single_hf.py \ --hf_model_dir ICICLE-AI/multimodal-yield-estimation \ --sample_json examples/sample_input.json \ --week 12 \ --output_json prediction.jsonRead the result, printed and saved to
prediction.json:{ "sample_id": "northwest_bftb_2024_plot101", "week": 12, "has_image_by_week": true, "image_age_weeks": 0, "yield_pred_bu_per_acre": 175.66 }
How-To Guides
Installation
pip install -r requirements.txt
Tested versions:
torch==2.12.1
torchvision==0.27.1
transformers==5.13.0
sentence-transformers==5.7.0
safetensors==0.8.0
numpy==2.4.4
pandas==3.0.3
scikit-learn==1.9.0
pillow==12.2.0
Prepare Input Data
Input format
Both inference scripts use the same per-plot sample format:
{
"sample_id": "plot_001",
"planting_date": "2024-05-16",
"week": 12,
"texts": [
"<caption for week 0>",
"<caption for week 1>",
"..."
],
"images": [
{"path": "/path/to/images/plot_001_20240530.tif", "date": "2024-05-30"},
{"path": "/path/to/images/plot_001_20240606.tif", "date": "2024-06-06"}
]
}
| Field | Required | Description |
|---|---|---|
planting_date |
yes | YYYY-MM-DD. Weeks are counted from this date. |
week |
yes, unless --week is passed |
Cutoff week, 1–22. |
texts |
yes | Weekly captions starting at week 0 (planting week). At least week captions are needed; at most 22 are used. |
images |
no | List of {path, date}. Order does not matter. An empty or missing list means no images. |
sample_id |
no | Copied to the output. |
- Single inference: one JSON object (see
examples/sample_input.json). - Batch inference: a JSONL file with one object per line (see
examples/sample_batch.jsonl).
Write the weekly captions
Week w covers days 7w to 7w+6 after planting; the last week (week 21) covers days 147–149. Each caption describes:
- 11 weekly weather variables (Daymet): day length, precipitation total, solar radiation, maximum, minimum and mean temperature, vapor pressure, GDD accumulation, and 3-, 7- and 14-day precipitation averages. Values are weekly means, except precipitation and GDD, which are weekly sums.
- 23 soil variables: pH, buffer pH, organic matter, CEC, P, K, Mg, Ca, K/Ca/Mg TE, S, Zn, Fe, Mn, Cu, B, Na, clay, sand, silt, bulk density and available water capacity.
- Topography: elevation and slope.
- Pre-plant soil moisture proxy: mean precipitation over the 14 days before planting.
- Hybrid/treatment identifier.
Each numeric value is converted to low, medium or high using tertile thresholds computed on the full corn caption dataset (not available when missing). The hybrid/treatment value is written as is.
Example caption (one week of one plot):
Weekly mean day length: medium. Weekly precipitation total: low. Weekly mean solar radiation: high. Weekly mean maximum temperature: high. Weekly mean minimum temperature: medium. Weekly mean vapor pressure: medium. Weekly mean temperature: high. Weekly GDD accumulation: high. Weekly mean 3-day precipitation average: medium. Weekly mean 7-day precipitation average: medium. Weekly mean 14-day precipitation average: high. Soil pH: low. Buffer pH: not available. Soil organic matter: low. Soil cation exchange capacity: high. Soil phosphorus: medium. Soil potassium: high. Soil magnesium: high. Soil calcium: high. Soil potassium TE: low. Soil calcium TE: high. Soil magnesium TE: low. Soil sulfur: high. Soil zinc: medium. Soil iron: high. Soil manganese: low. Soil copper: high. Soil boron: medium. Soil sodium: medium. Soil clay: high. Soil sand: medium. Soil silt: low. Soil bulk density: medium. Soil available water capacity: low. Elevation: low. Slope: low. Pre-plant soil moisture proxy: high. Hybrid/treatment: 100.0.
The model only learned this template. Captions should use the same wording, order and categories; free-form text produces embeddings the model has not seen. The caption generator and the tertile thresholds are not included in this repository.
Prepare the UAS plot tiles
- RGB plot tiles (
.tif) cropped from UAS orthomosaics, one tile per plot per flight date. - Any size works: the DINOv3 image processor resizes and normalizes each image.
- Each image needs its capture
date.
Run Inference
Both scripts take raw captions and image files, compute the Qwen3 and DINOv3 embeddings, place the images on the weekly grid and predict yield. --hf_model_dir accepts a local checkpoint folder (use . from the repository root) or a Hub repo ID. The model is trained on weather/soil text caption and UAS plot images with 29.8 bu/acre RMSE yield error at evaluation.
Single sample
python scripts/inference_single_hf.py \
--hf_model_dir ICICLE-AI/multimodal-yield-estimation \
--sample_json examples/sample_input.json \
--week 12 \
--output_json prediction.json
Batch
python scripts/inference_batch_hf.py \
--hf_model_dir ICICLE-AI/multimodal-yield-estimation \
--input_jsonl examples/sample_batch.jsonl \
--output_json predictions.json \
--output_csv predictions.csv
- The JSON output is a list with one object per sample, using the same fields as the single-sample output.
--output_csvis optional. - Each sample uses its own
week; pass--weekto use the same week for all samples.
| Option | Description |
|---|---|
--hf_model_dir |
Local checkpoint folder or Hub repo ID |
--sample_json / --input_jsonl |
Input file |
--week |
Cutoff week (1–22); overrides the sample's week |
--output_json |
Output JSON path (output folder must exist) |
--output_csv |
Batch only: optional CSV output |
--batch_size |
Batch only: model batch size (default 64) |
On a SLURM cluster
multimodal_yield_estimation.slurm runs both inference scripts on one GPU. Fill in --account, the conda paths and the repository path before submitting:
sbatch multimodal_yield_estimation.slurm
Load the Model from Python
The model uses custom code, so load it with trust_remote_code=True. The model takes embeddings, not raw text or images; use data/embeddings.py (as the inference scripts do) to build them. Run this from the repository root so data/ can be imported.
import torch
from transformers import AutoModel
from data.embeddings import ImageEmbedder, TextEmbedder, embed_samples
model = AutoModel.from_pretrained(
"ICICLE-AI/multimodal-yield-estimation", trust_remote_code=True
).eval()
x = embed_samples(
[sample], # dict in the input format above
TextEmbedder(model.config.text_model),
ImageEmbedder(model.config.image_model),
)
with torch.no_grad():
out = model(
text=torch.from_numpy(x["text"]),
image=torch.from_numpy(x["image"]),
img_mask=torch.from_numpy(x["img_mask"]),
img_age=torch.from_numpy(x["img_age"]),
cutoff=int(x["week"][0]),
)
print(out.predictions) # bu/acre
Model inputs:
| Input | Shape | Description |
|---|---|---|
text |
[B, T, 1024] |
Raw Qwen3 caption embeddings |
image |
[B, T, 768] |
Raw DINOv3 features (any value where img_mask is 0) |
img_mask |
[B, T] |
1 if an image is available at or before that week |
img_age |
[B, T] |
Weeks since that image was taken |
cutoff |
int | Number of weeks to use (defaults to T) |
Normalization statistics are stored in config.json, so inputs are passed un-normalized. The output has predictions (bu/acre) and logits (standardized yield).
Train on Your Own Data
Build an
.npzfile of embeddings with these arrays:Array Shape Description text[N, 22, 1024]float32Weekly Qwen3 caption embeddings image[N, 22, 768]float32DINOv3 feature of the latest image at or before each week (zeros where none) img_mask[N, 22]float321 if an image is available at or before that week img_age[N, 22]float32Weeks since that image was taken yield_[N]float32Yield (bu/acre) farm_field,plot,planting_date,row_key[N]Plot metadata The image arrays can be built with
images_to_weeks()indata/multimodal_dataset.py, which is the same function the inference scripts use.Train. Your data needs at least two
farm_fieldvalues; pass--test_field <name>to choose which one is held out.python scripts/train_multimodal_hf.py \ --data /path/to/embeddings.npz \ --out_dir checkpointsOther options:
--cutoffs,--epochs,--patience,--lr,--batch_size,--d_model,--num_layers,--dropout,--input_dropout,--seed.To train on SLURM, replace the inference commands in
multimodal_yield_estimation.slurmwith:python scripts/train_multimodal_hf.py --data /path/to/your.npz --out_dir checkpointsUse the new checkpoint by passing
--hf_model_dir checkpointsto the inference scripts.
Explanation
Problem Description
Given what is known about a corn plot up to a certain week of the season, estimate its final yield.
What is known each week is described in two ways:
- Text: a caption summarizing that week's weather and the plot's static soil, topography and treatment information.
- Image: UAS RGB images of the plot, taken on flight dates that do not line up with weeks (every 1–2 weeks during the season).
The model gives one yield estimate per plot for the requested cutoff week.
Placing Images on the Weekly Grid
The inference scripts do this automatically:
- Image week =
(date − planting_date).days // 7. Images before planting or after week 21 are ignored. - Each week uses the latest image taken at or before that week (forward-fill). If two images fall in the same week, the later one is used.
- Weeks before the first image have no image (
has_image = 0). - The scripts also compute the image age: the number of weeks since the image in use was taken (0 = new image that week).
Example (flights in weeks 2, 3, 4, 6, 7, 9, 11, 14 and 15):
week: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
has_image: 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
image_age: 0 0 0 0 0 1 0 0 1 0 1 0 1 2 0 0 1 2 3 4 5 6
Multi-Temporal Yield Estimation
- Each week after planting is represented by a Qwen3 embedding of that week's caption and a DINOv3 embedding of the most recent UAS image, concatenated with a has-image flag and the image age.
- The model reads tokens for weeks
0 … cutoff−1. A projection layer maps each token to 64 dimensions, then a week signal, a learned position embedding and a cutoff (horizon) embedding are added. - A 2-layer causal transformer encoder (4 heads) processes the sequence. The causal mask and forward-fill mean that no week sees later captions or images.
- The last token, together with the horizon embedding, goes to a small regression head.
- During training, the loss is averaged over all cutoffs (4, 8, 12, 16, 20, 22) for each batch, so one checkpoint serves every cutoff.
Features
- Text and image embeddings are concatenated per week.
- Inference starts from raw captions and UAS images: the scripts run Qwen3 and DINOv3, so no precomputed embeddings are needed.
- Missing weeks are handled with image forward-fill, a has-image flag and image age.
- One checkpoint predicts at any cutoff week from 1 to 22.
- The model is a standard Hugging Face
PreTrainedModeland loads withAutoModel.from_pretrained(..., trust_remote_code=True). - Single-sample and batch inference scripts write JSON (and optionally CSV) outputs.
Repository Structure
config.json model config, including normalization statistics
model.safetensors trained weights
configuration_multimodal.py HF config class (loaded by trust_remote_code)
modeling_multimodal.py HF model class (loaded by trust_remote_code)
multimodal_concat_tf.py temporal transformer module (loaded by trust_remote_code)
train_args.json arguments used to train the released checkpoint
hf/ same model code, imported by the local scripts
data/embeddings.py Qwen3 / DINOv3 embedding and input building
data/multimodal_dataset.py weekly image placement, dataset, split, normalization
scripts/inference_single_hf.py single-sample inference
scripts/inference_batch_hf.py batch inference (JSONL)
scripts/train_multimodal_hf.py training
examples/ sample_input.json, sample_batch.jsonl
multimodal_yield_estimation.slurm SLURM example
component-info.yaml ICICLE component catalog metadata
requirements.txt
LICENSE
Training Details
- Split: by field. One
farm_fieldis held out (--test_field, default: the last field in alphabetical order) and its plots are split 50/50 into validation and test; all other fields are used for training. - Loss: MSE on standardized yield, averaged over all training cutoffs for each batch.
- Defaults: AdamW, lr 3e-4, weight decay 1e-2, batch size 16, up to 300 epochs, early stopping on mean validation RMSE with patience 30.
- Checkpoint: the epoch with the best validation RMSE is saved in Hugging Face format to
--out_dir.
Evaluation
The training script evaluates the best checkpoint on the validation and test plots at every cutoff, and writes:
metrics.json: validation metrics at the best epochtest_metrics.json: validation and test RMSE, MAE, bias and R² per cutofftest_predictions.csv: per-plot predictions for the test plots