borjahernandez/wine-pricer-wine-pricer-modernbert-large-full-20260905-1229

wine-pricer · ModernBERT-large, full fine-tune

Predicts the price of a bottle of wine from its tasting note and a little geography. ModernBERT-large (395M) with a single-output regression head, fully fine-tuned on log1p(price).

Part of wine-pricer, which scores classical baselines, retrieval, RAG, QLoRA on Qwen2.5-3B, this model and Claude Opus 5 zero-shot on one fixed held-out set with the same evaluation harness.

Results

On the fixed 2,000-wine test split of borjahernandez/wine-pricer:

model	MAE	RMSLE	R²	hit rate*
This model	$12.43	0.367	62.2%	67.5%
Claude Opus 5, zero-shot	$13.31	0.390	45.3%	64.4%
Qwen2.5-3B, QLoRA	$13.57	0.425	54.5%	63.7%
TF-IDF + Ridge	$15.31	0.454	44.4%	58.9%

* within $10 or 20% of the true price.

The validation metrics further down (RMSLE 0.374) are from the validation split during training; the table above is the held-out test split.

Usage

The input is the prompt text up to and including Price is $, as built by the repo's Wine.test_prompt(). The model outputs log1p(price).

import numpy as np, torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "borjahernandez/wine-pricer-wine-pricer-modernbert-large-full-20260905-1229"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

text = "..."  # tasting note + geography, ending in "Price is $"
with torch.no_grad():
    log_price = model(**tok(text, return_tensors="pt", truncation=True, max_length=256)).logits.item()
print(np.expm1(np.clip(log_price, 0, np.log1p(1000))))

Limitations

• Trained on Wine Enthusiast reviews; prices are US retail at review time and are not inflation-adjusted.
• Weakest on expensive bottles (validation RMSLE on the expensive half: 0.421), as is every model in the project.
• Predictions are clipped to $0–$1,000.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 64
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 297
• num_epochs: 2

Training results

Training Loss	Epoch	Step	Validation Loss	Rmsle	Rmsle Expensive	Mae Dollars
0.5417	0.2016	500	0.2522	0.5022	0.4511	17.9103
0.4108	0.4032	1000	0.1945	0.4410	0.4667	15.8164
0.3416	0.6048	1500	0.1698	0.4121	0.4712	14.4232
0.3551	0.8065	2000	0.1579	0.3973	0.4417	13.9489
0.2930	1.0081	2500	0.1706	0.4131	0.4756	14.0374
0.2369	1.2097	3000	0.1499	0.3872	0.4420	13.4835
0.2258	1.4113	3500	0.1442	0.3797	0.4144	13.2492
0.2789	1.6129	4000	0.1418	0.3766	0.4270	13.0890
0.2483	1.8145	4500	0.1408	0.3752	0.4249	13.1100
0.2413	2.0	4960	0.1402	0.3744	0.4210	13.1054

Framework versions

• Transformers 5.16.1
• Pytorch 2.11.0+cu128
• Datasets 5.0.1
• Tokenizers 0.23.1