brezgis/lcc-metaphor-mdeberta

LCC metaphor classifier (mDeBERTa-v3, four languages)

A pair-level metaphor classifier. Given a sentence and a source and target word in it, it predicts whether that pair is used metaphorically ("the War on Poverty") or literally ("I pay more taxes"). It is microsoft/mdeberta-v3-base fine-tuned jointly on English, Spanish, Russian and Farsi from the LCC Metaphor Datasets (Mohler et al., 2016).

Code, data splits and the full set of experiments are at brezgis/metaphor-classifier-cosi114a, and the write-up is at brezgis.com.

Usage

The model has a small custom head, so load it with the included metaphor_pair.py:

from huggingface_hub import hf_hub_download
import importlib.util, sys

path = hf_hub_download("brezgis/lcc-metaphor-mdeberta", "metaphor_pair.py")
spec = importlib.util.spec_from_file_location("metaphor_pair", path)
mp = importlib.util.module_from_spec(spec); spec.loader.exec_module(mp)

model, tok = mp.load("brezgis/lcc-metaphor-mdeberta")
mp.predict(model, tok, [
    {"text": "Real enjoyment is at the heart of faith.", "source": "heart", "target": "faith"},
    {"text": "I work harder, get paid more, and pay more taxes.", "source": "pay", "target": "taxes"},
])
# -> [0.98, 0.002]   P(metaphor); call it a metaphor at >= 0.5

source and target can be substrings (first occurrence) or (start, end) character spans.

How it works

The pair is marked in the text as << source >> and [[ target ]], and the classifier reads the [CLS] vector together with the mean of the source tokens and the mean of the target tokens (a two-layer head on the concatenation). Training used the train splits of all four large LCC bundles, a class-weighted loss, AdamW (lr 2e-5, batch 32, linear warmup), max length 160, and 3 epochs, keeping the epoch with the best English development F1. This checkpoint is seed 0 of three.

Labels. LCC scores each pair 0–3 for metaphoricity (−1 = invalid relation). Pairs with mean score < 0 are dropped, and a pair counts as a metaphor if its mean score is ≥ 2.

Results

Test sets are split by document (English: the en_small test split). Metaphor is the positive class.

	this checkpoint (seed 0) F1	3 seeds, F1 mean ± sd	precision	recall	test pairs
English	0.826	0.827 ± 0.002	0.776	0.885	866
Spanish	0.740	0.737 ± 0.003	0.694	0.786	4,284
Russian	0.853	0.857 ± 0.003	0.833	0.883	2,199
Farsi	0.679	0.682 ± 0.003	0.618	0.760	2,652

Precision and recall are 3-seed means. For reference, one human annotator scored against another gets F1 0.66 / 0.60 / 0.82 / 0.39 (en / es / ru / fa) on doubly annotated pairs.

Limitations

• Domain. LCC is mostly web and political-forum text about government, taxes, guns, poverty and similar target concepts. Expect weaker results far from those topics.
• Farsi. The Farsi annotators barely agreed (Cohen's κ 0.12), so Farsi scores mean little.
• The pair must be given. The model judges a pair you point at; it does not find metaphors in running text, and it is not comparable to word-level benchmarks such as VUA.
• It leans metaphorical. Recall is higher than precision in every language.

License and citation

The training data, the LCC Metaphor Datasets, is released under CC BY-NC-SA 4.0, so these weights are shared under the same license (non-commercial, share-alike). The base model is MIT-licensed.

If you use the model, please cite the dataset:

@inproceedings{mohler-etal-2016-introducing,
  title     = {Introducing the {LCC} Metaphor Datasets},
  author    = {Mohler, Michael and Brunson, Mary and Rink, Bryan and Tomlinson, Marc},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation ({LREC}'16)},
  pages     = {4221--4227},
  year      = {2016}
}