README.md

metadata

title: ROUGE (rouge-score-rs)
emoji: 🦀
colorFrom: blue
colorTo: red
sdk: static
app_file: index.html
pinned: false
license: apache-2.0
tags:
  - evaluate
  - metric
description: >-
  ROUGE computed with rouge-score-rs, a Rust implementation of Google's
  rouge-score. Same inputs, options and scores as the `rouge` metric, computed
  faster.

Metric Card for ROUGE (rouge-score-rs)

Metric Description

ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, is a set of metrics and a software package used for evaluating automatic summarization and machine translation software in natural language processing. The metrics compare an automatically produced summary or translation against a reference or a set of references (human-produced) summary or translation.

Note that ROUGE is case insensitive, meaning that upper case letters are treated the same way as lower case letters.

This metric has the same inputs, options and outputs as the rouge metric and, for the inputs listed under Compatibility, returns identical scores. It computes them with rouge-score-rs (source), a Rust implementation of the Google Research reimplementation of ROUGE, instead of the pure Python rouge-score package.

This Space is maintained by SyntaxSpirits and is not affiliated with Google or Hugging Face. It is derived from the Apache-2.0 evaluate-metric/rouge Space by the HuggingFace Evaluate Authors; see LICENSE.

Install the dependencies before loading the metric (tested with rouge-score-rs 0.1.1):

pip install evaluate 'rouge-score-rs[aggregate]>=0.1.1'

How to Use

At minimum, this metric takes as input a list of predictions and a list of references:

>>> import evaluate
>>> rouge = evaluate.load('SyntaxSpirits/rouge')
>>> predictions = ["hello there", "general kenobi"]
>>> references = ["hello there", "general kenobi"]
>>> results = rouge.compute(predictions=predictions,
...                         references=references)
>>> print({key: round(value, 4) for key, value in results.items()})
{'rouge1': 1.0, 'rouge2': 1.0, 'rougeL': 1.0, 'rougeLsum': 1.0}

One can also pass a custom tokenizer, which is especially useful for non-Latin languages: the default tokenizer keeps only the ASCII letters a–z and digits.

>>> results = rouge.compute(predictions=predictions,
...                         references=references,
...                         tokenizer=lambda x: x.split())
>>> print({key: round(value, 4) for key, value in results.items()})
{'rouge1': 1.0, 'rouge2': 1.0, 'rougeL': 1.0, 'rougeLsum': 1.0}

It can also deal with lists of references for each prediction. For every ROUGE type, the reference with the highest F1 is used:

>>> rouge = evaluate.load('SyntaxSpirits/rouge', seed=0)
>>> predictions = ["hello there", "general kenobi"]
>>> references = [["hello", "there"], ["general kenobi", "general yoda"]]
>>> results = rouge.compute(predictions=predictions,
...                         references=references)
>>> print({key: round(value, 4) for key, value in results.items()})
{'rouge1': 0.8333, 'rouge2': 0.5, 'rougeL': 0.8333, 'rougeLsum': 0.8333}

Inputs

• predictions (list of str): predictions to score.
• references (list of str or list of list of str): one reference per prediction, or several references per prediction.
• rouge_types (list): ROUGE types to calculate. Defaults to ['rouge1', 'rouge2', 'rougeL', 'rougeLsum'].
  • "rouge1" to "rouge9": n-gram based scoring, e.g. "rouge2" for bigrams.
  • "rougeL": longest common subsequence based scoring.
  • "rougeLsum": summary-level longest common subsequence; sentences are separated by "\n". See here for more information.
• use_aggregator (boolean): if True, returns aggregated scores. Defaults to True.
• use_stemmer (boolean): if True, the default tokenizer applies the Porter stemmer to tokens longer than three characters. Defaults to False.
• tokenizer (callable, optional): a function that splits a string into a list of string tokens. Lowercasing and stemming are done only by the default tokenizer.

Output Values

The output is a dictionary with one entry for each ROUGE type in rouge_types; each value is an F1 score between 0 and 1.

If use_aggregator=True (the default), each entry is the median of 1,000 bootstrap means of the per-prediction F1 scores, as in the rouge metric. The bootstrap uses NumPy's random state; pass seed to evaluate.load to make it reproducible and comparable with the rouge metric loaded with the same seed.

If use_aggregator=False, each entry is a list with one F1 score per prediction.

Examples

An example without aggregation:

>>> rouge = evaluate.load('SyntaxSpirits/rouge')
>>> predictions = ["hello goodbye", "ankh morpork"]
>>> references = ["goodbye", "general kenobi"]
>>> results = rouge.compute(predictions=predictions,
...                         references=references,
...                         use_aggregator=False)
>>> print(list(results.keys()))
['rouge1', 'rouge2', 'rougeL', 'rougeLsum']
>>> print([round(score, 4) for score in results["rouge1"]])
[0.6667, 0.0]

The same example, but with aggregation:

>>> rouge = evaluate.load('SyntaxSpirits/rouge', seed=0)
>>> results = rouge.compute(predictions=predictions,
...                         references=references,
...                         use_aggregator=True)
>>> print(round(results["rouge1"], 4))
0.3333

The same example, but only calculating rouge1:

>>> results = rouge.compute(predictions=predictions,
...                         references=references,
...                         rouge_types=['rouge1'])
>>> print({key: round(value, 4) for key, value in results.items()})
{'rouge1': 0.3333}

Compatibility

Scores are bit-for-bit identical to the rouge metric for string inputs with the default tokenizer, with or without stemming, and with custom tokenizers that return lists of strings. This is checked against rouge-score 0.1.2 by the rouge-score-rs test suite and, for this metric, by comparing both metrics with the same seed on thousands of random examples.

Differences from the rouge metric:

• Tokens returned by a custom tokenizer must be hashable, such as strings.
• An unknown ROUGE type raises ValueError when the scorer is created, and malformed inputs (for example None instead of a string) can raise a different exception type.

Speed

Pair scoring runs in Rust. With the default tokenizer and one reference per prediction, all pairs are scored in one call that uses every CPU core unless RAYON_NUM_THREADS limits it (in a process forked after such a call, pairs are scored on one thread). Aggregation still runs the same NumPy bootstrap as the rouge metric, so it benefits less.

Limitations and Bias

See Schluter (2017) for an in-depth discussion of many of ROUGE's limits.

Citation

@inproceedings{lin-2004-rouge,
    title = "{ROUGE}: A Package for Automatic Evaluation of Summaries",
    author = "Lin, Chin-Yew",
    booktitle = "Text Summarization Branches Out",
    month = jul,
    year = "2004",
    address = "Barcelona, Spain",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/W04-1013",
    pages = "74--81",
}

Further References