Spaces:

SyntaxSpirits
/

rouge

Running

App Files Files Community

rouge / README.md

kholdrex

ROUGE metric computed with rouge-score-rs

ba9518e verified 21 days ago

preview code

Raw History Blame Contribute Delete

7.83 kB

---
title: ROUGE (rouge-score-rs)
emoji: 🦀
colorFrom: blue
colorTo: red
sdk: static
app_file: index.html
pinned: false
license: apache-2.0
tags:
- evaluate
- metric
description: >-
  ROUGE computed with rouge-score-rs, a Rust implementation of Google's rouge-score.
  Same inputs, options and scores as the `rouge` metric, computed faster.
---
# Metric Card for ROUGE (rouge-score-rs)

## Metric Description
ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, is a set of metrics and a software package used for evaluating automatic summarization and machine translation software in natural language processing. The metrics compare an automatically produced summary or translation against a reference or a set of references (human-produced) summary or translation.

Note that ROUGE is case insensitive, meaning that upper case letters are treated the same way as lower case letters.

This metric has the same inputs, options and outputs as the [`rouge`](https://huggingface.co/spaces/evaluate-metric/rouge) metric and, for the inputs listed under [Compatibility](#compatibility), returns identical scores. It computes them with [`rouge-score-rs`](https://pypi.org/project/rouge-score-rs/) ([source](https://github.com/SyntaxSpirits/rougers)), a Rust implementation of the [Google Research reimplementation of ROUGE](https://github.com/google-research/google-research/tree/master/rouge), instead of the pure Python `rouge-score` package.

This Space is maintained by SyntaxSpirits and is not affiliated with Google or Hugging Face. It is derived from the Apache-2.0 [`evaluate-metric/rouge`](https://huggingface.co/spaces/evaluate-metric/rouge) Space by the HuggingFace Evaluate Authors; see [LICENSE](LICENSE).

Install the dependencies before loading the metric (tested with `rouge-score-rs` 0.1.1):

```bash
pip install evaluate 'rouge-score-rs[aggregate]>=0.1.1'
```

## How to Use
At minimum, this metric takes as input a list of predictions and a list of references:
```python
>>> import evaluate
>>> rouge = evaluate.load('SyntaxSpirits/rouge')
>>> predictions = ["hello there", "general kenobi"]
>>> references = ["hello there", "general kenobi"]
>>> results = rouge.compute(predictions=predictions,
...                         references=references)
>>> print({key: round(value, 4) for key, value in results.items()})
{'rouge1': 1.0, 'rouge2': 1.0, 'rougeL': 1.0, 'rougeLsum': 1.0}
```

One can also pass a custom tokenizer, which is especially useful for non-Latin languages: the default tokenizer keeps only the ASCII letters `a`–`z` and digits.
```python
>>> results = rouge.compute(predictions=predictions,
...                         references=references,
...                         tokenizer=lambda x: x.split())
>>> print({key: round(value, 4) for key, value in results.items()})
{'rouge1': 1.0, 'rouge2': 1.0, 'rougeL': 1.0, 'rougeLsum': 1.0}
```

It can also deal with lists of references for each prediction. For every ROUGE type, the reference with the highest F1 is used:
```python
>>> rouge = evaluate.load('SyntaxSpirits/rouge', seed=0)
>>> predictions = ["hello there", "general kenobi"]
>>> references = [["hello", "there"], ["general kenobi", "general yoda"]]
>>> results = rouge.compute(predictions=predictions,
...                         references=references)
>>> print({key: round(value, 4) for key, value in results.items()})
{'rouge1': 0.8333, 'rouge2': 0.5, 'rougeL': 0.8333, 'rougeLsum': 0.8333}
```

### Inputs
- **predictions** (`list` of `str`): predictions to score.
- **references** (`list` of `str` or `list` of `list` of `str`): one reference per prediction, or several references per prediction.
- **rouge_types** (`list`): ROUGE types to calculate. Defaults to `['rouge1', 'rouge2', 'rougeL', 'rougeLsum']`.
    - `"rouge1"` to `"rouge9"`: n-gram based scoring, e.g. `"rouge2"` for bigrams.
    - `"rougeL"`: longest common subsequence based scoring.
    - `"rougeLsum"`: summary-level longest common subsequence; sentences are separated by `"\n"`. See [here](https://github.com/huggingface/datasets/issues/617) for more information.
- **use_aggregator** (`boolean`): if `True`, returns aggregated scores. Defaults to `True`.
- **use_stemmer** (`boolean`): if `True`, the default tokenizer applies the Porter stemmer to tokens longer than three characters. Defaults to `False`.
- **tokenizer** (callable, optional): a function that splits a string into a list of string tokens. Lowercasing and stemming are done only by the default tokenizer.
### Output Values
The output is a dictionary with one entry for each ROUGE type in `rouge_types`; each value is an F1 score between 0 and 1.
If `use_aggregator=True` (the default), each entry is the median of 1,000 bootstrap means of the per-prediction F1 scores, as in the `rouge` metric. The bootstrap uses NumPy's random state; pass `seed` to `evaluate.load` to make it reproducible and comparable with the `rouge` metric loaded with the same seed.
If `use_aggregator=False`, each entry is a list with one F1 score per prediction.
### Examples
An example without aggregation:
```python
>>> rouge = evaluate.load('SyntaxSpirits/rouge')
>>> predictions = ["hello goodbye", "ankh morpork"]
>>> references = ["goodbye", "general kenobi"]
>>> results = rouge.compute(predictions=predictions,
...                         references=references,
...                         use_aggregator=False)
>>> print(list(results.keys()))
['rouge1', 'rouge2', 'rougeL', 'rougeLsum']
>>> print([round(score, 4) for score in results["rouge1"]])
[0.6667, 0.0]
```
The same example, but with aggregation:
```python
>>> rouge = evaluate.load('SyntaxSpirits/rouge', seed=0)
>>> results = rouge.compute(predictions=predictions,
...                         references=references,
...                         use_aggregator=True)
>>> print(round(results["rouge1"], 4))
0.3333
```
The same example, but only calculating `rouge1`:
```python
>>> results = rouge.compute(predictions=predictions,
...                         references=references,
...                         rouge_types=['rouge1'])
>>> print({key: round(value, 4) for key, value in results.items()})
{'rouge1': 0.3333}
```
## Compatibility
Scores are bit-for-bit identical to the `rouge` metric for string inputs with the default tokenizer, with or without stemming, and with custom tokenizers that return lists of strings. This is checked against `rouge-score` 0.1.2 by the [rouge-score-rs test suite](https://github.com/SyntaxSpirits/rougers#compatibility) and, for this metric, by comparing both metrics with the same seed on thousands of random examples.
Differences from the `rouge` metric:
- Tokens returned by a custom tokenizer must be hashable, such as strings.
- An unknown ROUGE type raises `ValueError` when the scorer is created, and malformed inputs (for example `None` instead of a string) can raise a different exception type.
## Speed
Pair scoring runs in Rust. With the default tokenizer and one reference per prediction, all pairs are scored in one call that uses every CPU core unless `RAYON_NUM_THREADS` limits it (in a process forked after such a call, pairs are scored on one thread). Aggregation still runs the same NumPy bootstrap as the `rouge` metric, so it benefits less.
## Limitations and Bias
See [Schluter (2017)](https://aclanthology.org/E17-2007/) for an in-depth discussion of many of ROUGE's limits.
## Citation
```bibtex
@inproceedings{lin-2004-rouge,
    title = "{ROUGE}: A Package for Automatic Evaluation of Summaries",
    author = "Lin, Chin-Yew",
    booktitle = "Text Summarization Branches Out",
    month = jul,
    year = "2004",
    address = "Barcelona, Spain",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/W04-1013",
    pages = "74--81",
}
```
## Further References

	---
	title: ROUGE (rouge-score-rs)
	emoji: 🦀
	colorFrom: blue
	colorTo: red
	sdk: static
	app_file: index.html
	pinned: false
	license: apache-2.0
	tags:
	- evaluate
	- metric
	description: >-
	ROUGE computed with rouge-score-rs, a Rust implementation of Google's rouge-score.
	Same inputs, options and scores as the `rouge` metric, computed faster.
	---

	# Metric Card for ROUGE (rouge-score-rs)

	## Metric Description
	ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, is a set of metrics and a software package used for evaluating automatic summarization and machine translation software in natural language processing. The metrics compare an automatically produced summary or translation against a reference or a set of references (human-produced) summary or translation.

	Note that ROUGE is case insensitive, meaning that upper case letters are treated the same way as lower case letters.

	This metric has the same inputs, options and outputs as the [`rouge`](https://huggingface.co/spaces/evaluate-metric/rouge) metric and, for the inputs listed under [Compatibility](#compatibility), returns identical scores. It computes them with [`rouge-score-rs`](https://pypi.org/project/rouge-score-rs/) ([source](https://github.com/SyntaxSpirits/rougers)), a Rust implementation of the [Google Research reimplementation of ROUGE](https://github.com/google-research/google-research/tree/master/rouge), instead of the pure Python `rouge-score` package.

	This Space is maintained by SyntaxSpirits and is not affiliated with Google or Hugging Face. It is derived from the Apache-2.0 [`evaluate-metric/rouge`](https://huggingface.co/spaces/evaluate-metric/rouge) Space by the HuggingFace Evaluate Authors; see [LICENSE](LICENSE).

	Install the dependencies before loading the metric (tested with `rouge-score-rs` 0.1.1):

	```bash
	pip install evaluate 'rouge-score-rs[aggregate]>=0.1.1'
	```

	## How to Use
	At minimum, this metric takes as input a list of predictions and a list of references:
	```python
	>>> import evaluate
	>>> rouge = evaluate.load('SyntaxSpirits/rouge')
	>>> predictions = ["hello there", "general kenobi"]
	>>> references = ["hello there", "general kenobi"]
	>>> results = rouge.compute(predictions=predictions,
	... references=references)
	>>> print({key: round(value, 4) for key, value in results.items()})
	{'rouge1': 1.0, 'rouge2': 1.0, 'rougeL': 1.0, 'rougeLsum': 1.0}
	```

	One can also pass a custom tokenizer, which is especially useful for non-Latin languages: the default tokenizer keeps only the ASCII letters `a`–`z` and digits.
	```python
	>>> results = rouge.compute(predictions=predictions,
	... references=references,
	... tokenizer=lambda x: x.split())
	>>> print({key: round(value, 4) for key, value in results.items()})
	{'rouge1': 1.0, 'rouge2': 1.0, 'rougeL': 1.0, 'rougeLsum': 1.0}
	```

	It can also deal with lists of references for each prediction. For every ROUGE type, the reference with the highest F1 is used:
	```python
	>>> rouge = evaluate.load('SyntaxSpirits/rouge', seed=0)
	>>> predictions = ["hello there", "general kenobi"]
	>>> references = [["hello", "there"], ["general kenobi", "general yoda"]]
	>>> results = rouge.compute(predictions=predictions,
	... references=references)
	>>> print({key: round(value, 4) for key, value in results.items()})
	{'rouge1': 0.8333, 'rouge2': 0.5, 'rougeL': 0.8333, 'rougeLsum': 0.8333}
	```

	### Inputs
	- predictions (`list` of `str`): predictions to score.
	- references (`list` of `str` or `list` of `list` of `str`): one reference per prediction, or several references per prediction.
	- rouge_types (`list`): ROUGE types to calculate. Defaults to `['rouge1', 'rouge2', 'rougeL', 'rougeLsum']`.
	- `"rouge1"` to `"rouge9"`: n-gram based scoring, e.g. `"rouge2"` for bigrams.
	- `"rougeL"`: longest common subsequence based scoring.
	- `"rougeLsum"`: summary-level longest common subsequence; sentences are separated by `"\n"`. See [here](https://github.com/huggingface/datasets/issues/617) for more information.
	- use_aggregator (`boolean`): if `True`, returns aggregated scores. Defaults to `True`.
	- use_stemmer (`boolean`): if `True`, the default tokenizer applies the Porter stemmer to tokens longer than three characters. Defaults to `False`.
	- tokenizer (callable, optional): a function that splits a string into a list of string tokens. Lowercasing and stemming are done only by the default tokenizer.

	### Output Values
	The output is a dictionary with one entry for each ROUGE type in `rouge_types`; each value is an F1 score between 0 and 1.

	If `use_aggregator=True` (the default), each entry is the median of 1,000 bootstrap means of the per-prediction F1 scores, as in the `rouge` metric. The bootstrap uses NumPy's random state; pass `seed` to `evaluate.load` to make it reproducible and comparable with the `rouge` metric loaded with the same seed.

	If `use_aggregator=False`, each entry is a list with one F1 score per prediction.

	### Examples
	An example without aggregation:
	```python
	>>> rouge = evaluate.load('SyntaxSpirits/rouge')
	>>> predictions = ["hello goodbye", "ankh morpork"]
	>>> references = ["goodbye", "general kenobi"]
	>>> results = rouge.compute(predictions=predictions,
	... references=references,
	... use_aggregator=False)
	>>> print(list(results.keys()))
	['rouge1', 'rouge2', 'rougeL', 'rougeLsum']
	>>> print([round(score, 4) for score in results["rouge1"]])
	[0.6667, 0.0]
	```

	The same example, but with aggregation:
	```python
	>>> rouge = evaluate.load('SyntaxSpirits/rouge', seed=0)
	>>> results = rouge.compute(predictions=predictions,
	... references=references,
	... use_aggregator=True)
	>>> print(round(results["rouge1"], 4))
	0.3333
	```

	The same example, but only calculating `rouge1`:
	```python
	>>> results = rouge.compute(predictions=predictions,
	... references=references,
	... rouge_types=['rouge1'])
	>>> print({key: round(value, 4) for key, value in results.items()})
	{'rouge1': 0.3333}
	```

	## Compatibility
	Scores are bit-for-bit identical to the `rouge` metric for string inputs with the default tokenizer, with or without stemming, and with custom tokenizers that return lists of strings. This is checked against `rouge-score` 0.1.2 by the [rouge-score-rs test suite](https://github.com/SyntaxSpirits/rougers#compatibility) and, for this metric, by comparing both metrics with the same seed on thousands of random examples.

	Differences from the `rouge` metric:
	- Tokens returned by a custom tokenizer must be hashable, such as strings.
	- An unknown ROUGE type raises `ValueError` when the scorer is created, and malformed inputs (for example `None` instead of a string) can raise a different exception type.

	## Speed
	Pair scoring runs in Rust. With the default tokenizer and one reference per prediction, all pairs are scored in one call that uses every CPU core unless `RAYON_NUM_THREADS` limits it (in a process forked after such a call, pairs are scored on one thread). Aggregation still runs the same NumPy bootstrap as the `rouge` metric, so it benefits less.

	## Limitations and Bias
	See [Schluter (2017)](https://aclanthology.org/E17-2007/) for an in-depth discussion of many of ROUGE's limits.

	## Citation
	```bibtex
	@inproceedings{lin-2004-rouge,
	title = "{ROUGE}: A Package for Automatic Evaluation of Summaries",
	author = "Lin, Chin-Yew",
	booktitle = "Text Summarization Branches Out",
	month = jul,
	year = "2004",
	address = "Barcelona, Spain",
	publisher = "Association for Computational Linguistics",
	url = "https://www.aclweb.org/anthology/W04-1013",
	pages = "74--81",
	}
	```

	## Further References