Retire the BLEU pair that does not reproduce; add the strong-baseline and prompted-LLM comparison

050112d verified 2 months ago

8.78 kB

---
license: cc-by-sa-4.0
language:
  - en
base_model: facebook/bart-base
# No explicit pipeline_tag: "text2text-generation" is rejected by the Hub's current
# validation list (it renders a warning banner and mislabels the model as decoder-only
# "Text Generation"), and none of the accepted tags fit sentence simplification --
# "summarization" is wrong for a roughly length-preserving rewrite. Left unset so the
# Hub infers it from config.json's BartForConditionalGeneration instead.
tags:
  - text-simplification
  - accessibility
  - bart
datasets:
  - wikilarge
  - facebook/asset
metrics:
  - sari
  - bleu
model-index:
  - name: bart-base-wikilarge-simplification
    results:
      - task:
          type: text2text-generation
          name: Sentence-level text simplification
        dataset:
          name: ASSET (test split)
          type: facebook/asset
        metrics:
          - type: sari
            value: 37.80
          - type: bleu
            value: 88.30
          - type: fkgl
            value: 8.18
            name: FKGL (lower is better)
---
# bart-base-wikilarge-simplification

`facebook/bart-base` fine-tuned on **WikiLarge** for **sentence-level** English text
simplification. It rewrites one sentence at a time into simpler English while
preserving meaning.

Trained as part of a bachelor thesis on automated simplification of everyday English
web text, and served as the `local` model by the project's backend:
<https://github.com/yyvs/simple-website>

## Intended use

Simplifying a **single English sentence** of everyday prose. The companion project uses
it behind a browser extension that sentence-splits page text (`pysbd`) and sends each
sentence independently.

**Out of scope:** multi-sentence input, whole documents, languages other than English,
and any setting where a factual error would be harmful. This is a research artifact
from a student project — it hallucinates (see Limitations) and has had no human
evaluation.

For whole-document rewriting, use the companion model
[`yunvs/bart-base-dwikipedia-simplification-full`](https://huggingface.co/yunvs/bart-base-dwikipedia-simplification-full)
instead.

## Usage

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tok = AutoTokenizer.from_pretrained("yunvs/bart-base-wikilarge-simplification")
model = AutoModelForSeq2SeqLM.from_pretrained("yunvs/bart-base-wikilarge-simplification")
text = "Despite the multiplicity of antecedent conditions, the framework mandates uniform compliance."
inputs = tok(text, return_tensors="pt", truncation=True, max_length=64)
out = model.generate(**inputs, max_length=64, num_beams=4,
                     no_repeat_ngram_size=3, repetition_penalty=1.2)
print(tok.decode(out[0], skip_special_tokens=True))
```

**Use `max_length=64`.** That is the length the model was fine-tuned at, and generating
at BART's 512/1024 default degrades output. Input past 64 tokens is truncated, not
chunked — split into sentences first.
## Training
| | |
|---|---|
| Base model | `facebook/bart-base` |
| Dataset | WikiLarge, ~117k training pairs after filtering corrupted rows |
| Epochs | 5 (best checkpoint selected at epoch 4) |
| Batch size | 16 |
| Learning rate | 3e-5 |
| Weight decay | 0.01 |
| Max length | 64 tokens (WikiLarge p90 ≈ 52) |
| Seed | 42 |
| Selection | `load_best_model_at_end`, `metric_for_best_model="loss"`, early stopping (patience 4 quarter-epoch evals) |
| Final validation loss | 0.4568 |
Early stopping did not halt the run; epoch 5 (val loss 0.4590) did not improve on
epoch 4, and `load_best_model_at_end` selected epoch 4 correctly.
## Evaluation
Evaluated on the **ASSET** test split (359 sentences, 10 human references each) with
beam search (`num_beams=4`), against the un-fine-tuned `facebook/bart-base` as a
zero-shot baseline.

| | SARI ↑ | BLEU | FKGL ↓ | BERTScore | unchanged |
|---|---|---|---|---|---|
| **This model** | **37.80** | 88.30 | **8.18** | 97.65 | 11.7% |
| `facebook/bart-base` zero-shot | 21.39 | 91.64 | 10.02 | 98.58 | **93.9%** |

Paired bootstrap on the SARI improvement, 1000 resamples over sentence indices:
**ΔSARI +16.42, 95% CI [+15.38, +17.50]**, p < 0.001. The interval excluding zero by
fifteen points is the substantive claim; the p-value is the weaker statement.

Reproduced three times on different hardware (Apple Silicon MPS, and twice on an RTX
A5000 under a different torch build), agreeing to two decimals.

> ⚠️ **An earlier version of this card reported BLEU 67.26 / 89.89.** Those figures do
> not reproduce and have been retired. On generations giving identical SARI and FKGL,
> BLEU comes out 88.30 / 91.64 under three separate BLEU implementations
> (easse, sacrebleu`[13a]`, sacrebleu`[none]` → 85.31 — none near 67.26). The most
> likely cause is a reference-count mismatch in the original run: 67.26 lands near what
> **k=3** references produce. If you cited 67.26, use 88.30.

### Reading the BLEU number

**BLEU running opposite to SARI is expected, not a contradiction** — but note the gap is
small (3.34 points), not the 22-point chasm this card previously reported. BLEU is biased
toward conservative, low-edit output, and the zero-shot baseline is maximally
conservative: **93.9% of its outputs are byte-identical to the input.** It is being
rewarded for copying. SARI — which rewards appropriate edits against the source — and
FKGL both favour the fine-tuned model decisively (+16.42 SARI, −1.84 grade levels).

### Compared against a *strong* baseline, not just the un-fine-tuned one

The comparison above is against this model's own starting checkpoint, which is the
weakest available baseline. Measured on the same run, against an off-the-shelf
third-party simplifier and two prompted open-weight LLMs (3-shot, temperature 1.0,
three seeds each):

| Condition | Model | SARI ↑ | BLEU | FKGL ↓ | unchanged |
|---|---|---|---|---|---|
| zero-shot | `facebook/bart-base` | 21.39 | 91.64 | 10.02 | 93.9% |
| **this model** | `yunvs/bart-base-wikilarge-simplification` | **37.80** | 88.30 | 8.18 | 11.7% |
| off-the-shelf | `eilamc14/bart-large-text-simplification` | **38.17** | 88.58 | 7.85 | 13.4% |
| prompted | `qwen2.5:7b-instruct-q4_K_M` | 45.95 | 75.46 | 7.77 | 4.5% |
| prompted | `qwen2.5:3b-instruct-q4_K_M` | **47.03** | 68.21 | 8.22 | 1.2% |

⚠️ **Read this honestly: fine-tuning bought no measurable advantage over an
off-the-shelf simplifier** (37.80 vs 38.17, confidence intervals overlapping almost
entirely), and **both prompted LLMs beat it by 8–9 SARI.** The defensible claim for this
model is *efficiency*, not quality: it is a **bart-base** (139M parameters) matching a
**bart-large** roughly three times its size, and it runs locally on CPU without a
prompted-LLM sidecar. If output quality is what you need and you can run a 3B
instruction-tuned model, use that instead.

BLEU runs perfectly inverse to SARI across all five rows (91.64 → 88.58 → 88.30 → 75.46
→ 68.21 against SARI 21.39 → 38.17 → 37.80 → 45.95 → 47.03), which is the cleanest
demonstration of its unsuitability for this task that this project produced.

## Limitations

- **Hallucination.** Short or fragmentary inputs can produce unrelated text; the
  phrase `"Other websites"` (a Simple English Wikipedia section heading present in the
  training data) has been observed as output for several unrelated short inputs. The
  companion backend runs a quality guard that falls back to the original text in this
  case.
- **Conservative rewriting.** Observed output often drops a subordinate clause rather
  than rephrasing dense vocabulary — it does not reliably simplify words like
  "multiplicity" or "heterogeneous entities."
- **Outperformed by prompted LLMs and matched by an off-the-shelf model.** See
  Evaluation. Choose this model for size and local CPU inference, not for peak quality.
- **No human evaluation.** All reported numbers are automatic metrics. No readability,
  meaning-preservation, or accessibility study has been conducted.
- **Single sentences only.** Multi-sentence input is out of distribution and gets
  truncated at 64 tokens.
- **English only**, and trained on Wikipedia prose — encyclopedic register, with
  whatever demographic and topical bias that corpus carries.

## Training data provenance and licence

WikiLarge is derived from English Wikipedia and Simple English Wikipedia, which are
licensed **CC BY-SA**. This model is released under **CC BY-SA 4.0** accordingly.

WikiSmall was evaluated as an alternative and deliberately rejected: it carries
baked-in named-entity anonymization artifacts unsuitable for training.

## Citation

Produced for a bachelor thesis (2026). Please cite the project repository:
<https://github.com/yyvs/simple-website>

	---
	license: cc-by-sa-4.0
	language:
	- en
	base_model: facebook/bart-base
	# No explicit pipeline_tag: "text2text-generation" is rejected by the Hub's current
	# validation list (it renders a warning banner and mislabels the model as decoder-only
	# "Text Generation"), and none of the accepted tags fit sentence simplification --
	# "summarization" is wrong for a roughly length-preserving rewrite. Left unset so the
	# Hub infers it from config.json's BartForConditionalGeneration instead.
	tags:
	- text-simplification
	- accessibility
	- bart
	datasets:
	- wikilarge
	- facebook/asset
	metrics:
	- sari
	- bleu
	model-index:
	- name: bart-base-wikilarge-simplification
	results:
	- task:
	type: text2text-generation
	name: Sentence-level text simplification
	dataset:
	name: ASSET (test split)
	type: facebook/asset
	metrics:
	- type: sari
	value: 37.80
	- type: bleu
	value: 88.30
	- type: fkgl
	value: 8.18
	name: FKGL (lower is better)
	---

	# bart-base-wikilarge-simplification

	`facebook/bart-base` fine-tuned on WikiLarge for sentence-level English text
	simplification. It rewrites one sentence at a time into simpler English while
	preserving meaning.

	Trained as part of a bachelor thesis on automated simplification of everyday English
	web text, and served as the `local` model by the project's backend:
	<https://github.com/yyvs/simple-website>

	## Intended use

	Simplifying a single English sentence of everyday prose. The companion project uses
	it behind a browser extension that sentence-splits page text (`pysbd`) and sends each
	sentence independently.

	Out of scope: multi-sentence input, whole documents, languages other than English,
	and any setting where a factual error would be harmful. This is a research artifact
	from a student project — it hallucinates (see Limitations) and has had no human
	evaluation.

	For whole-document rewriting, use the companion model
	[`yunvs/bart-base-dwikipedia-simplification-full`](https://huggingface.co/yunvs/bart-base-dwikipedia-simplification-full)
	instead.

	## Usage

	```python
	from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

	tok = AutoTokenizer.from_pretrained("yunvs/bart-base-wikilarge-simplification")
	model = AutoModelForSeq2SeqLM.from_pretrained("yunvs/bart-base-wikilarge-simplification")

	text = "Despite the multiplicity of antecedent conditions, the framework mandates uniform compliance."
	inputs = tok(text, return_tensors="pt", truncation=True, max_length=64)
	out = model.generate(**inputs, max_length=64, num_beams=4,
	no_repeat_ngram_size=3, repetition_penalty=1.2)
	print(tok.decode(out[0], skip_special_tokens=True))
	```

	Use `max_length=64`. That is the length the model was fine-tuned at, and generating
	at BART's 512/1024 default degrades output. Input past 64 tokens is truncated, not
	chunked — split into sentences first.

	## Training

	\| \| \|
	\|---\|---\|
	\| Base model \| `facebook/bart-base` \|
	\| Dataset \| WikiLarge, ~117k training pairs after filtering corrupted rows \|
	\| Epochs \| 5 (best checkpoint selected at epoch 4) \|
	\| Batch size \| 16 \|
	\| Learning rate \| 3e-5 \|
	\| Weight decay \| 0.01 \|
	\| Max length \| 64 tokens (WikiLarge p90 ≈ 52) \|
	\| Seed \| 42 \|
	\| Selection \| `load_best_model_at_end`, `metric_for_best_model="loss"`, early stopping (patience 4 quarter-epoch evals) \|
	\| Final validation loss \| 0.4568 \|

	Early stopping did not halt the run; epoch 5 (val loss 0.4590) did not improve on
	epoch 4, and `load_best_model_at_end` selected epoch 4 correctly.

	## Evaluation

	Evaluated on the ASSET test split (359 sentences, 10 human references each) with
	beam search (`num_beams=4`), against the un-fine-tuned `facebook/bart-base` as a
	zero-shot baseline.

	\| \| SARI ↑ \| BLEU \| FKGL ↓ \| BERTScore \| unchanged \|
	\|---\|---\|---\|---\|---\|---\|
	\| This model \| 37.80 \| 88.30 \| 8.18 \| 97.65 \| 11.7% \|
	\| `facebook/bart-base` zero-shot \| 21.39 \| 91.64 \| 10.02 \| 98.58 \| 93.9% \|

	Paired bootstrap on the SARI improvement, 1000 resamples over sentence indices:
	ΔSARI +16.42, 95% CI [+15.38, +17.50], p < 0.001. The interval excluding zero by
	fifteen points is the substantive claim; the p-value is the weaker statement.

	Reproduced three times on different hardware (Apple Silicon MPS, and twice on an RTX
	A5000 under a different torch build), agreeing to two decimals.

	> ⚠️ An earlier version of this card reported BLEU 67.26 / 89.89. Those figures do
	> not reproduce and have been retired. On generations giving identical SARI and FKGL,
	> BLEU comes out 88.30 / 91.64 under three separate BLEU implementations
	> (easse, sacrebleu`[13a]`, sacrebleu`[none]` → 85.31 — none near 67.26). The most
	> likely cause is a reference-count mismatch in the original run: 67.26 lands near what
	> k=3 references produce. If you cited 67.26, use 88.30.

	### Reading the BLEU number

	BLEU running opposite to SARI is expected, not a contradiction — but note the gap is
	small (3.34 points), not the 22-point chasm this card previously reported. BLEU is biased
	toward conservative, low-edit output, and the zero-shot baseline is maximally
	conservative: 93.9% of its outputs are byte-identical to the input. It is being
	rewarded for copying. SARI — which rewards appropriate edits against the source — and
	FKGL both favour the fine-tuned model decisively (+16.42 SARI, −1.84 grade levels).

	### Compared against a strong baseline, not just the un-fine-tuned one

	The comparison above is against this model's own starting checkpoint, which is the
	weakest available baseline. Measured on the same run, against an off-the-shelf
	third-party simplifier and two prompted open-weight LLMs (3-shot, temperature 1.0,
	three seeds each):

	\| Condition \| Model \| SARI ↑ \| BLEU \| FKGL ↓ \| unchanged \|
	\|---\|---\|---\|---\|---\|---\|
	\| zero-shot \| `facebook/bart-base` \| 21.39 \| 91.64 \| 10.02 \| 93.9% \|
	\| this model \| `yunvs/bart-base-wikilarge-simplification` \| 37.80 \| 88.30 \| 8.18 \| 11.7% \|
	\| off-the-shelf \| `eilamc14/bart-large-text-simplification` \| 38.17 \| 88.58 \| 7.85 \| 13.4% \|
	\| prompted \| `qwen2.5:7b-instruct-q4_K_M` \| 45.95 \| 75.46 \| 7.77 \| 4.5% \|
	\| prompted \| `qwen2.5:3b-instruct-q4_K_M` \| 47.03 \| 68.21 \| 8.22 \| 1.2% \|

	⚠️ **Read this honestly: fine-tuning bought no measurable advantage over an
	off-the-shelf simplifier** (37.80 vs 38.17, confidence intervals overlapping almost
	entirely), and both prompted LLMs beat it by 8–9 SARI. The defensible claim for this
	model is efficiency, not quality: it is a bart-base (139M parameters) matching a
	bart-large roughly three times its size, and it runs locally on CPU without a
	prompted-LLM sidecar. If output quality is what you need and you can run a 3B
	instruction-tuned model, use that instead.

	BLEU runs perfectly inverse to SARI across all five rows (91.64 → 88.58 → 88.30 → 75.46
	→ 68.21 against SARI 21.39 → 38.17 → 37.80 → 45.95 → 47.03), which is the cleanest
	demonstration of its unsuitability for this task that this project produced.

	## Limitations

	- Hallucination. Short or fragmentary inputs can produce unrelated text; the
	phrase `"Other websites"` (a Simple English Wikipedia section heading present in the
	training data) has been observed as output for several unrelated short inputs. The
	companion backend runs a quality guard that falls back to the original text in this
	case.
	- Conservative rewriting. Observed output often drops a subordinate clause rather
	than rephrasing dense vocabulary — it does not reliably simplify words like
	"multiplicity" or "heterogeneous entities."
	- Outperformed by prompted LLMs and matched by an off-the-shelf model. See
	Evaluation. Choose this model for size and local CPU inference, not for peak quality.
	- No human evaluation. All reported numbers are automatic metrics. No readability,
	meaning-preservation, or accessibility study has been conducted.
	- Single sentences only. Multi-sentence input is out of distribution and gets
	truncated at 64 tokens.
	- English only, and trained on Wikipedia prose — encyclopedic register, with
	whatever demographic and topical bias that corpus carries.

	## Training data provenance and licence

	WikiLarge is derived from English Wikipedia and Simple English Wikipedia, which are
	licensed CC BY-SA. This model is released under CC BY-SA 4.0 accordingly.

	WikiSmall was evaluated as an alternative and deliberately rejected: it carries
	baked-in named-entity anonymization artifacts unsuitable for training.

	## Citation

	Produced for a bachelor thesis (2026). Please cite the project repository:
	<https://github.com/yyvs/simple-website>