Blame for README.md

---
license: cc-by-sa-4.0
language:
  - en
base_model: facebook/bart-base
# No explicit pipeline_tag: "text2text-generation" is rejected by the Hub's current
# validation list (it renders a warning banner and mislabels the model as decoder-only
# "Text Generation"), and none of the accepted tags fit sentence simplification --
# "summarization" is wrong for a roughly length-preserving rewrite. Left unset so the
# Hub infers it from config.json's BartForConditionalGeneration instead.
tags:
  - text-simplification
  - accessibility
  - bart
datasets:
  - wikilarge
  - facebook/asset
metrics:
  - sari
  - bleu
model-index:
  - name: bart-base-wikilarge-simplification
    results:
      - task:
          type: text2text-generation
          name: Sentence-level text simplification
        dataset:
          name: ASSET (test split)
          type: facebook/asset
        metrics:
          - type: sari
            value: 37.80
          - type: bleu
            value: 88.30
          - type: fkgl
            value: 8.18
            name: FKGL (lower is better)
---

# bart-base-wikilarge-simplification

`facebook/bart-base` fine-tuned on **WikiLarge** for **sentence-level** English text
simplification. It rewrites one sentence at a time into simpler English while
preserving meaning.

Trained as part of a bachelor thesis on automated simplification of everyday English
web text, and served as the `local` model by the project's backend:
<https://github.com/yyvs/simple-website>

## Intended use

Simplifying a **single English sentence** of everyday prose. The companion project uses
it behind a browser extension that sentence-splits page text (`pysbd`) and sends each
sentence independently.

**Out of scope:** multi-sentence input, whole documents, languages other than English,
and any setting where a factual error would be harmful. This is a research artifact
from a student project — it hallucinates (see Limitations) and has had no human
evaluation.

For whole-document rewriting, use the companion model
[`yunvs/bart-base-dwikipedia-simplification-full`](https://huggingface.co/yunvs/bart-base-dwikipedia-simplification-full)
instead.

## Usage

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tok = AutoTokenizer.from_pretrained("yunvs/bart-base-wikilarge-simplification")
model = AutoModelForSeq2SeqLM.from_pretrained("yunvs/bart-base-wikilarge-simplification")

text = "Despite the multiplicity of antecedent conditions, the framework mandates uniform compliance."
inputs = tok(text, return_tensors="pt", truncation=True, max_length=64)
out = model.generate(**inputs, max_length=64, num_beams=4,
                     no_repeat_ngram_size=3, repetition_penalty=1.2)
print(tok.decode(out[0], skip_special_tokens=True))
```

**Use `max_length=64`.** That is the length the model was fine-tuned at, and generating
at BART's 512/1024 default degrades output. Input past 64 tokens is truncated, not
chunked — split into sentences first.

## Training

| | |
|---|---|
| Base model | `facebook/bart-base` |
| Dataset | WikiLarge, ~117k training pairs after filtering corrupted rows |
| Epochs | 5 (best checkpoint selected at epoch 4) |
| Batch size | 16 |
| Learning rate | 3e-5 |
| Weight decay | 0.01 |
| Max length | 64 tokens (WikiLarge p90 ≈ 52) |
| Seed | 42 |
| Selection | `load_best_model_at_end`, `metric_for_best_model="loss"`, early stopping (patience 4 quarter-epoch evals) |
| Final validation loss | 0.4568 |

Early stopping did not halt the run; epoch 5 (val loss 0.4590) did not improve on
epoch 4, and `load_best_model_at_end` selected epoch 4 correctly.

## Evaluation

Evaluated on the **ASSET** test split (359 sentences, 10 human references each) with
beam search (`num_beams=4`), against the un-fine-tuned `facebook/bart-base` as a
zero-shot baseline.

| | SARI ↑ | BLEU | FKGL ↓ | BERTScore | unchanged |
|---|---|---|---|---|---|
| **This model** | **37.80** | 88.30 | **8.18** | 97.65 | 11.7% |
| `facebook/bart-base` zero-shot | 21.39 | 91.64 | 10.02 | 98.58 | **93.9%** |

Paired bootstrap on the SARI improvement, 1000 resamples over sentence indices:
**ΔSARI +16.42, 95% CI [+15.38, +17.50]**, p < 0.001. The interval excluding zero by
fifteen points is the substantive claim; the p-value is the weaker statement.

Reproduced three times on different hardware (Apple Silicon MPS, and twice on an RTX
A5000 under a different torch build), agreeing to two decimals.

> ⚠️ **An earlier version of this card reported BLEU 67.26 / 89.89.** Those figures do
> not reproduce and have been retired. On generations giving identical SARI and FKGL,
> BLEU comes out 88.30 / 91.64 under three separate BLEU implementations
> (easse, sacrebleu`[13a]`, sacrebleu`[none]` → 85.31 — none near 67.26). The most
> likely cause is a reference-count mismatch in the original run: 67.26 lands near what
> **k=3** references produce. If you cited 67.26, use 88.30.

### Reading the BLEU number

**BLEU running opposite to SARI is expected, not a contradiction** — but note the gap is
small (3.34 points), not the 22-point chasm this card previously reported. BLEU is biased
toward conservative, low-edit output, and the zero-shot baseline is maximally
conservative: **93.9% of its outputs are byte-identical to the input.** It is being
rewarded for copying. SARI — which rewards appropriate edits against the source — and
FKGL both favour the fine-tuned model decisively (+16.42 SARI, −1.84 grade levels).

### Compared against a *strong* baseline, not just the un-fine-tuned one

The comparison above is against this model's own starting checkpoint, which is the
weakest available baseline. Measured on the same run, against an off-the-shelf
third-party simplifier and two prompted open-weight LLMs (3-shot, temperature 1.0,
three seeds each):

| Condition | Model | SARI ↑ | BLEU | FKGL ↓ | unchanged |
|---|---|---|---|---|---|
| zero-shot | `facebook/bart-base` | 21.39 | 91.64 | 10.02 | 93.9% |
| **this model** | `yunvs/bart-base-wikilarge-simplification` | **37.80** | 88.30 | 8.18 | 11.7% |
| off-the-shelf | `eilamc14/bart-large-text-simplification` | **38.17** | 88.58 | 7.85 | 13.4% |
| prompted | `qwen2.5:7b-instruct-q4_K_M` | 45.95 | 75.46 | 7.77 | 4.5% |
| prompted | `qwen2.5:3b-instruct-q4_K_M` | **47.03** | 68.21 | 8.22 | 1.2% |

⚠️ **Read this honestly: fine-tuning bought no measurable advantage over an
off-the-shelf simplifier** (37.80 vs 38.17, confidence intervals overlapping almost
entirely), and **both prompted LLMs beat it by 8–9 SARI.** The defensible claim for this
model is *efficiency*, not quality: it is a **bart-base** (139M parameters) matching a
**bart-large** roughly three times its size, and it runs locally on CPU without a
prompted-LLM sidecar. If output quality is what you need and you can run a 3B
instruction-tuned model, use that instead.

BLEU runs perfectly inverse to SARI across all five rows (91.64 → 88.58 → 88.30 → 75.46
→ 68.21 against SARI 21.39 → 38.17 → 37.80 → 45.95 → 47.03), which is the cleanest
demonstration of its unsuitability for this task that this project produced.

## Limitations

- **Hallucination.** Short or fragmentary inputs can produce unrelated text; the
  phrase `"Other websites"` (a Simple English Wikipedia section heading present in the
  training data) has been observed as output for several unrelated short inputs. The
  companion backend runs a quality guard that falls back to the original text in this
  case.
- **Conservative rewriting.** Observed output often drops a subordinate clause rather
  than rephrasing dense vocabulary — it does not reliably simplify words like
  "multiplicity" or "heterogeneous entities."
- **Outperformed by prompted LLMs and matched by an off-the-shelf model.** See
  Evaluation. Choose this model for size and local CPU inference, not for peak quality.
- **No human evaluation.** All reported numbers are automatic metrics. No readability,
  meaning-preservation, or accessibility study has been conducted.
- **Single sentences only.** Multi-sentence input is out of distribution and gets
  truncated at 64 tokens.
- **English only**, and trained on Wikipedia prose — encyclopedic register, with
  whatever demographic and topical bias that corpus carries.

## Training data provenance and licence

WikiLarge is derived from English Wikipedia and Simple English Wikipedia, which are
licensed **CC BY-SA**. This model is released under **CC BY-SA 4.0** accordingly.

WikiSmall was evaluated as an alternative and deliberately rejected: it carries
baked-in named-entity anonymization artifacts unsuitable for training.

## Citation

Produced for a bachelor thesis (2026). Please cite the project repository:
<https://github.com/yyvs/simple-website>