coedit-paradetox-lora

A LoRA adapter for grammarly/coedit-large that rewrites English sentences containing toxic or offensive words into neutral ones. It is the core model of CleanSpeak, a college project on a sentence-cleaning bot: fix grammar and typos, replace slang, and remove inappropriate words while keeping the meaning.

This repository contains only the adapter weights (a few MB). You load the original CoEdIT-large and attach the adapter on top.

Status: student project, one training run, small evaluation sets. The results below are real but limited; see Limitations before relying on it.

Summary of results

Question	Answer
Does it remove toxic words better than the base model?	Yes. On 200 held-out ParaDetox sentences, words flagged by a profanity list dropped from 101 (input) → 45 (base CoEdIT) → 14 (this adapter)
Does it hurt plain grammar correction?	No visible regression on a 5-sentence check (identical outputs with the adapter off and on). This is weak evidence
Is it enough on its own?	No. It tends to delete insults instead of replacing them, so the tone is lost, and it ignores slang. The full CleanSpeak pipeline adds a slang dictionary, rule-based insult replacement, and a grammar pass

Intended use

• Research, coursework, and demos of detoxification and text-cleaning pipelines.
• Suggesting a cleaner rewrite of a single English sentence that a human can review.

Out of scope: automated moderation or enforcement decisions, rewriting other people's words without their knowledge, non-English text, long documents (trained on sentences up to 96 tokens), and any commercial use (the base model is non-commercial).

How to use

1. The adapter alone

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from peft import PeftModel

BASE = "grammarly/coedit-large"
ADAPTER = "Shinjan123/coedit-paradetox-lora"   # if the repo is private, run `hf auth login` first

tok = AutoTokenizer.from_pretrained(BASE)       # unchanged from CoEdIT: no tokens were added
base = AutoModelForSeq2SeqLM.from_pretrained(BASE)
model = PeftModel.from_pretrained(base, ADAPTER).eval()

def clean(text):
    # the adapter was trained with this exact instruction prefix
    enc = tok("Clean this sentence: " + text, return_tensors="pt", truncation=True, max_length=96)
    with torch.no_grad():
        out = model.generate(**enc, max_length=96, num_beams=4)
    return tok.decode(out[0], skip_special_tokens=True)

print(clean("you can have your stinking office building ."))
# -> You can have your office building.

# the original CoEdIT is still available underneath: switch the adapter off for grammar-only edits
with model.disable_adapter():
    enc = tok("Fix the grammar: She dont like apples", return_tensors="pt")
    print(tok.decode(model.generate(**enc, max_length=96)[0], skip_special_tokens=True))

Runs on CPU (slow) or GPU. CoEdIT-large is about 770M parameters; use fp32 (T5-family models are unstable in fp16).

2. The full CleanSpeak pipeline (recommended)

cleanspeak_cli.py in this repository wraps the adapter in a pipeline and adds an interactive terminal mode and an evaluation mode:

pip install torch transformers peft sentencepiece better-profanity sentence-transformers
python cleanspeak_cli.py               # type sentences, get cleaned output
python cleanspeak_cli.py --verbose     # show every stage
python cleanspeak_cli.py --eval        # run the built-in 39-sentence test set

The default and adopted pipeline is v1: slang dictionary → this adapter → hard-coded backstop for any bad word the model leaves → original CoEdIT grammar pass (adapter switched off) → mask as a last resort. The model itself decides how to remove or rewrite offensive words; hand-picked replacement words are only inserted when the model leaves a bad word behind.

Training

Data

Source	Pairs	Role
ParaDetox (toxic → neutral, human-written)	about 19k	Core task: remove toxicity, keep content
Synthetic slang/typo pairs (made here, rule-based)	4,000	Noisy sentence → clean original, so the meaning is preserved by construction
CoEdIT sample (replay)	3,000	Keeps the original editing skills alive (limits forgetting)

The total is about 25.9k pairs, consistent with 3,242 steps × batch 16 ÷ 2 epochs. The ParaDetox count is approximate (the exact figure was not recorded).

• Held-out test set: 500 unique toxic sentences were removed from ParaDetox before any other step, and every row sharing those sentences was excluded from training. Synthetic pairs were generated only from training-side sentences.
• Synthetic noise: reverse slang substitutions (going to → gonna, you → u, I do not know → idk, ...), dropped apostrophes, lowercasing, and an adjacent-letter swap as a typo. It only covers the slang in its dictionary.
• Prompt format: ParaDetox and synthetic pairs use Clean this sentence: <text>. CoEdIT replay pairs keep their original instructions.

Hyperparameters

Setting	Value
Base model	grammarly/coedit-large (about 770M parameters, FLAN-T5-large based)
Method	LoRA via PEFT (TaskType.SEQ_2_SEQ_LM)
LoRA rank / alpha / dropout	16 / 32 / 0.05
Target modules	q, v (attention query and value projections), bias="none"
Optimizer / schedule	Hugging Face Trainer defaults (AdamW, linear decay), no warmup set
Learning rate	5e-4
Epochs / batch size	2 / 16 (3,242 steps)
Max length (source and target)	96 tokens
Precision	fp32
Seed	0
Hardware / time	single NVIDIA T4 (Google Colab), 41 min 32 s

Training run

Training loss fell from 0.95 (step 50) to about 0.65 by step 650 and then drifted down slowly: mean 0.70 over epoch 1 and 0.63 over epoch 2, with a minimum of 0.584 at step 2,100. The curve is noisy and flattens in epoch 2, so a third epoch would probably add little.

• One run only. No hyperparameter search and no repeated seeds, so run-to-run variance is unknown.
• No intermediate checkpoints were saved (save_strategy="no"). The repository holds the final adapter only.
• The loss above is the training loss on the mixed dataset. No validation loss was tracked.

Evaluation

Metrics

• Bad words left: outputs containing a word from the better-profanity list plus 7 added mild insults (rascal, bastard, idiot, moron, clown, stupid, jerk), checked word by word so assess is not flagged.
• Meaning: cosine similarity of all-MiniLM-L6-v2 sentence embeddings. It is a rough proxy: it cannot tell a polite rewrite from a flipped attitude.
• Retention (pipeline eval only): the share of the input's ordinary words that survive, ignoring slang and flagged words.

A. Held-out ParaDetox, adapter only (first 200 test pairs)

System	Bad words left (of 200)	Meaning vs input	Meaning vs reference
Input	101	1.000	0.768
CoEdIT-large (Write in a more neutral way:)	45	0.923	0.788
CoEdIT-large + this adapter	14	0.877	0.808

The adapter removes far more toxic words than the base model and lands slightly closer to the human references. It also changes the sentences more (lower similarity to the input), which is expected.

Reading the 14 leftovers (11 unique sentences, since ParaDetox has several paraphrases per toxic sentence; counted by hand from the printed list):

• 3 false alarms of the word list: pot, kill, gay are not toxic in those sentences.
• 7 real misses: idiot, stupid (twice), sucked (twice), sucking, screwed. Mild insults and slang are not well covered.
• 1 regression: omg , stephens is so silly . became ... so stupid . (the model added an insult).

So the true remaining rate is probably closer to 11/200 than 14/200, but all three rows above use the same raw metric, so the comparison between them is consistent. A rerun with duplicates removed and a stricter ignore list has not been done yet.

B. Forgetting check

Five grammar-only sentences (She dont like apples, He go to school every day, Their going to the park tomorrow., plus two already-correct sentences) were run with the adapter off and on, using the original Fix the grammar: instruction. Outputs were identical. Five sentences is too few to rule out subtler drift.

C. Full pipeline v1 on a 39-sentence hand-written set

Category	n	Bad left	Meaning	Retention	Unchanged
clean (should not change)	8	0	1.00	0.98	7 / 8
grammar only	6	0	0.95	0.80	n/a
slang	7	0	0.81	0.97	n/a
insult	8	0	0.77	0.84	n/a
profanity	6	0	0.80	0.90	n/a
mixed	4	0	0.67	0.69	n/a

• "Bad left = 0" is partly by design: the backstop and the final mask guarantee it. It shows the safety net works, not that the model is good. Meaning and retention are the informative columns.
• Strong: clean text, grammar, slang, and expletives.
• Weak: insults and mixed sentences, where meaning and retention are lowest. The set is small and written by the author, so treat the numbers as directional.

Selected outputs (strong words masked here):

Input	Output
Bro, you b*****d, why the f*** are you doing this?	Why are you doing this?
tbh bro i kinda wanna go there cuz its lit af	Honestly, I kind of want to go there because it's really exciting.
shut the f*** up and listen to me	Shut up and listen to me.
she dont like apples (grammar)	She doesn't like apples.
you are a bastard and you never listen	You never listen. (insult deleted, tone lost)
only an idiot would believe that	Only a bad person would believe that. (meaning shifted)
wanna grab lunch, im starving af	want to grab lunch; I'm starving very badly. (slang mapping artifact)
bro u r such an idiot, idk why i even talk to u	You 're such a bad person; ... (spacing bug)

D. Pipeline v2 compared with v1 (same 39 sentences) - tried, not adopted

v2 replaced known insults with hand-picked milder words before the model (bastard → unpleasant person, idiot → fool, stupid → foolish, hell → heck), only ran the model when a bad word remained, and added a double-negative rule and spacing/capitalisation cleanup.

Category	Meaning v1 → v2	Retention v1 → v2
clean (7/8 unchanged in both)	1.00 → 1.00	0.98 → 0.98
grammar	0.95 → 0.95	0.80 → 0.80
slang	0.81 → 0.79	0.97 → 0.97
insult	0.77 → 0.79	0.84 → 0.91
profanity	0.80 → 0.82	0.90 → 0.90
mixed	0.67 → 0.75	0.69 → 0.75
all 39	0.844 → 0.859	0.880 → 0.901

The metrics moved only slightly, and v2 fixed several v1 bugs (You 're, starving very badly, a dropped me, the double negative doesn't know nothing). It also kept the attitude of insults (you are a bastard and you never listen → You are an unpleasant person, and you never listen.).

Why v1 was kept anyway: a human read of the outputs preferred v1. v2's replacement words are the author's own dictionary entries, not something the model produced, so the output reads as inserted vocabulary (unpleasant person, foolish, troublemaker). v1's cleanup sounds more natural, even though it scores slightly lower on the automatic metrics. Two further caveats for v2: its dictionary was built from the failures seen in this same test set (so the gains are partly by construction), and its model gate only fires on words the profanity list knows, so unlisted insults such as loser or scumbag would pass through unchanged (not yet tested). This is a small example of automatic similarity metrics and human judgement disagreeing.

Development history and previous approaches

1. Initial design (not benchmarked). A rule-based plan: LanguageTool for grammar, better-profanity for bad words, and a slang dictionary. It was dropped because dictionary swaps (bro → friend) cannot preserve tone.

2. Zero-shot model comparison. Eight test sentences, each targeting one failure mode (slang, insults, double negatives, false-positive trap assess, an already-clean sentence), run through four systems:

   Input	CoEdIT-large	Qwen2.5-1.5B-Instruct	Qwen2.5-3B-Instruct	s-nlp/t5-paranmt-detox
   you are a bastard and you never listen	... a liar ... (invented accusation)	you rarely listen (changed never)	You are rude, and you never listen.	You're a badass ... (insult turned into praise)
   that rascal stole my bike yesterday	That guy stole ...	That mischievous person took ...	That **** stole ...	unchanged, rascal kept
   tbh he's a clown, idk why u trust him	kept clown	amusing	funny (attitude flipped)	dropped the whole clause
   The meeting starts at 10 AM tomorrow.	dropped tomorrow	the day after today	unchanged	10: 00, dropped AM

   Findings: CoEdIT is the best grammar editor but ignores slang; the Qwen models over-edit clean text and can flip attitudes; the detox model sometimes inverts or deletes meaning. Qwen-3B with a strict prompt was the best of the zero-shot options.

3. Decision: fine-tune CoEdIT-large with LoRA on ParaDetox plus synthetic slang pairs, with CoEdIT replay data to protect grammar skills. Plain LoRA rather than QLoRA, because the model fits a 15 GB T4 in fp32 and QLoRA would add complexity for no benefit at this size.

4. Fine-tuning run (this adapter), then evaluation A, B, C above.

5. Pipeline v1: slang dictionary → adapter → backstop → grammar pass → mask (evaluated in C).

6. Pipeline v2 (evaluated, not adopted): known insults replaced by rules before the model, the model gated to sentences that still contain bad words, plus a double-negative rule and spacing/capitalisation cleanup. Slightly better metrics (section D), but its hand-picked replacement words felt invented, so v1 is the final pipeline.

Not tried: full fine-tuning, QLoRA, larger base models, multiple seeds or hyperparameter search, the GYAFC informal-to-formal corpus, a toxicity classifier as a gate.

Limitations and known failures

• It deletes instead of replacing. ParaDetox references often drop the offensive words, and the model learned that habit. Insults lose their attitude (you are a bastard and you never listen → You never listen.). Sometimes it generalises to a bad person, which changes the meaning.
• It does not handle slang. Slang was only in the synthetic data, and most of that data contained none of the dictionary phrases, so the model mostly learned to copy. A dictionary in the pipeline is what fixes slang.
• Mild insults are missed (stupid, idiot, sucked, screwed), and it occasionally adds one (silly → stupid).
• Grammar gaps: double negatives (doesn't know nothing) and subject-pronoun errors (Me and him was late) sometimes survive.
• Small, author-written evaluation sets, a single training run, and a similarity metric that cannot detect flipped tone. No human evaluation.
• The profanity list is a blunt instrument. It flags gay, pot, and kill even when they are not toxic, and misses obfuscations such as sh1t.
• English only. Sentences up to about 96 tokens.

Ethical considerations

ParaDetox and the test sets contain real toxic language from online comments, so examples in this card are partly masked. Detoxification systems can erase identity terms, change the meaning of reported speech, or be used to put cleaned-up words in someone's mouth. Use outputs as suggestions that a person reviews, not as automatic moderation. The identity-related words on a profanity list (for example gay) are not toxic by themselves, and automatic filters can mistreat them.

Reproducibility

• Seed 0 for the split, synthetic noise, and training. Split rule: shuffle the sorted unique toxic sentences, hold out the first 500.
• Training notebook: cleanspeak_finetune.ipynb. Pipeline and evaluation script: cleanspeak_cli.py. Raw evaluation output: eval/.
• The section C numbers were produced by an earlier revision of cleanspeak_cli.py; --pipeline v1 reproduces that logic (same slang and backstop dictionaries). Raw outputs are in eval/.
• Library versions were not pinned. Results may differ slightly on other versions.

License

CC BY-NC 4.0 (non-commercial), following the base model grammarly/coedit-large. The training data carries its own licences (see the ParaDetox and CoEdIT dataset cards).

References

• Raheja et al. (2023). CoEdIT: Text Editing by Task-Specific Instruction Tuning. https://arxiv.org/abs/2305.09857
• Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. https://arxiv.org/abs/2106.09685
• Logacheva et al. (2022). ParaDetox: Detoxification with Parallel Data. ACL 2022. Dataset and citation: https://huggingface.co/datasets/s-nlp/paradetox
• Dale et al. (2021). Text Detoxification using Large Pre-trained Neural Models. EMNLP 2021. https://arxiv.org/abs/2109.08914 (ParaGeDi and CondBERT; code in the s-nlp/detox repository)
• Raffel et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5). https://arxiv.org/abs/1910.10683
• Chung et al. (2022). Scaling Instruction-Finetuned Language Models (FLAN). https://arxiv.org/abs/2210.11416
• Reimers and Gurevych (2019). Sentence-BERT. https://arxiv.org/abs/1908.10084
• Rolnick et al. (2019). Experience Replay for Continual Learning. https://arxiv.org/abs/1811.11682 (the idea behind mixing in CoEdIT data)
• Rao and Tetreault (2018). Dear Sir or Madam, May I introduce the GYAFC Dataset. https://arxiv.org/abs/1803.06535 (related informal-to-formal data; not used here)
• Qwen Team (2024). Qwen2.5 Technical Report. https://arxiv.org/abs/2412.15115 (used only in the zero-shot comparison)
• Wolf et al. (2020). Transformers: State-of-the-Art Natural Language Processing. https://arxiv.org/abs/1910.03771
• PEFT library: https://github.com/huggingface/peft
• Zero-shot detox baseline: https://huggingface.co/s-nlp/t5-paranmt-detox

Citation

@misc{cleanspeak2026,
  title  = {CleanSpeak: a LoRA adapter for CoEdIT-large that detoxifies and cleans English sentences},
  author = {Shinjan123},
  year   = {2026},
  url    = {https://huggingface.co/Shinjan123/coedit-paradetox-lora}
}