Sefaria/BEREL_3.0-hebrew-error-detector

BEREL 3.0 Hebrew error detector

A token classifier that marks the words of a Rabbinic Hebrew passage that are probably errors in the digital text: OCR damage, typos, words run together or split apart, and stray characters. It is fine-tuned from dicta-il/BEREL_3.0 by Sefaria.

Purpose

Sefaria holds thousands of Hebrew texts, many of them OCR'd from printed editions, and their quality varies. The detector has two uses:

1. Point editors at likely errors in a text, so proofreading starts where it is needed.
2. Estimate the quality of a version (a text edition) from the share of its words the detector flags, to decide which versions need correcting before further processing such as machine translation. This use is still being validated; see Limitations.

It is meant for post-biblical works: Talmud, midrash, commentaries, codes, responsa and later Rabbinic literature. It is not suitable for Tanakh, Targum or manuscript-orthography texts (see Limitations).

How it was trained

Labels from real editing history. Sefaria keeps every edit made to the Hebrew of its texts. All ~1.05M edits were rebuilt into before/after pairs and classified by rules. Edits that fix an error became training data, and the word that was fixed is labelled as an error in the text before the edit:

Fix kind	What it is	Hand-checked precision (400-edit sample)
typo	changed words within edit distance 2	0.88
spacing	missing or extra space, same letters	0.94
ocr	a letter error together with a split or join, or an abbreviation mark read as a letter	0.76
junk	stray non-Hebrew characters removed	2 of 2

Excluded: word substitutions (mostly variant readings), added or dropped words, vowel-only edits, citations, formatting, edits by two bulk re-segmentation accounts, conventions such as joining words with a maqaf, and the Jerusalem Talmud (most of its edits are one editor's re-spelling of Aramaic). Inside a multi-word edit only the words that actually changed are labelled.

Context. Each edited segment is widened with its neighbouring segments up to BEREL's 510 word pieces, stopping at the edge of its Sefaria Passage (Tanakh, Bavli) or its section (everything else). This gives 22,553 windows, split 20,059 / 1,021 / 1,473 into train / validation / test by (version, book), so no book appears in two splits.

Errors nobody corrected. Hand-checking words the detector flagged but the data called correct found that two-thirds were real errors no editor had fixed yet. So the labels were corrected before training this model. Four detectors were trained, each on three-quarters of the training books, and each scored the quarter it had not seen. Every word scored 0.5 or more that the data labelled ok was judged by two LLMs (GPT-6 Luna and Claude Sonnet 5.5), with Gemini 3.8 Flash deciding where they disagreed. That flow agreed with the hand labels on 24 of 26 words. 8,810 training words were relabelled as errors, adding about a third to the editors' corrections. Validation and test were corrected the same way, using the flags of four detectors.

Clean pairs. Each real window is paired with its corrected version, every word labelled ok except judged errors, so the model does not learn that every passage contains an error.

Synthetic errors. 60,000 windows of train-split text with injected errors, drawn from the confusion statistics of the real corrections: single-letter substitutions (mostly visual OCR confusions such as ר→ד, ס→ם, ו→י, כ→ב), deletions, insertions, joins and splits; 15% have no error. Mixed in at half the weight of real windows.

Training. AutoModelForTokenClassification with two labels (ok, error), the first word piece of each word carrying its label. Cross-entropy with the error class weighted 10×, learning rate 3e-5, batch 64, 3 epochs of 120,000 chunks of up to 512 word pieces, bf16 on one A100. The weights with the best validation PR-AUC were kept (step 5,500 of 5,625). Sweeps chose this setting: synthetic weight 0.5 over 0 and 1.0, error weight 10 over 3, 20 and 30, and labelling the context words over leaving them unlabelled. Training longer did not help.

Results

Word-level, error class, on the test set (590,782 words of edited segments). Thresholds were chosen on validation and applied to test.

Test labels	Errors	This model PR-AUC	F1	Same setup trained on editor labels only, PR-AUC	F1
Editor corrections plus judged errors	4,273	0.746	0.725	0.647	0.648
Editor corrections only (judged words left out)	2,077	0.534	0.564	0.497	0.550

The two rows are scored on segment words. On the corrected labels, at a threshold of 0.536, the model is right 70% of the time it flags a word and finds 75% of the errors (all words, context included). Recall by kind: spacing 0.85, typo 0.51, ocr 0.47, uncorrected errors found by the judges 0.87. On editor corrections alone it still ranks better than the same setup trained on uncorrected labels, so the gain is not only from imitating the detectors that chose the judged words. For comparison, flagging words not seen in the corrected training text scores PR-AUC 0.019, and flagging words BEREL 3.0 finds unlikely (masked-LM) scores 0.014, both on the original labels.

Estimated error rate. In the test segments, which are segments an editor corrected, about 1.3% of words were errors before the edit. About 0.6% (roughly 6 per 1,000 words) remain after it, and about 0.27% in the neighbouring text nobody edited. These figures extrapolate the judged errors by the share of editor corrections that the detectors flag (63%), and assume uncorrected errors are as easy to flag.

How to use

The model expects the same normalization as training: remove HTML tags and niqqud/cantillation, split maqaf-joined words, keep punctuation and quote marks (geresh and gershayim help with abbreviations). Score the first piece of each word.

import re, torch
from transformers import AutoTokenizer, AutoModelForTokenClassification

repo = "Sefaria/BEREL_3.0-hebrew-error-detector"
tok = AutoTokenizer.from_pretrained(repo)          # the fast tokenizer; BertTokenizer gives bad results with BEREL 3.0
model = AutoModelForTokenClassification.from_pretrained(repo).eval()
THRESHOLD = 0.536                                  # best F1 on validation with corrected labels; raise it for higher precision

TAG = re.compile(r"&lt;.*?&gt;|<[^>]*>|&nbsp;|&[a-z]+;")
NIQQUD = re.compile("[֑-ׇֽֿׁׂׅׄ]")
def normalize(word):
    return TAG.sub("", NIQQUD.sub("", word)).replace("־", " ").strip()

def flag(text):
    words = [w for w in (normalize(t) for t in text.split()) if w]
    enc = tok(words, is_split_into_words=True, truncation=True, max_length=512, return_tensors="pt")
    with torch.no_grad():
        probs = model(**enc).logits.softmax(-1)[0, :, 1]
    out, seen = [], set()
    for pos, w in enumerate(enc.word_ids()):
        if w is not None and w not in seen:          # first piece of each word
            seen.add(w)
            out.append((words[w], round(probs[pos].item(), 3), probs[pos].item() >= THRESHOLD))
    return out

for word, p, is_error in flag("ואמר רבי יוחנן שהאדם צריך ללמודתורה בכל יום"):
    print(word, p, "ERROR" if is_error else "")

Passages longer than about 350 words should be split on word boundaries into chunks of at most 512 word pieces. The model was trained on windows that include neighbouring text, so give it some context around the words you want to check.

Limitations

• Rare valid forms look like errors. Almost all training errors come from Rabbinic prose, so the model flags unusual but correct forms. On Tanakh (Masoretic defective spellings), Targum (Aramaic forms) and manuscript-orthography editions it flags 15 to 24 words per 1,000, nearly all false. Do not use it on those texts.
• Edition conventions (abbreviations truncated with an apostrophe, joined של, defective spelling) can be flagged.
• Not a corrector. It says which words are suspicious, not what they should be.
• Version quality scores are not validated. Across 294 versions the flagged rate does not yet agree with how much each version has been corrected on Sefaria, and there is no ground truth for version quality. Treat per-version rates as a pointer for review, compared only between versions of the same genre.
• Vowel and cantillation errors are invisible by design (niqqud is removed).
• Judged labels come from LLMs, which make mistakes and favour standard spellings, so the model may lean that way too.
• Earlier version: the first release (256 word pieces, trained on editor labels only) is tagged v1-256 in this repository.

Training data

The training data was built from Sefaria's edit history and is not public (it quotes texts with mixed licenses). The training and evaluation code is in Sefaria/ML-Workflows (app/error_detection).

Citation

Please cite BEREL when using this model:

Avi Shmidman, Joshua Guedalia, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Eli Handel, Moshe Koppel, "Introducing BEREL: BERT Embeddings for Rabbinic-Encoded Language", 2022, arXiv:2208.01875.