README.md

---
license: mit
language:
  - en
  - he
library_name: transformers
pipeline_tag: token-classification
base_model: FacebookAI/xlm-roberta-base
datasets:
  - Hebrish/hebrew-latin-code-switching
tags:
  - code-switching
  - hebrew
  - transliteration
  - romanization
  - text-to-speech
metrics:
  - f1
  - precision
  - recall
model-index:
  - name: Hebrew-Latin-Token-Classifier
    results:
      - task:
          type: token-classification
        dataset:
          name: hebrew-latin-code-switching (test, term-disjoint)
          type: Hebrish/hebrew-latin-code-switching
          split: test
        metrics:
          - {type: f1, value: 0.941, name: span F1}
          - {type: precision, value: 0.928, name: span precision}
          - {type: recall, value: 0.953, name: span recall}
---

# Hebrew-in-Latin token classifier

> **Home moved to the Hebrish organisation (2026-10-20).** This repository is a copy of
> [`danielrosehill/hebrew-latin-token-classifier`](https://huggingface.co/danielrosehill/hebrew-latin-token-classifier)
> at revision `01f092f1`, the same weights. The original carries DOI
> [10.57967/hf/10997](https://doi.org/10.57967/hf/10997), and Hugging Face does not move a repository
> that has a DOI, so it was copied instead. New versions are published here.

Finds **Hebrew words written in Latin letters** inside English text.

```
I need to go to Bituach Leumi before the makolet closes.
                [Bituach Leumi]          [makolet]
```

**Try it:** [demo Space](https://huggingface.co/spaces/danielrosehill/hebrew-latin-token-classifier).
**API:** [Replicate](https://replicate.com/danielrosehill/hebrew-latin-token-classifier) (CPU;
inputs `text` and `threshold`, returns the span list shown below).

There was no model for this before. GlotLID, `lid.176` and `lid218e` have no
`heb_Latn` label, and no token-level code-switching model covers Hebrew. It was built
so a text-to-speech system can send these spans to a Hebrew voice instead of reading
*makolet* as English letters. The labelling test is **pronunciation, not etymology**.
Words English already says acceptably are deliberately **not** flagged: *Shabbat*,
*hummus*, *challah*, *kosher*, *aliyah*, and city names under the default posture.

**Model page: [hebrish.app/models/token-classifier](https://hebrish.app/models/token-classifier).** **Website: [hebrish.app](https://hebrish.app)**: get the Hebrish keyboard for Android and the desktop apps, see the results, and find every model, dataset and demo in one place. **Android updates through F-Droid:** add the [Hebrish F-Droid repository](https://fdroid.hebrish.app/repo?fingerprint=EF8C2D63D994A10CD489A978585510B47E47452EC09248591CB2C0F39C2BC1FA) (`https://fdroid.hebrish.app/repo`; [steps](https://hebrish.app/download#fdroid)).

**The whole TTS pipeline** (detect, transliterate, niqqud, tag, speak), with code, real outputs and a white paper: [Hebrish-ASR/English-Hebrew-Code-Switching-TTS-Pipeline](https://github.com/Hebrish-ASR/English-Hebrew-Code-Switching-TTS-Pipeline) · [PDF](https://hebrish.app/whitepaper/code-switching-tts-pipeline.pdf).

<!-- hebrish-listen:begin -->
## Listen: what Hebrish is and how it's made

A spoken introduction to the project by the person who makes it: why Hebrish exists, why stock
speech-to-text gets Hebrew words wrong, how the fine-tunes are trained, the model lines (Sabich,
Jachnun, Shakshuka, Meorav), how the training data is recorded, and the plans for the apps.

| | |
| --- | --- |
| Listen | [hebrish.app/updates](https://hebrish.app/updates): player, chapters, edited transcript, MP3 download |
| Author | Daniel Rosehill |
| Recorded | 2026-10-22 |
| Runtime | 1:07:03 |

Chapters (each opens the player at that point):
[0:00](https://hebrish.app/updates#t=0) Why this voice note ·
[1:36](https://hebrish.app/updates#t=96) Coming to speech to text ·
[6:36](https://hebrish.app/updates#t=396) Why Hebrew words go wrong ·
[9:23](https://hebrish.app/updates#t=563) How the fine-tune works ·
[12:27](https://hebrish.app/updates#t=747) Why Parakeet, and the model plan ·
[17:44](https://hebrish.app/updates#t=1064) Disfluencies, shekels, paragraphs ·
[25:40](https://hebrish.app/updates#t=1540) Tech vocabulary and Meorav ·
[29:07](https://hebrish.app/updates#t=1747) The kitchen sink: Sabich and Jachnun ·
[33:35](https://hebrish.app/updates#t=2015) Licensing, accents, noisy recordings ·
[39:10](https://hebrish.app/updates#t=2350) One recorder for ASR and TTS ·
[41:04](https://hebrish.app/updates#t=2464) Desktop shortcuts and the macro pad ·
[43:34](https://hebrish.app/updates#t=2614) Android: Bluetooth and one button ·
[45:35](https://hebrish.app/updates#t=2735) Streaming and Shakshuka ·
[50:28](https://hebrish.app/updates#t=3028) Long-form transcription ·
[56:00](https://hebrish.app/updates#t=3360) Output modes ·
[63:01](https://hebrish.app/updates#t=3781) Report a word, and more voices
<!-- hebrish-listen:end -->

## The author's note (audio)

Six and a half minutes on why this model exists, recorded the day after release.

<audio controls preload="none" src="https://huggingface.co/Hebrish/Hebrew-Latin-Token-Classifier/resolve/main/voice-note/why-this-model.mp3"></audio>

[Download the mp3](https://huggingface.co/Hebrish/Hebrew-Latin-Token-Classifier/resolve/main/voice-note/why-this-model.mp3)
· [Transcript](https://huggingface.co/Hebrish/Hebrew-Latin-Token-Classifier/blob/main/voice-note/transcript.md)

## Why: text-to-speech that stops botching Hebrew

English TTS reads romanized Hebrew as English: *makolet* comes out as "MACK-oh-let"
and *mashkanta* gets English stress. The fix is to find the Hebrew spans, convert them
to Hebrew script, and send each one to the TTS engine with the **Hebrew language
identifier**. For Chatterbox that is a separate `language_id="he"` call; for SSML
engines it is `<lang xml:lang="he-IL">`. The rest of the sentence stays English.

The first user is [**My Weird Prompts**](https://myweirdprompts.com), an AI-generated
podcast produced in Jerusalem whose episodes often contain Hebrew words that have long
been mispronounced. Its script-writing agent writes English, and when an episode is
about life in Israel the Hebrew comes out in Latin letters. Words common enough in
English (*Shabbat*, *kosher*) come out intelligibly. The long tail does not, and that
tail is where everyday Israeli life is: *Bituach Leumi*, *Misrad HaKalkala*, *arnona*.

Two obvious fixes don't work:

- **Always translate.** Many of these terms have no good English equivalent, and the
  translation is not what a listener in Israel would say.
- **Language ID.** A detector sees Latin script inside an English sentence and calls
  it English. Nothing expects the script itself to code-switch. That is the gap this
  model fills: it is pass 1 of a three-pass pipeline (detect → convert to Hebrew
  script → tag the span for a Hebrew voice), and the other two passes have existing
  tools.

**Where to draw the line** is a design choice, not a fact. *Shabbat* is Hebrew, but
flagging it sends it to a Hebrew voice mid-sentence when English already says it
fine. This release does not flag such words. A possible future version could add a
third label for "Hebrew that English already says", leaving the choice to the user.

The approach carries over to other immigrant and heritage-language
communities that mix their own words into a majority language; only the term list is
Hebrew-specific.

**Related work.** Hebrew-script NLP and speech are well covered by
[ivrit.ai](https://huggingface.co/ivrit-ai), [DICTA](https://huggingface.co/dicta-il),
[Phonikud](https://github.com/thewh1teagle/phonikud) and
[TaatikNet](https://huggingface.co/malper/taatiknet). The opposite direction,
foreign words written in Hebrew script, was studied by Goldberg & Elhadad (2008). No
language-ID system has a `heb_Latn` label, and before this there was no model for
Hebrew written in Latin letters inside English.

## Results

`xlm-roberta-base`, 8 epochs, about 3 minutes on one A10G. Scored on the
**term-disjoint** test split: no Hebrew term in test appears in training, so this
measures how well the model generalises to words it has never seen.

| Test slice | Sentences | Span precision | Span recall | Span F1 |
| --- | --- | --- | --- | --- |
| All | 468 | 0.928 | 0.953 | **0.941** |
| Generated | 430 | 0.928 | 0.948 | 0.938 |
| Dictated (human-chosen sentences) | 38 | 0.929 | 1.000 | 0.963 |
| Validation (model selection) | 467 | 0.950 | 0.948 | 0.949 |

![Precision, recall and F1 by evaluation slice](https://huggingface.co/Hebrish/Hebrew-Latin-Token-Classifier/resolve/main/figures/results-by-slice.png)

**Hard negatives: 0 false positives in 58.** These are sentences built to trap the
model: substring traps, the English sense of homographs (*dud*, *tor*, *salon*),
lookalikes, and anglicised loanwords. Of the 41 test sentences it gets wrong, the
false positives fall into two groups:

- **Real policy misses.** Place names the conservative policy excludes (*Dizengoff*,
  *Ramat Gan*, *Talpiot*), *mishnah*, and once *venv*.
- **Gold-label gaps.** Hebrew words in the active lexicon that the blind annotator
  refused and no human has yet re-reviewed: *msabbaha*, *lechem*, *sakin*, *kapsulot*,
  *od paam*, *mas hachnasa*, *yalla*, *babaganoush*. Here the model is arguably right.

![Breakdown of the 41 wrong test sentences by cause](https://huggingface.co/Hebrish/Hebrew-Latin-Token-Classifier/resolve/main/figures/error-breakdown.png)

Of the 41, **14 are label gaps** (grey) rather than model errors. Resolving them in
review would raise measured precision without retraining. The 280 open round-2 review
tasks are the same kind of gap in the training data.

Misses are mostly words that are also common Hebrew loanwords or short (*shakshuka*,
*sukkot*, *botz*, *chamutz*), and the third word of three-word compounds
(*heshbonit mas kabala*).

For comparison, the closest published recipe (EnTaCs, English–Tamil, arXiv 2603.26587)
reports F1 0.638 on its romanized class. The project planned for about that.

## Use

Pre-split words the way the training data was split, and take each word's label from
its **first sub-token**. This is the only one trained. `predict.py` in this repo does
both:

```python
# uv run predict.py "Grab some bamba from the makolet."   (or copy the class)
from predict import HebrewSpans
find = HebrewSpans()                       # Hebrish/Hebrew-Latin-Token-Classifier
find("My savta says the arnona went up again.")
# [{'start': 3, 'end': 8, 'text': 'savta', 'score': 1.0},
#  {'start': 18, 'end': 24, 'text': 'arnona', 'score': 1.0}]
find("...", threshold=0.8)                 # fewer false positives, fewer hits
```

The stock `pipeline` also works, with `aggregation_strategy="first"`:

```python
from transformers import pipeline
ner = pipeline("token-classification", model="Hebrish/Hebrew-Latin-Token-Classifier",
               aggregation_strategy="first")
```

It splits words on whitespace, though, so trailing punctuation attaches to a span
(`makolet?`). **Do not use `aggregation_strategy="simple"`**: it cuts words short
(`mako`).

Labels: `O`, `B-HE`, `I-HE`. Multi-word terms come out as one span (*Bituach Leumi*,
*kupat holim*).

## Training data

[`Hebrish/hebrew-latin-code-switching`](https://huggingface.co/datasets/Hebrish/hebrew-latin-code-switching):
4,737 sentences over 1,235 seed terms.

- **Generated sentences.** Each one was seeded with a term, then annotated blind by a
  second model family. Disagreements went to human review, along with a 10% random
  audit of agreements.
- **458 dictated sentences.** These are the text of sentences the author read aloud
  for a speech-recognition fine-tune.
- **Masked spans.** Spans nobody has ruled on, and Israeli person names, carry label
  `-100`: no loss, and not scored.

The data is largely synthetic. Treat the dictated slice as the closest thing to real
input.

## Limits

- It is English text with romanized Hebrew. Hebrew script is not the target and is
  not handled.
- Romanization varies. Common spelling variants were trained on, but an unusual
  spelling of an unseen word is the hardest case.
- **Person names are unlabelled** (masked in training), so their behaviour is
  undefined. **Place names** are negative by policy, but the model sometimes flags
  Hebrew-sounding ones.
- **Precision matters most** downstream: a false positive gets an English word read
  in Hebrew. Raise `threshold` if that matters more than recall.

## Provenance

Code, annotation policy and taxonomy:
[Hebrish-ASR/Hebrew-Latin-Token-Classifier](https://github.com/Hebrish-ASR/Hebrew-Latin-Token-Classifier).
The training script is `training/train.py`, and `eval_results.json` and
`test_errors.jsonl` in this repo hold the full test output. Trained 2026-10-15. MIT.

## Base model and licence notices

This model is a fine-tune of [`FacebookAI/xlm-roberta-base`](https://huggingface.co/FacebookAI/xlm-roberta-base), by Facebook AI, licensed under MIT. Hebrish modified the base model by fine-tuning it.

The licence stated at the top of this card covers Hebrish's contribution; the base model's licence keeps applying to the base model, and nothing here restricts what that licence allows for it. Terms of use: <https://hebrish.app/terms>.

Hebrish is not affiliated with or endorsed by Meta.