noamor commited on
Commit
c5f520f
·
verified ·
1 Parent(s): e583965

card: note the shoshan 0.4.0 text-handling fixes (weights unchanged)

Browse files
Files changed (1) hide show
  1. README.md +23 -1
README.md CHANGED
@@ -39,7 +39,7 @@ scales with the number of sentences, not tokens.
39
  ## Usage
40
 
41
  ```bash
42
- pip install git+https://github.com/ivrit/shoshan.git
43
  ```
44
 
45
  ```python
@@ -49,6 +49,28 @@ lz = Lemmatizer.from_pretrained() # pulls these weights, then caches
49
  lz.lemma("המחברות", "המורה חילקה את המחברות לתלמידים בכיתה.") # -> מחברת
50
  ```
51
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
52
  ## Results (out-of-domain, held-out registers)
53
 
54
  - Lemma accuracy **92.4%** out-of-domain (94.3% in-domain).
 
39
  ## Usage
40
 
41
  ```bash
42
+ pip install shoshan
43
  ```
44
 
45
  ```python
 
49
  lz.lemma("המחברות", "המורה חילקה את המחברות לתלמידים בכיתה.") # -> מחברת
50
  ```
51
 
52
+ ## `shoshan` 0.4.0 — update the software, not the weights
53
+
54
+ **These weights are unchanged**; nothing here needs re-downloading. 0.4.0 fixes the text
55
+ handling *around* the model, so the same weights now see your text as you wrote it.
56
+
57
+ It matters most for **text extracted from PDFs**, which commonly spells Hebrew with
58
+ presentation forms (U+FB1D-U+FB4F). Such a word used to be split apart before the model
59
+ saw it — `אנשים` ("people") written with U+FB2E became `נשים` ("women") — and, for the
60
+ width variants and the alef-lamed ligature, it also reached the encoder as a codepoint
61
+ DictaBERT has never seen. Both are fixed. `annotate()`'s character offsets now index the
62
+ string you passed in; their value changes on any input that normalization rewrites — text
63
+ that is not NFC, and text containing presentation forms.
64
+
65
+ ```bash
66
+ pip install -U shoshan
67
+ ```
68
+
69
+ If you have run this model over PDF-extracted Hebrew, the affected words were lemmatized
70
+ wrong rather than approximately — re-run rather than spot-check.
71
+ Measured effect and the full changelog:
72
+ <https://github.com/ivrit/shoshan/releases/tag/v0.4.0>
73
+
74
  ## Results (out-of-domain, held-out registers)
75
 
76
  - Lemma accuracy **92.4%** out-of-domain (94.3% in-domain).