english_us_arpa: the dictionary on Hugging Face has 54,848 words; the model card says 266,823
#1
by antimora - opened
The model card for MontrealCorpusTools/english_us_arpa on Hugging Face (revision 9c8d5d8) lists the english_us_arpa dictionary at 266,823 words, but dictionary/english_us_arpa.dict in the repository has 54,848 distinct words (61,698 lines).
For comparison, the GitHub release dictionary-english_us_arpa-v3.0.0 has 199,860 distinct words. All but 2 of the Hugging Face words are in it, and 145,014 of its words are not in the Hugging Face file.
Is the Hugging Face file reduced on purpose, for example to the training vocabulary? And will the full dictionary be published on Hugging Face?
(Counts: cut -f1 english_us_arpa.dict | sort -u | wc -l.)