Quagmire1
/

wiki-cased

Model card Files Files and versions

wiki-cased / tools /mosesdecoder /scripts /tokenizer

68.2 kB

Ctrl+K

Ctrl+K

1 contributor

History: 1 commit

Quagmire1's picture

Upload folder using huggingface_hub

41f6dd8 verified almost 2 years ago

mosestokenizer
Upload folder using huggingface_hub almost 2 years ago
basic-protected-patterns
267 Bytes
Upload folder using huggingface_hub almost 2 years ago
deescape-special-chars-PTB.perl
760 Bytes
Upload folder using huggingface_hub almost 2 years ago
deescape-special-chars.perl
729 Bytes
Upload folder using huggingface_hub almost 2 years ago
delete-long-words.perl
310 Bytes
Upload folder using huggingface_hub almost 2 years ago
detokenizer.perl
12.5 kB
Upload folder using huggingface_hub almost 2 years ago
escape-special-chars.perl
847 Bytes
Upload folder using huggingface_hub almost 2 years ago
lowercase.perl
383 Bytes
Upload folder using huggingface_hub almost 2 years ago
normalize-punctuation.perl
1.91 kB
Upload folder using huggingface_hub almost 2 years ago
pre-tok-clean.perl
1.43 kB
Upload folder using huggingface_hub almost 2 years ago
pre-tokenizer.perl
967 Bytes
Upload folder using huggingface_hub almost 2 years ago
pre_tokenize_cleaning.py
2.96 kB
Upload folder using huggingface_hub almost 2 years ago
remove-non-printing-char.perl
549 Bytes
Upload folder using huggingface_hub almost 2 years ago
replace-unicode-punctuation.perl
872 Bytes
Upload folder using huggingface_hub almost 2 years ago
tokenizer.perl
18.4 kB
Upload folder using huggingface_hub almost 2 years ago
tokenizer_PTB.perl
12.3 kB
Upload folder using huggingface_hub almost 2 years ago