Hugging Face
Models
Datasets
Spaces
Buckets
new
Docs
Enterprise
Pricing
Website
Tasks
HuggingChat
Collections
Languages
Organizations
Community
Blog
Posts
Daily Papers
Hardware
Learn
Discord
Forum
GitHub
Solutions
Team & Enterprise
Hugging Face PRO
Enterprise Support
Inference Providers
Inference Endpoints
Storage Buckets
Log In
Sign Up
picur
/
picur-tokenizer
Like
1
Follow
Picur
2
Hungarian
tokenizers
picur
tokenizer
hungarian
magyar
morphology
sentencepiece
unigram
License:
apache-2.0
Model card
Files
Files and versions
xet
Community
Copy to bucket
new
main
picur-tokenizer
4.38 MB
Ctrl+K
Ctrl+K
1 contributor
History:
8 commits
lazos
a2C vocabulary: refit on the cleaned, budget-proportional corpus
a858237
verified
25 days ago
.gitattributes
Safe
136 Bytes
picur-tokenizer a2x: 49,152-piece Hungarian Unigram vocabulary
about 1 month ago
LICENSE
Safe
11.3 kB
picur-tokenizer a2x: 49,152-piece Hungarian Unigram vocabulary
about 1 month ago
NOTICE
Safe
2.43 kB
picur-tokenizer a2x: 49,152-piece Hungarian Unigram vocabulary
about 1 month ago
README.md
Safe
9.3 kB
a2C vocabulary: refit on the cleaned, budget-proportional corpus
25 days ago
benchmark.py
Safe
4.63 kB
a2C vocabulary: refit on the cleaned, budget-proportional corpus
25 days ago
chat_template.jinja
Safe
2.04 kB
picur-tokenizer a2x: 49,152-piece Hungarian Unigram vocabulary
about 1 month ago
ties.json
Safe
21.3 kB
a2C vocabulary: refit on the cleaned, budget-proportional corpus
25 days ago
tokenizer.json
Safe
3.28 MB
a2C vocabulary: refit on the cleaned, budget-proportional corpus
25 days ago
tokenizer.model
Safe
1.04 MB
xet
a2C vocabulary: refit on the cleaned, budget-proportional corpus
25 days ago
tokenizer_config.json
Safe
776 Bytes
picur-tokenizer a2x: 49,152-piece Hungarian Unigram vocabulary
about 1 month ago
