Hugging Face's logo Hugging Face
  • Models
  • Datasets
  • Spaces
  • Buckets
  • Docs
  • Enterprise
  • Pricing
    • Website
      • Tasks
      • HuggingChat
      • Collections
      • Languages
      • Organizations
    • Community
      • Blog
      • Posts
      • Daily Papers
      • Hardware
      • Learn
      • Discord
      • Forum
      • GitHub
    • Solutions
      • Team & Enterprise
      • Hugging Face PRO
      • Enterprise Support
      • Inference Providers
      • Inference Endpoints
      • Storage Buckets

  • Log In
  • Sign Up

zhagaramlabs
/
ezhuthu-64k

Transformers
Tamil
English
tokenizer
tamil
bpe
Model card Files Files and versions
xet
Community

Instructions to use zhagaramlabs/ezhuthu-64k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

  • Libraries
  • Transformers

    How to use zhagaramlabs/ezhuthu-64k with Transformers:

    # pip install -U transformers accelerate
    # Load model directly
    from transformers import AutoModel
    model = AutoModel.from_pretrained("zhagaramlabs/ezhuthu-64k", device_map="auto")
  • Notebooks
  • Google Colab
  • Kaggle
ezhuthu-64k
5.14 MB
Ctrl+K
Ctrl+K
  • 1 contributor
History: 2 commits
gopalanj's picture
gopalanj
Ezhuthu 64k: whole-letter Tamil BPE tokenizer with stem-suffix cuts (r4-morph-64k)
78616b5 verified 7 days ago
  • .gitattributes
    1.52 kB
    initial commit 7 days ago
  • README.md
    15.8 kB
    Ezhuthu 64k: whole-letter Tamil BPE tokenizer with stem-suffix cuts (r4-morph-64k) 7 days ago
  • tokenizer.json
    5.12 MB
    Ezhuthu 64k: whole-letter Tamil BPE tokenizer with stem-suffix cuts (r4-morph-64k) 7 days ago
  • tokenizer_config.json
    237 Bytes
    Ezhuthu 64k: whole-letter Tamil BPE tokenizer with stem-suffix cuts (r4-morph-64k) 7 days ago