Hugging Face's logo Hugging Face
  • Models
  • Datasets
  • Spaces
  • Buckets new
  • Docs
  • Enterprise
  • Pricing
    • Website
      • Tasks
      • HuggingChat
      • Collections
      • Languages
      • Organizations
    • Community
      • Blog
      • Posts
      • Daily Papers
      • Hardware
      • Learn
      • Discord
      • Forum
      • GitHub
    • Solutions
      • Team & Enterprise
      • Hugging Face PRO
      • Enterprise Support
      • Inference Providers
      • Inference Endpoints
      • Storage Buckets

  • Log In
  • Sign Up

twinkle-ai
/
TwinkleTokenizer

Chinese
English
tokenizers
Taiwan
R.O.C
zhtw
tokenizer
BPE
traditional-chinese
Model card Files Files and versions
xet
Community

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

請說明您的使用目的。本詞表以台灣繁體中文為主要目標,開放供研究與開發使用。

Log in or Sign Up to review the conditions and access this model content.

Gated model
You can list files but not access them

Preview of files found in this repository
  • .gitattributes
    1.57 kB
    TwinkleTokenizer v1: Traditional Chinese (Taiwan) BPE tokenizer, 201,069 vocab about 1 month ago
  • README.md
    19.2 kB
    TwinkleTokenizer v1: Traditional Chinese (Taiwan) BPE tokenizer, 201,069 vocab about 1 month ago
  • README_EN.md
    20.1 kB
    TwinkleTokenizer v1: Traditional Chinese (Taiwan) BPE tokenizer, 201,069 vocab about 1 month ago
  • special_tokens_map.json
    951 Bytes
    TwinkleTokenizer v1: Traditional Chinese (Taiwan) BPE tokenizer, 201,069 vocab about 1 month ago
  • tokenizer.json
    17.7 MB
    xet
    TwinkleTokenizer v1: Traditional Chinese (Taiwan) BPE tokenizer, 201,069 vocab about 1 month ago
  • tokenizer_config.json
    2.3 kB
    TwinkleTokenizer v1: Traditional Chinese (Taiwan) BPE tokenizer, 201,069 vocab about 1 month ago