Hugging Face's logo Hugging Face
  • Models
  • Datasets
  • Spaces
  • Buckets new
  • Docs
  • Enterprise
  • Pricing
    • Website
      • Tasks
      • HuggingChat
      • Collections
      • Languages
      • Organizations
    • Community
      • Blog
      • Posts
      • Daily Papers
      • Hardware
      • Learn
      • Discord
      • Forum
      • GitHub
    • Solutions
      • Team & Enterprise
      • Hugging Face PRO
      • Enterprise Support
      • Inference Providers
      • Inference Endpoints
      • Storage Buckets

  • Log In
  • Sign Up

BeaverAI
/
gemma-2-tokenizer

Model card Files Files and versions
xet
Community
gemma-2-tokenizer / README.md
TheDrummer's picture
TheDrummer
Create README.md
157c743 verified about 2 years ago
preview code
|
Raw
History Blame Contribute Delete
248 Bytes

Tokenizer files for this script:

https://huggingface.co/docs/transformers.js/en/api/tokenizers#pretrainedtokenizerapplychattemplateconversation-options--code-string-code--code-tensor-code--code-array--number--code--code-array--array--number---code