Hugging Face's logo Hugging Face
  • Models
  • Datasets
  • Spaces
  • Buckets
  • Docs
  • Enterprise
  • Pricing
    • Website
      • Tasks
      • HuggingChat
      • Collections
      • Languages
      • Organizations
    • Community
      • Blog
      • Posts
      • Daily Papers
      • Hardware
      • Learn
      • Discord
      • Forum
      • GitHub
    • Solutions
      • Team & Enterprise
      • Hugging Face PRO
      • Enterprise Support
      • Inference Providers
      • Inference Endpoints
      • Storage Buckets

  • Log In
  • Sign Up

wonhui-labs
/
swt-tokenizer

Token Classification
multilingual
tokenizer
sentencepiece
byte-fallback
Model card Files Files and versions
xet
Community
1

You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

Gated model
You can list files but not access them

Preview of files found in this repository
  • .gitattributes
    1.57 kB
    Publish SWT fast tokenizer: byte-fallback BPE, 199k vocab, parity-validated with evaluation report 2 months ago
  • README.md
    2.43 kB
    Update README.md 2 months ago
  • requirements.txt
    79 Bytes
    Publish SWT fast tokenizer: byte-fallback BPE, 199k vocab, parity-validated with evaluation report 2 months ago
  • tokenizer.json
    21.5 MB
    xet
    Publish SWT fast tokenizer: byte-fallback BPE, 199k vocab, parity-validated with evaluation report 2 months ago
  • tokenizer_config.json
    18.5 kB
    Publish SWT fast tokenizer: byte-fallback BPE, 199k vocab, parity-validated with evaluation report 2 months ago
  • unified.model
    3.69 MB
    xet
    Publish SWT fast tokenizer: byte-fallback BPE, 199k vocab, parity-validated with evaluation report 2 months ago