Buckets:
Installation
🤗 Tokenizers is tested on Python 3.10+.
We highly recommend using uv for virtual environment installs.
Installation with pip
🤗 Tokenizers can be installed using pip as follows:
pip install tokenizers
Installation from sources
To use this method, you need to have the Rust language installed. You can follow the official guide for more information.
If you are using a unix based OS, the installation should be as simple as running:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Or you can easily update it with the following command:
rustup update
Once rust is installed, we can start retrieving the sources for 🤗 Tokenizers:
git clone https://github.com/huggingface/tokenizers
Then we go into the python bindings folder:
cd tokenizers/bindings/python
At this point you should have your virtual environment already activated. In order to compile 🤗 Tokenizers, you need to:
pip install -e .
or, if you want to dev you need to install maturin and run:
maturin dev
Crates.io
🤗 Tokenizers is available on crates.io.
You just need to add it to your Cargo.toml:
cargo add tokenizers
Installation with npm
npm install tokenizers
See the Node page for what the binding exposes and how to use it.
Xet Storage Details
- Size:
- 1.48 kB
- Xet hash:
- e65dea9411a05ebe424075eee4c11a7554fb53b357f45d512248d84944949c83
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.