Buckets:
| # Installation | |
| 🤗 Tokenizers is tested on Python 3.10+. | |
| We highly recommend using `uv` for virtual environment installs. | |
| ## Installation with pip | |
| 🤗 Tokenizers can be installed using pip as follows: | |
| ```bash | |
| pip install tokenizers | |
| ``` | |
| ## Installation from sources | |
| To use this method, you need to have the Rust language installed. You can follow [the official guide](https://www.rust-lang.org/learn/get-started) for more information. | |
| If you are using a unix based OS, the installation should be as simple as running: | |
| ```bash | |
| curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh | |
| ``` | |
| Or you can easily update it with the following command: | |
| ```bash | |
| rustup update | |
| ``` | |
| Once rust is installed, we can start retrieving the sources for 🤗 | |
| Tokenizers: | |
| ```bash | |
| git clone https://github.com/huggingface/tokenizers | |
| ``` | |
| Then we go into the python bindings folder: | |
| ```bash | |
| cd tokenizers/bindings/python | |
| ``` | |
| At this point you should have your [virtual environment]() already | |
| activated. In order to compile 🤗 Tokenizers, you need to: | |
| ```bash | |
| pip install -e . | |
| ``` | |
| or, if you want to dev you need to install `maturin` and run: | |
| ```bash | |
| maturin dev | |
| ``` | |
| ## Crates.io | |
| 🤗 Tokenizers is available on [crates.io](https://crates.io/crates/tokenizers). | |
| You just need to add it to your `Cargo.toml`: | |
| ```bash | |
| cargo add tokenizers | |
| ``` | |
| ## Installation with npm | |
| ```bash | |
| npm install tokenizers | |
| ``` | |
| See the [Node page](node) for what the binding exposes and how to use it. | |
Xet Storage Details
- Size:
- 1.48 kB
- Xet hash:
- e65dea9411a05ebe424075eee4c11a7554fb53b357f45d512248d84944949c83
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.