Buckets:

|
download
raw
1.48 kB
# Installation
🤗 Tokenizers is tested on Python 3.10+.
We highly recommend using `uv` for virtual environment installs.
## Installation with pip
🤗 Tokenizers can be installed using pip as follows:
```bash
pip install tokenizers
```
## Installation from sources
To use this method, you need to have the Rust language installed. You can follow [the official guide](https://www.rust-lang.org/learn/get-started) for more information.
If you are using a unix based OS, the installation should be as simple as running:
```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```
Or you can easily update it with the following command:
```bash
rustup update
```
Once rust is installed, we can start retrieving the sources for 🤗
Tokenizers:
```bash
git clone https://github.com/huggingface/tokenizers
```
Then we go into the python bindings folder:
```bash
cd tokenizers/bindings/python
```
At this point you should have your [virtual environment]() already
activated. In order to compile 🤗 Tokenizers, you need to:
```bash
pip install -e .
```
or, if you want to dev you need to install `maturin` and run:
```bash
maturin dev
```
## Crates.io
🤗 Tokenizers is available on [crates.io](https://crates.io/crates/tokenizers).
You just need to add it to your `Cargo.toml`:
```bash
cargo add tokenizers
```
## Installation with npm
```bash
npm install tokenizers
```
See the [Node page](node) for what the binding exposes and how to use it.

Xet Storage Details

Size:
1.48 kB
·
Xet hash:
e65dea9411a05ebe424075eee4c11a7554fb53b357f45d512248d84944949c83

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.