Buckets:
| # Node | |
| The Node binding loads a `tokenizer.json`, encodes text into token ids and | |
| decodes ids back into text. It does not build, edit, save or train tokenizers. | |
| ## Installation | |
| ```bash | |
| npm install tokenizers | |
| ``` | |
| ## Loading a tokenizer | |
| ```js | |
| const { Tokenizer } = require('tokenizers') | |
| const tokenizer = Tokenizer.fromFile('tokenizer.json') | |
| ``` | |
| The file is put through the legacy `1.0` → canonical `2.0` upgrade on the way in, | |
| so tokenizers already on disk keep loading. | |
| ## Encoding | |
| `encode` takes a text, or a pair of texts given as a two-item array, and returns | |
| an `Encoding`. The post-processor joins a pair, for example as | |
| `[CLS] A [SEP] B [SEP]`. | |
| ```js | |
| const encoding = tokenizer.encode("Hello, y'all! How are you 😁 ?") | |
| const pair = tokenizer.encode(["Hello, y'all!", 'How are you 😁 ?']) | |
| ``` | |
| | Property | Description | | |
| | --- | --- | | |
| | `ids` | The id of each token. | | |
| | `typeIds` | The type id of each token: 0 for the first text of a pair, 1 for the second. | | |
| | `attentionMask` | 1 for each token, 0 for each padding token. | | |
| Each property is a `Uint32Array` rather than a JS `Array`: an `Array` costs one | |
| napi value per token, which on token-dense input is 13x the encode itself. | |
| `encodeBatch` encodes a batch of texts and pairs in one call, and returns the | |
| encodings in input order. Batch padding pads every item to the longest one. | |
| ```js | |
| const batch = tokenizer.encodeBatch(["Hello, y'all!", ["Hello, y'all!", 'How are you 😁 ?']], { padding: {} }) | |
| ``` | |
| `tokenize` returns the token of each id, special tokens included. It is | |
| `decodeTokens(encode(input, options).ids)`. | |
| ```js | |
| const tokens = tokenizer.tokenize("Hello, y'all!") | |
| ``` | |
| For a tight loop, `encodeBytesInto` encodes one text and writes its ids into a | |
| `Uint32Array` you own. That drops two per-call costs: the JS string → UTF-8 copy, | |
| and a fresh `ArrayBuffer` for every call. It returns how many ids it wrote: | |
| ```js | |
| const out = new Uint32Array(512) | |
| const n = tokenizer.encodeBytesInto(Buffer.from("Hello, y'all!"), out) | |
| // out.subarray(0, n) holds the ids | |
| ``` | |
| ## Decoding | |
| `decode` turns ids back into text and skips special tokens unless its second | |
| argument is `false`. `decodeTokens` gives the token of each id without running the | |
| decoder, so a byte-level BPE token reads `Ġworld`, not ` world`. It keeps special | |
| tokens unless its second argument is `true`. | |
| ```js | |
| const text = tokenizer.decode(encoding.ids) | |
| const tokens = tokenizer.decodeTokens(encoding.ids) | |
| ``` | |
| ## Options | |
| Every encode method takes an optional last argument. A field left out keeps the | |
| tokenizer's own behaviour. | |
| ```js | |
| const encoding = tokenizer.encode("Hello, y'all!", { | |
| addSpecialTokens: true, | |
| encodeSpecialTokens: false, | |
| padding: { padId: 0, padToken: '[PAD]', length: 16 }, | |
| truncation: { maxLength: 512 }, | |
| }) | |
| ``` | |
| | Option | Default | Description | | |
| | --- | --- | --- | | |
| | `addSpecialTokens` | `true` | Whether the post-processor adds its special tokens, such as `[CLS]` and `[SEP]`. | | |
| | `encodeSpecialTokens` | `false` | Whether a special token written in the text goes through the model (`true`) or becomes its added-vocabulary id. | | |
| | `padding` | configured | Padding for this call, replacing the tokenizer's configured padding. `false` disables it. | | |
| | `truncation` | configured | Truncation for this call, replacing the tokenizer's configured truncation. `false` disables it. | | |
| ### `padding` | |
| Replaces the configured padding as a whole, so a field left out takes the default | |
| below, not the configured value. | |
| | Field | Default | Description | | |
| | --- | --- | --- | | |
| | `direction` | `right` | Whether pad tokens are appended right or prepended left. | | |
| | `padId` | `0` | The id of the padding token. | | |
| | `padTypeId` | `0` | The type id of the padding token. | | |
| | `padToken` | `[PAD]` | The text of the padding token. | | |
| | `length` | batch longest | Pads every encoding to exactly this many tokens. | | |
| | `padToMultipleOf` | none | Rounds the padded length up to a multiple of this. | | |
| ### `truncation` | |
| | Field | Default | Description | | |
| | --- | --- | --- | | |
| | `maxLength` | required | The maximum number of tokens, including special tokens, to keep. | | |
| | `strategy` | `longest_first` | Which sequence of a pair is truncated: `longest_first`, `only_first` or `only_second`. | | |
| | `direction` | `right` | Whether to truncate at the end of the sequence or at its beginning. | | |
Xet Storage Details
- Size:
- 4.35 kB
- Xet hash:
- b1868452bd709c99197e5beb2188642088b836857e83c34ce31eb67403ba909b
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.