Buckets:
Node
The Node binding loads a tokenizer.json, encodes text into token ids and
decodes ids back into text. It does not build, edit, save or train tokenizers.
Installation
npm install tokenizers
Loading a tokenizer
const { Tokenizer } = require('tokenizers')
const tokenizer = Tokenizer.fromFile('tokenizer.json')
The file is put through the legacy 1.0 → canonical 2.0 upgrade on the way in,
so tokenizers already on disk keep loading.
Encoding
encode takes a text, or a pair of texts given as a two-item array, and returns
an Encoding. The post-processor joins a pair, for example as
[CLS] A [SEP] B [SEP].
const encoding = tokenizer.encode("Hello, y'all! How are you 😁 ?")
const pair = tokenizer.encode(["Hello, y'all!", 'How are you 😁 ?'])
| Property | Description |
|---|---|
ids |
The id of each token. |
typeIds |
The type id of each token: 0 for the first text of a pair, 1 for the second. |
attentionMask |
1 for each token, 0 for each padding token. |
Each property is a Uint32Array rather than a JS Array: an Array costs one
napi value per token, which on token-dense input is 13x the encode itself.
encodeBatch encodes a batch of texts and pairs in one call, and returns the
encodings in input order. Batch padding pads every item to the longest one.
const batch = tokenizer.encodeBatch(["Hello, y'all!", ["Hello, y'all!", 'How are you 😁 ?']], { padding: {} })
tokenize returns the token of each id, special tokens included. It is
decodeTokens(encode(input, options).ids).
const tokens = tokenizer.tokenize("Hello, y'all!")
For a tight loop, encodeBytesInto encodes one text and writes its ids into a
Uint32Array you own. That drops two per-call costs: the JS string → UTF-8 copy,
and a fresh ArrayBuffer for every call. It returns how many ids it wrote:
const out = new Uint32Array(512)
const n = tokenizer.encodeBytesInto(Buffer.from("Hello, y'all!"), out)
// out.subarray(0, n) holds the ids
Decoding
decode turns ids back into text and skips special tokens unless its second
argument is false. decodeTokens gives the token of each id without running the
decoder, so a byte-level BPE token reads Ġworld, not world. It keeps special
tokens unless its second argument is true.
const text = tokenizer.decode(encoding.ids)
const tokens = tokenizer.decodeTokens(encoding.ids)
Options
Every encode method takes an optional last argument. A field left out keeps the tokenizer's own behaviour.
const encoding = tokenizer.encode("Hello, y'all!", {
addSpecialTokens: true,
encodeSpecialTokens: false,
padding: { padId: 0, padToken: '[PAD]', length: 16 },
truncation: { maxLength: 512 },
})
| Option | Default | Description |
|---|---|---|
addSpecialTokens |
true |
Whether the post-processor adds its special tokens, such as [CLS] and [SEP]. |
encodeSpecialTokens |
false |
Whether a special token written in the text goes through the model (true) or becomes its added-vocabulary id. |
padding |
configured | Padding for this call, replacing the tokenizer's configured padding. false disables it. |
truncation |
configured | Truncation for this call, replacing the tokenizer's configured truncation. false disables it. |
padding
Replaces the configured padding as a whole, so a field left out takes the default below, not the configured value.
| Field | Default | Description |
|---|---|---|
direction |
right |
Whether pad tokens are appended right or prepended left. |
padId |
0 |
The id of the padding token. |
padTypeId |
0 |
The type id of the padding token. |
padToken |
[PAD] |
The text of the padding token. |
length |
batch longest | Pads every encoding to exactly this many tokens. |
padToMultipleOf |
none | Rounds the padded length up to a multiple of this. |
truncation
| Field | Default | Description |
|---|---|---|
maxLength |
required | The maximum number of tokens, including special tokens, to keep. |
strategy |
longest_first |
Which sequence of a pair is truncated: longest_first, only_first or only_second. |
direction |
right |
Whether to truncate at the end of the sequence or at its beginning. |
Xet Storage Details
- Size:
- 4.35 kB
- Xet hash:
- b1868452bd709c99197e5beb2188642088b836857e83c34ce31eb67403ba909b
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.