Buckets:

|
download
raw
4.35 kB

Node

The Node binding loads a tokenizer.json, encodes text into token ids and decodes ids back into text. It does not build, edit, save or train tokenizers.

Installation

npm install tokenizers

Loading a tokenizer

const { Tokenizer } = require('tokenizers')

const tokenizer = Tokenizer.fromFile('tokenizer.json')

The file is put through the legacy 1.0 → canonical 2.0 upgrade on the way in, so tokenizers already on disk keep loading.

Encoding

encode takes a text, or a pair of texts given as a two-item array, and returns an Encoding. The post-processor joins a pair, for example as [CLS] A [SEP] B [SEP].

const encoding = tokenizer.encode("Hello, y'all! How are you 😁 ?")
const pair = tokenizer.encode(["Hello, y'all!", 'How are you 😁 ?'])
Property Description
ids The id of each token.
typeIds The type id of each token: 0 for the first text of a pair, 1 for the second.
attentionMask 1 for each token, 0 for each padding token.

Each property is a Uint32Array rather than a JS Array: an Array costs one napi value per token, which on token-dense input is 13x the encode itself.

encodeBatch encodes a batch of texts and pairs in one call, and returns the encodings in input order. Batch padding pads every item to the longest one.

const batch = tokenizer.encodeBatch(["Hello, y'all!", ["Hello, y'all!", 'How are you 😁 ?']], { padding: {} })

tokenize returns the token of each id, special tokens included. It is decodeTokens(encode(input, options).ids).

const tokens = tokenizer.tokenize("Hello, y'all!")

For a tight loop, encodeBytesInto encodes one text and writes its ids into a Uint32Array you own. That drops two per-call costs: the JS string → UTF-8 copy, and a fresh ArrayBuffer for every call. It returns how many ids it wrote:

const out = new Uint32Array(512)
const n = tokenizer.encodeBytesInto(Buffer.from("Hello, y'all!"), out)
// out.subarray(0, n) holds the ids

Decoding

decode turns ids back into text and skips special tokens unless its second argument is false. decodeTokens gives the token of each id without running the decoder, so a byte-level BPE token reads Ġworld, not world. It keeps special tokens unless its second argument is true.

const text = tokenizer.decode(encoding.ids)
const tokens = tokenizer.decodeTokens(encoding.ids)

Options

Every encode method takes an optional last argument. A field left out keeps the tokenizer's own behaviour.

const encoding = tokenizer.encode("Hello, y'all!", {
  addSpecialTokens: true,
  encodeSpecialTokens: false,
  padding: { padId: 0, padToken: '[PAD]', length: 16 },
  truncation: { maxLength: 512 },
})
Option Default Description
addSpecialTokens true Whether the post-processor adds its special tokens, such as [CLS] and [SEP].
encodeSpecialTokens false Whether a special token written in the text goes through the model (true) or becomes its added-vocabulary id.
padding configured Padding for this call, replacing the tokenizer's configured padding. false disables it.
truncation configured Truncation for this call, replacing the tokenizer's configured truncation. false disables it.

padding

Replaces the configured padding as a whole, so a field left out takes the default below, not the configured value.

Field Default Description
direction right Whether pad tokens are appended right or prepended left.
padId 0 The id of the padding token.
padTypeId 0 The type id of the padding token.
padToken [PAD] The text of the padding token.
length batch longest Pads every encoding to exactly this many tokens.
padToMultipleOf none Rounds the padded length up to a multiple of this.

truncation

Field Default Description
maxLength required The maximum number of tokens, including special tokens, to keep.
strategy longest_first Which sequence of a pair is truncated: longest_first, only_first or only_second.
direction right Whether to truncate at the end of the sequence or at its beginning.

Xet Storage Details

Size:
4.35 kB
·
Xet hash:
b1868452bd709c99197e5beb2188642088b836857e83c34ce31eb67403ba909b

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.