tokenizers/pr_2449/en/node.md

Node

The Node binding is encode-only. It loads a tokenizer.json and turns text into token ids; it does not decode, and it does not build, edit, save or train tokenizers.

Installation

npm install tokenizers

Loading a tokenizer

const { PipelineTokenizer } = require('tokenizers')

const tokenizer = PipelineTokenizer.fromFile('tokenizer.json')

The file is put through the legacy 1.0 → canonical 2.0 upgrade on the way in, so tokenizers already on disk keep loading.

Encoding

encode returns a Uint32Array of ids rather than an object: a JS Array costs one napi value per token, which on token-dense input is 13x the encode itself.

const ids = tokenizer.encode("Hello, y'all! How are you 😁 ?")

For a tight loop, encodeBytesInto writes into a Uint32Array you own. That drops the two remaining per-call costs — the JS string → UTF-8 copy, and a fresh ArrayBuffer every call — and returns how many ids it wrote:

const out = new Uint32Array(512)
const n = tokenizer.encodeBytesInto(Buffer.from("Hello, y'all!"), out)
// out.subarray(0, n) holds the ids

Options

Both methods take an optional second argument. A field left out keeps the tokenizer's own behaviour.

const ids = tokenizer.encode("Hello, y'all!", {
  addSpecialTokens: true,
  encodeSpecialTokens: false,
  padding: { padId: 0, padToken: '[PAD]', length: 16 },
  truncation: { maxLength: 512 },
})

Option	Default	Description
addSpecialTokens	true	Whether the post-processor adds its special tokens, such as [CLS] and [SEP].
encodeSpecialTokens	false	Whether a special token written in the text goes through the model (true) or becomes its added-vocabulary id.
padding	configured	Padding for this call, replacing the tokenizer's configured padding. false disables it.
truncation	configured	Truncation for this call, replacing the tokenizer's configured truncation. false disables it.

padding

Replaces the configured padding as a whole, so a field left out takes the default below, not the configured value.

Field	Default	Description
direction	right	Whether pad tokens are appended right or prepended left.
padId	0	The id of the padding token.
padTypeId	0	The type id of the padding token.
padToken	[PAD]	The text of the padding token.
length	batch longest	Pads every encoding to exactly this many tokens.
padToMultipleOf	—	Rounds the padded length up to a multiple of this.

truncation

Field	Default	Description
maxLength	required	The maximum number of tokens, including special tokens, to keep.
strategy	longest_first	Which sequence of a pair is truncated: longest_first, only_first or only_second.
direction	right	Whether to truncate at the end of the sequence or at its beginning.