DenCT's picture
|
download
raw
3.26 kB

1. Tokenization

Tokenization converts raw text into a sequence of numerical IDs that neural networks can process. Your task is to implement a word-level tokenizer where each word becomes a separate token. Your tokenizer needs to:

  • Build a vocabulary from training texts (mapping words to unique IDs)
  • Encode text into token IDs
  • Decode token IDs back to text
  • Handle special tokens: <PAD> for padding, <UNK> for unknown words, <BOS> for start, <EOS> for end
Examples

Input: "hello world" Output: [4, 5] hello→4, world→5 (based on vocab order)

Input: "hello unknown" Output: [4, 1] hello→4, "unknown" not in vocab → UNK=1

[!hint] Use a set to find unique words in the training text.

[!hint] Always add special tokens to the vocabulary first.

[!hint] When encoding, use the .get() method of the dictionary to handle unknown words.

Requirements

  1. Implement build_vocab: Create mappings between words and unique IDs.
  2. Implement encode: Convert strings to lists of IDs, handling unknown words with <UNK>.
  3. Implement decode: Convert lists of IDs back to strings.

Constraints • Special tokens: <PAD>=0, <UNK>=1, <BOS>=2, <EOS>=3 • Vocabulary should be built from the provided training texts.

Solution

import numpy as np
from typing import List, Dict

class SimpleTokenizer:
    """
    A word-level tokenizer with special tokens.
    """
    def __init__(self):
        self.word_to_id: Dict[str, int] = {}
        self.id_to_word: Dict[int, str] = {}
        self.vocab_size = 0
        
        # Special tokens
        self.pad_token = "<PAD>"
        self.unk_token = "<UNK>"
        self.bos_token = "<BOS>"
        self.eos_token = "<EOS>"

    def build_vocab(self, texts: List[str]) -> None:
        """
        Build vocabulary from a list of texts.
        Add special tokens first, then unique words.
        """
        # Reset mappings
        self.word_to_id.clear()
        self.id_to_word.clear()

        # Add special tokens first (fixed IDs)
        specials = [self.pad_token, self.unk_token, self.bos_token, self.eos_token]
        for idx, tok in enumerate(specials):
            self.word_to_id[tok] = idx
            self.id_to_word[idx] = tok

        # Collect unique words from training texts
        unique_words = set()
        for text in texts:
            for word in text.strip().split():
                unique_words.add(word)

        # Add words to vocabulary in a deterministic order
        for word in sorted(unique_words):
            if word in self.word_to_id:
                continue
            idx = len(self.word_to_id)
            self.word_to_id[word] = idx
            self.id_to_word[idx] = word

        self.vocab_size = len(self.word_to_id)
    def encode(self, text: str) -> List[int]:
        """
        Convert text to list of token IDs.
        Use UNK for unknown words.
        """
        unk_id = self.word_to_id.get(self.unk_token, 1)
        words = text.strip().split()
        return [self.word_to_id.get(w, unk_id) for w in words]
    def decode(self, ids: List[int]) -> str:
        """
        Convert list of token IDs back to text.
        """
        return " ".join(self.id_to_word.get(i, self.unk_token) for i in ids)

Xet Storage Details

Size:
3.26 kB
·
Xet hash:
67574b6366f0f14a5047b7ed83ad4a8e92890cbf607dd87c228615cd6e58eedc

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.