Buckets:
1. Tokenization
Tokenization converts raw text into a sequence of numerical IDs that neural networks can process. Your task is to implement a word-level tokenizer where each word becomes a separate token. Your tokenizer needs to:
- Build a vocabulary from training texts (mapping words to unique IDs)
- Encode text into token IDs
- Decode token IDs back to text
- Handle special tokens:
<PAD>for padding,<UNK>for unknown words,<BOS>for start,<EOS>for end
Examples
Input: "hello world" Output: [4, 5] hello→4, world→5 (based on vocab order)
Input: "hello unknown" Output: [4, 1] hello→4, "unknown" not in vocab → UNK=1
[!hint] Use a set to find unique words in the training text.
[!hint] Always add special tokens to the vocabulary first.
[!hint] When encoding, use the .get() method of the dictionary to handle unknown words.
Requirements
- Implement build_vocab: Create mappings between words and unique IDs.
- Implement encode: Convert strings to lists of IDs, handling unknown words with
<UNK>. - Implement decode: Convert lists of IDs back to strings.
Constraints
• Special tokens: <PAD>=0, <UNK>=1, <BOS>=2, <EOS>=3
• Vocabulary should be built from the provided training texts.
Solution
import numpy as np
from typing import List, Dict
class SimpleTokenizer:
"""
A word-level tokenizer with special tokens.
"""
def __init__(self):
self.word_to_id: Dict[str, int] = {}
self.id_to_word: Dict[int, str] = {}
self.vocab_size = 0
# Special tokens
self.pad_token = "<PAD>"
self.unk_token = "<UNK>"
self.bos_token = "<BOS>"
self.eos_token = "<EOS>"
def build_vocab(self, texts: List[str]) -> None:
"""
Build vocabulary from a list of texts.
Add special tokens first, then unique words.
"""
# Reset mappings
self.word_to_id.clear()
self.id_to_word.clear()
# Add special tokens first (fixed IDs)
specials = [self.pad_token, self.unk_token, self.bos_token, self.eos_token]
for idx, tok in enumerate(specials):
self.word_to_id[tok] = idx
self.id_to_word[idx] = tok
# Collect unique words from training texts
unique_words = set()
for text in texts:
for word in text.strip().split():
unique_words.add(word)
# Add words to vocabulary in a deterministic order
for word in sorted(unique_words):
if word in self.word_to_id:
continue
idx = len(self.word_to_id)
self.word_to_id[word] = idx
self.id_to_word[idx] = word
self.vocab_size = len(self.word_to_id)
def encode(self, text: str) -> List[int]:
"""
Convert text to list of token IDs.
Use UNK for unknown words.
"""
unk_id = self.word_to_id.get(self.unk_token, 1)
words = text.strip().split()
return [self.word_to_id.get(w, unk_id) for w in words]
def decode(self, ids: List[int]) -> str:
"""
Convert list of token IDs back to text.
"""
return " ".join(self.id_to_word.get(i, self.unk_token) for i in ids)
Xet Storage Details
- Size:
- 3.26 kB
- Xet hash:
- 67574b6366f0f14a5047b7ed83ad4a8e92890cbf607dd87c228615cd6e58eedc
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.