Buckets:
| ## 1. Tokenization | |
| Tokenization converts raw text into a sequence of numerical IDs that neural networks can process. | |
| Your task is to implement a word-level tokenizer where each word becomes a separate token. | |
| Your tokenizer needs to: | |
| - Build a vocabulary from training texts (mapping words to unique IDs) | |
| - Encode text into token IDs | |
| - Decode token IDs back to text | |
| - Handle special tokens: `<PAD>` for padding, `<UNK>` for unknown words, `<BOS>` for start, `<EOS>` for end | |
| ##### Examples | |
| Input: "hello world" | |
| Output: [4, 5] | |
| hello→4, world→5 (based on vocab order) | |
| Input: "hello unknown" | |
| Output: [4, 1] | |
| hello→4, "unknown" not in vocab → UNK=1 | |
| >[!hint] | |
| >Use a set to find unique words in the training text. | |
| > [!hint] | |
| Always add special tokens to the vocabulary first. | |
| >[!hint] | |
| When encoding, use the .get() method of the dictionary to handle unknown words. | |
| Requirements | |
| 1. Implement build_vocab: Create mappings between words and unique IDs. | |
| 2. Implement encode: Convert strings to lists of IDs, handling unknown words with `<UNK>`. | |
| 3. Implement decode: Convert lists of IDs back to strings. | |
| Constraints | |
| • Special tokens: `<PAD>=0`, `<UNK>=1`, `<BOS>=2`, `<EOS>=3` | |
| • Vocabulary should be built from the provided training texts. | |
| #### Solution | |
| ```python | |
| import numpy as np | |
| from typing import List, Dict | |
| class SimpleTokenizer: | |
| """ | |
| A word-level tokenizer with special tokens. | |
| """ | |
| def __init__(self): | |
| self.word_to_id: Dict[str, int] = {} | |
| self.id_to_word: Dict[int, str] = {} | |
| self.vocab_size = 0 | |
| # Special tokens | |
| self.pad_token = "<PAD>" | |
| self.unk_token = "<UNK>" | |
| self.bos_token = "<BOS>" | |
| self.eos_token = "<EOS>" | |
| def build_vocab(self, texts: List[str]) -> None: | |
| """ | |
| Build vocabulary from a list of texts. | |
| Add special tokens first, then unique words. | |
| """ | |
| # Reset mappings | |
| self.word_to_id.clear() | |
| self.id_to_word.clear() | |
| # Add special tokens first (fixed IDs) | |
| specials = [self.pad_token, self.unk_token, self.bos_token, self.eos_token] | |
| for idx, tok in enumerate(specials): | |
| self.word_to_id[tok] = idx | |
| self.id_to_word[idx] = tok | |
| # Collect unique words from training texts | |
| unique_words = set() | |
| for text in texts: | |
| for word in text.strip().split(): | |
| unique_words.add(word) | |
| # Add words to vocabulary in a deterministic order | |
| for word in sorted(unique_words): | |
| if word in self.word_to_id: | |
| continue | |
| idx = len(self.word_to_id) | |
| self.word_to_id[word] = idx | |
| self.id_to_word[idx] = word | |
| self.vocab_size = len(self.word_to_id) | |
| def encode(self, text: str) -> List[int]: | |
| """ | |
| Convert text to list of token IDs. | |
| Use UNK for unknown words. | |
| """ | |
| unk_id = self.word_to_id.get(self.unk_token, 1) | |
| words = text.strip().split() | |
| return [self.word_to_id.get(w, unk_id) for w in words] | |
| def decode(self, ids: List[int]) -> str: | |
| """ | |
| Convert list of token IDs back to text. | |
| """ | |
| return " ".join(self.id_to_word.get(i, self.unk_token) for i in ids) | |
| ``` | |
Xet Storage Details
- Size:
- 3.26 kB
- Xet hash:
- 67574b6366f0f14a5047b7ed83ad4a8e92890cbf607dd87c228615cd6e58eedc
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.