DenCT's picture
|
download
raw
3.26 kB
## 1. Tokenization
Tokenization converts raw text into a sequence of numerical IDs that neural networks can process.
Your task is to implement a word-level tokenizer where each word becomes a separate token.
Your tokenizer needs to:
- Build a vocabulary from training texts (mapping words to unique IDs)
- Encode text into token IDs
- Decode token IDs back to text
- Handle special tokens: `<PAD>` for padding, `<UNK>` for unknown words, `<BOS>` for start, `<EOS>` for end
##### Examples
Input: "hello world"
Output: [4, 5]
hello→4, world→5 (based on vocab order)
Input: "hello unknown"
Output: [4, 1]
hello→4, "unknown" not in vocab → UNK=1
>[!hint]
>Use a set to find unique words in the training text.
> [!hint]
Always add special tokens to the vocabulary first.
>[!hint]
When encoding, use the .get() method of the dictionary to handle unknown words.
Requirements
1. Implement build_vocab: Create mappings between words and unique IDs.
2. Implement encode: Convert strings to lists of IDs, handling unknown words with `<UNK>`.
3. Implement decode: Convert lists of IDs back to strings.
Constraints
• Special tokens: `<PAD>=0`, `<UNK>=1`, `<BOS>=2`, `<EOS>=3`
• Vocabulary should be built from the provided training texts.
#### Solution
```python
import numpy as np
from typing import List, Dict
class SimpleTokenizer:
"""
A word-level tokenizer with special tokens.
"""
def __init__(self):
self.word_to_id: Dict[str, int] = {}
self.id_to_word: Dict[int, str] = {}
self.vocab_size = 0
# Special tokens
self.pad_token = "<PAD>"
self.unk_token = "<UNK>"
self.bos_token = "<BOS>"
self.eos_token = "<EOS>"
def build_vocab(self, texts: List[str]) -> None:
"""
Build vocabulary from a list of texts.
Add special tokens first, then unique words.
"""
# Reset mappings
self.word_to_id.clear()
self.id_to_word.clear()
# Add special tokens first (fixed IDs)
specials = [self.pad_token, self.unk_token, self.bos_token, self.eos_token]
for idx, tok in enumerate(specials):
self.word_to_id[tok] = idx
self.id_to_word[idx] = tok
# Collect unique words from training texts
unique_words = set()
for text in texts:
for word in text.strip().split():
unique_words.add(word)
# Add words to vocabulary in a deterministic order
for word in sorted(unique_words):
if word in self.word_to_id:
continue
idx = len(self.word_to_id)
self.word_to_id[word] = idx
self.id_to_word[idx] = word
self.vocab_size = len(self.word_to_id)
def encode(self, text: str) -> List[int]:
"""
Convert text to list of token IDs.
Use UNK for unknown words.
"""
unk_id = self.word_to_id.get(self.unk_token, 1)
words = text.strip().split()
return [self.word_to_id.get(w, unk_id) for w in words]
def decode(self, ids: List[int]) -> str:
"""
Convert list of token IDs back to text.
"""
return " ".join(self.id_to_word.get(i, self.unk_token) for i in ids)
```

Xet Storage Details

Size:
3.26 kB
·
Xet hash:
67574b6366f0f14a5047b7ed83ad4a8e92890cbf607dd87c228615cd6e58eedc

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.