Model Card for Model ID
This is a tokenizer used in FAM model
Tokenizer Details
- Architecture: BPE
- Normalizer: BertNormalizer
- Pre-tokenizer: BertPreTokenizer
- Special tokens: [UNK], [PAD], [CLS], [SEP], [MASK]
- Dataset:
- Languages: English
Tokenizer training noteboook:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Sergeial1972-user/FAM-tokenizer", device_map="auto")