--- library_name: transformers pipeline_tag: feature-extraction tags: - tokenizer --- # gemma-tokenizer The text tokenizer used in Google's `Gemma` models. ## Usage ```py from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("much1na/gemma-tokenizer") ids = tokenizer("hello world").input_ids print(ids) # [2, 23391, 1902, 1] print(tokenizer.decode(ids, skip_special_tokens=True)) # hello world ```