How to use hmteams/teams-tokenizer with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForPreTraining tokenizer = AutoTokenizer.from_pretrained("hmteams/teams-tokenizer") model = AutoModelForPreTraining.from_pretrained("hmteams/teams-tokenizer", device_map="auto")
# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForPreTraining tokenizer = AutoTokenizer.from_pretrained("hmteams/teams-tokenizer") model = AutoModelForPreTraining.from_pretrained("hmteams/teams-tokenizer", device_map="auto")