Instructions to use otmanheddouch/shakespear-tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use otmanheddouch/shakespear-tokenizer with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("otmanheddouch/shakespear-tokenizer", device_map="auto") - Notebooks
- Google Colab
- Kaggle
File size: 549 Bytes
2412a74 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | from transformers import AutoTokenizer
class EndpointHandler():
def __init__(self, path=""):
self.tokenizer = AutoTokenizer.from_pretrained("otmanheddouch/shakespear-tokenizer")
def __call__(self, data: str] -> Dict[str, Any]]:
"""
data args:
data: text input
Return:
output (dict) : outputs with tokens and their ids
"""
tokens = self.tokenizer.tokenize()
tokens_ids = self.tokenizer.encode(example)
return {"tokens:"tokens, "ids":tokens_ids} |