How to use tensor-polinomics/tokenizer_python_52k with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("tensor-polinomics/tokenizer_python_52k", device_map="auto")
