How to use fktime/dbert_ai4p with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="fktime/dbert_ai4p")
# Load model directly from transformers import AutoTokenizer, AutoModelForTokenClassification tokenizer = AutoTokenizer.from_pretrained("fktime/dbert_ai4p") model = AutoModelForTokenClassification.from_pretrained("fktime/dbert_ai4p", device_map="auto")
https://huggingface.co/fktime/dbert_ai4p/resolve/main/tokenizer.json
hf download hf://fktime/dbert_ai4p/tokenizer.json
curl -L -o tokenizer.json https://huggingface.co/fktime/dbert_ai4p/resolve/main/tokenizer.json
