BioGptTokenizer vs AutoTokenizer
In the model card, the sample code uses BioGptTokenizer instead of the usual transformers.AutoTokenizer.
Comparing the encoded inputs of the two tokenizers, the IDs seem to be the same for the texts I used.
The model instantiated from BioGptForCausalLM.from_pretrained("microsoft/biogpt") takes the encoded input from either tokenize no problem but the output keys are different.
For AutoTokenizer I get odict_keys(['last_hidden_state', 'past_key_values']), but for BioGptTokenizer I get odict_keys(['logits', 'past_key_values']).
Are last_hidden_state and logits token embeddings or something else?
I get torch.Size([1, n, 1024]) torch.Size([1, n, 42384]) where n is the number of input tokens.
Hi,
If you use
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("microsoft/biogpt")
print(type(tokenizer))
you'll see that it automatically maps to a BioGptTokenizer. So both are equivalent (Auto tokenizer uses the fast tokenizer by default, if there is one).
Do you have a reproducible code snippet that shows that you get different keys in the output dictionary depending on the tokenizer? That shouldn't be the case.
Yes you are right. In the case with odict_keys(['last_hidden_state', 'past_key_values']) I was loading the bare model BioGptModel.from_pretrained("microsoft/biogpt") without the head. I mistakenly thought it was the tokenizer.
The bare model is sufficient for getting the input embeddings as I don't need it transformed back into the vocabulary. Is my understanding correct?
Yes, BioGptModel will only output the "last hidden states" (also called embeddings) for each of the text tokens (input_ids) being passed through it.
BioGptForCausalLM on the other hand will add a classifier head on top of BioGptModel that maps those last hidden states to token indices.
Thanks for clearing it up!