latincy

/

latin-bert

Like 1

Follow

LatinCy 27

In tokenizers==0.23 BERTProcessing signature changes, causing the tokenizer to fail

    tokenizer.post_processor = processors.BertProcessing(
                               ~~~~~~~~~~~~~~~~~~~~~~~~~^
        sep=("[SEP]", vocab["[SEP]"]),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        cls=("[CLS]", vocab["[CLS]"]),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
TypeError: BertProcessing.__new__() got an unexpected keyword argument 'cls'

I believe this can be remedied by changing the call in tokenization_latin_bert_fast.py to:

tokenizer.post_processor = processors.BertProcessing(
    ("[SEP]", vocab["[SEP]"]),
    ("[CLS]", vocab["[CLS]"]),
)