import json from pathlib import Path import pytest from transformers import AutoTokenizer from v4_prompt import messages_for from core import normalize_question,text from openjudgement import OpenJudgement ROOT=Path(__file__).resolve().parents[1] def test_real_tokenizer_and_examples(): tokenizer=AutoTokenizer.from_pretrained(ROOT,local_files_only=True) assert all(len(tokenizer.encode(str(i),add_special_tokens=False))==1 for i in range(10)) assert len(tokenizer.encode('10',add_special_tokens=False))>1 req=json.loads((ROOT/'examples/request.json').read_text()) for q in req['questions'].values(): kind,instruction,options,keys=normalize_question(q) row={'state':text(req['state']),'kind':kind,'instructions':instruction,'candidates':options,'keys':keys} tokens=tokenizer.apply_chat_template(messages_for(row),tokenize=True,add_generation_prompt=True,enable_thinking=False,return_dict=False) assert isinstance(tokens,list) and 0