Commit History
-
Document pretraining tokenizer behavior
Verified signatureAdd examples showing disabled BOS, explicit raw </s> tokenization, pretraining chat template rendering, and assistant token mask behavior.7a5473c -
Append </s> in pretraining chat template file
Verified signatureUpdate chat_template.jinja, which Transformers prioritizes over tokenizer_config.json, so the pretraining template emits </s> as EOD.f271e15 -
Use </s> as pretraining EOD token
Verified signatureSet tokenizer EOS/EOD metadata to </s> (id 2), restrict generation EOS to id 2, and append </s> in the pretraining chat template. config.json already used eos_token_id 2, so it is unchanged.cc75d08 -
Remove BOS from tokenizer
Verified signatureRemove bos_token metadata, clear bos_token_id, disable the BOS TemplateProcessing post-processor, and mark <s> as a normal token so the tokenizer no longer prepends or exposes BOS.afdb489
-
Add generation markers around answer content for optional SFT loss masking
Verified signature26f81d5 -
Update to simple question/answer pretraining template (all tokens trainable)
Verified signature7a287ad -
Add Nemotron-Nano tokenizer with generation markers for SFT loss masking
Verified signature356b6aa -
initial commit
Verified signature64abc35