mlboydaisuke commited on
Commit
5c548bc
·
verified ·
1 Parent(s): 525ef93

Add a known runtime issue: the stop token with a zero embedding from the second turn

Browse files
Files changed (1) hide show
  1. README.md +4 -0
README.md CHANGED
@@ -129,3 +129,7 @@ picture-region `<img>` tag — exactly its document convention.
129
 
130
  Conversion scripts and a step-by-step reproduction:
131
  [hf-to-litertlm](https://github.com/john-rocky/hf-to-litertlm) (`qwen35vl_work/`).
 
 
 
 
 
129
 
130
  Conversion scripts and a step-by-step reproduction:
131
  [hf-to-litertlm](https://github.com/john-rocky/hf-to-litertlm) (`qwen35vl_work/`).
132
+
133
+ ## Known runtime issues
134
+
135
+ From the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because this file holds the embedding table in its own section and the decoder takes embeddings, so later answers in a conversation can differ from the answer the model gives to the same messages in one prompt (read in the source of LiteRT-LM 0.16.0 to 0.18.1; measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on this file). One question per conversation avoids it, as does a new conversation that starts with the earlier messages and the new question.