Card: add a note on litert-lm's 10-minute send_message timeout
Browse files
README.md
CHANGED
|
@@ -75,6 +75,7 @@ Since the 2026-10-13 re-pack the default is a **direct answer**: the bundled tem
|
|
| 75 |
- `enable_thinking=false` (via `ThinkingConfig` or the conversation's extra context — both reach the template) switches the model to **direct answers**: two- to seven-token replies on the gate questions, ~10× faster turns, and the GSM8K numbers above. `enable_thinking=true` pre-fills the think opener explicitly.
|
| 76 |
- Sampling: OpenBMB recommends `temperature 1.0, top_p 0.95`; the gates above are greedy.
|
| 77 |
- Prompt format is ChatML (`<|im_start|>role\n…<|im_end|>\n`), 4096-token KV budget, six prefill signatures (1024, 256, 64, 16, 4, 1).
|
|
|
|
| 78 |
|
| 79 |
## Performance
|
| 80 |
|
|
|
|
| 75 |
- `enable_thinking=false` (via `ThinkingConfig` or the conversation's extra context — both reach the template) switches the model to **direct answers**: two- to seven-token replies on the gate questions, ~10× faster turns, and the GSM8K numbers above. `enable_thinking=true` pre-fills the think opener explicitly.
|
| 76 |
- Sampling: OpenBMB recommends `temperature 1.0, top_p 0.95`; the gates above are greedy.
|
| 77 |
- Prompt format is ChatML (`<|im_start|>role\n…<|im_end|>\n`), 4096-token KV budget, six prefill signatures (1024, 256, 64, 16, 4, 1).
|
| 78 |
+
- **Replies that take over 10 minutes.** On litert-lm 0.18.x and 0.19.0, a blocking `send_message` that is still generating after 10 minutes fails with `DEADLINE_EXCEEDED` (a fixed timeout with no setting), and the generation it leaves running can crash the process. For long replies on a slow backend (long thinking in particular), lower `max_output_tokens` or use the streaming `send_message_async`, which has no such limit.
|
| 79 |
|
| 80 |
## Performance
|
| 81 |
|