BDH 100M, bytes, chat v3
bdh-100m-bytes tuned for 800 steps on 6,960 short conversation-shaped documents: story requests
answered with a TinyStories-style story, stories followed by questions whose answers are only in the
story, list requests followed by position questions ("what was the second one?"), and a small set
of one-line tasks and stop markers. Loss was taken only on the assistant's replies. The tuning data
was written by an LLM from category prompts and is not in this release.
Format: turns are User: ... and Assistant: ... on their own lines. A reply ends with
<|endoftext|>. chat_stream.py in the code repository handles this and keeps the whole
conversation in the model's synaptic state with a 480-position sliding window.
What tuning changed
- Recall of a fact planted earlier in the context (the planted-fact probe, 30 trials per distance): 93 % at 128 characters against 17 % for the base model, then 13 % at 256 and 7 % at 384. The recall depends on the key the fact was planted under. For instance, with the word planted as "the dog's name" and the question still asking for the secret word, recall at 128 characters falls from 28 of 30 to 11 of 30.
- On a 60-item multi-turn exam (stories with three questions each, lists with position questions), 16 % of its story answers were judged supported by its own story, and it answered 79 % of "which was first" questions correctly, but 7 % of "which was second".
- Held-out bits per character rose from 1.042 to 1.340 (+29 %): the tune makes it a worse general language model, because the tuning text was never mixed with pretraining text.
Training
800 steps of 32 sequences, AdamW with weight decay 0.01, learning rate 1e-4 flat, dropout 0.1, bf16, about four passes over the data. One RTX 4090, 25 minutes.
Use
TEMP=0.3 python chat_stream.py bdh-100m-bytes-chat-v3.pt
Limitations
A 100M-parameter model tuned on a few thousand short dialogues. It answers in the register of children's stories, invents facts ("Water boils at 100°F"), and chooses the most recent item in a list when it cannot recall. Its 480-position window holds a short story and one or two questions. By the third question much of the story has left the window. Single training seed for the base. The exam numbers are the mean of three generation seeds.
Code: https://github.com/d3vmeh/small-dragon-hatchling. Paper: https://doi.org/10.5281/zenodo.23325933 (CC BY 4.0).
Model tree for d3vmeh/bdh-100m-bytes-chat-v3
Base model
d3vmeh/bdh-100m-bytes