Evaluation notes
Held-out loss
internal-eval.json records deterministic next-token loss measurements from
the three source-specific validation files and the mixed validation file.
These are corpus diagnostics, not standardized capability benchmarks.
Generation and context checks
behavior-checks.json contains four fixed-seed, 48-token samples at
temperature 0.8 and top-k 50, plus an exact 2,048-token forward-pass smoke
test.
Observed in this small sample:
- All four completions were non-empty and no German or English sample repeated a token trigram.
- The deliberately mixed prompt continued in English and had a 0.0217 repeated trigram fraction.
- German prompts stayed predominantly German and the English prompt stayed English, but this is manual inspection of four examples, not a language-ID benchmark.
- Text was locally fluent in places but semantically unreliable. For example, the Hamburg continuation incorrectly referred to northern England and the Rhine. This supports the model card's strong factuality warning.
- A forward pass at the trained limit of 2,048 tokens returned finite logits
with shape
[1, 2048, 24576].
The context smoke test proves implementation compatibility at 2,048 tokens, not long-context retrieval or reasoning. A later benchmark suite should add German and English likelihood tasks, multiple generation seeds, automatic language identification, repetition curves, and retrieval tests.
Broader generation suite
generation-suite.json contains 20 raw samples: ten German, English, mixed,
factual, recipe, story, and repetition-oriented prompts evaluated with two
seeds. The mean repeated token-trigram fraction was 0.0445; the maximum was
0.1613 and 15 of 20 samples had at least one repeated trigram.
Manual review found locally plausible German and English syntax, but severe semantic and factual failures. Berlin was incorrectly continued as the capital of southern France or Croatia; German federal history was assigned incorrect dates and events; and water was said to freeze below two degrees or at up to 40 degrees. Recipes and stories had recognizable surface structure but commonly lost coherence. The mixed German/English prompt continued in German in both seeds.
This result is appropriate for a tiny base-model research checkpoint: it has learned recognizable continuation patterns, but it is not a reliable factual model and has not learned assistant behavior.