| # Evaluation notes |
|
|
| ## Held-out loss |
|
|
| `internal-eval.json` records deterministic next-token loss measurements from |
| the three source-specific validation files and the mixed validation file. |
| These are corpus diagnostics, not standardized capability benchmarks. |
|
|
| ## Generation and context checks |
|
|
| `behavior-checks.json` contains four fixed-seed, 48-token samples at |
| temperature 0.8 and top-k 50, plus an exact 2,048-token forward-pass smoke |
| test. |
|
|
| Observed in this small sample: |
|
|
| - All four completions were non-empty and no German or English sample repeated |
| a token trigram. |
| - The deliberately mixed prompt continued in English and had a 0.0217 repeated |
| trigram fraction. |
| - German prompts stayed predominantly German and the English prompt stayed |
| English, but this is manual inspection of four examples, not a language-ID |
| benchmark. |
| - Text was locally fluent in places but semantically unreliable. For example, |
| the Hamburg continuation incorrectly referred to northern England and the |
| Rhine. This supports the model card's strong factuality warning. |
| - A forward pass at the trained limit of 2,048 tokens returned finite logits |
| with shape `[1, 2048, 24576]`. |
|
|
| The context smoke test proves implementation compatibility at 2,048 tokens, |
| not long-context retrieval or reasoning. A later benchmark suite should add |
| German and English likelihood tasks, multiple generation seeds, automatic |
| language identification, repetition curves, and retrieval tests. |
|
|
| ## Broader generation suite |
|
|
| `generation-suite.json` contains 20 raw samples: ten German, English, mixed, |
| factual, recipe, story, and repetition-oriented prompts evaluated with two |
| seeds. The mean repeated token-trigram fraction was 0.0445; the maximum was |
| 0.1613 and 15 of 20 samples had at least one repeated trigram. |
|
|
| Manual review found locally plausible German and English syntax, but severe |
| semantic and factual failures. Berlin was incorrectly continued as the |
| capital of southern France or Croatia; German federal history was assigned |
| incorrect dates and events; and water was said to freeze below two degrees or |
| at up to 40 degrees. Recipes and stories had recognizable surface structure |
| but commonly lost coherence. The mixed German/English prompt continued in |
| German in both seeds. |
|
|
| This result is appropriate for a tiny base-model research checkpoint: it has |
| learned recognizable continuation patterns, but it is not a reliable factual |
| model and has not learned assistant behavior. |
|
|
