Update card for v4: 100k steps, 1.64B tokens, val_loss 1.7603

README.md

@@ -12,13 +12,13 @@ tags:
 
 # TinyChat-5M
 
-A 5.1M parameter transformer trained from scratch on the [TinyChat](https://huggingface.co/datasets/starhopp3r/TinyChat) dataset.
+A 4.3M parameter transformer trained from scratch on the [TinyChat](https://huggingface.co/datasets/starhopp3r/TinyChat) dataset.
 
 ## Architecture
 
 | Parameter | Value |
 |-----------|-------|
-| Parameters | 5,114,304 |
+| Parameters | 4,327,872 |
 | Layers | 6 |
 | Hidden dim | 192 |
 | Attention heads | 3 |
@@ -32,37 +32,41 @@ A 5.1M parameter transformer trained from scratch on the [TinyChat](https://hugg
 
 ## Training
 
-- **Dataset**: starhopp3r/TinyChat (1M rows, ~170M tokens)
-- **Tokens seen**: 143.9M (90% train / 10% val split)
-- **Optimizer**: AdamW, LR 2e-3, cosine decay + warmup
+- **Dataset**: starhopp3r/TinyChat (1M rows, ~740M tokens)
+- **Tokens seen**: 1.64B (90% train / 10% val split, multi-epoch)
+- **Optimizer**: AdamW, LR 1e-3, cosine decay + warmup
 - **Batch**: 32 × 512 tokens
-- **Steps**: 45,000 (diverged to NaN at step 47,490; last stable checkpoint used)
+- **Steps**: 100,000 (completed cleanly)
 - **Hardware**: RTX 5090 (32 GB)
-- **Time**: ~25 minutes
+- **Time**: ~46 minutes
 
 ## Results
 
 | Metric | Value |
 |--------|-------|
-| Val loss (step 45k) | 1.9027 |
-| Val perplexity | 6.70 |
-| Best val loss (step 47k) | 1.8883 |
+| Final val loss (step 100k) | 1.7603 |
+| Final val perplexity | 5.81 |
+| Best val loss (step 96k) | 1.7531 |
+| Best val perplexity | 5.77 |
 
 ## Generation Samples
 
-> **Prompt**: [INST] What is the capital of France? [/INST]
-> **Output**: t time in a place me ant for ner ve and I feel quite nervous. But what if something goes wrong during those moments at night...
+> **Prompt**: Hello, how are you today?
+> **Output**: I feel quite disgusted by the mess in our office today. Yes, it is necessary to have a better work space for everyone. The clutter makes everyone a bit unwell and can affect their mood. It really does, especially when we just want things to be clean again. A tidy environment helps us enjoy our work more clearly each day.
 
-> **Prompt**: [INST] Tell me a joke. [/INST]
-> **Output**: very frustrating, but you are doing your best in the end. Thank you, it is nice to relax and share ideas with someone today...
+> **Prompt**: Once upon a time
+> **Output**: when the thread interrupts our plans and makes everything feel more tense. It is so annoying how simple things can turn into chaos, isn't it? Yes, it really is, like trying to find noise that won't stop bothering us. I wish we could just enjoy the sewing process without all these problems.
 
-> **Prompt**: [INST] Explain gravity in simple terms. [/INST]
-> **Output**: remind us of how we all have these smaller and more expressive feelings. I miss the days when everything felt bright and happy...
+> **Prompt**: The cat sat on the
+> **Output**: bench, waiting for lunch to arrive. That is surprising and cute! I hope it unfolds before they try to play outside. Yes, it will be a delightful surprise, but we must stay calm as her. Do you remember that day when the weather was so nice?
 
-The model produces coherent conversational English appropriate for its size. It does not answer factual questions correctly (expected at 5M params) but maintains consistent tone and topic.
+> **Prompt**: I think that
+> **Output**: is a good idea because sometimes sadness can feel so heavy and hard to handle. Yes, it feels like emotions are holding us down like a thin needle poking at our hearts. Do you believe sharing feelings helps lighten the load we carry in some way?
+
+The model produces coherent conversational English appropriate for its size. It does not answer factual questions correctly (expected at 4.3M params) but maintains consistent tone and topic.
 
 ## Notes
 
-- Training diverged to NaN at step 47,490 (likely LR too high for late training).
-- Checkpoint at step 45,000 is the last stable one before divergence.
+- v4 fixes v3's NaN divergence (LR reduced from 2e-3 to 1e-3, NaN detection added).
+- Training completed all 100k steps without divergence.
 - Model is for research/educational purposes. Not suitable for production use.