docs: update model card for Foundation Continued 30K

README.md

@@ -17,11 +17,19 @@ metrics:
 library_name: transformers
 ---
 
-# NeoLM-Small-Base (61M) — Stage 1: Foundation
+# NeoLM-Small-Base (61M) — Stage 1: Foundation Continued 30K
 
-**NeoLM-Small-Base** is an English-focused, decoder-only Transformer language model trained from scratch with PyTorch and Hugging Face Transformers. This repository publishes the **Stage 1 Foundation** base-model checkpoint in **bfloat16** format. It is a **base next-token prediction model**, not an instruction-tuned chat assistant.
+**NeoLM-Small-Base** is an English-focused, decoder-only Transformer language model trained from scratch with PyTorch and Hugging Face Transformers. The `main` branch publishes the **Stage 1 Foundation Continued 30K** base-model checkpoint in **bfloat16** (`safetensors`) format. This is a **base next-token prediction model**, not an instruction-tuned chat assistant.
 
-The fixed Foundation release can be loaded using the `stage1-foundation` revision.
+## Available revisions
+
+| Revision | Description |
+| --- | --- |
+| `main` | Current stable Base weights: Foundation continued for 30K additional optimizer steps; BF16 release |
+| `stage1-foundation` | Historical original Foundation checkpoint, preserved separately |
+| `stage1-foundation-continued-30k` | Planned immutable tag for this 30K release; created **after** this README is committed and the release is verified |
+
+The original Foundation checkpoint was trained for approximately **202K optimizer steps** before a system interruption. The selected continuation added **20K steps from that checkpoint**, followed by **10K more steps from the 20K checkpoint** (approximately **232K steps total along the selected training trajectory**). Independent 5K and 20K continuation experiments were also performed; they should not be summed together as one run. The original planned three-epoch Foundation schedule was not completed.
 
 ## Architecture
 
@@ -37,7 +45,7 @@ The fixed Foundation release can be loaded using the `stage1-foundation` revisio
 | Context length | 1,024 tokens |
 | Vocabulary | 49,152 (custom ByteLevel BPE) |
 | Special tokens | EOS ID 0, PAD ID 1; no BOS |
-| Weight format | BF16, safetensors |
+| Published weight format | BF16, safetensors |
 | Model implementation | Custom `NeoLMForCausalLM` (`trust_remote_code=True`) |
 
 ## Tokenizer
@@ -58,14 +66,14 @@ Special-token spellings and IDs (kept outside the Markdown table to preserve lit
 - EOS: `<|endoftext|>` — ID **0**
 - PAD: `<|pad|>` — ID **1**
 
-The tokenizer passed local encode/decode round-trip, special-token, and save/reload checks. It can also be loaded **without loading model weights**:
+The tokenizer passed local encode/decode round-trip, special-token, and save/reload checks. It can be loaded **without loading model weights**:
 
 ```python
 from transformers import AutoTokenizer
 
 tokenizer = AutoTokenizer.from_pretrained(
     "MaagDeveloper/NeoLM-Small-Base",
-    revision="stage1-foundation",
+    revision="main",
     trust_remote_code=True,
 )
 print(tokenizer.encode("Hello, NeoLM!"))
@@ -73,19 +81,39 @@ print(tokenizer.encode("Hello, NeoLM!"))
 
 ## Training data
 
-Foundation pretraining uses the **`foundation`** configuration of [NeoCorpus](https://huggingface.co/datasets/MaagDeveloper/NeoCorpus), derived from a filtered English-focused subset of [Karpathy ClimbMix Shuffle](https://huggingface.co/datasets/karpathy/climbmix-400b-shuffle), based on [NVIDIA Nemotron-ClimbMix](https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix). The train split contains **1,889,952 documents** and **1,147,218,931 tokens**, counted with this release's tokenizer (including document EOS separators). Validation and test splits are kept separate. See the dataset card for data preparation, attribution, and licensing information.
+Foundation pretraining uses the **`foundation`** configuration of [NeoCorpus](https://huggingface.co/datasets/MaagDeveloper/NeoCorpus), derived from a filtered English-focused subset of [Karpathy ClimbMix Shuffle](https://huggingface.co/datasets/karpathy/climbmix-400b-shuffle), based on [NVIDIA Nemotron-ClimbMix](https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix). The training split contains **1,889,952 documents / 1,147,218,931 tokens**, counted with this release's tokenizer (including document EOS separators). The validation split contains **5,000 documents / 3,084,132 tokens**; the test split contains **20,000 documents / 12,094,582 tokens**. Validation and test were held out from pretraining. See the dataset card for preparation, attribution, and licensing details.
+
+The tokenizer is trained on the Foundation training split, and pretraining consumes contiguous `uint16` token streams with EOS between documents and 1,024-token windows at stride 1,023 (one-token overlap).
+
+## Evaluation
+
+The **selected 30K continuation checkpoint** was evaluated on held-out Foundation token streams:
 
-This is a released training checkpoint rather than a claim of complete convergence or optimal training duration. The intended three-epoch schedule was not completed. Exact released-checkpoint step and full Foundation validation/test perplexity have **not been documented in this card**; downstream task results should not be confused with base-language-model perplexity.
+| Checkpoint | Full validation loss | Full validation PPL | Test loss | Test PPL |
+| --- | ---: | ---: | ---: | ---: |
+| Foundation continued 20K | 3.2685 | 26.27 | 3.2498 | 25.78 |
+| **Foundation continued 30K** | **3.2677** | **26.25** | **3.2489** | **25.76** |
+
+The 30K continuation improves the reported full validation loss over the 20K checkpoint by approximately **0.0008**; this is a small measured improvement, not evidence of a qualitative capability jump. Checkpoint selection should use validation rather than repeatedly consulted test results.
+
+For the BF16 release, the model was exported separately from the FP32 master, reloaded, and used successfully for text generation. A local **256-block quick validation comparison**, using the same BF16 autocast compute settings for both weight versions, reported:
+
+| Weight version | Quick validation loss | Quick validation PPL |
+| --- | ---: | ---: |
+| FP32 source | 3.236623 | 25.4476 |
+| BF16 export | 3.236627 | 25.4477 |
+
+The observed difference was **+0.000004 loss**. This is a *quick-subset parity check*, not a separate full-validation result or proof that every generated output will be identical. Published weights are BF16; the local FP32 master is retained separately.
 
 ## Load and generate text
 
-Install compatible versions of `torch` and `transformers`. The model requires custom Python code, so review the repository code before enabling `trust_remote_code=True`.
+Install compatible versions of `torch` and `transformers`. The model requires custom Python code, so review the repository code before enabling `trust_remote_code=True`. Use `revision="stage1-foundation"` to load the original historical release instead of the latest stable Base.
 
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
 
 repo_id = "MaagDeveloper/NeoLM-Small-Base"
-revision = "stage1-foundation"
+revision = "main"
 
 tokenizer = AutoTokenizer.from_pretrained(
     repo_id, revision=revision, trust_remote_code=True
@@ -105,9 +133,9 @@ inputs = tokenizer(
 
 outputs = model.generate(
     **inputs,
-    max_new_tokens=200, # max_length=1024
+    max_new_tokens=200,
     do_sample=True,
-    temperature=0.3, # 0.8
+    temperature=0.3,
     top_p=0.9,
     repetition_penalty=1.15,
     eos_token_id=tokenizer.eos_token_id,
@@ -117,16 +145,15 @@ outputs = model.generate(
 print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 ```
 
-
 ## Streaming Text Generation
 
-You can stream generated text progressively using Hugging Face Transformers' `TextStreamer`.
+Stream generated text progressively using Hugging Face Transformers' `TextStreamer`:
 
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer
 
 repo_id = "MaagDeveloper/NeoLM-Small-Base"
-revision = "stage1-foundation"
+revision = "main"
 
 tokenizer = AutoTokenizer.from_pretrained(
     repo_id,
@@ -153,9 +180,9 @@ inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
 _ = model.generate(
     **inputs,
     streamer=streamer,
-    max_new_tokens=100, # max_length=1024
+    max_new_tokens=100,
     do_sample=True,
-    temperature=0.3, # 0.8
+    temperature=0.3,
     top_p=0.9,
     repetition_penalty=1.15,
     eos_token_id=tokenizer.eos_token_id,
@@ -163,18 +190,18 @@ _ = model.generate(
 )
 ```
 
-The code above is **text completion**, not role-based chat. NeoLM-Small-Base may not reliably follow instructions, answer factual questions, or perform multi-step reasoning. A separate instruction-tuned model belongs in a distinct repository.
+Both examples are **text completion**, not role-based chat. For generation within the trained 1,024-token context, keep the total prompt-plus-generation length within the model's context window.
 
-## Evaluation and limitations
+## Capabilities and limitations
 
-The base model demonstrated decreasing training and held-out validation next-token loss during Foundation pretraining. Separate fine-tuning experiments have shown that this architecture can adapt to QA and text classification, but **those fine-tuned results are not performance claims for this Base checkpoint**.
+The Base checkpoint can generate fluent English passages and frequently follows broad topical cues in its prompt. These observations are **not** evidence of reliable instruction following, robust reasoning, or factual accuracy. Exploratory generation also exhibits repetition, invented facts, inconsistent entities, topic drift, and occasional malformed endings.
 
-Full Foundation validation/test loss and perplexity for the released checkpoint are not reported here until measured and verified. The model is small, English-focused, and can hallucinate, repeat text, produce harmful or biased content, and fail on complex reasoning, factual recall, or instruction-following tasks. Do not treat generated material as verified information.
+The model is small and English-focused; it can hallucinate, reproduce undesirable patterns, generate harmful or biased material, and fail on multi-step reasoning, factual recall, long-range coherence, or strict output formats. Do not treat generated material as verified information. A future instruction-tuned model, if released, will live in a separate repository.
 
 ## Provenance and licensing
 
 - Dataset: [MaagDeveloper/NeoCorpus](https://huggingface.co/datasets/MaagDeveloper/NeoCorpus), `foundation` configuration.
-- NeoCorpus's attribution and CC BY-NC 4.0 notice do not automatically establish a separate license for model weights. **Please verify the appropriate model-weight license and source-data obligations before commercial use.**
-- The `stage1-foundation` tag is intended to preserve an immutable reference to this release before later training stages.
+- NeoCorpus's attribution and CC BY-NC 4.0 notice do not automatically establish a separate license for model weights. **Verify the appropriate model-weight license and source-data obligations before commercial use.**
+- `stage1-foundation` preserves the historical original release. The 30K release tag is intended to preserve a separate immutable weights-and-documentation snapshot after the README update.
 
-NeoLM is an independent project and is not affiliated with NVIDIA, Andrej Karpathy, or Hugging Face.
+NeoLM is an independent project and is not affiliated with NVIDIA, Andrej Karpathy, or Hugging Face.