Publish corrected 4.96M-parameter Muon checkpoint

.gitattributes

@@ -1,35 +1,4 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,15 @@
+ISC License
+
+Copyright 2026 ApexDevelopment
+
+Permission to use, copy, modify, and/or distribute this software for any purpose
+with or without fee is hereby granted, provided that the above copyright notice
+and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
+REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
+FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
+INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
+OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
+TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF
+THIS SOFTWARE.

README.md

@@ -0,0 +1,131 @@
+---
+license: isc
+language:
+  - en
+library_name: transformers
+pipeline_tag: text-generation
+datasets:
+  - roneneldan/TinyStories
+tags:
+  - gemma4
+  - text-generation
+  - tiny-llm
+  - tinystories
+  - experimental
+model-index:
+  - name: tinygemma4
+    results:
+      - task:
+          type: text-generation
+          name: Text Generation
+        dataset:
+          type: roneneldan/TinyStories
+          name: TinyStories validation
+        metrics:
+          - type: loss
+            name: validation loss
+            value: 1.3825
+          - type: perplexity
+            name: validation perplexity
+            value: 3.98
+---
+
+# tinygemma4
+
+tinygemma4 is a deliberately tiny, text-only Gemma 4 model trained from scratch on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). It is intended for architecture compatibility checks, inference-engine testing, and small-scale language-model experiments.
+
+This is not a useful assistant model. It was trained on simple synthetic stories and should be expected to produce short, child-story-like completions with limited coherence.
+
+## Model Details
+
+- Architecture: `Gemma4ForCausalLM`
+- Parameters: 4,964,752
+- Vocabulary: 8192-token byte-level BPE
+- Context length in config: 2048
+- Training block size: 256
+- Hidden size: 128
+- Per-layer input hidden size: 16
+- Layers: 12
+- Attention heads: 4
+- KV heads: 1
+- Head dimension: 32
+- MLP intermediate size: 384
+- Sliding window: 128
+- Full attention layers: 4, 8, 12
+- Embeddings: tied
+- MoE: disabled
+- Multimodal components: none
+- Weight dtype: float32
+- Tensor format: safetensors
+
+The checkpoint is saved in ordinary Hugging Face Transformers format. Any runtime with a correct Gemma 4 text implementation and support for these small dimensions should be able to load it.
+
+## Training
+
+- Dataset: `roneneldan/TinyStories`
+- Training file: `TinyStoriesV2-GPT4-train.txt`
+- Validation file: `TinyStoriesV2-GPT4-valid.txt`
+- Final training step: 300000
+- Optimizer: Muon + AdamW (Muon for non-embedding matrix parameters; AdamW for embeddings, norms, and scalar parameters)
+- Learning-rate schedule: cosine decay from `3.5e-4` to `3.5e-5`, with 1,000 warmup steps
+- Batch size: 32
+- Random seed: 1337
+- Hardware: AMD Radeon RX 9070 XT, ROCm PyTorch for Windows
+- Training dtype: bf16 autocast where available
+
+## Evaluation
+
+Validation was measured during training on held-out TinyStories text with the local training script:
+
+- Validation loss: 1.3825
+- Validation perplexity: 3.98
+
+These numbers are only for this training setup. Evaluation used 50 shuffled validation batches of 32 sequences at a 256-token block size. They are not general language-understanding benchmarks.
+
+The final checkpoint's best tested sampling settings were temperature `0.45`, top-p `0.90`, top-k `50`, and repetition penalty `1.03`. On a small fixed sampling suite, the model remained fluent but still showed repetition, contradictions, pronoun errors, and weak long-range causality.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "ApexDevelopment/tinygemma4"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
+
+inputs = tokenizer("Once upon a time,", return_tensors="pt")
+outputs = model.generate(
+	**inputs,
+	max_new_tokens=80,
+	do_sample=True,
+	temperature=0.45,
+	top_p=0.90,
+	top_k=50,
+	repetition_penalty=1.03,
+	pad_token_id=tokenizer.pad_token_id,
+)
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Limitations
+
+- The model is tiny and heavily capacity-limited.
+- It is trained only on synthetic TinyStories text.
+- It is not instruction tuned.
+- It is not safety tuned.
+- It can repeat, contradict itself, confuse characters or objects, and produce malformed story fragments.
+- Its coherent range is usually much shorter than the configured 2,048-token context length.
+- It should be used for experimentation and testing, not production.
+
+## Data and License Notes
+
+The training dataset card lists TinyStories under `cdla-sharing-1.0`. This model was trained from scratch; it does not contain Gemma weights from Google or weights from TinyLLama-v0.
+
+Weights are released under the license declared in the metadata above. Users are responsible for checking whether their intended use is compatible with the dataset license and applicable law.
+
+## Inspiration
+
+This project was inspired by [`Maykeye/TinyLLama-v0`](https://huggingface.co/Maykeye/TinyLLama-v0), but uses a Gemma 4 text configuration instead of Llama.

config.json

@@ -0,0 +1,65 @@
+{
+  "architectures": [
+    "Gemma4ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attention_k_eq_v": false,
+  "bos_token_id": 2,
+  "dtype": "float32",
+  "enable_moe_block": false,
+  "eos_token_id": 1,
+  "final_logit_softcapping": null,
+  "global_head_dim": 32,
+  "head_dim": 32,
+  "hidden_activation": "gelu_pytorch_tanh",
+  "hidden_size": 128,
+  "hidden_size_per_layer_input": 16,
+  "initializer_range": 0.02,
+  "intermediate_size": 384,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 2048,
+  "model_type": "gemma4_text",
+  "moe_intermediate_size": null,
+  "num_attention_heads": 4,
+  "num_experts": 0,
+  "num_global_key_value_heads": 1,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 1,
+  "num_kv_shared_layers": 0,
+  "pad_token_id": 0,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "full_attention": {
+      "partial_rotary_factor": 1.0,
+      "rope_theta": 1000000.0,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 10000.0,
+      "rope_type": "default"
+    }
+  },
+  "sliding_window": 128,
+  "tie_word_embeddings": true,
+  "top_k_experts": null,
+  "transformers_version": "5.13.0",
+  "use_bidirectional_attention": null,
+  "use_cache": true,
+  "use_double_wide_mlp": false,
+  "vocab_size": 8192,
+  "vocab_size_per_layer_input": 8192
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "bos_token_id": 2,
+  "do_sample": true,
+  "eos_token_id": 1,
+  "pad_token_id": 0,
+  "repetition_penalty": 1.03,
+  "temperature": 0.45,
+  "top_p": 0.9,
+  "transformers_version": "5.13.0",
+  "top_k": 50
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4fc7d322c15ee9d8817e006b377d13fd4c8fc9a2e7a2e250924b35178b849339
+size 19881840

tokenizer_config.json

@@ -0,0 +1,11 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<bos>",
+  "eos_token": "<|endoftext|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 2048,
+  "pad_token": "<pad>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>"
+}

training_state.json

@@ -0,0 +1,47 @@
+{
+  "step": 300000,
+  "recipe": {
+    "vocab_size": 8192,
+    "num_hidden_layers": 12,
+    "hidden_size": 128,
+    "hidden_size_per_layer_input": 16,
+    "intermediate_size": 384,
+    "num_attention_heads": 4,
+    "num_key_value_heads": 1,
+    "head_dim": 32,
+    "global_head_dim": 32,
+    "sliding_window": 128,
+    "max_position_embeddings": 2048,
+    "full_attention_every": 4
+  },
+  "args": {
+    "output_dir": "runs\\tiny-gemma4-v3-muon-corrected",
+    "cache_dir": "runs\\tiny-gemma4-v3\\cache",
+    "init_from": null,
+    "data_dir": "data",
+    "tokenizer_dir": "runs\\tiny-gemma4-v3\\tokenizer",
+    "recipe": "v3",
+    "data_mode": null,
+    "max_steps": 300000,
+    "batch_size": 32,
+    "block_size": 256,
+    "gradient_accumulation_steps": 1,
+    "learning_rate": 0.00035,
+    "min_learning_rate": 3.5e-05,
+    "warmup_steps": 1000,
+    "schedule_start_step": 0,
+    "optimizer": "muon",
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "save_every": 10000,
+    "eval_every": 1000,
+    "eval_batches": 50,
+    "train_char_limit": null,
+    "valid_char_limit": null,
+    "num_workers": 0,
+    "seed": 1337,
+    "resume": false,
+    "compile": false
+  },
+  "time": "2026-07-22 21:53:23"
+}