Update README.md

README.md

@@ -9,23 +9,322 @@ tags:
 - refresh-gate
 - rose-x1
 license: apache-2.0
+pipeline_tag: text-generation
+---
+
+---
+
+license: mit
+language:
+
+* en
+  pipeline_tag: text-generation
+  tags:
+* causal-lm
+* custom-architecture
+* pytorch
+* transformers
+* rose-x1
+
 ---
 
 # Rose-Mini
 
-Rose X1 SLM (49.4M params).
-
-## Architecture
-| Property | Value |
-|---|---|
-| Layers | 14 |
-| Hidden | 512 |
-| Heads | 8 (kv=2) |
-| QK Norm | True |
-| Refresh Gate | [4, 9] |
-| Params | 49.443M |
-
-## Training
-- Tokens: 12,206,861,568
-- Val PPL: 6.14
-- Optimizer: muon_adamw
+## The First Bloom
+
+Every model teaches you something.
+Some teach optimization.
+Some teach patience.
+Some teach you that a single misplaced line of code can quietly destroy months of work.
+
+My previous projects taught me all three.
+
+**Archaea** taught me how to train language models from scratch.
+
+The **Ant** series taught me that lowering loss does not necessarily mean a model has learned anything useful.
+
+**Exp-1** humbled me by exposing a catastrophic implementation bug hidden inside my Transformer blocks.
+
+**Archaea-74M-V1.1** reminded me that refining a good model is sometimes more valuable than building a new one.
+
+Rose-Mini begins where those projects ended.
+Not with another Transformer.
+Not with another checkpoint.
+But with a question.
+
+> **What if I stopped modifying existing architectures and started building one of my own?**
+
+That question eventually became **Rose X1**.
+And Rose-Mini is the first public model built upon it.
+
+---
+
+# A Different Kind of Project
+
+Unlike my previous releases, Rose-Mini is not simply another pretrained language model.
+It represents my first successful custom architecture.
+For a long time I had wanted to move beyond reproducing existing Transformer implementations.
+Training models is fun.
+Understanding them is even more fun.
+Building one yourself...
+That is where the real adventure begins.
+Rose X1 became my playground for exploring architectural ideas while still remaining practical enough to train on consumer hardware.
+Fortunately...
+The architecture worked.
+
+---
+
+# Acknowledgements
+
+Before talking about Rose X1, I want to thank **Datdanboi25**.
+The **TX4** architecture was one of the first compact language model designs that genuinely made me stop scrolling and start reading code.
+Rather than simply scaling parameters, TX4 explored interesting architectural ideas for making smaller language models more capable.
+Rose X1 is **inspired by** many of those ideas while exploring its own design decisions and implementation.
+It is **not** a reproduction of TX4.
+It is my own architectural exploration built upon lessons learned from previous projects and inspired by the excellent work shared by the community.
+As an independent developer, openly published research and implementations are invaluable learning resources.
+
+So...
+
+Thank you, **Datdanboi25**.
+Without builders willing to share ideas, projects like Rose probably would never have existed.
+
+---
+
+# Meet Rose X1
+
+Rose X1 is a lightweight decoder-only architecture designed around one simple philosophy:
+
+> **Extract more capability from fewer parameters.**
+
+Instead of chasing larger parameter counts, Rose focuses on architectural efficiency and better utilization of compute.
+The architecture incorporates several modern design ideas including:
+
+* Grouped Query Attention (GQA)
+* RMSNorm
+* QK Normalization
+* Refresh Gates
+* Efficient decoder-only design
+* BF16 training
+
+The goal was never to reinvent the Transformer.
+The goal was to explore it.
+
+---
+
+# The Four Hour Battle
+
+Designing an architecture turned out to be easier than convincing Hugging Face that it actually existed.
+
+Training was straightforward.
+
+Writing the custom Hugging Face implementation...
+
+That was another story.
+
+Since Rose X1 is a custom architecture, I had to implement everything required for Hugging Face compatibility.
+
+Configuration classes.
+
+Model classes.
+
+Registration.
+
+Auto mapping.
+
+Generation support.
+
+Loading.
+
+Saving.
+
+Everything.
+
+I spent nearly **four hours** fighting the modeling file.
+
+Every time I fixed one error...
+
+Python politely introduced two more.
+
+Even AI models tried to help.
+
+They certainly made excellent suggestions.
+
+Unfortunately, they underestimated one critical variable.
+
+**Me.**
+
+I have a remarkable talent for introducing new bugs while fixing old ones.
+
+Some people have green fingers.
+
+Some have sticky fingers.
+
+I have **buggy fingers**.
+
+Eventually...
+
+The architecture loaded successfully.
+
+The checkpoints loaded.
+
+Generation worked.
+
+`AutoModelForCausalLM.from_pretrained()` finally stopped complaining.
+
+That tiny success was honestly one of the most satisfying moments of the entire project.
+
+---
+
+# Model Overview
+
+| Property         | Value          |
+| ---------------- | -------------- |
+| Model            | Rose-Mini      |
+| Architecture     | Rose X1        |
+| Parameters       | 49.443 Million |
+| Layers           | 14             |
+| Hidden Size      | 512            |
+| Attention Heads  | 8              |
+| KV Heads         | 2              |
+| QK Normalization | Enabled        |
+| Refresh Gates    | Layers 4 & 9   |
+| Training Tokens  | 12,206,861,568 |
+| Optimizer        | Muon AdamW     |
+
+---
+
+# Training
+
+Rose-Mini was trained on over **12.2 billion tokens**, representing my largest pretraining run so far.
+Rather than increasing parameter count, I chose to expose a compact model to significantly more data.
+Training small language models is an interesting balancing act.
+Too few tokens...
+The model memorizes very little.
+Too many...
+You begin wondering whether your SSD has developed feelings. #wirtten by Chat Gpt
+Fortunately, both survived.
+
+---
+
+# Comparing My Models
+
+One thing I enjoy about independent research is watching projects evolve over time.
+Each model solved a different problem.
+
+| Model            | Parameters |             Tokens | Architecture            | Purpose                               |
+| ---------------- | ---------: | -----------------: | ----------------------- | ------------------------------------- |
+| Archaea-74M      |        74M |              1.23B | LLaMA-style Transformer | First successful pretrained model     |
+| Archaea-74M-V1.1 |        74M | Continued Training | LLaMA-style Transformer | Refinement and benchmark improvements |
+| **Rose-Mini**    |  **49.4M** |          **12.2B** | **Rose X1**             | First custom architecture             |
+
+Rose-Mini contains roughly **33% fewer parameters** than Archaea-74M.
+Despite that, it demonstrates that architectural experimentation can remain competitive while dramatically reducing parameter count.
+For me, Rose-Mini is less about replacing Archaea.
+It is about exploring a different path.
+
+---
+
+# Benchmark Highlights
+
+Rose-Mini demonstrates encouraging performance for a model of its size.
+
+| Benchmark                     |           Score |
+| ----------------------------- | --------------: |
+| **BananaMind Base Bench 1.1** | **1037.37 Elo** |
+| **ArithMark-2.0**             |      **29.52%** |
+| **HellaSwag**                 |      **28.95%** |
+| **PIQA**                      |      **58.54%** |
+| **ARC-Easy**                  |      **36.62%** |
+| **ARC-Challenge**             |      **24.32%** |
+
+The BananaMind Base Bench result crossing **1000 Elo** was particularly satisfying.
+Watching a custom architecture compete after weeks of implementation work is an incredibly rewarding experience.
+
+---
+
+# Performance Compared to Archaea
+
+Rose-Mini was never intended to be a direct replacement for Archaea.
+Both models were built with different objectives.
+
+| Category            | Archaea-74M |   Archaea-74M-V1.1 |       Rose-Mini |
+| ------------------- | ----------: | -----------------: | --------------: |
+| Parameters          |         74M |                74M |       **49.4M** |
+| Training Tokens     |       1.23B | Continued Training |       **12.2B** |
+| Architecture        | LLaMA-style |        LLaMA-style |         Rose X1 |
+| Custom Architecture |          No |                 No |         **Yes** |
+| BananaMind          |           — |                936 | **1037.37 Elo** |
+| ArithMark-2.0       |      29.30% |         **37.37%** |      **29.52%** |
+
+
+
+---
+
+# Why Rose?
+
+The answer is simple.
+
+Curiosity.
+
+Training another standard Transformer would certainly have been easier.
+
+It also would have been considerably less interesting.
+
+Rose exists because experimentation is how progress happens.
+
+Sometimes the experiment works.
+
+Sometimes it becomes Exp-1.
+
+Fortunately...
+
+This one worked.
+
+---
+
+# Current Limitations
+
+Rose-Mini is still a relatively small language model.
+
+Like every compact model, it has limitations.
+
+* Complex reasoning remains challenging.
+* Long-context understanding can be improved.
+* Factual consistency still has room for growth.
+* Instruction following could be further refined.
+
+None of these are surprising.
+
+They're simply opportunities for the next version.
+
+---
+
+# Future Work
+
+Rose-Mini is only the beginning of the Rose family.
+Development is already underway for:
+
+## Rose-Medium
+
+The goal of Rose-Medium is not simply to increase parameter count.
+
+It is to continue exploring the Rose X1 architecture at a larger scale while preserving the design philosophy that made Rose-Mini successful.
+
+Better architecture.
+
+Better training.
+
+Better models.
+
+Hopefully fewer bugs.
+
+Although I refuse to make any promises regarding that last point.
+
+---
+
+# Final Thoughts
+
+Thank you for checking out Rose-Mini.
+
+I hope you enjoy experimenting with it as much as I enjoyed building it.