Create README.md

README.md

@@ -0,0 +1,84 @@
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-generation
+tags:
+- casual-lm
+- gqa
+- custom
+- refresh_gate
+- swiglu
+- qk-norm
+- rope
+---
+<img src="Banner.png" width="1920"/>
+# Rose-1.5-Pro
+
+Rose-1.5-Pro is a compact language model developed by **GODELEV**, built on the **Rose X1.5 architecture**. Trained on approximately **70 billion tokens**, it explores the scaling of the Rose X1.5 design with a wider hidden representation and a configuration intended to balance model capability with training efficiency.
+
+The model achieves an **Intelligence Index of 28.27**, with strong results across commonsense reasoning, science-question benchmarks, and general language understanding tasks. Its evaluation results provide another step in exploring how architectural choices and training configurations influence performance in small language models.
+
+## Model Overview
+
+| Property | Value |
+|---|---|
+| Model | Rose-1.5-Pro |
+| Developer | Mr Munk |
+| Architecture | Rose X1.5 |
+| Layers | 30 |
+| Hidden Size | 640 |
+| Attention Heads | 10 |
+| Key-Value Heads | 5 |
+| Head Dimension | 64 |
+| Intermediate Size | 1536 |
+| Vocabulary Size | 32,770 |
+| Context Length | 2,048 tokens |
+| Training Tokens | ~70B |
+| Activation | SiLU |
+| Normalization | RMSNorm |
+| QK Normalization | Enabled |
+| XSA | Enabled |
+| Refresh Gates | Layers 10 and 20 |
+
+## Evaluation Results
+
+Rose-1.5-Pro was evaluated across a set of benchmarks covering commonsense reasoning, scientific knowledge, and arithmetic-oriented tasks.
+
+| Benchmark | Score |
+|---|---:|
+| HellaSwag | 43.92% |
+| ARC-Easy | 58.29% |
+| ARC-Challenge | 30.72% |
+| Combined ARC | 44.50% |
+| PIQA | 69.48% |
+| ArithMark-3 | 40.00% |
+| ArithMark-2 | 34.52% |
+| **Intelligence Index** | **28.27** |
+
+The Intelligence Index aggregates selected benchmark results into a single normalized score. It is useful for comparing models evaluated under the same methodology, although it cannot represent every aspect of language-model capability.
+
+Rose-1.5-Pro's results show its performance across several evaluation categories, while the arithmetic benchmarks offer a separate view of its performance on structured numerical tasks.
+
+## Architecture and Training
+
+Rose-1.5-Pro continues the development of the Rose X1.5 architecture, using 30 layers and a hidden size of 640. It incorporates QK normalization, XSA, and refresh gates as part of its architectural configuration.
+
+The model was trained on approximately **70B tokens**.
+
+### Context Length
+
+The model uses a **2,048-token context window by design**. This decision prioritizes faster training and a more efficient training process rather than maximizing context length. Context length is an engineering trade-off, and a larger window is not automatically necessary for every training objective.
+
+## Position Within the Rose Series
+
+Rose-1.5-Pro builds on the experiments conducted with earlier Rose models. The series explores how changes in model scale, architecture, and training configuration affect benchmark performance.
+
+Rather than assuming that additional parameters or training tokens guarantee improvements on every task, the evaluation results provide a practical basis for identifying strengths and trade-offs across benchmarks.
+
+## Limitations
+
+- Benchmark scores measure performance on specific evaluation tasks and should not be treated as a complete measure of general intelligence.
+- Results depend on the evaluation datasets, prompts, scoring procedures, and implementation.
+- Performance may vary across tasks and domains not represented in the reported benchmarks.
+- The 2,048-token context window is a deliberate training-efficiency choice and may not suit tasks requiring longer input sequences.