Update README.md
Browse files
README.md
CHANGED
|
@@ -27,6 +27,35 @@ To ensure high-quality generation and robust linguistic understanding, the datas
|
|
| 27 |
* **FineWeb-2 (Khmer Extension):** Leveraging the massive, cleaned web-scale data from the FineWeb-2 initiative as our core foundation.
|
| 28 |
* **Custom Cleanup Pipelines:** We applied rigorous, proprietary filtering and deduplication methods tailored specifically to the Khmer script. This process stripped out machine-translated gibberish, HTML noise, and non-Khmer text, leaving a pristine dataset representing authentic language use.
|
| 29 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
---
|
| 31 |
|
| 32 |
## 🛠️ How to Use
|
|
@@ -38,7 +67,7 @@ import torch
|
|
| 38 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 39 |
|
| 40 |
# 1. Specify the model repository path
|
| 41 |
-
model_id = "
|
| 42 |
|
| 43 |
# 2. Load the custom tokenizer and optimized model weights
|
| 44 |
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
|
|
| 27 |
* **FineWeb-2 (Khmer Extension):** Leveraging the massive, cleaned web-scale data from the FineWeb-2 initiative as our core foundation.
|
| 28 |
* **Custom Cleanup Pipelines:** We applied rigorous, proprietary filtering and deduplication methods tailored specifically to the Khmer script. This process stripped out machine-translated gibberish, HTML noise, and non-Khmer text, leaving a pristine dataset representing authentic language use.
|
| 29 |
|
| 30 |
+
---
|
| 31 |
+
## 🏗️ Model Architecture
|
| 32 |
+
|
| 33 |
+
**mouy** is built on a highly optimized, deep-yet-narrow **Gemma-style** autoregressive decoder architecture. While many lightweight models sacrifice depth to reduce parameter counts, **mouy** prioritizes architectural depth (28 layers) to capture complex, long-range structural dependencies unique to the Khmer language, while maintaining a lean hidden dimension to stay incredibly fast and memory-efficient.
|
| 34 |
+
|
| 35 |
+
### Key Architectural Features
|
| 36 |
+
|
| 37 |
+
* **Grouped Query Attention (GQA):** Features 8 attention heads for queries but scales down to 2 heads for Keys and Values (KV). This significantly cuts down the KV-cache memory footprint during generation, allowing for faster inference and larger batch sizes.
|
| 38 |
+
* **GeGLU Activation:** The feed-forward network (MLP) utilizes Gated Linear Units with GELU activation functions (`gate_proj` paired with `up_proj` before projecting down), which has been shown to offer superior semantic representation over standard ReLU or vanilla GELU.
|
| 39 |
+
* **Rotary Position Embeddings (RoPE):** Implements dynamic rotary embeddings to inject positional context directly into the attention mechanism, supporting a context window of up to 2,048 tokens.
|
| 40 |
+
* **Root Mean Square Normalization (RMSNorm):** Applied at both the input and post-attention stages of every decoder layer to stabilize gradient flows and speed up training convergence without the computational overhead of standard LayerNorm.
|
| 41 |
+
|
| 42 |
+
---
|
| 43 |
+
|
| 44 |
+
### 📊 Hyperparameters at a Glance
|
| 45 |
+
|
| 46 |
+
| Hyperparameter | Value | Description |
|
| 47 |
+
| --- | --- | --- |
|
| 48 |
+
| **Parameters** | ~100M | Total trainable parameter count |
|
| 49 |
+
| **Layers (`num_hidden_layers`)** | 28 | Deep transformer stack for complex linguistic hierarchy |
|
| 50 |
+
| **Hidden Size (`hidden_size`)** | 512 | Width of the embedding and hidden states |
|
| 51 |
+
| **Intermediate Size** | 2,048 | Dimension of the GeGLU feed-forward layer |
|
| 52 |
+
| **Attention Heads ($Q$)** | 8 | Number of query heads |
|
| 53 |
+
| **Key-Value Heads ($K, V$)** | 2 | Grouped Query Attention (GQA) configuration |
|
| 54 |
+
| **Head Dimension** | 64 | Dimension per attention head |
|
| 55 |
+
| **Context Length (`max_position_embeddings`)** | 2,048 | Maximum sequence token window |
|
| 56 |
+
| **Vocabulary Size** | 5,000 | Custom localized Khmer-optimized vocabulary |
|
| 57 |
+
| **Rope Theta** | 10,000.0 | Base frequency for rotary position embeddings |
|
| 58 |
+
|
| 59 |
---
|
| 60 |
|
| 61 |
## 🛠️ How to Use
|
|
|
|
| 67 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 68 |
|
| 69 |
# 1. Specify the model repository path
|
| 70 |
+
model_id = "attentionlab/mouy"
|
| 71 |
|
| 72 |
# 2. Load the custom tokenizer and optimized model weights
|
| 73 |
tokenizer = AutoTokenizer.from_pretrained(model_id)
|