sovannpanhaseng commited on
Commit
0879e32
·
verified ·
1 Parent(s): 32d7e77

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +30 -1
README.md CHANGED
@@ -27,6 +27,35 @@ To ensure high-quality generation and robust linguistic understanding, the datas
27
  * **FineWeb-2 (Khmer Extension):** Leveraging the massive, cleaned web-scale data from the FineWeb-2 initiative as our core foundation.
28
  * **Custom Cleanup Pipelines:** We applied rigorous, proprietary filtering and deduplication methods tailored specifically to the Khmer script. This process stripped out machine-translated gibberish, HTML noise, and non-Khmer text, leaving a pristine dataset representing authentic language use.
29
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30
  ---
31
 
32
  ## 🛠️ How to Use
@@ -38,7 +67,7 @@ import torch
38
  from transformers import AutoModelForCausalLM, AutoTokenizer
39
 
40
  # 1. Specify the model repository path
41
- model_id = "your-username/mouy"
42
 
43
  # 2. Load the custom tokenizer and optimized model weights
44
  tokenizer = AutoTokenizer.from_pretrained(model_id)
 
27
  * **FineWeb-2 (Khmer Extension):** Leveraging the massive, cleaned web-scale data from the FineWeb-2 initiative as our core foundation.
28
  * **Custom Cleanup Pipelines:** We applied rigorous, proprietary filtering and deduplication methods tailored specifically to the Khmer script. This process stripped out machine-translated gibberish, HTML noise, and non-Khmer text, leaving a pristine dataset representing authentic language use.
29
 
30
+ ---
31
+ ## 🏗️ Model Architecture
32
+
33
+ **mouy** is built on a highly optimized, deep-yet-narrow **Gemma-style** autoregressive decoder architecture. While many lightweight models sacrifice depth to reduce parameter counts, **mouy** prioritizes architectural depth (28 layers) to capture complex, long-range structural dependencies unique to the Khmer language, while maintaining a lean hidden dimension to stay incredibly fast and memory-efficient.
34
+
35
+ ### Key Architectural Features
36
+
37
+ * **Grouped Query Attention (GQA):** Features 8 attention heads for queries but scales down to 2 heads for Keys and Values (KV). This significantly cuts down the KV-cache memory footprint during generation, allowing for faster inference and larger batch sizes.
38
+ * **GeGLU Activation:** The feed-forward network (MLP) utilizes Gated Linear Units with GELU activation functions (`gate_proj` paired with `up_proj` before projecting down), which has been shown to offer superior semantic representation over standard ReLU or vanilla GELU.
39
+ * **Rotary Position Embeddings (RoPE):** Implements dynamic rotary embeddings to inject positional context directly into the attention mechanism, supporting a context window of up to 2,048 tokens.
40
+ * **Root Mean Square Normalization (RMSNorm):** Applied at both the input and post-attention stages of every decoder layer to stabilize gradient flows and speed up training convergence without the computational overhead of standard LayerNorm.
41
+
42
+ ---
43
+
44
+ ### 📊 Hyperparameters at a Glance
45
+
46
+ | Hyperparameter | Value | Description |
47
+ | --- | --- | --- |
48
+ | **Parameters** | ~100M | Total trainable parameter count |
49
+ | **Layers (`num_hidden_layers`)** | 28 | Deep transformer stack for complex linguistic hierarchy |
50
+ | **Hidden Size (`hidden_size`)** | 512 | Width of the embedding and hidden states |
51
+ | **Intermediate Size** | 2,048 | Dimension of the GeGLU feed-forward layer |
52
+ | **Attention Heads ($Q$)** | 8 | Number of query heads |
53
+ | **Key-Value Heads ($K, V$)** | 2 | Grouped Query Attention (GQA) configuration |
54
+ | **Head Dimension** | 64 | Dimension per attention head |
55
+ | **Context Length (`max_position_embeddings`)** | 2,048 | Maximum sequence token window |
56
+ | **Vocabulary Size** | 5,000 | Custom localized Khmer-optimized vocabulary |
57
+ | **Rope Theta** | 10,000.0 | Base frequency for rotary position embeddings |
58
+
59
  ---
60
 
61
  ## 🛠️ How to Use
 
67
  from transformers import AutoModelForCausalLM, AutoTokenizer
68
 
69
  # 1. Specify the model repository path
70
+ model_id = "attentionlab/mouy"
71
 
72
  # 2. Load the custom tokenizer and optimized model weights
73
  tokenizer = AutoTokenizer.from_pretrained(model_id)