File size: 9,714 Bytes
3bcdbab
 
 
 
 
 
 
 
 
 
 
 
 
 
 
630adda
 
 
3bcdbab
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
630adda
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
---
language:
- my
- en
license: apache-2.0
library_name: transformers
tags:
- Translation
- Safetensors
- Burmese
- English
- myanmartranslator
- MyanmarLLM
- custom_code
- myanmar
datasets:
- freococo/myanmar-english-corpus
pipeline_tag: translation
---

# MyTranslation_300M_Base

**MyTranslation_300M_Base** is a **`298.76M`-parameter** (inference; `312.94M` training parameters including the Multi-Token Prediction head) bidirectional **English ↔ Burmese (`my`)** Neural Machine Translation encoder-decoder model trained from scratch on **~10B+ training tokens** using **9× NVIDIA GeForce RTX 5090 GPUs**.

> **Status:** Base checkpoint (`Stage 1 Pretraining + Stage 2 Gold Cooldown`), published prior to optional spoken/idiom Stage 3 polishing.

---

## 1. Architecture Highlights

| Component | Specification |
|---|---|
| **Inference Parameters** | `298,763,201` (`298.76M`, BF16 `model.safetensors` ~598 MB) |
| **Training Parameters** | `312,940,000` (`312.94M`, includes `MTPHeadBlock` dropped at export) |
| **Encoder** | `12` bidirectional layers, `d_model=1024`, `n_heads=16`, `n_kv_heads=4` (GQA, `head_dim=64`), `ffn_dim=3072` SwiGLU |
| **Decoder** | `6` causal layers, `d_model=1024`, `n_heads=16`, `n_kv_heads=4` (GQA), `ffn_dim=3072` SwiGLU |
| **Positional Encoding** | RoPE (`theta=50000.0`, `max_seq_len=2048` / trained at `1024`) |
| **Attention & Normalization** | `RMSNorm`, QK-Norm, Gated Attention (`sigmoid(gate_proj)`), Value Residual (`v0` mixing) |
| **Encoder-Decoder Routing** | Dense Layer-Wise Attention (`enc_dwa_logits`) + `3` learned `SharedCrossKVProj` mixtures (`enc_router_logits`) shared by decoder pairs `(0,1)`, `(2,3)`, `(4,5)` |
| **Tokenizer & Vocab** | `65,536`-piece SentencePiece (`UNK=0`, `BOS=1`, `EOS=2`, `PAD=3`, `<2my>=4`, `<2en>=5`, `<formal>=6`, `<spoken>=7`, `<bt>=8`, `<denoise>=9`, `<mask>=10`), tied embeddings (`embed_scale=32.0`) |

### Learned Cross-KV Encoder Mixture Weights (Post-Stage 2)
* **Cross-KV Pair 0 (Decoder 0–1):** Top encoder layers `[7, 8, 6, 2]` (`[0.226, 0.200, 0.187, 0.102]`)
* **Cross-KV Pair 1 (Decoder 2–3):** Top encoder layers `[9, 10, 8, 11]` (`[0.339, 0.206, 0.187, 0.124]`)
* **Cross-KV Pair 2 (Decoder 4–5):** Top encoder layers `[11, 9, 10, 8]` (`[0.340, 0.335, 0.278, 0.041]`)

---

## 2. Training Hardware, Data & Schedule

### Cluster & Precision Setup
* **GPUs:** **9× NVIDIA GeForce RTX 5090** (`32,607 MiB` VRAM each, `torchrun --nproc_per_node=9`, NCCL)
* **Precision & Sync:** `bfloat16` autocast, TF32 matmul/cuDNN, custom BF16 gradient `AllReduce` with `1024x` loss-scale protection, chunked cross-entropy (`chunk_sz=2048`) with Z-loss (`1e-4`), dynamic length-packed microbatches (`32,768` tokens/GPU, `~295k` budgeted tokens/step), load-balanced step grouping.
* **Throughput:** `~375k–400k tok/s` in Stage 1 (`98%–100%` utilization across all 9 GPUs), `~405k–429k tok/s` in Stage 2.

### Dataset Coverage
* **Raw Dataset (`tokenized_65k`):** `48,329,808` unique parallel/monolingual pairs (`train_en.bin` 7.9 GiB + `train_my.bin` 7.0 GiB ≈ `~8.0B` raw tokens) + `22,329` held-out validation pairs.
* **Tasks & Register Control:** Bidirectional `En -> My` (`<2my>`) and `My -> En` (`<2en>`), explicit `<formal>` / `<spoken>` style conditioning (`70%` dropout-free application on tagged subsets), `<bt>` marker for backtranslated `My -> En`, and span-corruption `<denoise>` (`Poisson(3.0)`, `<mask>`).

### Stage 1 (`stable` — 100% Coverage + Gold Replay)
* **Plan:** `48,329,808` unique base rows (`100%` coverage) + `12,000,000` gold replay = `60,329,808` planned rows (`60,331,713` rows trained across `58,916` steps, ~10B+ tokens).
* **Hyperparameters:** `AdamW` (`lr=5.0e-4` after 900-step warmup, `betas=(0.9, 0.95)`, `weight_decay=0.01`), `label_smoothing=0.1`, `MTP weight=0.3`, `grad_clip=1.0`.
* **Stage 1 Validation Curve (`eval_val_nll` on 1,024 uniform non-BT held-out pairs):**
  * Step `0`: `Comb NLL 20.1282`
  * Step `1,500`: `Comb NLL 3.0945` (**PPL `22.08`**)
  * Step `8,000`: `Comb NLL 1.7760` (**PPL `5.91`**)
  * Step `19,000`: `Comb NLL 1.5803` (**PPL `4.86`**)
  * Step `39,000`: `Comb NLL 1.4648` (**PPL `4.33`**)
  * Step `58,000`: `Comb NLL 1.4298` (**PPL `4.18`**)
  * **Stage 1 Final (Step `58,916`):** `Comb NLL 1.4316` (**PPL `4.19`** | `En->My: 1.7539` | `My->En: 1.1548`)

### Stage 2 (`cooldown` — Clean Gold Only, Unsmoothed)
* **Plan:** `2,706,748` unique non-BT base rows + `1,500,000` gold replay = `4,206,748` planned rows (`4,210,076` rows trained across `6,793` steps).
* **Hyperparameters:** `100`-step warmup to `3.0e-4` → cosine decay to `1.0e-5`, `label_smoothing=0.0`, `MTP weight=0.15`, `50/50 En↔My`.
* **Stage 2 Validation Curve:**
  * Step `250`: `Comb NLL 1.2945` (**PPL `3.65`**)
  * Step `1,500`: `Comb NLL 1.2403` (**PPL `3.46`**)
  * Step `3,500`: `Comb NLL 1.1941` (**PPL `3.30`**)
  * Step `5,500`: `Comb NLL 1.1687` (**PPL `3.22`**)
  * **Stage 2 Final (Step `6,793`):** **`Comb NLL 1.1640` (PPL `3.20`)** | `En->My: 1.4829` | `My->En: 0.8901` (`PPL 2.44`)

*(Note: PPL values are measured on the `tokenized_65k/val` split and are not directly comparable 1:1 against models evaluated on older splits/vocabularies.)*

---

## 3. Beam=4 Benchmark Outputs (Unseen + Audit Prompts)

```text
🏆 MyTranslation-300M Final Beam=4 Benchmark:
   [En->My Formal]: ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးကို တည်ငြိမ်စေရန် ဗဟိုဘဏ်သည် အတိုးနှုန်းကို မြှင့်တင်ခဲ့သည်။
   [En->My Spoken]: ရိုးရိုးသားသားပြောရရင် ဒီလိုဖြစ်လာမယ်လို့ ကျွန်တော် မမျှော်လင့်ထားပါဘူး၊ ဒါပေမယ့် စိတ်မပူပါနဲ့၊ အတူတူ အဖြေရှာကြည့်ပါမယ်။
   [My->En Formal]: The central bank raised interest rates to control inflation and stabilize the economy.
   [My->En Spoken]: It is a little late to go home because it is raining heavily in Yangon today
   [My->En Idiom]: Love, I have been deeply in love with you and myself, I know God, I know the sugar, I know the spirit, and I'm convinced by the envious deniers, so you're going crazy.
   [En->My News]: မုတ်သုံမိုးသည်းထန်စွာ ရွာသွန်းမှုကြောင့် မြို့နယ်အချို့တွင် ရေကြီးရေလျှံမှုများ ဆိုးရွားစွာ ဖြစ်ပေါ်ခဲ့ပြီး ထောင်နှင့်ချီသော ဒေသခံများသည် ယာယီခိုလှုံရေးစခန်းများသို့ ရွှေ့ပြောင်းခဲ့ရသည်။
   [En->My Health]: ရာသီတုပ်ကွေးကနေ မြန်မြန်သက်သာဖို့ သန့်ရှင်းတဲ့ရေ များများသောက်ပြီး လုံလုံလောက်လောက် အနားယူဖို့ ဆရာဝန်တွေက အကြံပြုထားပါတယ်။
   [En->My Spoken2]: ကျွန်တော် ဒီမနက် တစ်နာရီ ကျော်ကျော်လောက် ကားကြပ်နေလို့ ကျွန်တော် မပါဘဲ အစည်းအဝေးကို စတင်ပါ ။
   [My->En News]: Border trade checkpoints are being accelerated to improve international trade.
   [My->En Tech]: Artificial intelligence technology can help make processes more efficient in education, health, and business.
   [My->En Spoken2]: I have a need to go early tomorrow morning so I will go to bed early tonight so I will call me tomorrow
```

---

## 4. Usage Example (PyTorch + Transformers `trust_remote_code=True`)

```python
import torch, sentencepiece as spm
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSeq2SeqLM

repo_id = "freococo/MyTranslation_300M_Base"
dtype = torch.float16  # Use torch.bfloat16 on Ampere/Hopper/Blackwell, torch.float16 on T4

model = AutoModelForSeq2SeqLM.from_pretrained(repo_id, trust_remote_code=True, dtype=dtype).cuda().eval()
sp = spm.SentencePieceProcessor()
sp.load(hf_hub_download(repo_id, "tokenizer.model"))

print("En->My Formal:", model.translate(
    "The central bank raised interest rates to control inflation and stabilize the economy.",
    sp, direction="en2my", style="formal", num_beams=4
))
print("En->My Spoken:", model.translate(
    "Honestly, I didn't expect things to turn out like this, but don't worry—we'll figure it out together.",
    sp, direction="en2my", style="spoken", num_beams=4
))
print("My->En Formal:", model.translate(
    "ဗဟိုဘဏ်သည် ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးတည်ငြိမ်စေရန် အတိုးနှုန်းများကို မြှင့်တင်ခဲ့သည်။",
    sp, direction="my2en", style="formal", num_beams=4
))
```

---

## 5. Reproducibility Artifacts Included
* `training_codes.txt` — exact multi-GPU PyTorch training script (`train_300m.py`)
* `trainingcelloutputwitheval.txt` — complete Stage 1 + Stage 2 training logs & Beam=4 benchmark output
* `checkpoint-stage2-step-6793/` — FP32 `model.safetensors` + `training_state.pt` (`best_model.pt`)