freococo's picture
Update README.md
630adda verified
|
Raw History Blame Contribute Delete
9.71 kB
---
language:
- my
- en
license: apache-2.0
library_name: transformers
tags:
- Translation
- Safetensors
- Burmese
- English
- myanmartranslator
- MyanmarLLM
- custom_code
- myanmar
datasets:
- freococo/myanmar-english-corpus
pipeline_tag: translation
---
# MyTranslation_300M_Base
**MyTranslation_300M_Base** is a **`298.76M`-parameter** (inference; `312.94M` training parameters including the Multi-Token Prediction head) bidirectional **English ↔ Burmese (`my`)** Neural Machine Translation encoder-decoder model trained from scratch on **~10B+ training tokens** using **9× NVIDIA GeForce RTX 5090 GPUs**.
> **Status:** Base checkpoint (`Stage 1 Pretraining + Stage 2 Gold Cooldown`), published prior to optional spoken/idiom Stage 3 polishing.
---
## 1. Architecture Highlights
| Component | Specification |
|---|---|
| **Inference Parameters** | `298,763,201` (`298.76M`, BF16 `model.safetensors` ~598 MB) |
| **Training Parameters** | `312,940,000` (`312.94M`, includes `MTPHeadBlock` dropped at export) |
| **Encoder** | `12` bidirectional layers, `d_model=1024`, `n_heads=16`, `n_kv_heads=4` (GQA, `head_dim=64`), `ffn_dim=3072` SwiGLU |
| **Decoder** | `6` causal layers, `d_model=1024`, `n_heads=16`, `n_kv_heads=4` (GQA), `ffn_dim=3072` SwiGLU |
| **Positional Encoding** | RoPE (`theta=50000.0`, `max_seq_len=2048` / trained at `1024`) |
| **Attention & Normalization** | `RMSNorm`, QK-Norm, Gated Attention (`sigmoid(gate_proj)`), Value Residual (`v0` mixing) |
| **Encoder-Decoder Routing** | Dense Layer-Wise Attention (`enc_dwa_logits`) + `3` learned `SharedCrossKVProj` mixtures (`enc_router_logits`) shared by decoder pairs `(0,1)`, `(2,3)`, `(4,5)` |
| **Tokenizer & Vocab** | `65,536`-piece SentencePiece (`UNK=0`, `BOS=1`, `EOS=2`, `PAD=3`, `<2my>=4`, `<2en>=5`, `<formal>=6`, `<spoken>=7`, `<bt>=8`, `<denoise>=9`, `<mask>=10`), tied embeddings (`embed_scale=32.0`) |
### Learned Cross-KV Encoder Mixture Weights (Post-Stage 2)
* **Cross-KV Pair 0 (Decoder 0–1):** Top encoder layers `[7, 8, 6, 2]` (`[0.226, 0.200, 0.187, 0.102]`)
* **Cross-KV Pair 1 (Decoder 2–3):** Top encoder layers `[9, 10, 8, 11]` (`[0.339, 0.206, 0.187, 0.124]`)
* **Cross-KV Pair 2 (Decoder 4–5):** Top encoder layers `[11, 9, 10, 8]` (`[0.340, 0.335, 0.278, 0.041]`)
---
## 2. Training Hardware, Data & Schedule
### Cluster & Precision Setup
* **GPUs:** **9× NVIDIA GeForce RTX 5090** (`32,607 MiB` VRAM each, `torchrun --nproc_per_node=9`, NCCL)
* **Precision & Sync:** `bfloat16` autocast, TF32 matmul/cuDNN, custom BF16 gradient `AllReduce` with `1024x` loss-scale protection, chunked cross-entropy (`chunk_sz=2048`) with Z-loss (`1e-4`), dynamic length-packed microbatches (`32,768` tokens/GPU, `~295k` budgeted tokens/step), load-balanced step grouping.
* **Throughput:** `~375k–400k tok/s` in Stage 1 (`98%–100%` utilization across all 9 GPUs), `~405k–429k tok/s` in Stage 2.
### Dataset Coverage
* **Raw Dataset (`tokenized_65k`):** `48,329,808` unique parallel/monolingual pairs (`train_en.bin` 7.9 GiB + `train_my.bin` 7.0 GiB ≈ `~8.0B` raw tokens) + `22,329` held-out validation pairs.
* **Tasks & Register Control:** Bidirectional `En -> My` (`<2my>`) and `My -> En` (`<2en>`), explicit `<formal>` / `<spoken>` style conditioning (`70%` dropout-free application on tagged subsets), `<bt>` marker for backtranslated `My -> En`, and span-corruption `<denoise>` (`Poisson(3.0)`, `<mask>`).
### Stage 1 (`stable` — 100% Coverage + Gold Replay)
* **Plan:** `48,329,808` unique base rows (`100%` coverage) + `12,000,000` gold replay = `60,329,808` planned rows (`60,331,713` rows trained across `58,916` steps, ~10B+ tokens).
* **Hyperparameters:** `AdamW` (`lr=5.0e-4` after 900-step warmup, `betas=(0.9, 0.95)`, `weight_decay=0.01`), `label_smoothing=0.1`, `MTP weight=0.3`, `grad_clip=1.0`.
* **Stage 1 Validation Curve (`eval_val_nll` on 1,024 uniform non-BT held-out pairs):**
* Step `0`: `Comb NLL 20.1282`
* Step `1,500`: `Comb NLL 3.0945` (**PPL `22.08`**)
* Step `8,000`: `Comb NLL 1.7760` (**PPL `5.91`**)
* Step `19,000`: `Comb NLL 1.5803` (**PPL `4.86`**)
* Step `39,000`: `Comb NLL 1.4648` (**PPL `4.33`**)
* Step `58,000`: `Comb NLL 1.4298` (**PPL `4.18`**)
* **Stage 1 Final (Step `58,916`):** `Comb NLL 1.4316` (**PPL `4.19`** | `En->My: 1.7539` | `My->En: 1.1548`)
### Stage 2 (`cooldown` — Clean Gold Only, Unsmoothed)
* **Plan:** `2,706,748` unique non-BT base rows + `1,500,000` gold replay = `4,206,748` planned rows (`4,210,076` rows trained across `6,793` steps).
* **Hyperparameters:** `100`-step warmup to `3.0e-4` → cosine decay to `1.0e-5`, `label_smoothing=0.0`, `MTP weight=0.15`, `50/50 En↔My`.
* **Stage 2 Validation Curve:**
* Step `250`: `Comb NLL 1.2945` (**PPL `3.65`**)
* Step `1,500`: `Comb NLL 1.2403` (**PPL `3.46`**)
* Step `3,500`: `Comb NLL 1.1941` (**PPL `3.30`**)
* Step `5,500`: `Comb NLL 1.1687` (**PPL `3.22`**)
* **Stage 2 Final (Step `6,793`):** **`Comb NLL 1.1640` (PPL `3.20`)** | `En->My: 1.4829` | `My->En: 0.8901` (`PPL 2.44`)
*(Note: PPL values are measured on the `tokenized_65k/val` split and are not directly comparable 1:1 against models evaluated on older splits/vocabularies.)*
---
## 3. Beam=4 Benchmark Outputs (Unseen + Audit Prompts)
```text
🏆 MyTranslation-300M Final Beam=4 Benchmark:
[En->My Formal]: ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးကို တည်ငြိမ်စေရန် ဗဟိုဘဏ်သည် အတိုးနှုန်းကို မြှင့်တင်ခဲ့သည်။
[En->My Spoken]: ရိုးရိုးသားသားပြောရရင် ဒီလိုဖြစ်လာမယ်လို့ ကျွန်တော် မမျှော်လင့်ထားပါဘူး၊ ဒါပေမယ့် စိတ်မပူပါနဲ့၊ အတူတူ အဖြေရှာကြည့်ပါမယ်။
[My->En Formal]: The central bank raised interest rates to control inflation and stabilize the economy.
[My->En Spoken]: It is a little late to go home because it is raining heavily in Yangon today
[My->En Idiom]: Love, I have been deeply in love with you and myself, I know God, I know the sugar, I know the spirit, and I'm convinced by the envious deniers, so you're going crazy.
[En->My News]: မုတ်သုံမိုးသည်းထန်စွာ ရွာသွန်းမှုကြောင့် မြို့နယ်အချို့တွင် ရေကြီးရေလျှံမှုများ ဆိုးရွားစွာ ဖြစ်ပေါ်ခဲ့ပြီး ထောင်နှင့်ချီသော ဒေသခံများသည် ယာယီခိုလှုံရေးစခန်းများသို့ ရွှေ့ပြောင်းခဲ့ရသည်။
[En->My Health]: ရာသီတုပ်ကွေးကနေ မြန်မြန်သက်သာဖို့ သန့်ရှင်းတဲ့ရေ များများသောက်ပြီး လုံလုံလောက်လောက် အနားယူဖို့ ဆရာဝန်တွေက အကြံပြုထားပါတယ်။
[En->My Spoken2]: ကျွန်တော် ဒီမနက် တစ်နာရီ ကျော်ကျော်လောက် ကားကြပ်နေလို့ ကျွန်တော် မပါဘဲ အစည်းအဝေးကို စတင်ပါ ။
[My->En News]: Border trade checkpoints are being accelerated to improve international trade.
[My->En Tech]: Artificial intelligence technology can help make processes more efficient in education, health, and business.
[My->En Spoken2]: I have a need to go early tomorrow morning so I will go to bed early tonight so I will call me tomorrow
```
---
## 4. Usage Example (PyTorch + Transformers `trust_remote_code=True`)
```python
import torch, sentencepiece as spm
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSeq2SeqLM
repo_id = "freococo/MyTranslation_300M_Base"
dtype = torch.float16 # Use torch.bfloat16 on Ampere/Hopper/Blackwell, torch.float16 on T4
model = AutoModelForSeq2SeqLM.from_pretrained(repo_id, trust_remote_code=True, dtype=dtype).cuda().eval()
sp = spm.SentencePieceProcessor()
sp.load(hf_hub_download(repo_id, "tokenizer.model"))
print("En->My Formal:", model.translate(
"The central bank raised interest rates to control inflation and stabilize the economy.",
sp, direction="en2my", style="formal", num_beams=4
))
print("En->My Spoken:", model.translate(
"Honestly, I didn't expect things to turn out like this, but don't worry—we'll figure it out together.",
sp, direction="en2my", style="spoken", num_beams=4
))
print("My->En Formal:", model.translate(
"ဗဟိုဘဏ်သည် ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးတည်ငြိမ်စေရန် အတိုးနှုန်းများကို မြှင့်တင်ခဲ့သည်။",
sp, direction="my2en", style="formal", num_beams=4
))
```
---
## 5. Reproducibility Artifacts Included
* `training_codes.txt` — exact multi-GPU PyTorch training script (`train_300m.py`)
* `trainingcelloutputwitheval.txt` — complete Stage 1 + Stage 2 training logs & Beam=4 benchmark output
* `checkpoint-stage2-step-6793/` — FP32 `model.safetensors` + `training_state.pt` (`best_model.pt`)