Translation
Transformers
Safetensors
Burmese
English
mytranslation
text2text-generation
Translation
Safetensors
Burmese
English
myanmartranslator
MyanmarLLM
custom_code
myanmar
Instructions to use freococo/MyTranslation_300M_Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use freococo/MyTranslation_300M_Base with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("translation", model="freococo/MyTranslation_300M_Base", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForSeq2SeqLM model = AutoModelForSeq2SeqLM.from_pretrained("freococo/MyTranslation_300M_Base", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
File size: 9,714 Bytes
3bcdbab 630adda 3bcdbab 630adda | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 | ---
language:
- my
- en
license: apache-2.0
library_name: transformers
tags:
- Translation
- Safetensors
- Burmese
- English
- myanmartranslator
- MyanmarLLM
- custom_code
- myanmar
datasets:
- freococo/myanmar-english-corpus
pipeline_tag: translation
---
# MyTranslation_300M_Base
**MyTranslation_300M_Base** is a **`298.76M`-parameter** (inference; `312.94M` training parameters including the Multi-Token Prediction head) bidirectional **English ↔ Burmese (`my`)** Neural Machine Translation encoder-decoder model trained from scratch on **~10B+ training tokens** using **9× NVIDIA GeForce RTX 5090 GPUs**.
> **Status:** Base checkpoint (`Stage 1 Pretraining + Stage 2 Gold Cooldown`), published prior to optional spoken/idiom Stage 3 polishing.
---
## 1. Architecture Highlights
| Component | Specification |
|---|---|
| **Inference Parameters** | `298,763,201` (`298.76M`, BF16 `model.safetensors` ~598 MB) |
| **Training Parameters** | `312,940,000` (`312.94M`, includes `MTPHeadBlock` dropped at export) |
| **Encoder** | `12` bidirectional layers, `d_model=1024`, `n_heads=16`, `n_kv_heads=4` (GQA, `head_dim=64`), `ffn_dim=3072` SwiGLU |
| **Decoder** | `6` causal layers, `d_model=1024`, `n_heads=16`, `n_kv_heads=4` (GQA), `ffn_dim=3072` SwiGLU |
| **Positional Encoding** | RoPE (`theta=50000.0`, `max_seq_len=2048` / trained at `1024`) |
| **Attention & Normalization** | `RMSNorm`, QK-Norm, Gated Attention (`sigmoid(gate_proj)`), Value Residual (`v0` mixing) |
| **Encoder-Decoder Routing** | Dense Layer-Wise Attention (`enc_dwa_logits`) + `3` learned `SharedCrossKVProj` mixtures (`enc_router_logits`) shared by decoder pairs `(0,1)`, `(2,3)`, `(4,5)` |
| **Tokenizer & Vocab** | `65,536`-piece SentencePiece (`UNK=0`, `BOS=1`, `EOS=2`, `PAD=3`, `<2my>=4`, `<2en>=5`, `<formal>=6`, `<spoken>=7`, `<bt>=8`, `<denoise>=9`, `<mask>=10`), tied embeddings (`embed_scale=32.0`) |
### Learned Cross-KV Encoder Mixture Weights (Post-Stage 2)
* **Cross-KV Pair 0 (Decoder 0–1):** Top encoder layers `[7, 8, 6, 2]` (`[0.226, 0.200, 0.187, 0.102]`)
* **Cross-KV Pair 1 (Decoder 2–3):** Top encoder layers `[9, 10, 8, 11]` (`[0.339, 0.206, 0.187, 0.124]`)
* **Cross-KV Pair 2 (Decoder 4–5):** Top encoder layers `[11, 9, 10, 8]` (`[0.340, 0.335, 0.278, 0.041]`)
---
## 2. Training Hardware, Data & Schedule
### Cluster & Precision Setup
* **GPUs:** **9× NVIDIA GeForce RTX 5090** (`32,607 MiB` VRAM each, `torchrun --nproc_per_node=9`, NCCL)
* **Precision & Sync:** `bfloat16` autocast, TF32 matmul/cuDNN, custom BF16 gradient `AllReduce` with `1024x` loss-scale protection, chunked cross-entropy (`chunk_sz=2048`) with Z-loss (`1e-4`), dynamic length-packed microbatches (`32,768` tokens/GPU, `~295k` budgeted tokens/step), load-balanced step grouping.
* **Throughput:** `~375k–400k tok/s` in Stage 1 (`98%–100%` utilization across all 9 GPUs), `~405k–429k tok/s` in Stage 2.
### Dataset Coverage
* **Raw Dataset (`tokenized_65k`):** `48,329,808` unique parallel/monolingual pairs (`train_en.bin` 7.9 GiB + `train_my.bin` 7.0 GiB ≈ `~8.0B` raw tokens) + `22,329` held-out validation pairs.
* **Tasks & Register Control:** Bidirectional `En -> My` (`<2my>`) and `My -> En` (`<2en>`), explicit `<formal>` / `<spoken>` style conditioning (`70%` dropout-free application on tagged subsets), `<bt>` marker for backtranslated `My -> En`, and span-corruption `<denoise>` (`Poisson(3.0)`, `<mask>`).
### Stage 1 (`stable` — 100% Coverage + Gold Replay)
* **Plan:** `48,329,808` unique base rows (`100%` coverage) + `12,000,000` gold replay = `60,329,808` planned rows (`60,331,713` rows trained across `58,916` steps, ~10B+ tokens).
* **Hyperparameters:** `AdamW` (`lr=5.0e-4` after 900-step warmup, `betas=(0.9, 0.95)`, `weight_decay=0.01`), `label_smoothing=0.1`, `MTP weight=0.3`, `grad_clip=1.0`.
* **Stage 1 Validation Curve (`eval_val_nll` on 1,024 uniform non-BT held-out pairs):**
* Step `0`: `Comb NLL 20.1282`
* Step `1,500`: `Comb NLL 3.0945` (**PPL `22.08`**)
* Step `8,000`: `Comb NLL 1.7760` (**PPL `5.91`**)
* Step `19,000`: `Comb NLL 1.5803` (**PPL `4.86`**)
* Step `39,000`: `Comb NLL 1.4648` (**PPL `4.33`**)
* Step `58,000`: `Comb NLL 1.4298` (**PPL `4.18`**)
* **Stage 1 Final (Step `58,916`):** `Comb NLL 1.4316` (**PPL `4.19`** | `En->My: 1.7539` | `My->En: 1.1548`)
### Stage 2 (`cooldown` — Clean Gold Only, Unsmoothed)
* **Plan:** `2,706,748` unique non-BT base rows + `1,500,000` gold replay = `4,206,748` planned rows (`4,210,076` rows trained across `6,793` steps).
* **Hyperparameters:** `100`-step warmup to `3.0e-4` → cosine decay to `1.0e-5`, `label_smoothing=0.0`, `MTP weight=0.15`, `50/50 En↔My`.
* **Stage 2 Validation Curve:**
* Step `250`: `Comb NLL 1.2945` (**PPL `3.65`**)
* Step `1,500`: `Comb NLL 1.2403` (**PPL `3.46`**)
* Step `3,500`: `Comb NLL 1.1941` (**PPL `3.30`**)
* Step `5,500`: `Comb NLL 1.1687` (**PPL `3.22`**)
* **Stage 2 Final (Step `6,793`):** **`Comb NLL 1.1640` (PPL `3.20`)** | `En->My: 1.4829` | `My->En: 0.8901` (`PPL 2.44`)
*(Note: PPL values are measured on the `tokenized_65k/val` split and are not directly comparable 1:1 against models evaluated on older splits/vocabularies.)*
---
## 3. Beam=4 Benchmark Outputs (Unseen + Audit Prompts)
```text
🏆 MyTranslation-300M Final Beam=4 Benchmark:
[En->My Formal]: ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးကို တည်ငြိမ်စေရန် ဗဟိုဘဏ်သည် အတိုးနှုန်းကို မြှင့်တင်ခဲ့သည်။
[En->My Spoken]: ရိုးရိုးသားသားပြောရရင် ဒီလိုဖြစ်လာမယ်လို့ ကျွန်တော် မမျှော်လင့်ထားပါဘူး၊ ဒါပေမယ့် စိတ်မပူပါနဲ့၊ အတူတူ အဖြေရှာကြည့်ပါမယ်။
[My->En Formal]: The central bank raised interest rates to control inflation and stabilize the economy.
[My->En Spoken]: It is a little late to go home because it is raining heavily in Yangon today
[My->En Idiom]: Love, I have been deeply in love with you and myself, I know God, I know the sugar, I know the spirit, and I'm convinced by the envious deniers, so you're going crazy.
[En->My News]: မုတ်သုံမိုးသည်းထန်စွာ ရွာသွန်းမှုကြောင့် မြို့နယ်အချို့တွင် ရေကြီးရေလျှံမှုများ ဆိုးရွားစွာ ဖြစ်ပေါ်ခဲ့ပြီး ထောင်နှင့်ချီသော ဒေသခံများသည် ယာယီခိုလှုံရေးစခန်းများသို့ ရွှေ့ပြောင်းခဲ့ရသည်။
[En->My Health]: ရာသီတုပ်ကွေးကနေ မြန်မြန်သက်သာဖို့ သန့်ရှင်းတဲ့ရေ များများသောက်ပြီး လုံလုံလောက်လောက် အနားယူဖို့ ဆရာဝန်တွေက အကြံပြုထားပါတယ်။
[En->My Spoken2]: ကျွန်တော် ဒီမနက် တစ်နာရီ ကျော်ကျော်လောက် ကားကြပ်နေလို့ ကျွန်တော် မပါဘဲ အစည်းအဝေးကို စတင်ပါ ။
[My->En News]: Border trade checkpoints are being accelerated to improve international trade.
[My->En Tech]: Artificial intelligence technology can help make processes more efficient in education, health, and business.
[My->En Spoken2]: I have a need to go early tomorrow morning so I will go to bed early tonight so I will call me tomorrow
```
---
## 4. Usage Example (PyTorch + Transformers `trust_remote_code=True`)
```python
import torch, sentencepiece as spm
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSeq2SeqLM
repo_id = "freococo/MyTranslation_300M_Base"
dtype = torch.float16 # Use torch.bfloat16 on Ampere/Hopper/Blackwell, torch.float16 on T4
model = AutoModelForSeq2SeqLM.from_pretrained(repo_id, trust_remote_code=True, dtype=dtype).cuda().eval()
sp = spm.SentencePieceProcessor()
sp.load(hf_hub_download(repo_id, "tokenizer.model"))
print("En->My Formal:", model.translate(
"The central bank raised interest rates to control inflation and stabilize the economy.",
sp, direction="en2my", style="formal", num_beams=4
))
print("En->My Spoken:", model.translate(
"Honestly, I didn't expect things to turn out like this, but don't worry—we'll figure it out together.",
sp, direction="en2my", style="spoken", num_beams=4
))
print("My->En Formal:", model.translate(
"ဗဟိုဘဏ်သည် ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးတည်ငြိမ်စေရန် အတိုးနှုန်းများကို မြှင့်တင်ခဲ့သည်။",
sp, direction="my2en", style="formal", num_beams=4
))
```
---
## 5. Reproducibility Artifacts Included
* `training_codes.txt` — exact multi-GPU PyTorch training script (`train_300m.py`)
* `trainingcelloutputwitheval.txt` — complete Stage 1 + Stage 2 training logs & Beam=4 benchmark output
* `checkpoint-stage2-step-6793/` — FP32 `model.safetensors` + `training_state.pt` (`best_model.pt`) |