--- language: - my - en license: apache-2.0 library_name: transformers tags: - Translation - Safetensors - Burmese - English - myanmartranslator - MyanmarLLM - custom_code - myanmar datasets: - freococo/myanmar-english-corpus pipeline_tag: translation --- # MyTranslation_300M_Base **MyTranslation_300M_Base** is a **`298.76M`-parameter** (inference; `312.94M` training parameters including the Multi-Token Prediction head) bidirectional **English ↔ Burmese (`my`)** Neural Machine Translation encoder-decoder model trained from scratch on **~10B+ training tokens** using **9× NVIDIA GeForce RTX 5090 GPUs**. > **Status:** Base checkpoint (`Stage 1 Pretraining + Stage 2 Gold Cooldown`), published prior to optional spoken/idiom Stage 3 polishing. --- ## 1. Architecture Highlights | Component | Specification | |---|---| | **Inference Parameters** | `298,763,201` (`298.76M`, BF16 `model.safetensors` ~598 MB) | | **Training Parameters** | `312,940,000` (`312.94M`, includes `MTPHeadBlock` dropped at export) | | **Encoder** | `12` bidirectional layers, `d_model=1024`, `n_heads=16`, `n_kv_heads=4` (GQA, `head_dim=64`), `ffn_dim=3072` SwiGLU | | **Decoder** | `6` causal layers, `d_model=1024`, `n_heads=16`, `n_kv_heads=4` (GQA), `ffn_dim=3072` SwiGLU | | **Positional Encoding** | RoPE (`theta=50000.0`, `max_seq_len=2048` / trained at `1024`) | | **Attention & Normalization** | `RMSNorm`, QK-Norm, Gated Attention (`sigmoid(gate_proj)`), Value Residual (`v0` mixing) | | **Encoder-Decoder Routing** | Dense Layer-Wise Attention (`enc_dwa_logits`) + `3` learned `SharedCrossKVProj` mixtures (`enc_router_logits`) shared by decoder pairs `(0,1)`, `(2,3)`, `(4,5)` | | **Tokenizer & Vocab** | `65,536`-piece SentencePiece (`UNK=0`, `BOS=1`, `EOS=2`, `PAD=3`, `<2my>=4`, `<2en>=5`, `=6`, `=7`, `=8`, `=9`, `=10`), tied embeddings (`embed_scale=32.0`) | ### Learned Cross-KV Encoder Mixture Weights (Post-Stage 2) * **Cross-KV Pair 0 (Decoder 0–1):** Top encoder layers `[7, 8, 6, 2]` (`[0.226, 0.200, 0.187, 0.102]`) * **Cross-KV Pair 1 (Decoder 2–3):** Top encoder layers `[9, 10, 8, 11]` (`[0.339, 0.206, 0.187, 0.124]`) * **Cross-KV Pair 2 (Decoder 4–5):** Top encoder layers `[11, 9, 10, 8]` (`[0.340, 0.335, 0.278, 0.041]`) --- ## 2. Training Hardware, Data & Schedule ### Cluster & Precision Setup * **GPUs:** **9× NVIDIA GeForce RTX 5090** (`32,607 MiB` VRAM each, `torchrun --nproc_per_node=9`, NCCL) * **Precision & Sync:** `bfloat16` autocast, TF32 matmul/cuDNN, custom BF16 gradient `AllReduce` with `1024x` loss-scale protection, chunked cross-entropy (`chunk_sz=2048`) with Z-loss (`1e-4`), dynamic length-packed microbatches (`32,768` tokens/GPU, `~295k` budgeted tokens/step), load-balanced step grouping. * **Throughput:** `~375k–400k tok/s` in Stage 1 (`98%–100%` utilization across all 9 GPUs), `~405k–429k tok/s` in Stage 2. ### Dataset Coverage * **Raw Dataset (`tokenized_65k`):** `48,329,808` unique parallel/monolingual pairs (`train_en.bin` 7.9 GiB + `train_my.bin` 7.0 GiB ≈ `~8.0B` raw tokens) + `22,329` held-out validation pairs. * **Tasks & Register Control:** Bidirectional `En -> My` (`<2my>`) and `My -> En` (`<2en>`), explicit `` / `` style conditioning (`70%` dropout-free application on tagged subsets), `` marker for backtranslated `My -> En`, and span-corruption `` (`Poisson(3.0)`, ``). ### Stage 1 (`stable` — 100% Coverage + Gold Replay) * **Plan:** `48,329,808` unique base rows (`100%` coverage) + `12,000,000` gold replay = `60,329,808` planned rows (`60,331,713` rows trained across `58,916` steps, ~10B+ tokens). * **Hyperparameters:** `AdamW` (`lr=5.0e-4` after 900-step warmup, `betas=(0.9, 0.95)`, `weight_decay=0.01`), `label_smoothing=0.1`, `MTP weight=0.3`, `grad_clip=1.0`. * **Stage 1 Validation Curve (`eval_val_nll` on 1,024 uniform non-BT held-out pairs):** * Step `0`: `Comb NLL 20.1282` * Step `1,500`: `Comb NLL 3.0945` (**PPL `22.08`**) * Step `8,000`: `Comb NLL 1.7760` (**PPL `5.91`**) * Step `19,000`: `Comb NLL 1.5803` (**PPL `4.86`**) * Step `39,000`: `Comb NLL 1.4648` (**PPL `4.33`**) * Step `58,000`: `Comb NLL 1.4298` (**PPL `4.18`**) * **Stage 1 Final (Step `58,916`):** `Comb NLL 1.4316` (**PPL `4.19`** | `En->My: 1.7539` | `My->En: 1.1548`) ### Stage 2 (`cooldown` — Clean Gold Only, Unsmoothed) * **Plan:** `2,706,748` unique non-BT base rows + `1,500,000` gold replay = `4,206,748` planned rows (`4,210,076` rows trained across `6,793` steps). * **Hyperparameters:** `100`-step warmup to `3.0e-4` → cosine decay to `1.0e-5`, `label_smoothing=0.0`, `MTP weight=0.15`, `50/50 En↔My`. * **Stage 2 Validation Curve:** * Step `250`: `Comb NLL 1.2945` (**PPL `3.65`**) * Step `1,500`: `Comb NLL 1.2403` (**PPL `3.46`**) * Step `3,500`: `Comb NLL 1.1941` (**PPL `3.30`**) * Step `5,500`: `Comb NLL 1.1687` (**PPL `3.22`**) * **Stage 2 Final (Step `6,793`):** **`Comb NLL 1.1640` (PPL `3.20`)** | `En->My: 1.4829` | `My->En: 0.8901` (`PPL 2.44`) *(Note: PPL values are measured on the `tokenized_65k/val` split and are not directly comparable 1:1 against models evaluated on older splits/vocabularies.)* --- ## 3. Beam=4 Benchmark Outputs (Unseen + Audit Prompts) ```text 🏆 MyTranslation-300M Final Beam=4 Benchmark: [En->My Formal]: ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးကို တည်ငြိမ်စေရန် ဗဟိုဘဏ်သည် အတိုးနှုန်းကို မြှင့်တင်ခဲ့သည်။ [En->My Spoken]: ရိုးရိုးသားသားပြောရရင် ဒီလိုဖြစ်လာမယ်လို့ ကျွန်တော် မမျှော်လင့်ထားပါဘူး၊ ဒါပေမယ့် စိတ်မပူပါနဲ့၊ အတူတူ အဖြေရှာကြည့်ပါမယ်။ [My->En Formal]: The central bank raised interest rates to control inflation and stabilize the economy. [My->En Spoken]: It is a little late to go home because it is raining heavily in Yangon today [My->En Idiom]: Love, I have been deeply in love with you and myself, I know God, I know the sugar, I know the spirit, and I'm convinced by the envious deniers, so you're going crazy. [En->My News]: မုတ်သုံမိုးသည်းထန်စွာ ရွာသွန်းမှုကြောင့် မြို့နယ်အချို့တွင် ရေကြီးရေလျှံမှုများ ဆိုးရွားစွာ ဖြစ်ပေါ်ခဲ့ပြီး ထောင်နှင့်ချီသော ဒေသခံများသည် ယာယီခိုလှုံရေးစခန်းများသို့ ရွှေ့ပြောင်းခဲ့ရသည်။ [En->My Health]: ရာသီတုပ်ကွေးကနေ မြန်မြန်သက်သာဖို့ သန့်ရှင်းတဲ့ရေ များများသောက်ပြီး လုံလုံလောက်လောက် အနားယူဖို့ ဆရာဝန်တွေက အကြံပြုထားပါတယ်။ [En->My Spoken2]: ကျွန်တော် ဒီမနက် တစ်နာရီ ကျော်ကျော်လောက် ကားကြပ်နေလို့ ကျွန်တော် မပါဘဲ အစည်းအဝေးကို စတင်ပါ ။ [My->En News]: Border trade checkpoints are being accelerated to improve international trade. [My->En Tech]: Artificial intelligence technology can help make processes more efficient in education, health, and business. [My->En Spoken2]: I have a need to go early tomorrow morning so I will go to bed early tonight so I will call me tomorrow ``` --- ## 4. Usage Example (PyTorch + Transformers `trust_remote_code=True`) ```python import torch, sentencepiece as spm from huggingface_hub import hf_hub_download from transformers import AutoModelForSeq2SeqLM repo_id = "freococo/MyTranslation_300M_Base" dtype = torch.float16 # Use torch.bfloat16 on Ampere/Hopper/Blackwell, torch.float16 on T4 model = AutoModelForSeq2SeqLM.from_pretrained(repo_id, trust_remote_code=True, dtype=dtype).cuda().eval() sp = spm.SentencePieceProcessor() sp.load(hf_hub_download(repo_id, "tokenizer.model")) print("En->My Formal:", model.translate( "The central bank raised interest rates to control inflation and stabilize the economy.", sp, direction="en2my", style="formal", num_beams=4 )) print("En->My Spoken:", model.translate( "Honestly, I didn't expect things to turn out like this, but don't worry—we'll figure it out together.", sp, direction="en2my", style="spoken", num_beams=4 )) print("My->En Formal:", model.translate( "ဗဟိုဘဏ်သည် ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးတည်ငြိမ်စေရန် အတိုးနှုန်းများကို မြှင့်တင်ခဲ့သည်။", sp, direction="my2en", style="formal", num_beams=4 )) ``` --- ## 5. Reproducibility Artifacts Included * `training_codes.txt` — exact multi-GPU PyTorch training script (`train_300m.py`) * `trainingcelloutputwitheval.txt` — complete Stage 1 + Stage 2 training logs & Beam=4 benchmark output * `checkpoint-stage2-step-6793/` — FP32 `model.safetensors` + `training_state.pt` (`best_model.pt`)