Translation
Transformers
Safetensors
Burmese
English
mytranslation
text2text-generation
Translation
Safetensors
Burmese
English
myanmartranslator
MyanmarLLM
custom_code
myanmar
Instructions to use freococo/MyTranslation_300M_Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use freococo/MyTranslation_300M_Base with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("translation", model="freococo/MyTranslation_300M_Base", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForSeq2SeqLM model = AutoModelForSeq2SeqLM.from_pretrained("freococo/MyTranslation_300M_Base", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from freococo/MyTranslation_300M_Base: direct link, hf CLI and curl.
- Browser
- Download file 9.71 kB
-
https://huggingface.co/freococo/MyTranslation_300M_Base/resolve/main/README.md
- Command line
-
hf download hf://freococo/MyTranslation_300M_Base/README.md
-
curl -L -o README.md https://huggingface.co/freococo/MyTranslation_300M_Base/resolve/main/README.md
9.71 kB
metadata
language:
- my
- en
license: apache-2.0
library_name: transformers
tags:
- Translation
- Safetensors
- Burmese
- English
- myanmartranslator
- MyanmarLLM
- custom_code
- myanmar
datasets:
- freococo/myanmar-english-corpus
pipeline_tag: translation
MyTranslation_300M_Base
MyTranslation_300M_Base is a 298.76M-parameter (inference; 312.94M training parameters including the Multi-Token Prediction head) bidirectional English ↔ Burmese (my) Neural Machine Translation encoder-decoder model trained from scratch on ~10B+ training tokens using 9× NVIDIA GeForce RTX 5090 GPUs.
Status: Base checkpoint (
Stage 1 Pretraining + Stage 2 Gold Cooldown), published prior to optional spoken/idiom Stage 3 polishing.
1. Architecture Highlights
| Component | Specification |
|---|---|
| Inference Parameters | 298,763,201 (298.76M, BF16 model.safetensors ~598 MB) |
| Training Parameters | 312,940,000 (312.94M, includes MTPHeadBlock dropped at export) |
| Encoder | 12 bidirectional layers, d_model=1024, n_heads=16, n_kv_heads=4 (GQA, head_dim=64), ffn_dim=3072 SwiGLU |
| Decoder | 6 causal layers, d_model=1024, n_heads=16, n_kv_heads=4 (GQA), ffn_dim=3072 SwiGLU |
| Positional Encoding | RoPE (theta=50000.0, max_seq_len=2048 / trained at 1024) |
| Attention & Normalization | RMSNorm, QK-Norm, Gated Attention (sigmoid(gate_proj)), Value Residual (v0 mixing) |
| Encoder-Decoder Routing | Dense Layer-Wise Attention (enc_dwa_logits) + 3 learned SharedCrossKVProj mixtures (enc_router_logits) shared by decoder pairs (0,1), (2,3), (4,5) |
| Tokenizer & Vocab | 65,536-piece SentencePiece (UNK=0, BOS=1, EOS=2, PAD=3, <2my>=4, <2en>=5, <formal>=6, <spoken>=7, <bt>=8, <denoise>=9, <mask>=10), tied embeddings (embed_scale=32.0) |
Learned Cross-KV Encoder Mixture Weights (Post-Stage 2)
- Cross-KV Pair 0 (Decoder 0–1): Top encoder layers
[7, 8, 6, 2]([0.226, 0.200, 0.187, 0.102]) - Cross-KV Pair 1 (Decoder 2–3): Top encoder layers
[9, 10, 8, 11]([0.339, 0.206, 0.187, 0.124]) - Cross-KV Pair 2 (Decoder 4–5): Top encoder layers
[11, 9, 10, 8]([0.340, 0.335, 0.278, 0.041])
2. Training Hardware, Data & Schedule
Cluster & Precision Setup
- GPUs: 9× NVIDIA GeForce RTX 5090 (
32,607 MiBVRAM each,torchrun --nproc_per_node=9, NCCL) - Precision & Sync:
bfloat16autocast, TF32 matmul/cuDNN, custom BF16 gradientAllReducewith1024xloss-scale protection, chunked cross-entropy (chunk_sz=2048) with Z-loss (1e-4), dynamic length-packed microbatches (32,768tokens/GPU,~295kbudgeted tokens/step), load-balanced step grouping. - Throughput:
~375k–400k tok/sin Stage 1 (98%–100%utilization across all 9 GPUs),~405k–429k tok/sin Stage 2.
Dataset Coverage
- Raw Dataset (
tokenized_65k):48,329,808unique parallel/monolingual pairs (train_en.bin7.9 GiB +train_my.bin7.0 GiB ≈~8.0Braw tokens) +22,329held-out validation pairs. - Tasks & Register Control: Bidirectional
En -> My(<2my>) andMy -> En(<2en>), explicit<formal>/<spoken>style conditioning (70%dropout-free application on tagged subsets),<bt>marker for backtranslatedMy -> En, and span-corruption<denoise>(Poisson(3.0),<mask>).
Stage 1 (stable — 100% Coverage + Gold Replay)
- Plan:
48,329,808unique base rows (100%coverage) +12,000,000gold replay =60,329,808planned rows (60,331,713rows trained across58,916steps, ~10B+ tokens). - Hyperparameters:
AdamW(lr=5.0e-4after 900-step warmup,betas=(0.9, 0.95),weight_decay=0.01),label_smoothing=0.1,MTP weight=0.3,grad_clip=1.0. - Stage 1 Validation Curve (
eval_val_nllon 1,024 uniform non-BT held-out pairs):- Step
0:Comb NLL 20.1282 - Step
1,500:Comb NLL 3.0945(PPL22.08) - Step
8,000:Comb NLL 1.7760(PPL5.91) - Step
19,000:Comb NLL 1.5803(PPL4.86) - Step
39,000:Comb NLL 1.4648(PPL4.33) - Step
58,000:Comb NLL 1.4298(PPL4.18) - Stage 1 Final (Step
58,916):Comb NLL 1.4316(PPL4.19|En->My: 1.7539|My->En: 1.1548)
- Step
Stage 2 (cooldown — Clean Gold Only, Unsmoothed)
- Plan:
2,706,748unique non-BT base rows +1,500,000gold replay =4,206,748planned rows (4,210,076rows trained across6,793steps). - Hyperparameters:
100-step warmup to3.0e-4→ cosine decay to1.0e-5,label_smoothing=0.0,MTP weight=0.15,50/50 En↔My. - Stage 2 Validation Curve:
- Step
250:Comb NLL 1.2945(PPL3.65) - Step
1,500:Comb NLL 1.2403(PPL3.46) - Step
3,500:Comb NLL 1.1941(PPL3.30) - Step
5,500:Comb NLL 1.1687(PPL3.22) - Stage 2 Final (Step
6,793):Comb NLL 1.1640(PPL3.20) |En->My: 1.4829|My->En: 0.8901(PPL 2.44)
- Step
(Note: PPL values are measured on the tokenized_65k/val split and are not directly comparable 1:1 against models evaluated on older splits/vocabularies.)
3. Beam=4 Benchmark Outputs (Unseen + Audit Prompts)
🏆 MyTranslation-300M Final Beam=4 Benchmark:
[En->My Formal]: ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးကို တည်ငြိမ်စေရန် ဗဟိုဘဏ်သည် အတိုးနှုန်းကို မြှင့်တင်ခဲ့သည်။
[En->My Spoken]: ရိုးရိုးသားသားပြောရရင် ဒီလိုဖြစ်လာမယ်လို့ ကျွန်တော် မမျှော်လင့်ထားပါဘူး၊ ဒါပေမယ့် စိတ်မပူပါနဲ့၊ အတူတူ အဖြေရှာကြည့်ပါမယ်။
[My->En Formal]: The central bank raised interest rates to control inflation and stabilize the economy.
[My->En Spoken]: It is a little late to go home because it is raining heavily in Yangon today
[My->En Idiom]: Love, I have been deeply in love with you and myself, I know God, I know the sugar, I know the spirit, and I'm convinced by the envious deniers, so you're going crazy.
[En->My News]: မုတ်သုံမိုးသည်းထန်စွာ ရွာသွန်းမှုကြောင့် မြို့နယ်အချို့တွင် ရေကြီးရေလျှံမှုများ ဆိုးရွားစွာ ဖြစ်ပေါ်ခဲ့ပြီး ထောင်နှင့်ချီသော ဒေသခံများသည် ယာယီခိုလှုံရေးစခန်းများသို့ ရွှေ့ပြောင်းခဲ့ရသည်။
[En->My Health]: ရာသီတုပ်ကွေးကနေ မြန်မြန်သက်သာဖို့ သန့်ရှင်းတဲ့ရေ များများသောက်ပြီး လုံလုံလောက်လောက် အနားယူဖို့ ဆရာဝန်တွေက အကြံပြုထားပါတယ်။
[En->My Spoken2]: ကျွန်တော် ဒီမနက် တစ်နာရီ ကျော်ကျော်လောက် ကားကြပ်နေလို့ ကျွန်တော် မပါဘဲ အစည်းအဝေးကို စတင်ပါ ။
[My->En News]: Border trade checkpoints are being accelerated to improve international trade.
[My->En Tech]: Artificial intelligence technology can help make processes more efficient in education, health, and business.
[My->En Spoken2]: I have a need to go early tomorrow morning so I will go to bed early tonight so I will call me tomorrow
4. Usage Example (PyTorch + Transformers trust_remote_code=True)
import torch, sentencepiece as spm
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSeq2SeqLM
repo_id = "freococo/MyTranslation_300M_Base"
dtype = torch.float16 # Use torch.bfloat16 on Ampere/Hopper/Blackwell, torch.float16 on T4
model = AutoModelForSeq2SeqLM.from_pretrained(repo_id, trust_remote_code=True, dtype=dtype).cuda().eval()
sp = spm.SentencePieceProcessor()
sp.load(hf_hub_download(repo_id, "tokenizer.model"))
print("En->My Formal:", model.translate(
"The central bank raised interest rates to control inflation and stabilize the economy.",
sp, direction="en2my", style="formal", num_beams=4
))
print("En->My Spoken:", model.translate(
"Honestly, I didn't expect things to turn out like this, but don't worry—we'll figure it out together.",
sp, direction="en2my", style="spoken", num_beams=4
))
print("My->En Formal:", model.translate(
"ဗဟိုဘဏ်သည် ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးတည်ငြိမ်စေရန် အတိုးနှုန်းများကို မြှင့်တင်ခဲ့သည်။",
sp, direction="my2en", style="formal", num_beams=4
))
5. Reproducibility Artifacts Included
training_codes.txt— exact multi-GPU PyTorch training script (train_300m.py)trainingcelloutputwitheval.txt— complete Stage 1 + Stage 2 training logs & Beam=4 benchmark outputcheckpoint-stage2-step-6793/— FP32model.safetensors+training_state.pt(best_model.pt)