Translation
Transformers
Safetensors
Burmese
English
mytranslation
text2text-generation
Translation
Safetensors
Burmese
English
myanmartranslator
MyanmarLLM
custom_code
myanmar
Instructions to use freococo/MyTranslation_300M_Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use freococo/MyTranslation_300M_Base with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("translation", model="freococo/MyTranslation_300M_Base", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForSeq2SeqLM model = AutoModelForSeq2SeqLM.from_pretrained("freococo/MyTranslation_300M_Base", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from freococo/MyTranslation_300M_Base: direct link, hf CLI and curl.
- Browser
- Download file 9.71 kB
-
https://huggingface.co/freococo/MyTranslation_300M_Base/resolve/main/README.md
- Command line
-
hf download hf://freococo/MyTranslation_300M_Base/README.md
-
curl -L -o README.md https://huggingface.co/freococo/MyTranslation_300M_Base/resolve/main/README.md
9.71 kB
| language: | |
| - my | |
| - en | |
| license: apache-2.0 | |
| library_name: transformers | |
| tags: | |
| - Translation | |
| - Safetensors | |
| - Burmese | |
| - English | |
| - myanmartranslator | |
| - MyanmarLLM | |
| - custom_code | |
| - myanmar | |
| datasets: | |
| - freococo/myanmar-english-corpus | |
| pipeline_tag: translation | |
| # MyTranslation_300M_Base | |
| **MyTranslation_300M_Base** is a **`298.76M`-parameter** (inference; `312.94M` training parameters including the Multi-Token Prediction head) bidirectional **English ↔ Burmese (`my`)** Neural Machine Translation encoder-decoder model trained from scratch on **~10B+ training tokens** using **9× NVIDIA GeForce RTX 5090 GPUs**. | |
| > **Status:** Base checkpoint (`Stage 1 Pretraining + Stage 2 Gold Cooldown`), published prior to optional spoken/idiom Stage 3 polishing. | |
| --- | |
| ## 1. Architecture Highlights | |
| | Component | Specification | | |
| |---|---| | |
| | **Inference Parameters** | `298,763,201` (`298.76M`, BF16 `model.safetensors` ~598 MB) | | |
| | **Training Parameters** | `312,940,000` (`312.94M`, includes `MTPHeadBlock` dropped at export) | | |
| | **Encoder** | `12` bidirectional layers, `d_model=1024`, `n_heads=16`, `n_kv_heads=4` (GQA, `head_dim=64`), `ffn_dim=3072` SwiGLU | | |
| | **Decoder** | `6` causal layers, `d_model=1024`, `n_heads=16`, `n_kv_heads=4` (GQA), `ffn_dim=3072` SwiGLU | | |
| | **Positional Encoding** | RoPE (`theta=50000.0`, `max_seq_len=2048` / trained at `1024`) | | |
| | **Attention & Normalization** | `RMSNorm`, QK-Norm, Gated Attention (`sigmoid(gate_proj)`), Value Residual (`v0` mixing) | | |
| | **Encoder-Decoder Routing** | Dense Layer-Wise Attention (`enc_dwa_logits`) + `3` learned `SharedCrossKVProj` mixtures (`enc_router_logits`) shared by decoder pairs `(0,1)`, `(2,3)`, `(4,5)` | | |
| | **Tokenizer & Vocab** | `65,536`-piece SentencePiece (`UNK=0`, `BOS=1`, `EOS=2`, `PAD=3`, `<2my>=4`, `<2en>=5`, `<formal>=6`, `<spoken>=7`, `<bt>=8`, `<denoise>=9`, `<mask>=10`), tied embeddings (`embed_scale=32.0`) | | |
| ### Learned Cross-KV Encoder Mixture Weights (Post-Stage 2) | |
| * **Cross-KV Pair 0 (Decoder 0–1):** Top encoder layers `[7, 8, 6, 2]` (`[0.226, 0.200, 0.187, 0.102]`) | |
| * **Cross-KV Pair 1 (Decoder 2–3):** Top encoder layers `[9, 10, 8, 11]` (`[0.339, 0.206, 0.187, 0.124]`) | |
| * **Cross-KV Pair 2 (Decoder 4–5):** Top encoder layers `[11, 9, 10, 8]` (`[0.340, 0.335, 0.278, 0.041]`) | |
| --- | |
| ## 2. Training Hardware, Data & Schedule | |
| ### Cluster & Precision Setup | |
| * **GPUs:** **9× NVIDIA GeForce RTX 5090** (`32,607 MiB` VRAM each, `torchrun --nproc_per_node=9`, NCCL) | |
| * **Precision & Sync:** `bfloat16` autocast, TF32 matmul/cuDNN, custom BF16 gradient `AllReduce` with `1024x` loss-scale protection, chunked cross-entropy (`chunk_sz=2048`) with Z-loss (`1e-4`), dynamic length-packed microbatches (`32,768` tokens/GPU, `~295k` budgeted tokens/step), load-balanced step grouping. | |
| * **Throughput:** `~375k–400k tok/s` in Stage 1 (`98%–100%` utilization across all 9 GPUs), `~405k–429k tok/s` in Stage 2. | |
| ### Dataset Coverage | |
| * **Raw Dataset (`tokenized_65k`):** `48,329,808` unique parallel/monolingual pairs (`train_en.bin` 7.9 GiB + `train_my.bin` 7.0 GiB ≈ `~8.0B` raw tokens) + `22,329` held-out validation pairs. | |
| * **Tasks & Register Control:** Bidirectional `En -> My` (`<2my>`) and `My -> En` (`<2en>`), explicit `<formal>` / `<spoken>` style conditioning (`70%` dropout-free application on tagged subsets), `<bt>` marker for backtranslated `My -> En`, and span-corruption `<denoise>` (`Poisson(3.0)`, `<mask>`). | |
| ### Stage 1 (`stable` — 100% Coverage + Gold Replay) | |
| * **Plan:** `48,329,808` unique base rows (`100%` coverage) + `12,000,000` gold replay = `60,329,808` planned rows (`60,331,713` rows trained across `58,916` steps, ~10B+ tokens). | |
| * **Hyperparameters:** `AdamW` (`lr=5.0e-4` after 900-step warmup, `betas=(0.9, 0.95)`, `weight_decay=0.01`), `label_smoothing=0.1`, `MTP weight=0.3`, `grad_clip=1.0`. | |
| * **Stage 1 Validation Curve (`eval_val_nll` on 1,024 uniform non-BT held-out pairs):** | |
| * Step `0`: `Comb NLL 20.1282` | |
| * Step `1,500`: `Comb NLL 3.0945` (**PPL `22.08`**) | |
| * Step `8,000`: `Comb NLL 1.7760` (**PPL `5.91`**) | |
| * Step `19,000`: `Comb NLL 1.5803` (**PPL `4.86`**) | |
| * Step `39,000`: `Comb NLL 1.4648` (**PPL `4.33`**) | |
| * Step `58,000`: `Comb NLL 1.4298` (**PPL `4.18`**) | |
| * **Stage 1 Final (Step `58,916`):** `Comb NLL 1.4316` (**PPL `4.19`** | `En->My: 1.7539` | `My->En: 1.1548`) | |
| ### Stage 2 (`cooldown` — Clean Gold Only, Unsmoothed) | |
| * **Plan:** `2,706,748` unique non-BT base rows + `1,500,000` gold replay = `4,206,748` planned rows (`4,210,076` rows trained across `6,793` steps). | |
| * **Hyperparameters:** `100`-step warmup to `3.0e-4` → cosine decay to `1.0e-5`, `label_smoothing=0.0`, `MTP weight=0.15`, `50/50 En↔My`. | |
| * **Stage 2 Validation Curve:** | |
| * Step `250`: `Comb NLL 1.2945` (**PPL `3.65`**) | |
| * Step `1,500`: `Comb NLL 1.2403` (**PPL `3.46`**) | |
| * Step `3,500`: `Comb NLL 1.1941` (**PPL `3.30`**) | |
| * Step `5,500`: `Comb NLL 1.1687` (**PPL `3.22`**) | |
| * **Stage 2 Final (Step `6,793`):** **`Comb NLL 1.1640` (PPL `3.20`)** | `En->My: 1.4829` | `My->En: 0.8901` (`PPL 2.44`) | |
| *(Note: PPL values are measured on the `tokenized_65k/val` split and are not directly comparable 1:1 against models evaluated on older splits/vocabularies.)* | |
| --- | |
| ## 3. Beam=4 Benchmark Outputs (Unseen + Audit Prompts) | |
| ```text | |
| 🏆 MyTranslation-300M Final Beam=4 Benchmark: | |
| [En->My Formal]: ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးကို တည်ငြိမ်စေရန် ဗဟိုဘဏ်သည် အတိုးနှုန်းကို မြှင့်တင်ခဲ့သည်။ | |
| [En->My Spoken]: ရိုးရိုးသားသားပြောရရင် ဒီလိုဖြစ်လာမယ်လို့ ကျွန်တော် မမျှော်လင့်ထားပါဘူး၊ ဒါပေမယ့် စိတ်မပူပါနဲ့၊ အတူတူ အဖြေရှာကြည့်ပါမယ်။ | |
| [My->En Formal]: The central bank raised interest rates to control inflation and stabilize the economy. | |
| [My->En Spoken]: It is a little late to go home because it is raining heavily in Yangon today | |
| [My->En Idiom]: Love, I have been deeply in love with you and myself, I know God, I know the sugar, I know the spirit, and I'm convinced by the envious deniers, so you're going crazy. | |
| [En->My News]: မုတ်သုံမိုးသည်းထန်စွာ ရွာသွန်းမှုကြောင့် မြို့နယ်အချို့တွင် ရေကြီးရေလျှံမှုများ ဆိုးရွားစွာ ဖြစ်ပေါ်ခဲ့ပြီး ထောင်နှင့်ချီသော ဒေသခံများသည် ယာယီခိုလှုံရေးစခန်းများသို့ ရွှေ့ပြောင်းခဲ့ရသည်။ | |
| [En->My Health]: ရာသီတုပ်ကွေးကနေ မြန်မြန်သက်သာဖို့ သန့်ရှင်းတဲ့ရေ များများသောက်ပြီး လုံလုံလောက်လောက် အနားယူဖို့ ဆရာဝန်တွေက အကြံပြုထားပါတယ်။ | |
| [En->My Spoken2]: ကျွန်တော် ဒီမနက် တစ်နာရီ ကျော်ကျော်လောက် ကားကြပ်နေလို့ ကျွန်တော် မပါဘဲ အစည်းအဝေးကို စတင်ပါ ။ | |
| [My->En News]: Border trade checkpoints are being accelerated to improve international trade. | |
| [My->En Tech]: Artificial intelligence technology can help make processes more efficient in education, health, and business. | |
| [My->En Spoken2]: I have a need to go early tomorrow morning so I will go to bed early tonight so I will call me tomorrow | |
| ``` | |
| --- | |
| ## 4. Usage Example (PyTorch + Transformers `trust_remote_code=True`) | |
| ```python | |
| import torch, sentencepiece as spm | |
| from huggingface_hub import hf_hub_download | |
| from transformers import AutoModelForSeq2SeqLM | |
| repo_id = "freococo/MyTranslation_300M_Base" | |
| dtype = torch.float16 # Use torch.bfloat16 on Ampere/Hopper/Blackwell, torch.float16 on T4 | |
| model = AutoModelForSeq2SeqLM.from_pretrained(repo_id, trust_remote_code=True, dtype=dtype).cuda().eval() | |
| sp = spm.SentencePieceProcessor() | |
| sp.load(hf_hub_download(repo_id, "tokenizer.model")) | |
| print("En->My Formal:", model.translate( | |
| "The central bank raised interest rates to control inflation and stabilize the economy.", | |
| sp, direction="en2my", style="formal", num_beams=4 | |
| )) | |
| print("En->My Spoken:", model.translate( | |
| "Honestly, I didn't expect things to turn out like this, but don't worry—we'll figure it out together.", | |
| sp, direction="en2my", style="spoken", num_beams=4 | |
| )) | |
| print("My->En Formal:", model.translate( | |
| "ဗဟိုဘဏ်သည် ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးတည်ငြိမ်စေရန် အတိုးနှုန်းများကို မြှင့်တင်ခဲ့သည်။", | |
| sp, direction="my2en", style="formal", num_beams=4 | |
| )) | |
| ``` | |
| --- | |
| ## 5. Reproducibility Artifacts Included | |
| * `training_codes.txt` — exact multi-GPU PyTorch training script (`train_300m.py`) | |
| * `trainingcelloutputwitheval.txt` — complete Stage 1 + Stage 2 training logs & Beam=4 benchmark output | |
| * `checkpoint-stage2-step-6793/` — FP32 `model.safetensors` + `training_state.pt` (`best_model.pt`) |