Translation
Transformers
Safetensors
Burmese
English
mytranslation
text2text-generation
Translation
Safetensors
Burmese
English
myanmartranslator
MyanmarLLM
custom_code
myanmar
Instructions to use freococo/MyTranslation_300M_Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use freococo/MyTranslation_300M_Base with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("translation", model="freococo/MyTranslation_300M_Base", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForSeq2SeqLM model = AutoModelForSeq2SeqLM.from_pretrained("freococo/MyTranslation_300M_Base", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Release MyTranslation_300M_Base (298.76M inf / 9x RTX 5090, Val PPL 3.20 + custom_code, code & logs)
Browse files- README.md +135 -0
- checkpoint-stage2-step-6793/config.json +32 -0
- checkpoint-stage2-step-6793/configuration_mytranslation.py +42 -0
- checkpoint-stage2-step-6793/generation_config.json +10 -0
- checkpoint-stage2-step-6793/model.safetensors +3 -0
- checkpoint-stage2-step-6793/modeling_mytranslation.py +372 -0
- checkpoint-stage2-step-6793/special_tokens_map.json +15 -0
- checkpoint-stage2-step-6793/tokenizer.model +3 -0
- checkpoint-stage2-step-6793/tokenizer_config.json +25 -0
- checkpoint-stage2-step-6793/training_codes.txt +1214 -0
- checkpoint-stage2-step-6793/training_state.pt +3 -0
- checkpoint-stage2-step-6793/trainingcelloutputwitheval.txt +0 -0
- config.json +32 -0
- configuration_mytranslation.py +42 -0
- generation_config.json +10 -0
- model.safetensors +3 -0
- modeling_mytranslation.py +372 -0
- special_tokens_map.json +15 -0
- tokenizer.model +3 -0
- tokenizer_config.json +25 -0
- training_codes.txt +1214 -0
- trainingcelloutputwitheval.txt +0 -0
README.md
ADDED
|
@@ -0,0 +1,135 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- my
|
| 4 |
+
- en
|
| 5 |
+
license: apache-2.0
|
| 6 |
+
library_name: transformers
|
| 7 |
+
tags:
|
| 8 |
+
- Translation
|
| 9 |
+
- Safetensors
|
| 10 |
+
- Burmese
|
| 11 |
+
- English
|
| 12 |
+
- myanmartranslator
|
| 13 |
+
- MyanmarLLM
|
| 14 |
+
- custom_code
|
| 15 |
+
- myanmar
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
# MyTranslation_300M_Base
|
| 19 |
+
|
| 20 |
+
**MyTranslation_300M_Base** is a **`298.76M`-parameter** (inference; `312.94M` training parameters including the Multi-Token Prediction head) bidirectional **English ↔ Burmese (`my`)** Neural Machine Translation encoder-decoder model trained from scratch on **~10B+ training tokens** using **9× NVIDIA GeForce RTX 5090 GPUs**.
|
| 21 |
+
|
| 22 |
+
> **Status:** Base checkpoint (`Stage 1 Pretraining + Stage 2 Gold Cooldown`), published prior to optional spoken/idiom Stage 3 polishing.
|
| 23 |
+
|
| 24 |
+
---
|
| 25 |
+
|
| 26 |
+
## 1. Architecture Highlights
|
| 27 |
+
|
| 28 |
+
| Component | Specification |
|
| 29 |
+
|---|---|
|
| 30 |
+
| **Inference Parameters** | `298,763,201` (`298.76M`, BF16 `model.safetensors` ~598 MB) |
|
| 31 |
+
| **Training Parameters** | `312,940,000` (`312.94M`, includes `MTPHeadBlock` dropped at export) |
|
| 32 |
+
| **Encoder** | `12` bidirectional layers, `d_model=1024`, `n_heads=16`, `n_kv_heads=4` (GQA, `head_dim=64`), `ffn_dim=3072` SwiGLU |
|
| 33 |
+
| **Decoder** | `6` causal layers, `d_model=1024`, `n_heads=16`, `n_kv_heads=4` (GQA), `ffn_dim=3072` SwiGLU |
|
| 34 |
+
| **Positional Encoding** | RoPE (`theta=50000.0`, `max_seq_len=2048` / trained at `1024`) |
|
| 35 |
+
| **Attention & Normalization** | `RMSNorm`, QK-Norm, Gated Attention (`sigmoid(gate_proj)`), Value Residual (`v0` mixing) |
|
| 36 |
+
| **Encoder-Decoder Routing** | Dense Layer-Wise Attention (`enc_dwa_logits`) + `3` learned `SharedCrossKVProj` mixtures (`enc_router_logits`) shared by decoder pairs `(0,1)`, `(2,3)`, `(4,5)` |
|
| 37 |
+
| **Tokenizer & Vocab** | `65,536`-piece SentencePiece (`UNK=0`, `BOS=1`, `EOS=2`, `PAD=3`, `<2my>=4`, `<2en>=5`, `<formal>=6`, `<spoken>=7`, `<bt>=8`, `<denoise>=9`, `<mask>=10`), tied embeddings (`embed_scale=32.0`) |
|
| 38 |
+
|
| 39 |
+
### Learned Cross-KV Encoder Mixture Weights (Post-Stage 2)
|
| 40 |
+
* **Cross-KV Pair 0 (Decoder 0–1):** Top encoder layers `[7, 8, 6, 2]` (`[0.226, 0.200, 0.187, 0.102]`)
|
| 41 |
+
* **Cross-KV Pair 1 (Decoder 2–3):** Top encoder layers `[9, 10, 8, 11]` (`[0.339, 0.206, 0.187, 0.124]`)
|
| 42 |
+
* **Cross-KV Pair 2 (Decoder 4–5):** Top encoder layers `[11, 9, 10, 8]` (`[0.340, 0.335, 0.278, 0.041]`)
|
| 43 |
+
|
| 44 |
+
---
|
| 45 |
+
|
| 46 |
+
## 2. Training Hardware, Data & Schedule
|
| 47 |
+
|
| 48 |
+
### Cluster & Precision Setup
|
| 49 |
+
* **GPUs:** **9× NVIDIA GeForce RTX 5090** (`32,607 MiB` VRAM each, `torchrun --nproc_per_node=9`, NCCL)
|
| 50 |
+
* **Precision & Sync:** `bfloat16` autocast, TF32 matmul/cuDNN, custom BF16 gradient `AllReduce` with `1024x` loss-scale protection, chunked cross-entropy (`chunk_sz=2048`) with Z-loss (`1e-4`), dynamic length-packed microbatches (`32,768` tokens/GPU, `~295k` budgeted tokens/step), load-balanced step grouping.
|
| 51 |
+
* **Throughput:** `~375k–400k tok/s` in Stage 1 (`98%–100%` utilization across all 9 GPUs), `~405k–429k tok/s` in Stage 2.
|
| 52 |
+
|
| 53 |
+
### Dataset Coverage
|
| 54 |
+
* **Raw Dataset (`tokenized_65k`):** `48,329,808` unique parallel/monolingual pairs (`train_en.bin` 7.9 GiB + `train_my.bin` 7.0 GiB ≈ `~8.0B` raw tokens) + `22,329` held-out validation pairs.
|
| 55 |
+
* **Tasks & Register Control:** Bidirectional `En -> My` (`<2my>`) and `My -> En` (`<2en>`), explicit `<formal>` / `<spoken>` style conditioning (`70%` dropout-free application on tagged subsets), `<bt>` marker for backtranslated `My -> En`, and span-corruption `<denoise>` (`Poisson(3.0)`, `<mask>`).
|
| 56 |
+
|
| 57 |
+
### Stage 1 (`stable` — 100% Coverage + Gold Replay)
|
| 58 |
+
* **Plan:** `48,329,808` unique base rows (`100%` coverage) + `12,000,000` gold replay = `60,329,808` planned rows (`60,331,713` rows trained across `58,916` steps, ~10B+ tokens).
|
| 59 |
+
* **Hyperparameters:** `AdamW` (`lr=5.0e-4` after 900-step warmup, `betas=(0.9, 0.95)`, `weight_decay=0.01`), `label_smoothing=0.1`, `MTP weight=0.3`, `grad_clip=1.0`.
|
| 60 |
+
* **Stage 1 Validation Curve (`eval_val_nll` on 1,024 uniform non-BT held-out pairs):**
|
| 61 |
+
* Step `0`: `Comb NLL 20.1282`
|
| 62 |
+
* Step `1,500`: `Comb NLL 3.0945` (**PPL `22.08`**)
|
| 63 |
+
* Step `8,000`: `Comb NLL 1.7760` (**PPL `5.91`**)
|
| 64 |
+
* Step `19,000`: `Comb NLL 1.5803` (**PPL `4.86`**)
|
| 65 |
+
* Step `39,000`: `Comb NLL 1.4648` (**PPL `4.33`**)
|
| 66 |
+
* Step `58,000`: `Comb NLL 1.4298` (**PPL `4.18`**)
|
| 67 |
+
* **Stage 1 Final (Step `58,916`):** `Comb NLL 1.4316` (**PPL `4.19`** | `En->My: 1.7539` | `My->En: 1.1548`)
|
| 68 |
+
|
| 69 |
+
### Stage 2 (`cooldown` — Clean Gold Only, Unsmoothed)
|
| 70 |
+
* **Plan:** `2,706,748` unique non-BT base rows + `1,500,000` gold replay = `4,206,748` planned rows (`4,210,076` rows trained across `6,793` steps).
|
| 71 |
+
* **Hyperparameters:** `100`-step warmup to `3.0e-4` → cosine decay to `1.0e-5`, `label_smoothing=0.0`, `MTP weight=0.15`, `50/50 En↔My`.
|
| 72 |
+
* **Stage 2 Validation Curve:**
|
| 73 |
+
* Step `250`: `Comb NLL 1.2945` (**PPL `3.65`**)
|
| 74 |
+
* Step `1,500`: `Comb NLL 1.2403` (**PPL `3.46`**)
|
| 75 |
+
* Step `3,500`: `Comb NLL 1.1941` (**PPL `3.30`**)
|
| 76 |
+
* Step `5,500`: `Comb NLL 1.1687` (**PPL `3.22`**)
|
| 77 |
+
* **Stage 2 Final (Step `6,793`):** **`Comb NLL 1.1640` (PPL `3.20`)** | `En->My: 1.4829` | `My->En: 0.8901` (`PPL 2.44`)
|
| 78 |
+
|
| 79 |
+
*(Note: PPL values are measured on the `tokenized_65k/val` split and are not directly comparable 1:1 against models evaluated on older splits/vocabularies.)*
|
| 80 |
+
|
| 81 |
+
---
|
| 82 |
+
|
| 83 |
+
## 3. Beam=4 Benchmark Outputs (Unseen + Audit Prompts)
|
| 84 |
+
|
| 85 |
+
```text
|
| 86 |
+
🏆 MyTranslation-300M Final Beam=4 Benchmark:
|
| 87 |
+
[En->My Formal]: ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးကို တည်ငြိမ်စေရန် ဗဟိုဘဏ်သည် အတိုးနှုန်းကို မြှင့်တင်ခဲ့သည်။
|
| 88 |
+
[En->My Spoken]: ရိုးရိုးသားသားပြောရရင် ဒီလိုဖြစ်လာမယ်လို့ ကျွန်တော် မမျှော်လင့်ထားပါဘူး၊ ဒါပေမယ့် စိတ်မပူပါနဲ့၊ အတူတူ အဖြေရှာကြည့်ပါမယ်။
|
| 89 |
+
[My->En Formal]: The central bank raised interest rates to control inflation and stabilize the economy.
|
| 90 |
+
[My->En Spoken]: It is a little late to go home because it is raining heavily in Yangon today
|
| 91 |
+
[My->En Idiom]: Love, I have been deeply in love with you and myself, I know God, I know the sugar, I know the spirit, and I'm convinced by the envious deniers, so you're going crazy.
|
| 92 |
+
[En->My News]: မုတ်သုံမိုးသည်းထန်စွာ ရွာသွန်းမှုကြောင့် မြို့နယ်အချို့တွင် ရေကြီးရေလျှံမှုများ ဆိုးရွားစွာ ဖြစ်ပေါ်ခဲ့ပြီး ထောင်နှင့်ချီသော ဒေသခံများသည် ယာယီခိုလှုံရေးစခန်းများသို့ ရွှေ့ပြောင်းခဲ့ရသည်။
|
| 93 |
+
[En->My Health]: ရာသီတုပ်ကွေးကနေ မြန်မြန်သက်သာဖို့ သန့်ရှင်းတဲ့ရေ များများသောက်ပြီး လုံလုံလောက်လောက် အနားယူဖို့ ဆရာဝန်တွေက အကြံပြုထားပါတယ်။
|
| 94 |
+
[En->My Spoken2]: ကျွန်တော် ဒီမနက် တစ်နာရီ ကျော်ကျော်လောက် ကားကြပ်နေလို့ ကျွန်တော် မပါဘဲ အစည်းအဝေးကို စတင်ပါ ။
|
| 95 |
+
[My->En News]: Border trade checkpoints are being accelerated to improve international trade.
|
| 96 |
+
[My->En Tech]: Artificial intelligence technology can help make processes more efficient in education, health, and business.
|
| 97 |
+
[My->En Spoken2]: I have a need to go early tomorrow morning so I will go to bed early tonight so I will call me tomorrow
|
| 98 |
+
```
|
| 99 |
+
|
| 100 |
+
---
|
| 101 |
+
|
| 102 |
+
## 4. Usage Example (PyTorch + Transformers `trust_remote_code=True`)
|
| 103 |
+
|
| 104 |
+
```python
|
| 105 |
+
import torch, sentencepiece as spm
|
| 106 |
+
from huggingface_hub import hf_hub_download
|
| 107 |
+
from transformers import AutoModelForSeq2SeqLM
|
| 108 |
+
|
| 109 |
+
repo_id = "freococo/MyTranslation_300M_Base"
|
| 110 |
+
dtype = torch.float16 # Use torch.bfloat16 on Ampere/Hopper/Blackwell, torch.float16 on T4
|
| 111 |
+
|
| 112 |
+
model = AutoModelForSeq2SeqLM.from_pretrained(repo_id, trust_remote_code=True, dtype=dtype).cuda().eval()
|
| 113 |
+
sp = spm.SentencePieceProcessor()
|
| 114 |
+
sp.load(hf_hub_download(repo_id, "tokenizer.model"))
|
| 115 |
+
|
| 116 |
+
print("En->My Formal:", model.translate(
|
| 117 |
+
"The central bank raised interest rates to control inflation and stabilize the economy.",
|
| 118 |
+
sp, direction="en2my", style="formal", num_beams=4
|
| 119 |
+
))
|
| 120 |
+
print("En->My Spoken:", model.translate(
|
| 121 |
+
"Honestly, I didn't expect things to turn out like this, but don't worry—we'll figure it out together.",
|
| 122 |
+
sp, direction="en2my", style="spoken", num_beams=4
|
| 123 |
+
))
|
| 124 |
+
print("My->En Formal:", model.translate(
|
| 125 |
+
"ဗဟိုဘဏ်သည် ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးတည်ငြိမ်စေရန် အတိုးနှုန်းများကို မြှင့်တင်ခဲ့သည်။",
|
| 126 |
+
sp, direction="my2en", style="formal", num_beams=4
|
| 127 |
+
))
|
| 128 |
+
```
|
| 129 |
+
|
| 130 |
+
---
|
| 131 |
+
|
| 132 |
+
## 5. Reproducibility Artifacts Included
|
| 133 |
+
* `training_codes.txt` — exact multi-GPU PyTorch training script (`train_300m.py`)
|
| 134 |
+
* `trainingcelloutputwitheval.txt` — complete Stage 1 + Stage 2 training logs & Beam=4 benchmark output
|
| 135 |
+
* `checkpoint-stage2-step-6793/` — FP32 `model.safetensors` + `training_state.pt` (`best_model.pt`)
|
checkpoint-stage2-step-6793/config.json
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"MyTranslationForConditionalGeneration"
|
| 4 |
+
],
|
| 5 |
+
"auto_map": {
|
| 6 |
+
"AutoConfig": "configuration_mytranslation.MyTranslationConfig",
|
| 7 |
+
"AutoModelForSeq2SeqLM": "modeling_mytranslation.MyTranslationForConditionalGeneration"
|
| 8 |
+
},
|
| 9 |
+
"model_type": "mytranslation",
|
| 10 |
+
"is_encoder_decoder": true,
|
| 11 |
+
"tie_word_embeddings": true,
|
| 12 |
+
"vocab_size": 65536,
|
| 13 |
+
"d_model": 1024,
|
| 14 |
+
"n_heads": 16,
|
| 15 |
+
"n_kv_heads": 4,
|
| 16 |
+
"enc_layers": 12,
|
| 17 |
+
"dec_layers": 6,
|
| 18 |
+
"ffn_dim": 3072,
|
| 19 |
+
"max_seq_len": 2048,
|
| 20 |
+
"rope_theta": 50000.0,
|
| 21 |
+
"unk_token_id": 0,
|
| 22 |
+
"bos_token_id": 1,
|
| 23 |
+
"eos_token_id": 2,
|
| 24 |
+
"pad_token_id": 3,
|
| 25 |
+
"stage1_final_val_nll": 1.4316,
|
| 26 |
+
"stage1_final_val_ppl": 4.19,
|
| 27 |
+
"val_nll": 1.164,
|
| 28 |
+
"val_ppl": 3.2,
|
| 29 |
+
"en2my_val_nll": 1.4829,
|
| 30 |
+
"my2en_val_nll": 0.8901,
|
| 31 |
+
"torch_dtype": "float32"
|
| 32 |
+
}
|
checkpoint-stage2-step-6793/configuration_mytranslation.py
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from transformers import PretrainedConfig
|
| 2 |
+
|
| 3 |
+
class MyTranslationConfig(PretrainedConfig):
|
| 4 |
+
model_type = "mytranslation"
|
| 5 |
+
|
| 6 |
+
def __init__(
|
| 7 |
+
self,
|
| 8 |
+
vocab_size=65536,
|
| 9 |
+
d_model=1024,
|
| 10 |
+
n_heads=16,
|
| 11 |
+
n_kv_heads=4,
|
| 12 |
+
enc_layers=12,
|
| 13 |
+
dec_layers=6,
|
| 14 |
+
ffn_dim=3072,
|
| 15 |
+
max_seq_len=2048,
|
| 16 |
+
rope_theta=50000.0,
|
| 17 |
+
unk_token_id=0,
|
| 18 |
+
bos_token_id=1,
|
| 19 |
+
eos_token_id=2,
|
| 20 |
+
pad_token_id=3,
|
| 21 |
+
tie_word_embeddings=True,
|
| 22 |
+
is_encoder_decoder=True,
|
| 23 |
+
**kwargs,
|
| 24 |
+
):
|
| 25 |
+
self.vocab_size = vocab_size
|
| 26 |
+
self.d_model = d_model
|
| 27 |
+
self.n_heads = n_heads
|
| 28 |
+
self.n_kv_heads = n_kv_heads
|
| 29 |
+
self.enc_layers = enc_layers
|
| 30 |
+
self.dec_layers = dec_layers
|
| 31 |
+
self.ffn_dim = ffn_dim
|
| 32 |
+
self.max_seq_len = max_seq_len
|
| 33 |
+
self.rope_theta = rope_theta
|
| 34 |
+
super().__init__(
|
| 35 |
+
unk_token_id=unk_token_id,
|
| 36 |
+
bos_token_id=bos_token_id,
|
| 37 |
+
eos_token_id=eos_token_id,
|
| 38 |
+
pad_token_id=pad_token_id,
|
| 39 |
+
tie_word_embeddings=tie_word_embeddings,
|
| 40 |
+
is_encoder_decoder=is_encoder_decoder,
|
| 41 |
+
**kwargs,
|
| 42 |
+
)
|
checkpoint-stage2-step-6793/generation_config.json
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token_id": 1,
|
| 3 |
+
"eos_token_id": 2,
|
| 4 |
+
"pad_token_id": 3,
|
| 5 |
+
"decoder_start_token_id": 1,
|
| 6 |
+
"max_new_tokens": 192,
|
| 7 |
+
"num_beams": 4,
|
| 8 |
+
"length_penalty": 0.85,
|
| 9 |
+
"repetition_penalty": 1.12
|
| 10 |
+
}
|
checkpoint-stage2-step-6793/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:90bd4db2e504e2c0ea42f3b5fb2c9fbfa6e657e84fa21fbe560faa947644c20e
|
| 3 |
+
size 1195078172
|
checkpoint-stage2-step-6793/modeling_mytranslation.py
ADDED
|
@@ -0,0 +1,372 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import math
|
| 2 |
+
import torch
|
| 3 |
+
import torch.nn as nn
|
| 4 |
+
import torch.nn.functional as F
|
| 5 |
+
import transformers
|
| 6 |
+
from transformers import PreTrainedModel
|
| 7 |
+
from transformers.modeling_outputs import Seq2SeqLMOutput
|
| 8 |
+
from .configuration_mytranslation import MyTranslationConfig
|
| 9 |
+
|
| 10 |
+
PAD_ID = 3
|
| 11 |
+
BOS_ID = 1
|
| 12 |
+
EOS_ID = 2
|
| 13 |
+
TOK_2MY, TOK_2EN, TOK_FORMAL, TOK_SPOKEN = 4, 5, 6, 7
|
| 14 |
+
_IS_TF5 = int(transformers.__version__.split(".")[0]) >= 5
|
| 15 |
+
|
| 16 |
+
def _sdpa(q, k, v, attn_mask=None, is_causal=False):
|
| 17 |
+
if q.size(1) != k.size(1):
|
| 18 |
+
try:
|
| 19 |
+
return F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask, is_causal=is_causal, enable_gqa=True)
|
| 20 |
+
except TypeError:
|
| 21 |
+
rep = q.size(1) // k.size(1)
|
| 22 |
+
k = k.repeat_interleave(rep, dim=1)
|
| 23 |
+
v = v.repeat_interleave(rep, dim=1)
|
| 24 |
+
return F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask, is_causal=is_causal)
|
| 25 |
+
|
| 26 |
+
class RMSNorm(nn.Module):
|
| 27 |
+
def __init__(self, dim: int, eps: float = 1e-6):
|
| 28 |
+
super().__init__()
|
| 29 |
+
self.dim = (dim,)
|
| 30 |
+
self.eps = eps
|
| 31 |
+
self.weight = nn.Parameter(torch.ones(dim))
|
| 32 |
+
|
| 33 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 34 |
+
if hasattr(F, "rms_norm"):
|
| 35 |
+
return F.rms_norm(x, self.dim, self.weight, self.eps)
|
| 36 |
+
xf = x.float()
|
| 37 |
+
y = xf * torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + self.eps)
|
| 38 |
+
return (y * self.weight.float()).to(dtype=x.dtype)
|
| 39 |
+
|
| 40 |
+
class RotaryEmbedding(nn.Module):
|
| 41 |
+
def __init__(self, dim: int, max_seq_len: int = 2048, theta: float = 50000.0):
|
| 42 |
+
super().__init__()
|
| 43 |
+
inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))
|
| 44 |
+
t = torch.arange(max_seq_len, dtype=torch.float32)
|
| 45 |
+
freqs = torch.outer(t, inv_freq)
|
| 46 |
+
self.register_buffer("cos_cached", freqs.cos()[None, None, :, :], persistent=False)
|
| 47 |
+
self.register_buffer("sin_cached", freqs.sin()[None, None, :, :], persistent=False)
|
| 48 |
+
|
| 49 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 50 |
+
L = x.size(2)
|
| 51 |
+
cos = self.cos_cached[:, :, :L, :].to(dtype=x.dtype)
|
| 52 |
+
sin = self.sin_cached[:, :, :L, :].to(dtype=x.dtype)
|
| 53 |
+
x1, x2 = x[..., ::2], x[..., 1::2]
|
| 54 |
+
return torch.cat((x1 * cos - x2 * sin, x1 * sin + x2 * cos), dim=-1)
|
| 55 |
+
|
| 56 |
+
class SwiGLU(nn.Module):
|
| 57 |
+
def __init__(self, d_model: int, ffn_dim: int):
|
| 58 |
+
super().__init__()
|
| 59 |
+
self.gate_up = nn.Linear(d_model, 2 * ffn_dim, bias=False)
|
| 60 |
+
self.down = nn.Linear(ffn_dim, d_model, bias=False)
|
| 61 |
+
|
| 62 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 63 |
+
gate, up = self.gate_up(x).chunk(2, dim=-1)
|
| 64 |
+
return self.down(F.silu(gate) * up)
|
| 65 |
+
|
| 66 |
+
class GatedGQASelfAttention(nn.Module):
|
| 67 |
+
def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, is_causal: bool = False, has_v_res: bool = False):
|
| 68 |
+
super().__init__()
|
| 69 |
+
self.n_heads = n_heads
|
| 70 |
+
self.n_kv_heads = n_kv_heads
|
| 71 |
+
self.head_dim = d_model // n_heads
|
| 72 |
+
self.is_causal = is_causal
|
| 73 |
+
self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
|
| 74 |
+
self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * self.head_dim, bias=False)
|
| 75 |
+
self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
|
| 76 |
+
self.out = nn.Linear(d_model, d_model, bias=False)
|
| 77 |
+
self.q_norm = RMSNorm(self.head_dim)
|
| 78 |
+
self.k_norm = RMSNorm(self.head_dim)
|
| 79 |
+
self.v_res_logit = nn.Parameter(torch.tensor(-2.0)) if has_v_res else None
|
| 80 |
+
|
| 81 |
+
def forward(self, x: torch.Tensor, rope: RotaryEmbedding, attn_mask: torch.Tensor = None, v0: torch.Tensor = None):
|
| 82 |
+
B, L, D = x.shape
|
| 83 |
+
q = self.q_norm(self.q_proj(x).view(B, L, self.n_heads, self.head_dim).transpose(1, 2))
|
| 84 |
+
k, v = self.kv_proj(x).chunk(2, dim=-1)
|
| 85 |
+
k = self.k_norm(k.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2))
|
| 86 |
+
v = v.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
| 87 |
+
v_raw = v
|
| 88 |
+
if self.v_res_logit is not None and v0 is not None:
|
| 89 |
+
alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
|
| 90 |
+
v = (1.0 - alpha) * v + alpha * v0
|
| 91 |
+
q, k = rope(q), rope(k)
|
| 92 |
+
y = _sdpa(q, k, v, attn_mask=None if self.is_causal else attn_mask, is_causal=self.is_causal)
|
| 93 |
+
gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
|
| 94 |
+
y = (y * gate).transpose(1, 2).contiguous().view(B, L, D)
|
| 95 |
+
return self.out(y), v_raw
|
| 96 |
+
|
| 97 |
+
def step_kv(self, x_step, cos_p, sin_p, pos, k_cache, v_cache, v0=None):
|
| 98 |
+
B, _, D = x_step.shape
|
| 99 |
+
q = self.q_norm(self.q_proj(x_step).view(B, 1, self.n_heads, self.head_dim).transpose(1, 2))
|
| 100 |
+
k, v = self.kv_proj(x_step).chunk(2, dim=-1)
|
| 101 |
+
k = self.k_norm(k.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2))
|
| 102 |
+
v = v.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
| 103 |
+
v_raw = v
|
| 104 |
+
if self.v_res_logit is not None and v0 is not None:
|
| 105 |
+
alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
|
| 106 |
+
v = (1.0 - alpha) * v + alpha * v0
|
| 107 |
+
q1, q2 = q[..., ::2], q[..., 1::2]
|
| 108 |
+
k1, k2 = k[..., ::2], k[..., 1::2]
|
| 109 |
+
q = torch.cat((q1 * cos_p - q2 * sin_p, q1 * sin_p + q2 * cos_p), dim=-1)
|
| 110 |
+
k = torch.cat((k1 * cos_p - k2 * sin_p, k1 * sin_p + k2 * cos_p), dim=-1)
|
| 111 |
+
k_cache[:, :, pos : pos + 1, :] = k
|
| 112 |
+
v_cache[:, :, pos : pos + 1, :] = v
|
| 113 |
+
y = _sdpa(q, k_cache[:, :, : pos + 1, :], v_cache[:, :, : pos + 1, :], is_causal=False)
|
| 114 |
+
gate = torch.sigmoid(self.gate_proj(x_step)).transpose(1, 2).unsqueeze(-1)
|
| 115 |
+
y = (y * gate).transpose(1, 2).contiguous().view(B, 1, D)
|
| 116 |
+
return self.out(y), v_raw
|
| 117 |
+
|
| 118 |
+
class SharedCrossKVProj(nn.Module):
|
| 119 |
+
def __init__(self, d_model: int, n_kv_heads: int, head_dim: int):
|
| 120 |
+
super().__init__()
|
| 121 |
+
self.n_kv_heads = n_kv_heads
|
| 122 |
+
self.head_dim = head_dim
|
| 123 |
+
self.norm = RMSNorm(d_model)
|
| 124 |
+
self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * head_dim, bias=False)
|
| 125 |
+
self.k_norm = RMSNorm(head_dim)
|
| 126 |
+
|
| 127 |
+
def forward(self, enc_mix: torch.Tensor):
|
| 128 |
+
B, S, _ = enc_mix.shape
|
| 129 |
+
k, v = self.kv_proj(self.norm(enc_mix)).chunk(2, dim=-1)
|
| 130 |
+
k = self.k_norm(k.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2))
|
| 131 |
+
v = v.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
| 132 |
+
return k, v
|
| 133 |
+
|
| 134 |
+
class GatedGQACrossAttention(nn.Module):
|
| 135 |
+
def __init__(self, d_model: int, n_heads: int):
|
| 136 |
+
super().__init__()
|
| 137 |
+
self.n_heads = n_heads
|
| 138 |
+
self.head_dim = d_model // n_heads
|
| 139 |
+
self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
|
| 140 |
+
self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
|
| 141 |
+
self.out = nn.Linear(d_model, d_model, bias=False)
|
| 142 |
+
self.q_norm = RMSNorm(self.head_dim)
|
| 143 |
+
|
| 144 |
+
def forward(self, x: torch.Tensor, k_cross: torch.Tensor, v_cross: torch.Tensor, enc_mask: torch.Tensor):
|
| 145 |
+
B, T, D = x.shape
|
| 146 |
+
q = self.q_norm(self.q_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2))
|
| 147 |
+
y = _sdpa(q, k_cross, v_cross, attn_mask=enc_mask, is_causal=False)
|
| 148 |
+
gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
|
| 149 |
+
y = (y * gate).transpose(1, 2).contiguous().view(B, T, D)
|
| 150 |
+
return self.out(y)
|
| 151 |
+
|
| 152 |
+
class EncoderLayer(nn.Module):
|
| 153 |
+
def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
|
| 154 |
+
super().__init__()
|
| 155 |
+
self.norm1 = RMSNorm(d_model)
|
| 156 |
+
self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=False, has_v_res=has_v_res)
|
| 157 |
+
self.norm2 = RMSNorm(d_model)
|
| 158 |
+
self.mlp = SwiGLU(d_model, ffn_dim)
|
| 159 |
+
|
| 160 |
+
def forward(self, x, rope, enc_mask, v0=None):
|
| 161 |
+
attn_out, v_raw = self.self_attn(self.norm1(x), rope, enc_mask, v0=v0)
|
| 162 |
+
x = x + attn_out
|
| 163 |
+
x = x + self.mlp(self.norm2(x))
|
| 164 |
+
return x, v_raw
|
| 165 |
+
|
| 166 |
+
class DecoderLayer(nn.Module):
|
| 167 |
+
def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
|
| 168 |
+
super().__init__()
|
| 169 |
+
self.norm1 = RMSNorm(d_model)
|
| 170 |
+
self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=True, has_v_res=has_v_res)
|
| 171 |
+
self.norm2 = RMSNorm(d_model)
|
| 172 |
+
self.cross_attn = GatedGQACrossAttention(d_model, n_heads)
|
| 173 |
+
self.norm3 = RMSNorm(d_model)
|
| 174 |
+
self.mlp = SwiGLU(d_model, ffn_dim)
|
| 175 |
+
|
| 176 |
+
def forward(self, x, k_cross, v_cross, rope, enc_mask, v0=None):
|
| 177 |
+
attn_out, v_raw = self.self_attn(self.norm1(x), rope, None, v0=v0)
|
| 178 |
+
x = x + attn_out
|
| 179 |
+
x = x + self.cross_attn(self.norm2(x), k_cross, v_cross, enc_mask)
|
| 180 |
+
x = x + self.mlp(self.norm3(x))
|
| 181 |
+
return x, v_raw
|
| 182 |
+
|
| 183 |
+
def step_kv(self, x_step, cos_p, sin_p, pos, k_self, v_self, k_cross, v_cross, enc_mask, v0=None):
|
| 184 |
+
attn_out, v_raw = self.self_attn.step_kv(self.norm1(x_step), cos_p, sin_p, pos, k_self, v_self, v0=v0)
|
| 185 |
+
x_step = x_step + attn_out
|
| 186 |
+
x_step = x_step + self.cross_attn(self.norm2(x_step), k_cross, v_cross, enc_mask)
|
| 187 |
+
x_step = x_step + self.mlp(self.norm3(x_step))
|
| 188 |
+
return x_step, v_raw
|
| 189 |
+
|
| 190 |
+
class MyTranslationForConditionalGeneration(PreTrainedModel):
|
| 191 |
+
config_class = MyTranslationConfig
|
| 192 |
+
base_model_prefix = ""
|
| 193 |
+
_tied_weights_keys = {"lm_head.weight": "embed_tokens.weight"} if hasattr(PreTrainedModel, "get_expanded_tied_weights_keys") else ["lm_head.weight"]
|
| 194 |
+
|
| 195 |
+
def __init__(self, config: MyTranslationConfig):
|
| 196 |
+
super().__init__(config)
|
| 197 |
+
self.d_model = config.d_model
|
| 198 |
+
self.n_heads = config.n_heads
|
| 199 |
+
self.n_kv_heads = config.n_kv_heads
|
| 200 |
+
self.head_dim = config.d_model // config.n_heads
|
| 201 |
+
self.enc_layers = config.enc_layers
|
| 202 |
+
self.dec_layers = config.dec_layers
|
| 203 |
+
self.n_cross_pairs = config.dec_layers // 2
|
| 204 |
+
self.embed_scale = math.sqrt(config.d_model)
|
| 205 |
+
|
| 206 |
+
self.embed_tokens = nn.Embedding(config.vocab_size, config.d_model, padding_idx=config.pad_token_id)
|
| 207 |
+
self.rope = RotaryEmbedding(self.head_dim, max_seq_len=config.max_seq_len, theta=config.rope_theta)
|
| 208 |
+
self.encoder = nn.ModuleList([
|
| 209 |
+
EncoderLayer(config.d_model, config.n_heads, config.n_kv_heads, config.ffn_dim, has_v_res=(i > 0))
|
| 210 |
+
for i in range(config.enc_layers)
|
| 211 |
+
])
|
| 212 |
+
self.enc_dwa_logits = nn.ParameterList([
|
| 213 |
+
nn.Parameter(torch.tensor([-4.0] * i + [0.0], dtype=torch.float32))
|
| 214 |
+
for i in range(1, config.enc_layers)
|
| 215 |
+
])
|
| 216 |
+
self.enc_router_logits = nn.Parameter(torch.zeros(self.n_cross_pairs, config.enc_layers))
|
| 217 |
+
self.cross_kv_projs = nn.ModuleList([
|
| 218 |
+
SharedCrossKVProj(config.d_model, config.n_kv_heads, self.head_dim)
|
| 219 |
+
for _ in range(self.n_cross_pairs)
|
| 220 |
+
])
|
| 221 |
+
self.decoder = nn.ModuleList([
|
| 222 |
+
DecoderLayer(config.d_model, config.n_heads, config.n_kv_heads, config.ffn_dim, has_v_res=(i > 0))
|
| 223 |
+
for i in range(config.dec_layers)
|
| 224 |
+
])
|
| 225 |
+
self.dec_norm = RMSNorm(config.d_model)
|
| 226 |
+
self.lm_head = nn.Linear(config.d_model, config.vocab_size, bias=False)
|
| 227 |
+
self.post_init()
|
| 228 |
+
self.tie_weights()
|
| 229 |
+
|
| 230 |
+
def get_input_embeddings(self):
|
| 231 |
+
return self.embed_tokens
|
| 232 |
+
|
| 233 |
+
def set_input_embeddings(self, value):
|
| 234 |
+
self.embed_tokens = value
|
| 235 |
+
|
| 236 |
+
def get_output_embeddings(self):
|
| 237 |
+
return self.lm_head
|
| 238 |
+
|
| 239 |
+
def set_output_embeddings(self, new_embeddings):
|
| 240 |
+
self.lm_head = new_embeddings
|
| 241 |
+
|
| 242 |
+
def tie_weights(self, **kwargs):
|
| 243 |
+
self.lm_head.weight = self.embed_tokens.weight
|
| 244 |
+
|
| 245 |
+
def encode_and_route(self, src_ids: torch.Tensor):
|
| 246 |
+
enc_mask = (src_ids != self.config.pad_token_id).unsqueeze(1).unsqueeze(2)
|
| 247 |
+
x = self.embed_tokens(src_ids) * self.embed_scale
|
| 248 |
+
raw_states = [x]
|
| 249 |
+
enc_outputs = []
|
| 250 |
+
v0 = None
|
| 251 |
+
for i, layer in enumerate(self.encoder):
|
| 252 |
+
if i > 0:
|
| 253 |
+
w_dwa = F.softmax(self.enc_dwa_logits[i - 1], dim=0).to(dtype=x.dtype)
|
| 254 |
+
x_in = torch.einsum("j,jbsd->bsd", w_dwa, torch.stack(raw_states, dim=0))
|
| 255 |
+
else:
|
| 256 |
+
x_in = x
|
| 257 |
+
x_out, v_raw = layer(x_in, self.rope, enc_mask, v0=v0)
|
| 258 |
+
if i == 0:
|
| 259 |
+
v0 = v_raw
|
| 260 |
+
raw_states.append(x_out)
|
| 261 |
+
enc_outputs.append(x_out)
|
| 262 |
+
stacked_enc = torch.stack(enc_outputs, dim=0)
|
| 263 |
+
router_w = F.softmax(self.enc_router_logits, dim=-1).to(dtype=stacked_enc.dtype)
|
| 264 |
+
cross_kv = [
|
| 265 |
+
self.cross_kv_projs[p](torch.einsum("k,kbsd->bsd", router_w[p], stacked_enc))
|
| 266 |
+
for p in range(self.n_cross_pairs)
|
| 267 |
+
]
|
| 268 |
+
return cross_kv, enc_mask
|
| 269 |
+
|
| 270 |
+
def decode_seq(self, tgt_in: torch.Tensor, cross_kv: list, enc_mask: torch.Tensor):
|
| 271 |
+
x = self.embed_tokens(tgt_in) * self.embed_scale
|
| 272 |
+
v0 = None
|
| 273 |
+
for i, layer in enumerate(self.decoder):
|
| 274 |
+
kc, vc = cross_kv[i // 2]
|
| 275 |
+
x, v_raw = layer(x, kc, vc, self.rope, enc_mask, v0=v0)
|
| 276 |
+
if i == 0:
|
| 277 |
+
v0 = v_raw
|
| 278 |
+
return self.dec_norm(x)
|
| 279 |
+
|
| 280 |
+
def forward(self, input_ids=None, decoder_input_ids=None, labels=None, **kwargs):
|
| 281 |
+
if decoder_input_ids is None:
|
| 282 |
+
decoder_input_ids = torch.full((input_ids.size(0), 1), BOS_ID, dtype=torch.long, device=input_ids.device)
|
| 283 |
+
cross_kv, enc_mask = self.encode_and_route(input_ids)
|
| 284 |
+
h_dec = self.decode_seq(decoder_input_ids, cross_kv, enc_mask)
|
| 285 |
+
logits = self.lm_head(h_dec)
|
| 286 |
+
loss = None
|
| 287 |
+
if labels is not None:
|
| 288 |
+
loss = F.cross_entropy(logits.view(-1, logits.size(-1)), labels.view(-1), ignore_index=-100)
|
| 289 |
+
return Seq2SeqLMOutput(loss=loss, logits=logits)
|
| 290 |
+
|
| 291 |
+
@torch.inference_mode()
|
| 292 |
+
def translate(
|
| 293 |
+
self,
|
| 294 |
+
text: str,
|
| 295 |
+
sp_proc,
|
| 296 |
+
direction: str = "en2my",
|
| 297 |
+
style: str = "auto",
|
| 298 |
+
num_beams: int = 4,
|
| 299 |
+
max_new_tokens: int = 192,
|
| 300 |
+
length_penalty: float = 0.85,
|
| 301 |
+
rep_penalty: float = 1.12,
|
| 302 |
+
) -> str:
|
| 303 |
+
self.eval()
|
| 304 |
+
dev = self.embed_tokens.weight.device
|
| 305 |
+
dt = self.embed_tokens.weight.dtype
|
| 306 |
+
pref = [TOK_2MY if direction == "en2my" else TOK_2EN]
|
| 307 |
+
if style == "formal":
|
| 308 |
+
pref.append(TOK_FORMAL)
|
| 309 |
+
elif style == "spoken":
|
| 310 |
+
pref.append(TOK_SPOKEN)
|
| 311 |
+
src_ids = pref + sp_proc.encode_as_ids(text.strip())[: self.config.max_seq_len - 8] + [EOS_ID]
|
| 312 |
+
src_t = torch.tensor([src_ids], dtype=torch.long, device=dev)
|
| 313 |
+
cross_kv, enc_mask = self.encode_and_route(src_t)
|
| 314 |
+
|
| 315 |
+
if num_beams <= 1:
|
| 316 |
+
k_cache = torch.empty((self.dec_layers, 1, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=dt, device=dev)
|
| 317 |
+
v_cache = torch.empty((self.dec_layers, 1, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=dt, device=dev)
|
| 318 |
+
cur_tok = torch.tensor([[BOS_ID]], dtype=torch.long, device=dev)
|
| 319 |
+
cos_all = self.rope.cos_cached.to(dtype=dt, device=dev)
|
| 320 |
+
sin_all = self.rope.sin_cached.to(dtype=dt, device=dev)
|
| 321 |
+
ys = []
|
| 322 |
+
for pos in range(max_new_tokens):
|
| 323 |
+
x_dec = self.embed_tokens(cur_tok) * self.embed_scale
|
| 324 |
+
v0 = None
|
| 325 |
+
cos_p, sin_p = cos_all[:, :, pos : pos + 1, :], sin_all[:, :, pos : pos + 1, :]
|
| 326 |
+
for l_idx, layer in enumerate(self.decoder):
|
| 327 |
+
kc, vc = cross_kv[l_idx // 2]
|
| 328 |
+
x_dec, v_raw = layer.step_kv(x_dec, cos_p, sin_p, pos, k_cache[l_idx], v_cache[l_idx], kc, vc, enc_mask, v0=v0)
|
| 329 |
+
if l_idx == 0:
|
| 330 |
+
v0 = v_raw
|
| 331 |
+
logits = self.lm_head(self.dec_norm(x_dec)[:, 0, :]).float()[0]
|
| 332 |
+
logits[[0, 1, 3, 4, 5, 6, 7, 8, 9, 10]] = -1e9
|
| 333 |
+
if rep_penalty != 1.0 and ys:
|
| 334 |
+
seen = torch.tensor(list(set(ys)), dtype=torch.long, device=dev)
|
| 335 |
+
sl = logits[seen]
|
| 336 |
+
logits[seen] = torch.where(sl > 0, sl / rep_penalty, sl * rep_penalty)
|
| 337 |
+
nxt = int(logits.argmax().item())
|
| 338 |
+
if nxt == EOS_ID:
|
| 339 |
+
break
|
| 340 |
+
ys.append(nxt)
|
| 341 |
+
cur_tok = torch.tensor([[nxt]], dtype=torch.long, device=dev)
|
| 342 |
+
return sp_proc.decode_ids([x for x in ys if x >= 11]).strip()
|
| 343 |
+
|
| 344 |
+
beams = [(0.0, [BOS_ID], False)]
|
| 345 |
+
for _ in range(max_new_tokens):
|
| 346 |
+
active = [(sc, seq) for sc, seq, done in beams if not done]
|
| 347 |
+
if not active:
|
| 348 |
+
break
|
| 349 |
+
B_act = len(active)
|
| 350 |
+
tgt_t = torch.tensor([seq for _, seq in active], dtype=torch.long, device=dev)
|
| 351 |
+
ckv_exp = [(kc.expand(B_act, -1, -1, -1), vc.expand(B_act, -1, -1, -1)) for kc, vc in cross_kv]
|
| 352 |
+
h_dec = self.decode_seq(tgt_t, ckv_exp, enc_mask.expand(B_act, -1, -1, -1))
|
| 353 |
+
logits = self.lm_head(h_dec[:, -1, :]).float()
|
| 354 |
+
logits[:, [0, 1, 3, 4, 5, 6, 7, 8, 9, 10]] = -1e9
|
| 355 |
+
for b_i, (_, seq) in enumerate(active):
|
| 356 |
+
if rep_penalty != 1.0 and len(seq) > 1:
|
| 357 |
+
seen = torch.tensor(list(set(seq[1:])), dtype=torch.long, device=dev)
|
| 358 |
+
sl = logits[b_i, seen]
|
| 359 |
+
logits[b_i, seen] = torch.where(sl > 0, sl / rep_penalty, sl * rep_penalty)
|
| 360 |
+
lp = F.log_softmax(logits, dim=-1)
|
| 361 |
+
topk_lp, topk_ids = torch.topk(lp, num_beams * 2, dim=-1)
|
| 362 |
+
cands = [(sc, seq, True) for sc, seq, done in beams if done]
|
| 363 |
+
for b_i, (sc, seq) in enumerate(active):
|
| 364 |
+
for k in range(num_beams * 2):
|
| 365 |
+
tid = int(topk_ids[b_i, k].item())
|
| 366 |
+
n_sc = sc + float(topk_lp[b_i, k].item())
|
| 367 |
+
cands.append((n_sc, seq if tid == EOS_ID else seq + [tid], tid == EOS_ID))
|
| 368 |
+
cands.sort(key=lambda x: x[0] / (max(1, len(x[1]) - 1) ** length_penalty), reverse=True)
|
| 369 |
+
beams = cands[:num_beams]
|
| 370 |
+
if all(done for _, _, done in beams):
|
| 371 |
+
break
|
| 372 |
+
return sp_proc.decode_ids([x for x in beams[0][1][1:] if x >= 11]).strip()
|
checkpoint-stage2-step-6793/special_tokens_map.json
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"unk_token": "<unk>",
|
| 3 |
+
"bos_token": "<s>",
|
| 4 |
+
"eos_token": "</s>",
|
| 5 |
+
"pad_token": "<pad>",
|
| 6 |
+
"additional_special_tokens": [
|
| 7 |
+
"<2my>",
|
| 8 |
+
"<2en>",
|
| 9 |
+
"<formal>",
|
| 10 |
+
"<spoken>",
|
| 11 |
+
"<bt>",
|
| 12 |
+
"<denoise>",
|
| 13 |
+
"<mask>"
|
| 14 |
+
]
|
| 15 |
+
}
|
checkpoint-stage2-step-6793/tokenizer.model
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9aaeeaf36272c5c158d11ed0987ca5930686f1e02713292558003909bb147bc1
|
| 3 |
+
size 1840017
|
checkpoint-stage2-step-6793/tokenizer_config.json
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_max_length": 1024,
|
| 3 |
+
"unk_token": "<unk>",
|
| 4 |
+
"bos_token": "<s>",
|
| 5 |
+
"eos_token": "</s>",
|
| 6 |
+
"pad_token": "<pad>",
|
| 7 |
+
"additional_special_tokens": [
|
| 8 |
+
"<2my>",
|
| 9 |
+
"<2en>",
|
| 10 |
+
"<formal>",
|
| 11 |
+
"<spoken>",
|
| 12 |
+
"<bt>",
|
| 13 |
+
"<denoise>",
|
| 14 |
+
"<mask>"
|
| 15 |
+
],
|
| 16 |
+
"special_tokens_map": {
|
| 17 |
+
"2my": 4,
|
| 18 |
+
"2en": 5,
|
| 19 |
+
"formal": 6,
|
| 20 |
+
"spoken": 7,
|
| 21 |
+
"bt": 8,
|
| 22 |
+
"denoise": 9,
|
| 23 |
+
"mask": 10
|
| 24 |
+
}
|
| 25 |
+
}
|
checkpoint-stage2-step-6793/training_codes.txt
ADDED
|
@@ -0,0 +1,1214 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import os
|
| 2 |
+
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"
|
| 3 |
+
|
| 4 |
+
import argparse, json, math, time
|
| 5 |
+
from datetime import timedelta
|
| 6 |
+
import numpy as np
|
| 7 |
+
import sentencepiece as spm
|
| 8 |
+
import torch
|
| 9 |
+
import torch.distributed as dist
|
| 10 |
+
import torch.nn as nn
|
| 11 |
+
import torch.nn.functional as F
|
| 12 |
+
from torch.utils.checkpoint import checkpoint as grad_ckpt
|
| 13 |
+
from torch.utils.data import Dataset, DataLoader
|
| 14 |
+
|
| 15 |
+
torch.backends.cuda.matmul.allow_tf32 = True
|
| 16 |
+
torch.backends.cudnn.allow_tf32 = True
|
| 17 |
+
torch._dynamo.config.cache_size_limit = 64
|
| 18 |
+
|
| 19 |
+
DATA_DIR = "/workspace/mytrans_dl/tokenized_65k"
|
| 20 |
+
CKPT_DIR = "/workspace/ckpt_300m"
|
| 21 |
+
|
| 22 |
+
VOCAB_SIZE = 65536
|
| 23 |
+
UNK_ID, BOS_ID, EOS_ID, PAD_ID = 0, 1, 2, 3
|
| 24 |
+
TOK_2MY, TOK_2EN, TOK_FORMAL, TOK_SPOKEN, TOK_BT, TOK_DENOISE, TOK_MASK = 4, 5, 6, 7, 8, 9, 10
|
| 25 |
+
FLAG_BT, FLAG_FORMAL, FLAG_SPOKEN = 1, 2, 4
|
| 26 |
+
|
| 27 |
+
BULK_SUBSETS = {
|
| 28 |
+
"MyanmarTranslator_works/myanmar_spoken_corpus",
|
| 29 |
+
"MyanmarTranslator_works/myanmar-cc100-dataset",
|
| 30 |
+
"MyanmarTranslator_works/myanmar-written-corpus",
|
| 31 |
+
}
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def load_and_verify_contracts(data_dir: str):
|
| 35 |
+
with open(os.path.join(data_dir, "meta.json"), "r") as f:
|
| 36 |
+
meta = json.load(f)
|
| 37 |
+
sp = spm.SentencePieceProcessor()
|
| 38 |
+
sp.load(os.path.join(data_dir, "tokenizer.model"))
|
| 39 |
+
|
| 40 |
+
assert sp.get_piece_size() == VOCAB_SIZE
|
| 41 |
+
assert sp.unk_id() == UNK_ID and sp.bos_id() == BOS_ID and sp.eos_id() == EOS_ID and sp.pad_id() == PAD_ID
|
| 42 |
+
expected = {4: "<2my>", 5: "<2en>", 6: "<formal>", 7: "<spoken>", 8: "<bt>", 9: "<denoise>", 10: "<mask>"}
|
| 43 |
+
for tid, piece in expected.items():
|
| 44 |
+
assert sp.id_to_piece(tid) == piece, f"Token {tid} mismatch"
|
| 45 |
+
if "special_tokens" in meta:
|
| 46 |
+
st = meta["special_tokens"]
|
| 47 |
+
assert st["2my"] == TOK_2MY and st["2en"] == TOK_2EN
|
| 48 |
+
assert st["formal"] == TOK_FORMAL and st["spoken"] == TOK_SPOKEN
|
| 49 |
+
assert st["bt"] == TOK_BT and st["denoise"] == TOK_DENOISE and st["mask"] == TOK_MASK
|
| 50 |
+
return meta, sp
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
# ==============================================================================
|
| 54 |
+
# 1. MyTranslation-300M Architecture
|
| 55 |
+
# ==============================================================================
|
| 56 |
+
class RMSNorm(nn.Module):
|
| 57 |
+
def __init__(self, dim: int, eps: float = 1e-6):
|
| 58 |
+
super().__init__()
|
| 59 |
+
self.dim = (dim,)
|
| 60 |
+
self.eps = eps
|
| 61 |
+
self.weight = nn.Parameter(torch.ones(dim))
|
| 62 |
+
|
| 63 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 64 |
+
return F.rms_norm(x, self.dim, self.weight, self.eps)
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
class RotaryEmbedding(nn.Module):
|
| 68 |
+
def __init__(self, dim: int, max_seq_len: int = 2048, theta: float = 50000.0):
|
| 69 |
+
super().__init__()
|
| 70 |
+
inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))
|
| 71 |
+
t = torch.arange(max_seq_len, dtype=torch.float32)
|
| 72 |
+
freqs = torch.outer(t, inv_freq)
|
| 73 |
+
self.register_buffer("cos_cached", freqs.cos()[None, None, :, :], persistent=False)
|
| 74 |
+
self.register_buffer("sin_cached", freqs.sin()[None, None, :, :], persistent=False)
|
| 75 |
+
|
| 76 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 77 |
+
L = x.size(2)
|
| 78 |
+
cos = self.cos_cached[:, :, :L, :].to(dtype=x.dtype)
|
| 79 |
+
sin = self.sin_cached[:, :, :L, :].to(dtype=x.dtype)
|
| 80 |
+
x1, x2 = x[..., ::2], x[..., 1::2]
|
| 81 |
+
return torch.cat((x1 * cos - x2 * sin, x1 * sin + x2 * cos), dim=-1)
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
class SwiGLU(nn.Module):
|
| 85 |
+
def __init__(self, d_model: int, ffn_dim: int):
|
| 86 |
+
super().__init__()
|
| 87 |
+
self.gate_up = nn.Linear(d_model, 2 * ffn_dim, bias=False)
|
| 88 |
+
self.down = nn.Linear(ffn_dim, d_model, bias=False)
|
| 89 |
+
|
| 90 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 91 |
+
gate, up = self.gate_up(x).chunk(2, dim=-1)
|
| 92 |
+
return self.down(F.silu(gate) * up)
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
class GatedGQASelfAttention(nn.Module):
|
| 96 |
+
def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, is_causal: bool = False, has_v_res: bool = False):
|
| 97 |
+
super().__init__()
|
| 98 |
+
self.n_heads = n_heads
|
| 99 |
+
self.n_kv_heads = n_kv_heads
|
| 100 |
+
self.head_dim = d_model // n_heads
|
| 101 |
+
self.is_causal = is_causal
|
| 102 |
+
self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
|
| 103 |
+
self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * self.head_dim, bias=False)
|
| 104 |
+
self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
|
| 105 |
+
self.out = nn.Linear(d_model, d_model, bias=False)
|
| 106 |
+
self.q_norm = RMSNorm(self.head_dim)
|
| 107 |
+
self.k_norm = RMSNorm(self.head_dim)
|
| 108 |
+
self.v_res_logit = nn.Parameter(torch.tensor(-2.0)) if has_v_res else None
|
| 109 |
+
|
| 110 |
+
def forward(self, x: torch.Tensor, rope: RotaryEmbedding, attn_mask: torch.Tensor = None, v0: torch.Tensor = None):
|
| 111 |
+
B, L, D = x.shape
|
| 112 |
+
q = self.q_norm(self.q_proj(x).view(B, L, self.n_heads, self.head_dim).transpose(1, 2))
|
| 113 |
+
k, v = self.kv_proj(x).chunk(2, dim=-1)
|
| 114 |
+
k = self.k_norm(k.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2))
|
| 115 |
+
v = v.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
| 116 |
+
v_raw = v
|
| 117 |
+
if self.v_res_logit is not None and v0 is not None:
|
| 118 |
+
alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
|
| 119 |
+
v = (1.0 - alpha) * v + alpha * v0
|
| 120 |
+
q, k = rope(q), rope(k)
|
| 121 |
+
y = F.scaled_dot_product_attention(
|
| 122 |
+
q, k, v,
|
| 123 |
+
attn_mask=None if self.is_causal else attn_mask,
|
| 124 |
+
is_causal=self.is_causal,
|
| 125 |
+
enable_gqa=True,
|
| 126 |
+
)
|
| 127 |
+
gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
|
| 128 |
+
y = (y * gate).transpose(1, 2).contiguous().view(B, L, D)
|
| 129 |
+
return self.out(y), v_raw
|
| 130 |
+
|
| 131 |
+
def step_kv(self, x_step, cos_p, sin_p, pos, k_cache, v_cache, v0=None):
|
| 132 |
+
B, _, D = x_step.shape
|
| 133 |
+
q = self.q_norm(self.q_proj(x_step).view(B, 1, self.n_heads, self.head_dim).transpose(1, 2))
|
| 134 |
+
k, v = self.kv_proj(x_step).chunk(2, dim=-1)
|
| 135 |
+
k = self.k_norm(k.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2))
|
| 136 |
+
v = v.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
| 137 |
+
v_raw = v
|
| 138 |
+
if self.v_res_logit is not None and v0 is not None:
|
| 139 |
+
alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
|
| 140 |
+
v = (1.0 - alpha) * v + alpha * v0
|
| 141 |
+
q1, q2 = q[..., ::2], q[..., 1::2]
|
| 142 |
+
k1, k2 = k[..., ::2], k[..., 1::2]
|
| 143 |
+
q = torch.cat((q1 * cos_p - q2 * sin_p, q1 * sin_p + q2 * cos_p), dim=-1)
|
| 144 |
+
k = torch.cat((k1 * cos_p - k2 * sin_p, k1 * sin_p + k2 * cos_p), dim=-1)
|
| 145 |
+
k_cache[:, :, pos : pos + 1, :] = k
|
| 146 |
+
v_cache[:, :, pos : pos + 1, :] = v
|
| 147 |
+
y = F.scaled_dot_product_attention(
|
| 148 |
+
q, k_cache[:, :, : pos + 1, :], v_cache[:, :, : pos + 1, :],
|
| 149 |
+
is_causal=False, enable_gqa=True,
|
| 150 |
+
)
|
| 151 |
+
gate = torch.sigmoid(self.gate_proj(x_step)).transpose(1, 2).unsqueeze(-1)
|
| 152 |
+
y = (y * gate).transpose(1, 2).contiguous().view(B, 1, D)
|
| 153 |
+
return self.out(y), v_raw
|
| 154 |
+
|
| 155 |
+
|
| 156 |
+
class SharedCrossKVProj(nn.Module):
|
| 157 |
+
def __init__(self, d_model: int, n_kv_heads: int, head_dim: int):
|
| 158 |
+
super().__init__()
|
| 159 |
+
self.n_kv_heads = n_kv_heads
|
| 160 |
+
self.head_dim = head_dim
|
| 161 |
+
self.norm = RMSNorm(d_model)
|
| 162 |
+
self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * head_dim, bias=False)
|
| 163 |
+
self.k_norm = RMSNorm(head_dim)
|
| 164 |
+
|
| 165 |
+
def forward(self, enc_mix: torch.Tensor):
|
| 166 |
+
B, S, _ = enc_mix.shape
|
| 167 |
+
k, v = self.kv_proj(self.norm(enc_mix)).chunk(2, dim=-1)
|
| 168 |
+
k = self.k_norm(k.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2))
|
| 169 |
+
v = v.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
| 170 |
+
return k, v
|
| 171 |
+
|
| 172 |
+
|
| 173 |
+
class GatedGQACrossAttention(nn.Module):
|
| 174 |
+
def __init__(self, d_model: int, n_heads: int):
|
| 175 |
+
super().__init__()
|
| 176 |
+
self.n_heads = n_heads
|
| 177 |
+
self.head_dim = d_model // n_heads
|
| 178 |
+
self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
|
| 179 |
+
self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
|
| 180 |
+
self.out = nn.Linear(d_model, d_model, bias=False)
|
| 181 |
+
self.q_norm = RMSNorm(self.head_dim)
|
| 182 |
+
|
| 183 |
+
def forward(self, x: torch.Tensor, k_cross: torch.Tensor, v_cross: torch.Tensor, enc_mask: torch.Tensor):
|
| 184 |
+
B, T, D = x.shape
|
| 185 |
+
q = self.q_norm(self.q_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2))
|
| 186 |
+
y = F.scaled_dot_product_attention(q, k_cross, v_cross, attn_mask=enc_mask, is_causal=False, enable_gqa=True)
|
| 187 |
+
gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
|
| 188 |
+
y = (y * gate).transpose(1, 2).contiguous().view(B, T, D)
|
| 189 |
+
return self.out(y)
|
| 190 |
+
|
| 191 |
+
|
| 192 |
+
class EncoderLayer(nn.Module):
|
| 193 |
+
def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
|
| 194 |
+
super().__init__()
|
| 195 |
+
self.norm1 = RMSNorm(d_model)
|
| 196 |
+
self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=False, has_v_res=has_v_res)
|
| 197 |
+
self.norm2 = RMSNorm(d_model)
|
| 198 |
+
self.mlp = SwiGLU(d_model, ffn_dim)
|
| 199 |
+
|
| 200 |
+
def forward(self, x, rope, enc_mask, v0=None):
|
| 201 |
+
attn_out, v_raw = self.self_attn(self.norm1(x), rope, enc_mask, v0=v0)
|
| 202 |
+
x = x + attn_out
|
| 203 |
+
x = x + self.mlp(self.norm2(x))
|
| 204 |
+
return x, v_raw
|
| 205 |
+
|
| 206 |
+
|
| 207 |
+
class DecoderLayer(nn.Module):
|
| 208 |
+
def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
|
| 209 |
+
super().__init__()
|
| 210 |
+
self.norm1 = RMSNorm(d_model)
|
| 211 |
+
self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=True, has_v_res=has_v_res)
|
| 212 |
+
self.norm2 = RMSNorm(d_model)
|
| 213 |
+
self.cross_attn = GatedGQACrossAttention(d_model, n_heads)
|
| 214 |
+
self.norm3 = RMSNorm(d_model)
|
| 215 |
+
self.mlp = SwiGLU(d_model, ffn_dim)
|
| 216 |
+
|
| 217 |
+
def forward(self, x, k_cross, v_cross, rope, enc_mask, v0=None):
|
| 218 |
+
attn_out, v_raw = self.self_attn(self.norm1(x), rope, None, v0=v0)
|
| 219 |
+
x = x + attn_out
|
| 220 |
+
x = x + self.cross_attn(self.norm2(x), k_cross, v_cross, enc_mask)
|
| 221 |
+
x = x + self.mlp(self.norm3(x))
|
| 222 |
+
return x, v_raw
|
| 223 |
+
|
| 224 |
+
def step_kv(self, x_step, cos_p, sin_p, pos, k_self, v_self, k_cross, v_cross, enc_mask, v0=None):
|
| 225 |
+
attn_out, v_raw = self.self_attn.step_kv(self.norm1(x_step), cos_p, sin_p, pos, k_self, v_self, v0=v0)
|
| 226 |
+
x_step = x_step + attn_out
|
| 227 |
+
x_step = x_step + self.cross_attn(self.norm2(x_step), k_cross, v_cross, enc_mask)
|
| 228 |
+
x_step = x_step + self.mlp(self.norm3(x_step))
|
| 229 |
+
return x_step, v_raw
|
| 230 |
+
|
| 231 |
+
|
| 232 |
+
class MTPHeadBlock(nn.Module):
|
| 233 |
+
def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int):
|
| 234 |
+
super().__init__()
|
| 235 |
+
self.norm_h = RMSNorm(d_model)
|
| 236 |
+
self.norm_e = RMSNorm(d_model)
|
| 237 |
+
self.proj = nn.Linear(2 * d_model, d_model, bias=False)
|
| 238 |
+
self.norm1 = RMSNorm(d_model)
|
| 239 |
+
self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=True, has_v_res=False)
|
| 240 |
+
self.norm2 = RMSNorm(d_model)
|
| 241 |
+
self.mlp = SwiGLU(d_model, ffn_dim)
|
| 242 |
+
self.out_norm = RMSNorm(d_model)
|
| 243 |
+
|
| 244 |
+
def forward(self, h_main: torch.Tensor, next_tok_emb: torch.Tensor, rope: RotaryEmbedding):
|
| 245 |
+
x = self.proj(torch.cat([self.norm_h(h_main), self.norm_e(next_tok_emb)], dim=-1))
|
| 246 |
+
attn_out, _ = self.self_attn(self.norm1(x), rope, None)
|
| 247 |
+
x = x + attn_out
|
| 248 |
+
x = x + self.mlp(self.norm2(x))
|
| 249 |
+
return self.out_norm(x)
|
| 250 |
+
|
| 251 |
+
|
| 252 |
+
class MyTranslation300M(nn.Module):
|
| 253 |
+
def __init__(
|
| 254 |
+
self,
|
| 255 |
+
vocab_size=65536,
|
| 256 |
+
d_model=1024,
|
| 257 |
+
n_heads=16,
|
| 258 |
+
n_kv_heads=4,
|
| 259 |
+
enc_layers=12,
|
| 260 |
+
dec_layers=6,
|
| 261 |
+
ffn_dim=3072,
|
| 262 |
+
max_seq_len=2048,
|
| 263 |
+
):
|
| 264 |
+
super().__init__()
|
| 265 |
+
assert d_model % n_heads == 0 and n_heads % n_kv_heads == 0 and dec_layers % 2 == 0
|
| 266 |
+
self.config = dict(
|
| 267 |
+
vocab_size=vocab_size, d_model=d_model, n_heads=n_heads, n_kv_heads=n_kv_heads,
|
| 268 |
+
enc_layers=enc_layers, dec_layers=dec_layers, ffn_dim=ffn_dim, max_seq_len=max_seq_len,
|
| 269 |
+
)
|
| 270 |
+
self.d_model = d_model
|
| 271 |
+
self.n_heads = n_heads
|
| 272 |
+
self.n_kv_heads = n_kv_heads
|
| 273 |
+
self.head_dim = d_model // n_heads
|
| 274 |
+
self.enc_layers = enc_layers
|
| 275 |
+
self.dec_layers = dec_layers
|
| 276 |
+
self.n_cross_pairs = dec_layers // 2
|
| 277 |
+
self.embed_scale = math.sqrt(d_model)
|
| 278 |
+
|
| 279 |
+
self.embed_tokens = nn.Embedding(vocab_size, d_model, padding_idx=PAD_ID)
|
| 280 |
+
self.rope = RotaryEmbedding(self.head_dim, max_seq_len=max_seq_len, theta=50000.0)
|
| 281 |
+
|
| 282 |
+
self.encoder = nn.ModuleList([
|
| 283 |
+
EncoderLayer(d_model, n_heads, n_kv_heads, ffn_dim, has_v_res=(i > 0))
|
| 284 |
+
for i in range(enc_layers)
|
| 285 |
+
])
|
| 286 |
+
self.enc_dwa_logits = nn.ParameterList([
|
| 287 |
+
nn.Parameter(torch.tensor([-4.0] * i + [0.0], dtype=torch.float32))
|
| 288 |
+
for i in range(1, enc_layers)
|
| 289 |
+
])
|
| 290 |
+
|
| 291 |
+
self.enc_router_logits = nn.Parameter(torch.zeros(self.n_cross_pairs, enc_layers))
|
| 292 |
+
with torch.no_grad():
|
| 293 |
+
for p in range(self.n_cross_pairs):
|
| 294 |
+
center = (p + 0.5) * (enc_layers / self.n_cross_pairs) - 0.5
|
| 295 |
+
for k in range(enc_layers):
|
| 296 |
+
self.enc_router_logits[p, k] = -0.35 * ((k - center) ** 2)
|
| 297 |
+
|
| 298 |
+
self.cross_kv_projs = nn.ModuleList([
|
| 299 |
+
SharedCrossKVProj(d_model, n_kv_heads, self.head_dim)
|
| 300 |
+
for _ in range(self.n_cross_pairs)
|
| 301 |
+
])
|
| 302 |
+
|
| 303 |
+
self.decoder = nn.ModuleList([
|
| 304 |
+
DecoderLayer(d_model, n_heads, n_kv_heads, ffn_dim, has_v_res=(i > 0))
|
| 305 |
+
for i in range(dec_layers)
|
| 306 |
+
])
|
| 307 |
+
self.dec_norm = RMSNorm(d_model)
|
| 308 |
+
self.lm_head = nn.Linear(d_model, vocab_size, bias=False)
|
| 309 |
+
self.lm_head.weight = self.embed_tokens.weight
|
| 310 |
+
|
| 311 |
+
self.mtp_head = MTPHeadBlock(d_model, n_heads, n_kv_heads, ffn_dim)
|
| 312 |
+
self._init_weights(enc_layers + dec_layers)
|
| 313 |
+
|
| 314 |
+
def _init_weights(self, total_layers: int):
|
| 315 |
+
res_std = 0.02 / math.sqrt(2 * total_layers)
|
| 316 |
+
for name, m in self.named_modules():
|
| 317 |
+
if isinstance(m, nn.Linear):
|
| 318 |
+
if name.endswith(".out") or name.endswith(".down"):
|
| 319 |
+
nn.init.normal_(m.weight, mean=0.0, std=res_std)
|
| 320 |
+
elif name.endswith(".gate_proj"):
|
| 321 |
+
nn.init.normal_(m.weight, mean=0.0, std=0.01)
|
| 322 |
+
else:
|
| 323 |
+
nn.init.normal_(m.weight, mean=0.0, std=0.02)
|
| 324 |
+
elif isinstance(m, nn.Embedding):
|
| 325 |
+
nn.init.normal_(m.weight, mean=0.0, std=0.02)
|
| 326 |
+
if m.padding_idx is not None:
|
| 327 |
+
m.weight.data[m.padding_idx].zero_()
|
| 328 |
+
|
| 329 |
+
def compile_blocks(self):
|
| 330 |
+
for i in range(len(self.encoder)):
|
| 331 |
+
self.encoder[i] = torch.compile(self.encoder[i], dynamic=True)
|
| 332 |
+
for i in range(len(self.cross_kv_projs)):
|
| 333 |
+
self.cross_kv_projs[i] = torch.compile(self.cross_kv_projs[i], dynamic=True)
|
| 334 |
+
for i in range(len(self.decoder)):
|
| 335 |
+
self.decoder[i] = torch.compile(self.decoder[i], dynamic=True)
|
| 336 |
+
self.mtp_head = torch.compile(self.mtp_head, dynamic=True)
|
| 337 |
+
|
| 338 |
+
def encode_and_route(self, src_ids: torch.Tensor, ckpt_mode: str = "none"):
|
| 339 |
+
enc_mask = (src_ids != PAD_ID).unsqueeze(1).unsqueeze(2)
|
| 340 |
+
x = self.embed_tokens(src_ids) * self.embed_scale
|
| 341 |
+
raw_states = [x]
|
| 342 |
+
enc_outputs = []
|
| 343 |
+
v0 = None
|
| 344 |
+
|
| 345 |
+
for i, layer in enumerate(self.encoder):
|
| 346 |
+
if i > 0:
|
| 347 |
+
w_dwa = F.softmax(self.enc_dwa_logits[i - 1], dim=0).to(dtype=x.dtype)
|
| 348 |
+
if self.training and ckpt_mode != "none":
|
| 349 |
+
x_in = grad_ckpt(
|
| 350 |
+
lambda w, *st: torch.einsum("j,jbsd->bsd", w, torch.stack(st, dim=0)),
|
| 351 |
+
w_dwa, *raw_states, use_reentrant=False
|
| 352 |
+
)
|
| 353 |
+
else:
|
| 354 |
+
x_in = torch.einsum("j,jbsd->bsd", w_dwa, torch.stack(raw_states, dim=0))
|
| 355 |
+
else:
|
| 356 |
+
x_in = x
|
| 357 |
+
|
| 358 |
+
do_ckpt = self.training and (
|
| 359 |
+
(ckpt_mode == "heavy" and (i % 3 != 0)) or
|
| 360 |
+
(ckpt_mode == "light" and (i in (3, 7, 11)))
|
| 361 |
+
)
|
| 362 |
+
if do_ckpt:
|
| 363 |
+
x_out, v_raw = grad_ckpt(layer, x_in, self.rope, enc_mask, v0, use_reentrant=False)
|
| 364 |
+
else:
|
| 365 |
+
x_out, v_raw = layer(x_in, self.rope, enc_mask, v0=v0)
|
| 366 |
+
|
| 367 |
+
if i == 0:
|
| 368 |
+
v0 = v_raw
|
| 369 |
+
raw_states.append(x_out)
|
| 370 |
+
enc_outputs.append(x_out)
|
| 371 |
+
|
| 372 |
+
stacked_enc = torch.stack(enc_outputs, dim=0)
|
| 373 |
+
router_w = F.softmax(self.enc_router_logits, dim=-1).to(dtype=stacked_enc.dtype)
|
| 374 |
+
cross_kv = []
|
| 375 |
+
for p in range(self.n_cross_pairs):
|
| 376 |
+
enc_mix = torch.einsum("k,kbsd->bsd", router_w[p], stacked_enc)
|
| 377 |
+
cross_kv.append(self.cross_kv_projs[p](enc_mix))
|
| 378 |
+
return cross_kv, enc_mask
|
| 379 |
+
|
| 380 |
+
def decode_seq(self, tgt_in: torch.Tensor, cross_kv: list, enc_mask: torch.Tensor, ckpt_mode: str = "none"):
|
| 381 |
+
x = self.embed_tokens(tgt_in) * self.embed_scale
|
| 382 |
+
v0 = None
|
| 383 |
+
for i, layer in enumerate(self.decoder):
|
| 384 |
+
kc, vc = cross_kv[i // 2]
|
| 385 |
+
do_ckpt = self.training and (
|
| 386 |
+
(ckpt_mode == "heavy" and (i % 2 == 1)) or
|
| 387 |
+
(ckpt_mode == "light" and (i == 3))
|
| 388 |
+
)
|
| 389 |
+
if do_ckpt:
|
| 390 |
+
x, v_raw = grad_ckpt(layer, x, kc, vc, self.rope, enc_mask, v0, use_reentrant=False)
|
| 391 |
+
else:
|
| 392 |
+
x, v_raw = layer(x, kc, vc, self.rope, enc_mask, v0=v0)
|
| 393 |
+
if i == 0:
|
| 394 |
+
v0 = v_raw
|
| 395 |
+
return self.dec_norm(x)
|
| 396 |
+
|
| 397 |
+
def forward_and_backward(
|
| 398 |
+
self,
|
| 399 |
+
src_ids: torch.Tensor,
|
| 400 |
+
tgt_in: torch.Tensor,
|
| 401 |
+
labels: torch.Tensor,
|
| 402 |
+
main_denom: float,
|
| 403 |
+
mtp_denom: float,
|
| 404 |
+
label_smoothing: float = 0.1,
|
| 405 |
+
z_loss_weight: float = 1e-4,
|
| 406 |
+
mtp_weight: float = 0.3,
|
| 407 |
+
):
|
| 408 |
+
tot_tok = src_ids.numel() + tgt_in.numel()
|
| 409 |
+
max_len = max(src_ids.size(1), tgt_in.size(1))
|
| 410 |
+
if tot_tok <= 14336:
|
| 411 |
+
ckpt_mode = "none"
|
| 412 |
+
elif max_len <= 256:
|
| 413 |
+
ckpt_mode = "light"
|
| 414 |
+
else:
|
| 415 |
+
ckpt_mode = "heavy"
|
| 416 |
+
|
| 417 |
+
with torch.amp.autocast("cuda", dtype=torch.bfloat16):
|
| 418 |
+
cross_kv, enc_mask = self.encode_and_route(src_ids, ckpt_mode=ckpt_mode)
|
| 419 |
+
h_main = self.decode_seq(tgt_in, cross_kv, enc_mask, ckpt_mode=ckpt_mode)
|
| 420 |
+
|
| 421 |
+
valid_mask = labels.ne(-100)
|
| 422 |
+
valid_y = labels[valid_mask]
|
| 423 |
+
n_valid = valid_y.numel()
|
| 424 |
+
valid_h_leaf = h_main[valid_mask].detach().requires_grad_(True)
|
| 425 |
+
|
| 426 |
+
tot_main_ce_t = torch.zeros((), dtype=torch.float64, device=h_main.device)
|
| 427 |
+
tot_z_loss_t = torch.zeros((), dtype=torch.float64, device=h_main.device)
|
| 428 |
+
tot_mtp_ce_t = torch.zeros((), dtype=torch.float64, device=h_main.device)
|
| 429 |
+
chunk_sz = 2048
|
| 430 |
+
|
| 431 |
+
for i in range(0, n_valid, chunk_sz):
|
| 432 |
+
hc = valid_h_leaf[i : i + chunk_sz]
|
| 433 |
+
yc = valid_y[i : i + chunk_sz]
|
| 434 |
+
with torch.amp.autocast("cuda", dtype=torch.bfloat16):
|
| 435 |
+
logits_c = self.lm_head(hc)
|
| 436 |
+
ce_c = F.cross_entropy(logits_c, yc, label_smoothing=label_smoothing, reduction="sum")
|
| 437 |
+
lse = torch.logsumexp(logits_c.float(), dim=-1)
|
| 438 |
+
z_c = (lse * lse).sum()
|
| 439 |
+
loss_c = (ce_c + z_loss_weight * z_c) / main_denom
|
| 440 |
+
loss_c.backward()
|
| 441 |
+
tot_main_ce_t.add_(ce_c.detach().to(torch.float64))
|
| 442 |
+
tot_z_loss_t.add_(z_c.detach().to(torch.float64))
|
| 443 |
+
del logits_c, ce_c, z_c, loss_c
|
| 444 |
+
|
| 445 |
+
grad_h_main = torch.zeros_like(h_main)
|
| 446 |
+
if valid_h_leaf.grad is not None:
|
| 447 |
+
grad_h_main[valid_mask] = valid_h_leaf.grad
|
| 448 |
+
|
| 449 |
+
n_mtp = 0
|
| 450 |
+
if mtp_weight > 0.0 and tgt_in.size(1) > 1:
|
| 451 |
+
mtp_mask = labels[:, 1:].ne(-100) & labels[:, :-1].ne(-100)
|
| 452 |
+
valid_ym = labels[:, 1:][mtp_mask]
|
| 453 |
+
n_mtp = valid_ym.numel()
|
| 454 |
+
if n_mtp > 0:
|
| 455 |
+
h_main_leaf = h_main.detach().requires_grad_(True)
|
| 456 |
+
with torch.amp.autocast("cuda", dtype=torch.bfloat16):
|
| 457 |
+
next_emb = self.embed_tokens(tgt_in[:, 1:]) * self.embed_scale
|
| 458 |
+
h_mtp = self.mtp_head(h_main_leaf[:, :-1, :], next_emb, self.rope)
|
| 459 |
+
valid_hm_leaf = h_mtp[mtp_mask].detach().requires_grad_(True)
|
| 460 |
+
|
| 461 |
+
for i in range(0, n_mtp, chunk_sz):
|
| 462 |
+
hmc = valid_hm_leaf[i : i + chunk_sz]
|
| 463 |
+
ymc = valid_ym[i : i + chunk_sz]
|
| 464 |
+
with torch.amp.autocast("cuda", dtype=torch.bfloat16):
|
| 465 |
+
logits_m = self.lm_head(hmc)
|
| 466 |
+
ce_m = F.cross_entropy(logits_m, ymc, label_smoothing=label_smoothing, reduction="sum")
|
| 467 |
+
loss_m = (mtp_weight * ce_m) / mtp_denom
|
| 468 |
+
loss_m.backward()
|
| 469 |
+
tot_mtp_ce_t.add_(ce_m.detach().to(torch.float64))
|
| 470 |
+
del logits_m, ce_m, loss_m
|
| 471 |
+
|
| 472 |
+
grad_h_mtp = torch.zeros_like(h_mtp)
|
| 473 |
+
if valid_hm_leaf.grad is not None:
|
| 474 |
+
grad_h_mtp[mtp_mask] = valid_hm_leaf.grad
|
| 475 |
+
h_mtp.backward(grad_h_mtp)
|
| 476 |
+
if h_main_leaf.grad is not None:
|
| 477 |
+
grad_h_main.add_(h_main_leaf.grad)
|
| 478 |
+
|
| 479 |
+
h_main.backward(grad_h_main)
|
| 480 |
+
ce_v, mtp_v, z_v = torch.stack([tot_main_ce_t, tot_mtp_ce_t, tot_z_loss_t]).tolist()
|
| 481 |
+
return ce_v, mtp_v, z_v, n_valid, n_mtp
|
| 482 |
+
|
| 483 |
+
@torch.no_grad()
|
| 484 |
+
def eval_nll(self, src_ids: torch.Tensor, tgt_in: torch.Tensor, labels: torch.Tensor):
|
| 485 |
+
with torch.amp.autocast("cuda", dtype=torch.bfloat16):
|
| 486 |
+
cross_kv, enc_mask = self.encode_and_route(src_ids, ckpt_mode="none")
|
| 487 |
+
h_main = self.decode_seq(tgt_in, cross_kv, enc_mask, ckpt_mode="none")
|
| 488 |
+
valid_mask = labels.ne(-100)
|
| 489 |
+
valid_h = h_main[valid_mask]
|
| 490 |
+
valid_y = labels[valid_mask]
|
| 491 |
+
n_valid = valid_y.numel()
|
| 492 |
+
tot_nll = 0.0
|
| 493 |
+
for i in range(0, n_valid, 4096):
|
| 494 |
+
logits_c = self.lm_head(valid_h[i : i + 4096])
|
| 495 |
+
tot_nll += float(F.cross_entropy(logits_c, valid_y[i : i + 4096], reduction="sum").item())
|
| 496 |
+
return tot_nll, n_valid
|
| 497 |
+
|
| 498 |
+
@torch.inference_mode()
|
| 499 |
+
def translate_kv(self, prompt_ids_list: list[list[int]], sp_proc, max_new_tokens: int = 160, rep_penalty: float = 1.12) -> list[str]:
|
| 500 |
+
assert len(prompt_ids_list) > 0 and all(len(s) > 0 for s in prompt_ids_list), "Empty prompt batch!"
|
| 501 |
+
max_rope = self.rope.cos_cached.size(2)
|
| 502 |
+
assert 0 < max_new_tokens <= max_rope, f"max_new_tokens={max_new_tokens} exceeds RoPE={max_rope}"
|
| 503 |
+
|
| 504 |
+
self.eval()
|
| 505 |
+
dev = self.embed_tokens.weight.device
|
| 506 |
+
B = len(prompt_ids_list)
|
| 507 |
+
trimmed = [s[:max_rope] for s in prompt_ids_list]
|
| 508 |
+
max_s = min(max_rope, ((max(len(s) for s in trimmed) + 7) // 8) * 8)
|
| 509 |
+
src_ids = torch.full((B, max_s), PAD_ID, dtype=torch.long, device=dev)
|
| 510 |
+
for i, ids in enumerate(trimmed):
|
| 511 |
+
src_ids[i, : len(ids)] = torch.tensor(ids, dtype=torch.long, device=dev)
|
| 512 |
+
|
| 513 |
+
with torch.amp.autocast("cuda", dtype=torch.bfloat16):
|
| 514 |
+
cross_kv, enc_mask = self.encode_and_route(src_ids, ckpt_mode="none")
|
| 515 |
+
k_cache = torch.empty((self.dec_layers, B, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=torch.bfloat16, device=dev)
|
| 516 |
+
v_cache = torch.empty((self.dec_layers, B, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=torch.bfloat16, device=dev)
|
| 517 |
+
|
| 518 |
+
cur_tok = torch.full((B, 1), BOS_ID, dtype=torch.long, device=dev)
|
| 519 |
+
gen_ids = torch.full((B, max_new_tokens), EOS_ID, dtype=torch.long, device=dev)
|
| 520 |
+
done = torch.zeros(B, dtype=torch.bool, device=dev)
|
| 521 |
+
cos_all, sin_all = self.rope.cos_cached.to(torch.bfloat16), self.rope.sin_cached.to(torch.bfloat16)
|
| 522 |
+
|
| 523 |
+
for pos in range(max_new_tokens):
|
| 524 |
+
cos_p, sin_p = cos_all[:, :, pos : pos + 1, :], sin_all[:, :, pos : pos + 1, :]
|
| 525 |
+
x_dec = self.embed_tokens(cur_tok) * self.embed_scale
|
| 526 |
+
v0 = None
|
| 527 |
+
for l_idx, layer in enumerate(self.decoder):
|
| 528 |
+
kc, vc = cross_kv[l_idx // 2]
|
| 529 |
+
x_dec, v_raw = layer.step_kv(x_dec, cos_p, sin_p, pos, k_cache[l_idx], v_cache[l_idx], kc, vc, enc_mask, v0=v0)
|
| 530 |
+
if l_idx == 0:
|
| 531 |
+
v0 = v_raw
|
| 532 |
+
|
| 533 |
+
logits = self.lm_head(self.dec_norm(x_dec)[:, 0, :]).float()
|
| 534 |
+
logits[:, [0, 1, 3, 4, 5, 6, 7, 8, 9, 10]] = -1e9
|
| 535 |
+
if rep_penalty != 1.0 and pos > 0:
|
| 536 |
+
prev = gen_ids[:, :pos]
|
| 537 |
+
s_l = torch.gather(logits, 1, prev)
|
| 538 |
+
s_l = torch.where(s_l > 0, s_l / rep_penalty, s_l * rep_penalty)
|
| 539 |
+
logits.scatter_(1, prev, s_l)
|
| 540 |
+
|
| 541 |
+
nxt = torch.where(done, torch.tensor(EOS_ID, device=dev), logits.argmax(dim=-1))
|
| 542 |
+
gen_ids[:, pos] = nxt
|
| 543 |
+
done = done | (nxt == EOS_ID)
|
| 544 |
+
cur_tok = nxt.unsqueeze(1)
|
| 545 |
+
if bool(done.all()):
|
| 546 |
+
break
|
| 547 |
+
|
| 548 |
+
out_texts = []
|
| 549 |
+
for row in gen_ids.cpu().tolist():
|
| 550 |
+
if EOS_ID in row:
|
| 551 |
+
row = row[: row.index(EOS_ID)]
|
| 552 |
+
out_texts.append(sp_proc.decode_ids([x for x in row if x >= 11]).strip())
|
| 553 |
+
return out_texts
|
| 554 |
+
|
| 555 |
+
|
| 556 |
+
# ==============================================================================
|
| 557 |
+
# 2. 100%-Coverage Plan + Tight-Padding Dynamic Packing + Distributed Sharding
|
| 558 |
+
# ==============================================================================
|
| 559 |
+
class CanonicalMemmapStore:
|
| 560 |
+
def __init__(self, data_dir: str, split: str):
|
| 561 |
+
self.my_path = os.path.join(data_dir, f"{split}_my.bin")
|
| 562 |
+
self.en_path = os.path.join(data_dir, f"{split}_en.bin")
|
| 563 |
+
self.index = np.load(os.path.join(data_dir, f"{split}_index.npy"), mmap_mode="r")
|
| 564 |
+
self.my_mm = None
|
| 565 |
+
self.en_mm = None
|
| 566 |
+
|
| 567 |
+
def ensure_open(self):
|
| 568 |
+
if self.my_mm is None:
|
| 569 |
+
self.my_mm = np.memmap(self.my_path, dtype=np.uint16, mode="r")
|
| 570 |
+
self.en_mm = np.memmap(self.en_path, dtype=np.uint16, mode="r")
|
| 571 |
+
|
| 572 |
+
def get_pair(self, idx: int):
|
| 573 |
+
self.ensure_open()
|
| 574 |
+
rec = self.index[idx]
|
| 575 |
+
m = self.my_mm[int(rec["my_off"]) : int(rec["my_off"]) + int(rec["my_len"])].astype(np.int64)
|
| 576 |
+
e = self.en_mm[int(rec["en_off"]) : int(rec["en_off"]) + int(rec["en_len"])].astype(np.int64)
|
| 577 |
+
return m, e, int(rec["flags"])
|
| 578 |
+
|
| 579 |
+
|
| 580 |
+
def build_or_load_epoch_plan(train_index: np.ndarray, meta: dict, stage: str, plan_path: str, is_main: bool, world_size: int, seed: int = 2026) -> np.ndarray:
|
| 581 |
+
if is_main and not os.path.exists(plan_path):
|
| 582 |
+
rng = np.random.RandomState(seed)
|
| 583 |
+
n_subsets = len(meta["subset_to_id"])
|
| 584 |
+
sids = np.asarray(train_index["subset_id"])
|
| 585 |
+
assert int(sids.max()) < n_subsets and int(sids.min()) >= 0
|
| 586 |
+
|
| 587 |
+
if stage == "stable":
|
| 588 |
+
base_rows = rng.permutation(len(train_index)).astype(np.uint32)
|
| 589 |
+
replay_n = 12_000_000
|
| 590 |
+
eligible_mask = np.ones(len(train_index), dtype=bool)
|
| 591 |
+
for k, name in meta["id_to_subset"].items():
|
| 592 |
+
if name in BULK_SUBSETS:
|
| 593 |
+
eligible_mask[sids == int(k)] = False
|
| 594 |
+
else:
|
| 595 |
+
eligible_mask = (np.asarray(train_index["flags"]) & FLAG_BT) == 0
|
| 596 |
+
base_rows = np.flatnonzero(eligible_mask).astype(np.uint32)
|
| 597 |
+
rng.shuffle(base_rows)
|
| 598 |
+
replay_n = 1_500_000
|
| 599 |
+
|
| 600 |
+
elig_pos = np.flatnonzero(eligible_mask).astype(np.uint32)
|
| 601 |
+
elig_sids = sids[elig_pos]
|
| 602 |
+
order = elig_pos[np.argsort(elig_sids, kind="mergesort")]
|
| 603 |
+
sorted_sids = sids[order]
|
| 604 |
+
counts = np.bincount(sorted_sids, minlength=n_subsets)
|
| 605 |
+
starts = np.zeros(n_subsets, dtype=np.int64)
|
| 606 |
+
starts[1:] = np.cumsum(counts[:-1])
|
| 607 |
+
|
| 608 |
+
weights = np.zeros(n_subsets, dtype=np.float64)
|
| 609 |
+
for k, name in meta["id_to_subset"].items():
|
| 610 |
+
sid = int(k)
|
| 611 |
+
if counts[sid] == 0:
|
| 612 |
+
continue
|
| 613 |
+
if stage == "stable" and name in BULK_SUBSETS:
|
| 614 |
+
continue
|
| 615 |
+
if stage == "cooldown" and name == "non_myanmartranslator/finetranslations":
|
| 616 |
+
w = float(counts[sid]) ** 0.35 * 0.25
|
| 617 |
+
else:
|
| 618 |
+
w = float(counts[sid]) ** 0.50
|
| 619 |
+
if stage == "stable" and name.startswith("non_myanmartranslator/"):
|
| 620 |
+
w *= 4.0
|
| 621 |
+
weights[sid] = w
|
| 622 |
+
|
| 623 |
+
assert weights.sum() > 0, "Empty replay weights!"
|
| 624 |
+
probs = weights / weights.sum()
|
| 625 |
+
|
| 626 |
+
chosen_sids = rng.choice(n_subsets, size=replay_n, p=probs)
|
| 627 |
+
replay_rows = np.empty(replay_n, dtype=np.uint32)
|
| 628 |
+
for sid in range(n_subsets):
|
| 629 |
+
m = (chosen_sids == sid)
|
| 630 |
+
npick = int(m.sum())
|
| 631 |
+
if npick > 0:
|
| 632 |
+
replay_rows[m] = order[starts[sid] + rng.randint(0, counts[sid], size=npick)]
|
| 633 |
+
|
| 634 |
+
full_plan = np.concatenate([base_rows, replay_rows])
|
| 635 |
+
rng.shuffle(full_plan)
|
| 636 |
+
tmp_plan = plan_path + ".tmp.npy"
|
| 637 |
+
np.save(tmp_plan, full_plan)
|
| 638 |
+
os.replace(tmp_plan, plan_path)
|
| 639 |
+
print(
|
| 640 |
+
f"📋 Built {stage} 100%-coverage plan: {len(base_rows):,} unique base rows (100% coverage) "
|
| 641 |
+
f"+ {len(replay_rows):,} gold replay = {len(full_plan):,} total rows"
|
| 642 |
+
)
|
| 643 |
+
|
| 644 |
+
if world_size > 1:
|
| 645 |
+
dist.barrier()
|
| 646 |
+
plan = np.load(plan_path, mmap_mode="r")
|
| 647 |
+
if is_main:
|
| 648 |
+
print(f"📋 Ready {stage} plan: {len(plan):,} rows")
|
| 649 |
+
return plan
|
| 650 |
+
|
| 651 |
+
|
| 652 |
+
def pack_chunk_batches(
|
| 653 |
+
row_slice: np.ndarray,
|
| 654 |
+
train_index: np.ndarray,
|
| 655 |
+
max_tokens_combined: int,
|
| 656 |
+
max_seq_len: int,
|
| 657 |
+
step_group: int,
|
| 658 |
+
seed: int,
|
| 659 |
+
):
|
| 660 |
+
rng = np.random.RandomState(seed)
|
| 661 |
+
my_lens = np.minimum(max_seq_len, train_index["my_len"][row_slice].astype(np.int32))
|
| 662 |
+
en_lens = np.minimum(max_seq_len, train_index["en_len"][row_slice].astype(np.int32))
|
| 663 |
+
max_side = np.maximum(my_lens, en_lens)
|
| 664 |
+
comb_lens = my_lens + en_lens
|
| 665 |
+
|
| 666 |
+
mega = 65536
|
| 667 |
+
batches_cost = []
|
| 668 |
+
for i in range(0, len(row_slice), mega):
|
| 669 |
+
sub_r = row_slice[i : i + mega]
|
| 670 |
+
sub_ms = max_side[i : i + mega]
|
| 671 |
+
sub_cl = comb_lens[i : i + mega]
|
| 672 |
+
sub_r = sub_r[np.lexsort((sub_cl, sub_ms))]
|
| 673 |
+
|
| 674 |
+
cur_b, cur_s_max, cur_t_max = [], 0, 0
|
| 675 |
+
for r in sub_r:
|
| 676 |
+
ri = int(r)
|
| 677 |
+
ml = min(max_seq_len, int(train_index["my_len"][ri]))
|
| 678 |
+
el = min(max_seq_len, int(train_index["en_len"][ri]))
|
| 679 |
+
s_est = max(el, ml) + 4
|
| 680 |
+
t_est = max(ml, el) + 2
|
| 681 |
+
new_s = max(cur_s_max, s_est)
|
| 682 |
+
new_t = max(cur_t_max, t_est)
|
| 683 |
+
if cur_b and (len(cur_b) >= 256 or (len(cur_b) + 1) * (new_s + new_t) > max_tokens_combined):
|
| 684 |
+
batches_cost.append(((max(cur_s_max, cur_t_max), len(cur_b) * (cur_s_max + cur_t_max)), cur_b))
|
| 685 |
+
cur_b, cur_s_max, cur_t_max = [ri], s_est, t_est
|
| 686 |
+
else:
|
| 687 |
+
cur_b.append(ri)
|
| 688 |
+
cur_s_max, cur_t_max = new_s, new_t
|
| 689 |
+
if cur_b:
|
| 690 |
+
batches_cost.append(((max(cur_s_max, cur_t_max), len(cur_b) * (cur_s_max + cur_t_max)), cur_b))
|
| 691 |
+
|
| 692 |
+
# Load-balanced step grouping by (max_seq, tok_cost) + safe small-chunk pad
|
| 693 |
+
batches_cost.sort(key=lambda x: x[0])
|
| 694 |
+
batches = [b for _, b in batches_cost]
|
| 695 |
+
rem = len(batches) % step_group
|
| 696 |
+
if rem != 0 and len(batches) > 0:
|
| 697 |
+
need = step_group - rem
|
| 698 |
+
batches.extend((batches * ((need // len(batches)) + 1))[:need])
|
| 699 |
+
groups = [batches[j : j + step_group] for j in range(0, len(batches), step_group)]
|
| 700 |
+
rng.shuffle(groups)
|
| 701 |
+
return [b for g in groups for b in g]
|
| 702 |
+
|
| 703 |
+
|
| 704 |
+
class DynamicStreamDataset(Dataset):
|
| 705 |
+
def __init__(self, store: CanonicalMemmapStore, batches: list[list[int]], stage: str, max_seq_len: int, seed: int):
|
| 706 |
+
self.store = store
|
| 707 |
+
self.batches = batches
|
| 708 |
+
self.stage = stage
|
| 709 |
+
self.max_seq_len = max_seq_len
|
| 710 |
+
self.base_seed = seed
|
| 711 |
+
self.rng = np.random.RandomState(seed)
|
| 712 |
+
|
| 713 |
+
def __len__(self):
|
| 714 |
+
return len(self.batches)
|
| 715 |
+
|
| 716 |
+
def _corrupt_span(self, toks: np.ndarray):
|
| 717 |
+
n = len(toks)
|
| 718 |
+
if n < 4:
|
| 719 |
+
return toks
|
| 720 |
+
out = []
|
| 721 |
+
i = 0
|
| 722 |
+
masked_any = False
|
| 723 |
+
while i < n:
|
| 724 |
+
if self.rng.rand() < 0.15:
|
| 725 |
+
span_len = min(n - i, max(1, int(self.rng.poisson(3.0))))
|
| 726 |
+
out.append(TOK_MASK)
|
| 727 |
+
i += span_len
|
| 728 |
+
masked_any = True
|
| 729 |
+
else:
|
| 730 |
+
out.append(int(toks[i]))
|
| 731 |
+
i += 1
|
| 732 |
+
if not masked_any:
|
| 733 |
+
idx = int(self.rng.randint(0, n))
|
| 734 |
+
out = list(toks[:idx]) + [TOK_MASK] + list(toks[idx + min(2, n - idx):])
|
| 735 |
+
return np.asarray(out, dtype=np.int64)
|
| 736 |
+
|
| 737 |
+
def __getitem__(self, b_idx: int):
|
| 738 |
+
row_ids = self.batches[b_idx]
|
| 739 |
+
srcs, tgts = [], []
|
| 740 |
+
for r_id in row_ids:
|
| 741 |
+
m, e, flg = self.store.get_pair(r_id)
|
| 742 |
+
m = m[: self.max_seq_len - 4]
|
| 743 |
+
e = e[: self.max_seq_len - 4]
|
| 744 |
+
is_bt = bool(flg & FLAG_BT)
|
| 745 |
+
is_formal = bool(flg & FLAG_FORMAL)
|
| 746 |
+
is_spoken = bool(flg & FLAG_SPOKEN)
|
| 747 |
+
|
| 748 |
+
if self.stage == "cooldown":
|
| 749 |
+
mode = "en2my" if self.rng.rand() < 0.50 else "my2en"
|
| 750 |
+
else:
|
| 751 |
+
p = self.rng.rand()
|
| 752 |
+
if is_bt:
|
| 753 |
+
mode = "en2my" if p < 0.72 else ("my2en_bt" if p < 0.90 else "denoise")
|
| 754 |
+
else:
|
| 755 |
+
mode = "en2my" if p < 0.47 else ("my2en" if p < 0.94 else "denoise")
|
| 756 |
+
|
| 757 |
+
style_tok = None
|
| 758 |
+
if is_formal and self.rng.rand() < 0.70:
|
| 759 |
+
style_tok = TOK_FORMAL
|
| 760 |
+
elif is_spoken and self.rng.rand() < 0.70:
|
| 761 |
+
style_tok = TOK_SPOKEN
|
| 762 |
+
|
| 763 |
+
if mode == "en2my":
|
| 764 |
+
pref = [TOK_2MY] + ([style_tok] if style_tok is not None else [])
|
| 765 |
+
src_arr = np.concatenate([np.asarray(pref, dtype=np.int64), e, [EOS_ID]])
|
| 766 |
+
tgt_arr = np.concatenate([[BOS_ID], m, [EOS_ID]])
|
| 767 |
+
elif mode in ("my2en", "my2en_bt"):
|
| 768 |
+
pref = [TOK_2EN] + ([TOK_BT] if mode == "my2en_bt" else []) + ([style_tok] if style_tok is not None else [])
|
| 769 |
+
src_arr = np.concatenate([np.asarray(pref, dtype=np.int64), m, [EOS_ID]])
|
| 770 |
+
tgt_arr = np.concatenate([[BOS_ID], e, [EOS_ID]])
|
| 771 |
+
else:
|
| 772 |
+
base = m if (is_bt or self.rng.rand() < 0.5) else e
|
| 773 |
+
noisy = self._corrupt_span(base)
|
| 774 |
+
src_arr = np.concatenate([[TOK_DENOISE], noisy, [EOS_ID]])
|
| 775 |
+
tgt_arr = np.concatenate([[BOS_ID], base, [EOS_ID]])
|
| 776 |
+
|
| 777 |
+
srcs.append(src_arr)
|
| 778 |
+
tgts.append(tgt_arr)
|
| 779 |
+
return srcs, tgts
|
| 780 |
+
|
| 781 |
+
|
| 782 |
+
def worker_init_fn(worker_id: int):
|
| 783 |
+
info = torch.utils.data.get_worker_info()
|
| 784 |
+
if info is not None and hasattr(info.dataset, "rng"):
|
| 785 |
+
info.dataset.rng = np.random.RandomState((info.dataset.base_seed + worker_id * 9973) % (2**31 - 1))
|
| 786 |
+
|
| 787 |
+
|
| 788 |
+
def collate_prebatched(batch_list):
|
| 789 |
+
srcs, tgts = batch_list[0]
|
| 790 |
+
B = len(srcs)
|
| 791 |
+
max_s = min(2048, ((max(len(x) for x in srcs) + 31) // 32) * 32)
|
| 792 |
+
max_t = min(2048, ((max(len(x) - 1 for x in tgts) + 31) // 32) * 32)
|
| 793 |
+
src_ids = torch.full((B, max_s), PAD_ID, dtype=torch.long)
|
| 794 |
+
tgt_in = torch.full((B, max_t), PAD_ID, dtype=torch.long)
|
| 795 |
+
labels = torch.full((B, max_t), -100, dtype=torch.long)
|
| 796 |
+
for i, (s, t) in enumerate(zip(srcs, tgts)):
|
| 797 |
+
sl = min(len(s), max_s)
|
| 798 |
+
tl = min(len(t) - 1, max_t)
|
| 799 |
+
src_ids[i, :sl] = torch.from_numpy(s[:sl])
|
| 800 |
+
tgt_in[i, :tl] = torch.from_numpy(t[:tl])
|
| 801 |
+
labels[i, :tl] = torch.from_numpy(t[1 : tl + 1])
|
| 802 |
+
return src_ids, tgt_in, labels
|
| 803 |
+
|
| 804 |
+
|
| 805 |
+
def clean_state_dict(model: nn.Module):
|
| 806 |
+
return {k.replace("._orig_mod.", "."): v.detach().cpu() for k, v in model.state_dict().items()}
|
| 807 |
+
|
| 808 |
+
|
| 809 |
+
def broadcast_model(model: nn.Module, world_size: int):
|
| 810 |
+
if world_size == 1:
|
| 811 |
+
return
|
| 812 |
+
for p in model.parameters():
|
| 813 |
+
dist.broadcast(p.data, src=0)
|
| 814 |
+
for b in model.buffers():
|
| 815 |
+
dist.broadcast(b.data, src=0)
|
| 816 |
+
|
| 817 |
+
|
| 818 |
+
def allreduce_grads(model: nn.Module, world_size: int, grad_scale: float = 1024.0):
|
| 819 |
+
if world_size == 1:
|
| 820 |
+
return
|
| 821 |
+
grads = []
|
| 822 |
+
for p in model.parameters():
|
| 823 |
+
if p.requires_grad:
|
| 824 |
+
if p.grad is None:
|
| 825 |
+
p.grad = torch.zeros_like(p.data)
|
| 826 |
+
grads.append(p.grad)
|
| 827 |
+
b_grads = [g.bfloat16() for g in grads]
|
| 828 |
+
flat = torch._utils._flatten_dense_tensors(b_grads)
|
| 829 |
+
dist.all_reduce(flat, op=dist.ReduceOp.SUM)
|
| 830 |
+
inv_scale = 1.0 / grad_scale
|
| 831 |
+
for orig, synced in zip(grads, torch._utils._unflatten_dense_tensors(flat, b_grads)):
|
| 832 |
+
orig.copy_(synced.float() * inv_scale)
|
| 833 |
+
|
| 834 |
+
|
| 835 |
+
# ==============================================================================
|
| 836 |
+
# 3. Main Multi-GPU Training Loop
|
| 837 |
+
# ==============================================================================
|
| 838 |
+
def main():
|
| 839 |
+
parser = argparse.ArgumentParser()
|
| 840 |
+
parser.add_argument("--stage", choices=["stable", "cooldown"], default="stable")
|
| 841 |
+
parser.add_argument("--verify-steps", type=int, default=0)
|
| 842 |
+
parser.add_argument("--grad-accum", type=int, default=1)
|
| 843 |
+
parser.add_argument("--max-tok-micro", type=int, default=32768)
|
| 844 |
+
parser.add_argument("--max-seq-len", type=int, default=1024)
|
| 845 |
+
parser.add_argument("--chunk-rows", type=int, default=4000000)
|
| 846 |
+
parser.add_argument("--peak-lr", type=float, default=5.0e-4)
|
| 847 |
+
parser.add_argument("--warmup-steps", type=int, default=900)
|
| 848 |
+
parser.add_argument("--eval-every", type=int, default=500)
|
| 849 |
+
parser.add_argument("--save-every", type=int, default=500)
|
| 850 |
+
parser.add_argument("--milestone-every", type=int, default=5000)
|
| 851 |
+
parser.add_argument("--compile", action="store_true")
|
| 852 |
+
parser.add_argument("--fresh", action="store_true")
|
| 853 |
+
parser.add_argument("--init-weights", type=str, default="")
|
| 854 |
+
args = parser.parse_args()
|
| 855 |
+
|
| 856 |
+
world_size = int(os.environ.get("WORLD_SIZE", 1))
|
| 857 |
+
rank = int(os.environ.get("RANK", 0))
|
| 858 |
+
local_rank = int(os.environ.get("LOCAL_RANK", 0))
|
| 859 |
+
is_main = (rank == 0)
|
| 860 |
+
|
| 861 |
+
if world_size > 1:
|
| 862 |
+
torch.cuda.set_device(local_rank)
|
| 863 |
+
dist.init_process_group(backend="nccl", timeout=timedelta(minutes=30), device_id=torch.device(f"cuda:{local_rank}"))
|
| 864 |
+
device = torch.device("cuda", local_rank)
|
| 865 |
+
else:
|
| 866 |
+
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 867 |
+
|
| 868 |
+
if is_main:
|
| 869 |
+
os.makedirs(CKPT_DIR, exist_ok=True)
|
| 870 |
+
if world_size > 1:
|
| 871 |
+
dist.barrier()
|
| 872 |
+
|
| 873 |
+
meta, sp = load_and_verify_contracts(DATA_DIR)
|
| 874 |
+
train_store = CanonicalMemmapStore(DATA_DIR, "train")
|
| 875 |
+
val_store = CanonicalMemmapStore(DATA_DIR, "val")
|
| 876 |
+
|
| 877 |
+
all_val_non_bt = np.flatnonzero((val_store.index["flags"] & FLAG_BT) == 0)
|
| 878 |
+
assert len(all_val_non_bt) > 0, "Validation set has zero non-BT rows!"
|
| 879 |
+
val_rows = all_val_non_bt[::max(1, len(all_val_non_bt) // 1024)][:1024]
|
| 880 |
+
val_batches = [val_rows[i : i + 64].tolist() for i in range(0, len(val_rows), 64)]
|
| 881 |
+
|
| 882 |
+
plan_path = os.path.join(CKPT_DIR, f"{args.stage}_plan.npy")
|
| 883 |
+
latest_path_early = os.path.join(CKPT_DIR, "latest.pt")
|
| 884 |
+
if args.fresh and is_main:
|
| 885 |
+
if os.path.exists(plan_path):
|
| 886 |
+
os.remove(plan_path)
|
| 887 |
+
if os.path.exists(latest_path_early):
|
| 888 |
+
os.replace(latest_path_early, latest_path_early + ".bak")
|
| 889 |
+
if world_size > 1:
|
| 890 |
+
dist.barrier()
|
| 891 |
+
full_plan = build_or_load_epoch_plan(train_store.index, meta, args.stage, plan_path, is_main, world_size, seed=2026)
|
| 892 |
+
num_chunks = (len(full_plan) + args.chunk_rows - 1) // args.chunk_rows
|
| 893 |
+
|
| 894 |
+
model = MyTranslation300M().to(device)
|
| 895 |
+
decay_p = [p for n, p in model.named_parameters() if p.requires_grad and p.ndim >= 2]
|
| 896 |
+
nodecay_p = [p for n, p in model.named_parameters() if p.requires_grad and p.ndim < 2]
|
| 897 |
+
opt = torch.optim.AdamW(
|
| 898 |
+
[{"params": decay_p, "weight_decay": 0.01}, {"params": nodecay_p, "weight_decay": 0.0}],
|
| 899 |
+
lr=args.peak_lr, betas=(0.9, 0.95), eps=1e-8, fused=True,
|
| 900 |
+
)
|
| 901 |
+
|
| 902 |
+
step = 0
|
| 903 |
+
chunk_idx = 0
|
| 904 |
+
batch_cursor = 0
|
| 905 |
+
rows_seen = 0
|
| 906 |
+
best_val_nll = float("inf")
|
| 907 |
+
latest_path = os.path.join(CKPT_DIR, "latest.pt")
|
| 908 |
+
|
| 909 |
+
if args.init_weights:
|
| 910 |
+
if is_main:
|
| 911 |
+
print(f"📥 Loading weights from {args.init_weights}")
|
| 912 |
+
model.load_state_dict(torch.load(args.init_weights, map_location="cpu"), strict=True)
|
| 913 |
+
elif (not args.fresh) and os.path.exists(latest_path):
|
| 914 |
+
if is_main:
|
| 915 |
+
print(f"🔄 Resuming from {latest_path}")
|
| 916 |
+
ckpt = torch.load(latest_path, map_location="cpu")
|
| 917 |
+
model.load_state_dict(ckpt["model"], strict=True)
|
| 918 |
+
if args.stage == ckpt.get("stage", "stable"):
|
| 919 |
+
opt.load_state_dict(ckpt["opt"])
|
| 920 |
+
step = int(ckpt["step"])
|
| 921 |
+
chunk_idx = int(ckpt.get("chunk_idx", 0))
|
| 922 |
+
batch_cursor = int(ckpt.get("batch_cursor", 0))
|
| 923 |
+
rows_seen = int(ckpt.get("rows_seen", 0))
|
| 924 |
+
best_val_nll = float(ckpt.get("best_val_nll", float("inf")))
|
| 925 |
+
elif is_main:
|
| 926 |
+
print(f"➡️ Stage transition {ckpt.get('stage')} -> {args.stage}")
|
| 927 |
+
|
| 928 |
+
broadcast_model(model, world_size)
|
| 929 |
+
|
| 930 |
+
if args.compile:
|
| 931 |
+
if is_main:
|
| 932 |
+
print("⚡ Compiling encoder/cross_kv/decoder/MTP blocks with torch.compile(dynamic=True)...")
|
| 933 |
+
model.compile_blocks()
|
| 934 |
+
|
| 935 |
+
if is_main:
|
| 936 |
+
tot_params = sum(p.numel() for p in model.parameters())
|
| 937 |
+
inf_params = tot_params - sum(p.numel() for p in model.mtp_head.parameters())
|
| 938 |
+
print(f"🏛️ MyTranslation-300M | GPUs: {world_size}x | Stage: {args.stage} | Inf: {inf_params/1e6:.2f}M | Train: {tot_params/1e6:.2f}M")
|
| 939 |
+
|
| 940 |
+
label_smoothing = 0.1 if args.stage == "stable" else 0.0
|
| 941 |
+
mtp_weight = 0.3 if args.stage == "stable" else 0.15
|
| 942 |
+
|
| 943 |
+
STYLE_EVAL_PROMPTS = [
|
| 944 |
+
("En->My Formal", [TOK_2MY, TOK_FORMAL], "The central bank raised interest rates to control inflation and stabilize the economy."),
|
| 945 |
+
("En->My Spoken", [TOK_2MY, TOK_SPOKEN], "Honestly, I didn't expect things to turn out like this, but don't worry—we'll figure it out together."),
|
| 946 |
+
("My->En Formal", [TOK_2EN, TOK_FORMAL], "ဗဟိုဘဏ်သည် ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးတည်ငြိမ်စေရန် အတိုးနှုန်းများကို မြှင့်တင်ခဲ့သည်။"),
|
| 947 |
+
("My->En Spoken", [TOK_2EN, TOK_SPOKEN], "ဒီနေ့ ရန်ကုန်မှာ မိုးတွေ အရမ်းရွာနေလို့ အိမ်ပြန်ရတာ နည်းနည်းနောက်ကျမယ်နော်။"),
|
| 948 |
+
]
|
| 949 |
+
|
| 950 |
+
@torch.no_grad()
|
| 951 |
+
def eval_val_nll():
|
| 952 |
+
model.eval()
|
| 953 |
+
en2my_nll, en2my_tok = 0.0, 0
|
| 954 |
+
my2en_nll, my2en_tok = 0.0, 0
|
| 955 |
+
for b in val_batches:
|
| 956 |
+
for direction in ("en2my", "my2en"):
|
| 957 |
+
srcs, tgts = [], []
|
| 958 |
+
for r_id in b:
|
| 959 |
+
m, e, _ = val_store.get_pair(int(r_id))
|
| 960 |
+
m, e = m[: args.max_seq_len - 4], e[: args.max_seq_len - 4]
|
| 961 |
+
if direction == "en2my":
|
| 962 |
+
srcs.append(np.concatenate([[TOK_2MY], e, [EOS_ID]]))
|
| 963 |
+
tgts.append(np.concatenate([[BOS_ID], m, [EOS_ID]]))
|
| 964 |
+
else:
|
| 965 |
+
srcs.append(np.concatenate([[TOK_2EN], m, [EOS_ID]]))
|
| 966 |
+
tgts.append(np.concatenate([[BOS_ID], e, [EOS_ID]]))
|
| 967 |
+
src_ids, tgt_in, labels = collate_prebatched([(srcs, tgts)])
|
| 968 |
+
nll, nv = model.eval_nll(src_ids.to(device), tgt_in.to(device), labels.to(device))
|
| 969 |
+
if direction == "en2my":
|
| 970 |
+
en2my_nll += nll
|
| 971 |
+
en2my_tok += nv
|
| 972 |
+
else:
|
| 973 |
+
my2en_nll += nll
|
| 974 |
+
my2en_tok += nv
|
| 975 |
+
e2m = en2my_nll / max(1, en2my_tok)
|
| 976 |
+
m2e = my2en_nll / max(1, my2en_tok)
|
| 977 |
+
comb = (en2my_nll + my2en_nll) / max(1, en2my_tok + my2en_tok)
|
| 978 |
+
return comb, e2m, m2e
|
| 979 |
+
|
| 980 |
+
def run_style_generations(step_num: int):
|
| 981 |
+
prompts = [pref + sp.encode_as_ids(txt)[:500] + [EOS_ID] for _, pref, txt in STYLE_EVAL_PROMPTS]
|
| 982 |
+
preds = model.translate_kv(prompts, sp, max_new_tokens=96)
|
| 983 |
+
print(f" 🗣️ Style Generations @ Step {step_num}:")
|
| 984 |
+
for (tag, _, _), out in zip(STYLE_EVAL_PROMPTS, preds):
|
| 985 |
+
print(f" [{tag}]: {out[:180]}")
|
| 986 |
+
|
| 987 |
+
def make_chunk_loader(c_idx: int, skip_steps: int = 0):
|
| 988 |
+
s_off = c_idx * args.chunk_rows
|
| 989 |
+
e_off = min(len(full_plan), s_off + args.chunk_rows)
|
| 990 |
+
row_slice = np.asarray(full_plan[s_off:e_off])
|
| 991 |
+
step_group = world_size * args.grad_accum
|
| 992 |
+
all_batches = pack_chunk_batches(
|
| 993 |
+
row_slice, train_store.index,
|
| 994 |
+
max_tokens_combined=args.max_tok_micro,
|
| 995 |
+
max_seq_len=args.max_seq_len,
|
| 996 |
+
step_group=step_group,
|
| 997 |
+
seed=2026 + c_idx * 17,
|
| 998 |
+
)
|
| 999 |
+
rank_batches = all_batches[rank::world_size]
|
| 1000 |
+
if skip_steps > 0:
|
| 1001 |
+
rank_batches = rank_batches[skip_steps * args.grad_accum :]
|
| 1002 |
+
n_windows = len(rank_batches) // args.grad_accum
|
| 1003 |
+
rank_batches = rank_batches[: n_windows * args.grad_accum]
|
| 1004 |
+
ds = DynamicStreamDataset(
|
| 1005 |
+
train_store, rank_batches, args.stage, args.max_seq_len,
|
| 1006 |
+
seed=10000 + c_idx * 997 + rank * 131 + skip_steps,
|
| 1007 |
+
)
|
| 1008 |
+
loader = DataLoader(
|
| 1009 |
+
ds, batch_size=1, shuffle=False, collate_fn=collate_prebatched,
|
| 1010 |
+
num_workers=4, pin_memory=True, worker_init_fn=worker_init_fn, persistent_workers=False,
|
| 1011 |
+
)
|
| 1012 |
+
return loader, n_windows
|
| 1013 |
+
|
| 1014 |
+
if is_main:
|
| 1015 |
+
comb, e2m, m2e = eval_val_nll()
|
| 1016 |
+
print(f"🔎 Step {step:6d} | Val Comb: {comb:.4f} | En->My: {e2m:.4f} | My->En: {m2e:.4f}")
|
| 1017 |
+
if world_size > 1:
|
| 1018 |
+
dist.barrier()
|
| 1019 |
+
|
| 1020 |
+
model.train()
|
| 1021 |
+
opt.zero_grad(set_to_none=True)
|
| 1022 |
+
acc_ce = acc_mtp = acc_z = 0.0
|
| 1023 |
+
acc_valid_tok = acc_mtp_tok = acc_tot_tok = 0
|
| 1024 |
+
t0 = time.time()
|
| 1025 |
+
|
| 1026 |
+
while chunk_idx < num_chunks:
|
| 1027 |
+
train_loader, n_windows = make_chunk_loader(chunk_idx, skip_steps=batch_cursor)
|
| 1028 |
+
train_iter = iter(train_loader)
|
| 1029 |
+
done_flag = False
|
| 1030 |
+
|
| 1031 |
+
for _ in range(n_windows):
|
| 1032 |
+
window = [next(train_iter) for _ in range(args.grad_accum)]
|
| 1033 |
+
batch_cursor += 1
|
| 1034 |
+
|
| 1035 |
+
loc_valid = sum(int((lb != -100).sum().item()) for _, _, lb in window)
|
| 1036 |
+
loc_mtp = sum(
|
| 1037 |
+
int(((lb[:, 1:] != -100) & (lb[:, :-1] != -100)).sum().item()) if lb.size(1) > 1 else 0
|
| 1038 |
+
for _, _, lb in window
|
| 1039 |
+
)
|
| 1040 |
+
loc_rows = sum(s.size(0) for s, _, _ in window)
|
| 1041 |
+
|
| 1042 |
+
if world_size > 1:
|
| 1043 |
+
counts_t = torch.tensor([loc_valid, loc_mtp, loc_rows], dtype=torch.long, device=device)
|
| 1044 |
+
dist.all_reduce(counts_t, op=dist.ReduceOp.SUM)
|
| 1045 |
+
glob_valid, glob_mtp, glob_rows = int(counts_t[0].item()), int(counts_t[1].item()), int(counts_t[2].item())
|
| 1046 |
+
else:
|
| 1047 |
+
glob_valid, glob_mtp, glob_rows = loc_valid, loc_mtp, loc_rows
|
| 1048 |
+
|
| 1049 |
+
g_scale = 1024.0 if world_size > 1 else 1.0
|
| 1050 |
+
main_denom = float(max(1, glob_valid)) / g_scale
|
| 1051 |
+
mtp_denom = float(max(1, glob_mtp)) / g_scale
|
| 1052 |
+
|
| 1053 |
+
loc_ce = loc_mtp_ce = loc_z = 0.0
|
| 1054 |
+
loc_tok = sum(int((s != PAD_ID).sum().item()) for s, _, _ in window)
|
| 1055 |
+
for src_ids, tgt_in, labels in window:
|
| 1056 |
+
src_ids = src_ids.to(device, non_blocking=True)
|
| 1057 |
+
tgt_in = tgt_in.to(device, non_blocking=True)
|
| 1058 |
+
labels = labels.to(device, non_blocking=True)
|
| 1059 |
+
ce_s, mtp_s, z_s, nv, nm = model.forward_and_backward(
|
| 1060 |
+
src_ids, tgt_in, labels,
|
| 1061 |
+
main_denom=main_denom, mtp_denom=mtp_denom,
|
| 1062 |
+
label_smoothing=label_smoothing, mtp_weight=mtp_weight,
|
| 1063 |
+
)
|
| 1064 |
+
loc_ce += ce_s
|
| 1065 |
+
loc_mtp_ce += mtp_s
|
| 1066 |
+
loc_z += z_s
|
| 1067 |
+
loc_tok += nv
|
| 1068 |
+
|
| 1069 |
+
allreduce_grads(model, world_size)
|
| 1070 |
+
rows_seen += glob_rows
|
| 1071 |
+
|
| 1072 |
+
if args.stage == "stable":
|
| 1073 |
+
lr = args.peak_lr * min(1.0, (step + 1) / max(1, args.warmup_steps))
|
| 1074 |
+
else:
|
| 1075 |
+
if step < 100:
|
| 1076 |
+
lr = args.peak_lr * (step + 1) / 100.0
|
| 1077 |
+
else:
|
| 1078 |
+
progress = min(1.0, rows_seen / max(1, len(full_plan)))
|
| 1079 |
+
lr = 1.0e-5 + 0.5 * (args.peak_lr - 1.0e-5) * (1.0 + math.cos(math.pi * progress))
|
| 1080 |
+
|
| 1081 |
+
for pg in opt.param_groups:
|
| 1082 |
+
pg["lr"] = lr
|
| 1083 |
+
grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
|
| 1084 |
+
opt.step()
|
| 1085 |
+
opt.zero_grad(set_to_none=True)
|
| 1086 |
+
step += 1
|
| 1087 |
+
|
| 1088 |
+
if world_size > 1:
|
| 1089 |
+
stats_t = torch.tensor([loc_ce, loc_mtp_ce, loc_z, float(loc_tok)], dtype=torch.float64, device=device)
|
| 1090 |
+
dist.all_reduce(stats_t, op=dist.ReduceOp.SUM)
|
| 1091 |
+
acc_ce += float(stats_t[0].item())
|
| 1092 |
+
acc_mtp += float(stats_t[1].item())
|
| 1093 |
+
acc_z += float(stats_t[2].item())
|
| 1094 |
+
acc_tot_tok += int(stats_t[3].item())
|
| 1095 |
+
else:
|
| 1096 |
+
acc_ce += loc_ce
|
| 1097 |
+
acc_mtp += loc_mtp_ce
|
| 1098 |
+
acc_z += loc_z
|
| 1099 |
+
acc_tot_tok += loc_tok
|
| 1100 |
+
acc_valid_tok += glob_valid
|
| 1101 |
+
acc_mtp_tok += glob_mtp
|
| 1102 |
+
|
| 1103 |
+
log_int = 10 if args.verify_steps > 0 else 50
|
| 1104 |
+
if step % log_int == 0:
|
| 1105 |
+
if is_main:
|
| 1106 |
+
dt = time.time() - t0
|
| 1107 |
+
tok_s = acc_tot_tok / max(1e-3, dt)
|
| 1108 |
+
vram_g = torch.cuda.max_memory_allocated(device) / 1e9
|
| 1109 |
+
pct = 100.0 * min(rows_seen, len(full_plan)) / len(full_plan)
|
| 1110 |
+
print(
|
| 1111 |
+
f"Step {step:6d} ({pct:5.1f}% | Chunk {chunk_idx+1}/{num_chunks}) | "
|
| 1112 |
+
f"Main: {acc_ce/max(1,acc_valid_tok):.4f} | MTP2: {acc_mtp/max(1,acc_mtp_tok):.4f} | "
|
| 1113 |
+
f"Z: {acc_z/max(1,acc_valid_tok):.2f} | LR: {lr:.2e} | Grad: {grad_norm:.2f} | "
|
| 1114 |
+
f"{tok_s/1e3:.1f}k tok/s | VRAM: {vram_g:.1f}G"
|
| 1115 |
+
)
|
| 1116 |
+
acc_ce = acc_mtp = acc_z = 0.0
|
| 1117 |
+
acc_valid_tok = acc_mtp_tok = acc_tot_tok = 0
|
| 1118 |
+
t0 = time.time()
|
| 1119 |
+
|
| 1120 |
+
if args.verify_steps > 0 and step >= args.verify_steps:
|
| 1121 |
+
done_flag = True
|
| 1122 |
+
break
|
| 1123 |
+
|
| 1124 |
+
if step % args.eval_every == 0:
|
| 1125 |
+
if is_main:
|
| 1126 |
+
comb, e2m, m2e = eval_val_nll()
|
| 1127 |
+
print("-" * 96)
|
| 1128 |
+
print(f"🔎 EVAL @ Step {step:6d} | Comb NLL: {comb:.4f} (PPL {math.exp(min(comb, 30.0)):.2f}) | En->My: {e2m:.4f} | My->En: {m2e:.4f}")
|
| 1129 |
+
run_style_generations(step)
|
| 1130 |
+
if comb < best_val_nll:
|
| 1131 |
+
best_val_nll = comb
|
| 1132 |
+
best_sd = {k: v.to(torch.bfloat16) for k, v in clean_state_dict(model).items()}
|
| 1133 |
+
torch.save(best_sd, os.path.join(CKPT_DIR, "best_model.pt"))
|
| 1134 |
+
print(f" ⭐ New Best Model saved! Comb NLL: {best_val_nll:.4f}")
|
| 1135 |
+
print("-" * 96)
|
| 1136 |
+
model.train()
|
| 1137 |
+
if world_size > 1:
|
| 1138 |
+
dist.barrier()
|
| 1139 |
+
t0 = time.time()
|
| 1140 |
+
|
| 1141 |
+
if step % args.save_every == 0:
|
| 1142 |
+
if is_main:
|
| 1143 |
+
tmp_p = latest_path + ".tmp"
|
| 1144 |
+
torch.save({
|
| 1145 |
+
"model": clean_state_dict(model),
|
| 1146 |
+
"opt": opt.state_dict(),
|
| 1147 |
+
"step": step,
|
| 1148 |
+
"stage": args.stage,
|
| 1149 |
+
"chunk_idx": chunk_idx,
|
| 1150 |
+
"batch_cursor": batch_cursor,
|
| 1151 |
+
"rows_seen": rows_seen,
|
| 1152 |
+
"best_val_nll": best_val_nll,
|
| 1153 |
+
}, tmp_p)
|
| 1154 |
+
os.replace(tmp_p, latest_path)
|
| 1155 |
+
if world_size > 1:
|
| 1156 |
+
dist.barrier()
|
| 1157 |
+
|
| 1158 |
+
if step % args.milestone_every == 0:
|
| 1159 |
+
if is_main:
|
| 1160 |
+
ms_sd = {k: v.to(torch.bfloat16) for k, v in clean_state_dict(model).items()}
|
| 1161 |
+
torch.save(ms_sd, os.path.join(CKPT_DIR, f"{args.stage}_step_{step:06d}.pt"))
|
| 1162 |
+
if world_size > 1:
|
| 1163 |
+
dist.barrier()
|
| 1164 |
+
|
| 1165 |
+
if done_flag:
|
| 1166 |
+
break
|
| 1167 |
+
chunk_idx += 1
|
| 1168 |
+
batch_cursor = 0
|
| 1169 |
+
|
| 1170 |
+
if args.verify_steps > 0:
|
| 1171 |
+
if is_main:
|
| 1172 |
+
print(f"✅ {world_size}-GPU VERIFY PASSED ({step} steps)! Ready for launch.")
|
| 1173 |
+
if world_size > 1:
|
| 1174 |
+
dist.barrier()
|
| 1175 |
+
dist.destroy_process_group()
|
| 1176 |
+
return
|
| 1177 |
+
|
| 1178 |
+
if is_main:
|
| 1179 |
+
comb, e2m, m2e = eval_val_nll()
|
| 1180 |
+
print("-" * 96)
|
| 1181 |
+
print(f"🏁 FINAL EVAL @ Step {step:6d} | Comb NLL: {comb:.4f} (PPL {math.exp(min(comb, 30.0)):.2f}) | En->My: {e2m:.4f} | My->En: {m2e:.4f}")
|
| 1182 |
+
run_style_generations(step)
|
| 1183 |
+
if comb < best_val_nll:
|
| 1184 |
+
best_val_nll = comb
|
| 1185 |
+
best_sd = {k: v.to(torch.bfloat16) for k, v in clean_state_dict(model).items()}
|
| 1186 |
+
torch.save(best_sd, os.path.join(CKPT_DIR, "best_model.pt"))
|
| 1187 |
+
tmp_p = latest_path + ".tmp"
|
| 1188 |
+
torch.save({
|
| 1189 |
+
"model": clean_state_dict(model),
|
| 1190 |
+
"opt": opt.state_dict(),
|
| 1191 |
+
"step": step,
|
| 1192 |
+
"stage": args.stage,
|
| 1193 |
+
"chunk_idx": chunk_idx,
|
| 1194 |
+
"batch_cursor": batch_cursor,
|
| 1195 |
+
"rows_seen": rows_seen,
|
| 1196 |
+
"best_val_nll": best_val_nll,
|
| 1197 |
+
}, tmp_p)
|
| 1198 |
+
os.replace(tmp_p, latest_path)
|
| 1199 |
+
|
| 1200 |
+
with torch.no_grad():
|
| 1201 |
+
rw = F.softmax(model.enc_router_logits, dim=-1).cpu().numpy()
|
| 1202 |
+
for p in range(model.n_cross_pairs):
|
| 1203 |
+
top_layers = np.argsort(rw[p])[::-1][:4].tolist()
|
| 1204 |
+
print(f" • Cross-KV Pair {p} top encoder mixture weights: {top_layers} ({[round(float(rw[p][k]), 3) for k in top_layers]})")
|
| 1205 |
+
|
| 1206 |
+
print(f"🎉 Stage '{args.stage}' 100% COMPLETE ({rows_seen:,} rows trained across {step:,} steps)!")
|
| 1207 |
+
|
| 1208 |
+
if world_size > 1:
|
| 1209 |
+
dist.barrier()
|
| 1210 |
+
dist.destroy_process_group()
|
| 1211 |
+
|
| 1212 |
+
|
| 1213 |
+
if __name__ == "__main__":
|
| 1214 |
+
main()
|
checkpoint-stage2-step-6793/training_state.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4508a7fcc2c1bf57e6d1ed1bfe89eba8826af3cb740665f9a8d0a6bf0ca2a22b
|
| 3 |
+
size 760180633
|
checkpoint-stage2-step-6793/trainingcelloutputwitheval.txt
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
config.json
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"MyTranslationForConditionalGeneration"
|
| 4 |
+
],
|
| 5 |
+
"auto_map": {
|
| 6 |
+
"AutoConfig": "configuration_mytranslation.MyTranslationConfig",
|
| 7 |
+
"AutoModelForSeq2SeqLM": "modeling_mytranslation.MyTranslationForConditionalGeneration"
|
| 8 |
+
},
|
| 9 |
+
"model_type": "mytranslation",
|
| 10 |
+
"is_encoder_decoder": true,
|
| 11 |
+
"tie_word_embeddings": true,
|
| 12 |
+
"vocab_size": 65536,
|
| 13 |
+
"d_model": 1024,
|
| 14 |
+
"n_heads": 16,
|
| 15 |
+
"n_kv_heads": 4,
|
| 16 |
+
"enc_layers": 12,
|
| 17 |
+
"dec_layers": 6,
|
| 18 |
+
"ffn_dim": 3072,
|
| 19 |
+
"max_seq_len": 2048,
|
| 20 |
+
"rope_theta": 50000.0,
|
| 21 |
+
"unk_token_id": 0,
|
| 22 |
+
"bos_token_id": 1,
|
| 23 |
+
"eos_token_id": 2,
|
| 24 |
+
"pad_token_id": 3,
|
| 25 |
+
"stage1_final_val_nll": 1.4316,
|
| 26 |
+
"stage1_final_val_ppl": 4.19,
|
| 27 |
+
"val_nll": 1.164,
|
| 28 |
+
"val_ppl": 3.2,
|
| 29 |
+
"en2my_val_nll": 1.4829,
|
| 30 |
+
"my2en_val_nll": 0.8901,
|
| 31 |
+
"torch_dtype": "bfloat16"
|
| 32 |
+
}
|
configuration_mytranslation.py
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from transformers import PretrainedConfig
|
| 2 |
+
|
| 3 |
+
class MyTranslationConfig(PretrainedConfig):
|
| 4 |
+
model_type = "mytranslation"
|
| 5 |
+
|
| 6 |
+
def __init__(
|
| 7 |
+
self,
|
| 8 |
+
vocab_size=65536,
|
| 9 |
+
d_model=1024,
|
| 10 |
+
n_heads=16,
|
| 11 |
+
n_kv_heads=4,
|
| 12 |
+
enc_layers=12,
|
| 13 |
+
dec_layers=6,
|
| 14 |
+
ffn_dim=3072,
|
| 15 |
+
max_seq_len=2048,
|
| 16 |
+
rope_theta=50000.0,
|
| 17 |
+
unk_token_id=0,
|
| 18 |
+
bos_token_id=1,
|
| 19 |
+
eos_token_id=2,
|
| 20 |
+
pad_token_id=3,
|
| 21 |
+
tie_word_embeddings=True,
|
| 22 |
+
is_encoder_decoder=True,
|
| 23 |
+
**kwargs,
|
| 24 |
+
):
|
| 25 |
+
self.vocab_size = vocab_size
|
| 26 |
+
self.d_model = d_model
|
| 27 |
+
self.n_heads = n_heads
|
| 28 |
+
self.n_kv_heads = n_kv_heads
|
| 29 |
+
self.enc_layers = enc_layers
|
| 30 |
+
self.dec_layers = dec_layers
|
| 31 |
+
self.ffn_dim = ffn_dim
|
| 32 |
+
self.max_seq_len = max_seq_len
|
| 33 |
+
self.rope_theta = rope_theta
|
| 34 |
+
super().__init__(
|
| 35 |
+
unk_token_id=unk_token_id,
|
| 36 |
+
bos_token_id=bos_token_id,
|
| 37 |
+
eos_token_id=eos_token_id,
|
| 38 |
+
pad_token_id=pad_token_id,
|
| 39 |
+
tie_word_embeddings=tie_word_embeddings,
|
| 40 |
+
is_encoder_decoder=is_encoder_decoder,
|
| 41 |
+
**kwargs,
|
| 42 |
+
)
|
generation_config.json
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token_id": 1,
|
| 3 |
+
"eos_token_id": 2,
|
| 4 |
+
"pad_token_id": 3,
|
| 5 |
+
"decoder_start_token_id": 1,
|
| 6 |
+
"max_new_tokens": 192,
|
| 7 |
+
"num_beams": 4,
|
| 8 |
+
"length_penalty": 0.85,
|
| 9 |
+
"repetition_penalty": 1.12
|
| 10 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:58cf818fcf629f0c9ab4d5981a7a3ceadfbcb3a28af5141e8859ebd3102b4da3
|
| 3 |
+
size 597551858
|
modeling_mytranslation.py
ADDED
|
@@ -0,0 +1,372 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import math
|
| 2 |
+
import torch
|
| 3 |
+
import torch.nn as nn
|
| 4 |
+
import torch.nn.functional as F
|
| 5 |
+
import transformers
|
| 6 |
+
from transformers import PreTrainedModel
|
| 7 |
+
from transformers.modeling_outputs import Seq2SeqLMOutput
|
| 8 |
+
from .configuration_mytranslation import MyTranslationConfig
|
| 9 |
+
|
| 10 |
+
PAD_ID = 3
|
| 11 |
+
BOS_ID = 1
|
| 12 |
+
EOS_ID = 2
|
| 13 |
+
TOK_2MY, TOK_2EN, TOK_FORMAL, TOK_SPOKEN = 4, 5, 6, 7
|
| 14 |
+
_IS_TF5 = int(transformers.__version__.split(".")[0]) >= 5
|
| 15 |
+
|
| 16 |
+
def _sdpa(q, k, v, attn_mask=None, is_causal=False):
|
| 17 |
+
if q.size(1) != k.size(1):
|
| 18 |
+
try:
|
| 19 |
+
return F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask, is_causal=is_causal, enable_gqa=True)
|
| 20 |
+
except TypeError:
|
| 21 |
+
rep = q.size(1) // k.size(1)
|
| 22 |
+
k = k.repeat_interleave(rep, dim=1)
|
| 23 |
+
v = v.repeat_interleave(rep, dim=1)
|
| 24 |
+
return F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask, is_causal=is_causal)
|
| 25 |
+
|
| 26 |
+
class RMSNorm(nn.Module):
|
| 27 |
+
def __init__(self, dim: int, eps: float = 1e-6):
|
| 28 |
+
super().__init__()
|
| 29 |
+
self.dim = (dim,)
|
| 30 |
+
self.eps = eps
|
| 31 |
+
self.weight = nn.Parameter(torch.ones(dim))
|
| 32 |
+
|
| 33 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 34 |
+
if hasattr(F, "rms_norm"):
|
| 35 |
+
return F.rms_norm(x, self.dim, self.weight, self.eps)
|
| 36 |
+
xf = x.float()
|
| 37 |
+
y = xf * torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + self.eps)
|
| 38 |
+
return (y * self.weight.float()).to(dtype=x.dtype)
|
| 39 |
+
|
| 40 |
+
class RotaryEmbedding(nn.Module):
|
| 41 |
+
def __init__(self, dim: int, max_seq_len: int = 2048, theta: float = 50000.0):
|
| 42 |
+
super().__init__()
|
| 43 |
+
inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))
|
| 44 |
+
t = torch.arange(max_seq_len, dtype=torch.float32)
|
| 45 |
+
freqs = torch.outer(t, inv_freq)
|
| 46 |
+
self.register_buffer("cos_cached", freqs.cos()[None, None, :, :], persistent=False)
|
| 47 |
+
self.register_buffer("sin_cached", freqs.sin()[None, None, :, :], persistent=False)
|
| 48 |
+
|
| 49 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 50 |
+
L = x.size(2)
|
| 51 |
+
cos = self.cos_cached[:, :, :L, :].to(dtype=x.dtype)
|
| 52 |
+
sin = self.sin_cached[:, :, :L, :].to(dtype=x.dtype)
|
| 53 |
+
x1, x2 = x[..., ::2], x[..., 1::2]
|
| 54 |
+
return torch.cat((x1 * cos - x2 * sin, x1 * sin + x2 * cos), dim=-1)
|
| 55 |
+
|
| 56 |
+
class SwiGLU(nn.Module):
|
| 57 |
+
def __init__(self, d_model: int, ffn_dim: int):
|
| 58 |
+
super().__init__()
|
| 59 |
+
self.gate_up = nn.Linear(d_model, 2 * ffn_dim, bias=False)
|
| 60 |
+
self.down = nn.Linear(ffn_dim, d_model, bias=False)
|
| 61 |
+
|
| 62 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 63 |
+
gate, up = self.gate_up(x).chunk(2, dim=-1)
|
| 64 |
+
return self.down(F.silu(gate) * up)
|
| 65 |
+
|
| 66 |
+
class GatedGQASelfAttention(nn.Module):
|
| 67 |
+
def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, is_causal: bool = False, has_v_res: bool = False):
|
| 68 |
+
super().__init__()
|
| 69 |
+
self.n_heads = n_heads
|
| 70 |
+
self.n_kv_heads = n_kv_heads
|
| 71 |
+
self.head_dim = d_model // n_heads
|
| 72 |
+
self.is_causal = is_causal
|
| 73 |
+
self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
|
| 74 |
+
self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * self.head_dim, bias=False)
|
| 75 |
+
self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
|
| 76 |
+
self.out = nn.Linear(d_model, d_model, bias=False)
|
| 77 |
+
self.q_norm = RMSNorm(self.head_dim)
|
| 78 |
+
self.k_norm = RMSNorm(self.head_dim)
|
| 79 |
+
self.v_res_logit = nn.Parameter(torch.tensor(-2.0)) if has_v_res else None
|
| 80 |
+
|
| 81 |
+
def forward(self, x: torch.Tensor, rope: RotaryEmbedding, attn_mask: torch.Tensor = None, v0: torch.Tensor = None):
|
| 82 |
+
B, L, D = x.shape
|
| 83 |
+
q = self.q_norm(self.q_proj(x).view(B, L, self.n_heads, self.head_dim).transpose(1, 2))
|
| 84 |
+
k, v = self.kv_proj(x).chunk(2, dim=-1)
|
| 85 |
+
k = self.k_norm(k.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2))
|
| 86 |
+
v = v.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
| 87 |
+
v_raw = v
|
| 88 |
+
if self.v_res_logit is not None and v0 is not None:
|
| 89 |
+
alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
|
| 90 |
+
v = (1.0 - alpha) * v + alpha * v0
|
| 91 |
+
q, k = rope(q), rope(k)
|
| 92 |
+
y = _sdpa(q, k, v, attn_mask=None if self.is_causal else attn_mask, is_causal=self.is_causal)
|
| 93 |
+
gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
|
| 94 |
+
y = (y * gate).transpose(1, 2).contiguous().view(B, L, D)
|
| 95 |
+
return self.out(y), v_raw
|
| 96 |
+
|
| 97 |
+
def step_kv(self, x_step, cos_p, sin_p, pos, k_cache, v_cache, v0=None):
|
| 98 |
+
B, _, D = x_step.shape
|
| 99 |
+
q = self.q_norm(self.q_proj(x_step).view(B, 1, self.n_heads, self.head_dim).transpose(1, 2))
|
| 100 |
+
k, v = self.kv_proj(x_step).chunk(2, dim=-1)
|
| 101 |
+
k = self.k_norm(k.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2))
|
| 102 |
+
v = v.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
| 103 |
+
v_raw = v
|
| 104 |
+
if self.v_res_logit is not None and v0 is not None:
|
| 105 |
+
alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
|
| 106 |
+
v = (1.0 - alpha) * v + alpha * v0
|
| 107 |
+
q1, q2 = q[..., ::2], q[..., 1::2]
|
| 108 |
+
k1, k2 = k[..., ::2], k[..., 1::2]
|
| 109 |
+
q = torch.cat((q1 * cos_p - q2 * sin_p, q1 * sin_p + q2 * cos_p), dim=-1)
|
| 110 |
+
k = torch.cat((k1 * cos_p - k2 * sin_p, k1 * sin_p + k2 * cos_p), dim=-1)
|
| 111 |
+
k_cache[:, :, pos : pos + 1, :] = k
|
| 112 |
+
v_cache[:, :, pos : pos + 1, :] = v
|
| 113 |
+
y = _sdpa(q, k_cache[:, :, : pos + 1, :], v_cache[:, :, : pos + 1, :], is_causal=False)
|
| 114 |
+
gate = torch.sigmoid(self.gate_proj(x_step)).transpose(1, 2).unsqueeze(-1)
|
| 115 |
+
y = (y * gate).transpose(1, 2).contiguous().view(B, 1, D)
|
| 116 |
+
return self.out(y), v_raw
|
| 117 |
+
|
| 118 |
+
class SharedCrossKVProj(nn.Module):
|
| 119 |
+
def __init__(self, d_model: int, n_kv_heads: int, head_dim: int):
|
| 120 |
+
super().__init__()
|
| 121 |
+
self.n_kv_heads = n_kv_heads
|
| 122 |
+
self.head_dim = head_dim
|
| 123 |
+
self.norm = RMSNorm(d_model)
|
| 124 |
+
self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * head_dim, bias=False)
|
| 125 |
+
self.k_norm = RMSNorm(head_dim)
|
| 126 |
+
|
| 127 |
+
def forward(self, enc_mix: torch.Tensor):
|
| 128 |
+
B, S, _ = enc_mix.shape
|
| 129 |
+
k, v = self.kv_proj(self.norm(enc_mix)).chunk(2, dim=-1)
|
| 130 |
+
k = self.k_norm(k.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2))
|
| 131 |
+
v = v.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
| 132 |
+
return k, v
|
| 133 |
+
|
| 134 |
+
class GatedGQACrossAttention(nn.Module):
|
| 135 |
+
def __init__(self, d_model: int, n_heads: int):
|
| 136 |
+
super().__init__()
|
| 137 |
+
self.n_heads = n_heads
|
| 138 |
+
self.head_dim = d_model // n_heads
|
| 139 |
+
self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
|
| 140 |
+
self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
|
| 141 |
+
self.out = nn.Linear(d_model, d_model, bias=False)
|
| 142 |
+
self.q_norm = RMSNorm(self.head_dim)
|
| 143 |
+
|
| 144 |
+
def forward(self, x: torch.Tensor, k_cross: torch.Tensor, v_cross: torch.Tensor, enc_mask: torch.Tensor):
|
| 145 |
+
B, T, D = x.shape
|
| 146 |
+
q = self.q_norm(self.q_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2))
|
| 147 |
+
y = _sdpa(q, k_cross, v_cross, attn_mask=enc_mask, is_causal=False)
|
| 148 |
+
gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
|
| 149 |
+
y = (y * gate).transpose(1, 2).contiguous().view(B, T, D)
|
| 150 |
+
return self.out(y)
|
| 151 |
+
|
| 152 |
+
class EncoderLayer(nn.Module):
|
| 153 |
+
def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
|
| 154 |
+
super().__init__()
|
| 155 |
+
self.norm1 = RMSNorm(d_model)
|
| 156 |
+
self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=False, has_v_res=has_v_res)
|
| 157 |
+
self.norm2 = RMSNorm(d_model)
|
| 158 |
+
self.mlp = SwiGLU(d_model, ffn_dim)
|
| 159 |
+
|
| 160 |
+
def forward(self, x, rope, enc_mask, v0=None):
|
| 161 |
+
attn_out, v_raw = self.self_attn(self.norm1(x), rope, enc_mask, v0=v0)
|
| 162 |
+
x = x + attn_out
|
| 163 |
+
x = x + self.mlp(self.norm2(x))
|
| 164 |
+
return x, v_raw
|
| 165 |
+
|
| 166 |
+
class DecoderLayer(nn.Module):
|
| 167 |
+
def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
|
| 168 |
+
super().__init__()
|
| 169 |
+
self.norm1 = RMSNorm(d_model)
|
| 170 |
+
self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=True, has_v_res=has_v_res)
|
| 171 |
+
self.norm2 = RMSNorm(d_model)
|
| 172 |
+
self.cross_attn = GatedGQACrossAttention(d_model, n_heads)
|
| 173 |
+
self.norm3 = RMSNorm(d_model)
|
| 174 |
+
self.mlp = SwiGLU(d_model, ffn_dim)
|
| 175 |
+
|
| 176 |
+
def forward(self, x, k_cross, v_cross, rope, enc_mask, v0=None):
|
| 177 |
+
attn_out, v_raw = self.self_attn(self.norm1(x), rope, None, v0=v0)
|
| 178 |
+
x = x + attn_out
|
| 179 |
+
x = x + self.cross_attn(self.norm2(x), k_cross, v_cross, enc_mask)
|
| 180 |
+
x = x + self.mlp(self.norm3(x))
|
| 181 |
+
return x, v_raw
|
| 182 |
+
|
| 183 |
+
def step_kv(self, x_step, cos_p, sin_p, pos, k_self, v_self, k_cross, v_cross, enc_mask, v0=None):
|
| 184 |
+
attn_out, v_raw = self.self_attn.step_kv(self.norm1(x_step), cos_p, sin_p, pos, k_self, v_self, v0=v0)
|
| 185 |
+
x_step = x_step + attn_out
|
| 186 |
+
x_step = x_step + self.cross_attn(self.norm2(x_step), k_cross, v_cross, enc_mask)
|
| 187 |
+
x_step = x_step + self.mlp(self.norm3(x_step))
|
| 188 |
+
return x_step, v_raw
|
| 189 |
+
|
| 190 |
+
class MyTranslationForConditionalGeneration(PreTrainedModel):
|
| 191 |
+
config_class = MyTranslationConfig
|
| 192 |
+
base_model_prefix = ""
|
| 193 |
+
_tied_weights_keys = {"lm_head.weight": "embed_tokens.weight"} if hasattr(PreTrainedModel, "get_expanded_tied_weights_keys") else ["lm_head.weight"]
|
| 194 |
+
|
| 195 |
+
def __init__(self, config: MyTranslationConfig):
|
| 196 |
+
super().__init__(config)
|
| 197 |
+
self.d_model = config.d_model
|
| 198 |
+
self.n_heads = config.n_heads
|
| 199 |
+
self.n_kv_heads = config.n_kv_heads
|
| 200 |
+
self.head_dim = config.d_model // config.n_heads
|
| 201 |
+
self.enc_layers = config.enc_layers
|
| 202 |
+
self.dec_layers = config.dec_layers
|
| 203 |
+
self.n_cross_pairs = config.dec_layers // 2
|
| 204 |
+
self.embed_scale = math.sqrt(config.d_model)
|
| 205 |
+
|
| 206 |
+
self.embed_tokens = nn.Embedding(config.vocab_size, config.d_model, padding_idx=config.pad_token_id)
|
| 207 |
+
self.rope = RotaryEmbedding(self.head_dim, max_seq_len=config.max_seq_len, theta=config.rope_theta)
|
| 208 |
+
self.encoder = nn.ModuleList([
|
| 209 |
+
EncoderLayer(config.d_model, config.n_heads, config.n_kv_heads, config.ffn_dim, has_v_res=(i > 0))
|
| 210 |
+
for i in range(config.enc_layers)
|
| 211 |
+
])
|
| 212 |
+
self.enc_dwa_logits = nn.ParameterList([
|
| 213 |
+
nn.Parameter(torch.tensor([-4.0] * i + [0.0], dtype=torch.float32))
|
| 214 |
+
for i in range(1, config.enc_layers)
|
| 215 |
+
])
|
| 216 |
+
self.enc_router_logits = nn.Parameter(torch.zeros(self.n_cross_pairs, config.enc_layers))
|
| 217 |
+
self.cross_kv_projs = nn.ModuleList([
|
| 218 |
+
SharedCrossKVProj(config.d_model, config.n_kv_heads, self.head_dim)
|
| 219 |
+
for _ in range(self.n_cross_pairs)
|
| 220 |
+
])
|
| 221 |
+
self.decoder = nn.ModuleList([
|
| 222 |
+
DecoderLayer(config.d_model, config.n_heads, config.n_kv_heads, config.ffn_dim, has_v_res=(i > 0))
|
| 223 |
+
for i in range(config.dec_layers)
|
| 224 |
+
])
|
| 225 |
+
self.dec_norm = RMSNorm(config.d_model)
|
| 226 |
+
self.lm_head = nn.Linear(config.d_model, config.vocab_size, bias=False)
|
| 227 |
+
self.post_init()
|
| 228 |
+
self.tie_weights()
|
| 229 |
+
|
| 230 |
+
def get_input_embeddings(self):
|
| 231 |
+
return self.embed_tokens
|
| 232 |
+
|
| 233 |
+
def set_input_embeddings(self, value):
|
| 234 |
+
self.embed_tokens = value
|
| 235 |
+
|
| 236 |
+
def get_output_embeddings(self):
|
| 237 |
+
return self.lm_head
|
| 238 |
+
|
| 239 |
+
def set_output_embeddings(self, new_embeddings):
|
| 240 |
+
self.lm_head = new_embeddings
|
| 241 |
+
|
| 242 |
+
def tie_weights(self, **kwargs):
|
| 243 |
+
self.lm_head.weight = self.embed_tokens.weight
|
| 244 |
+
|
| 245 |
+
def encode_and_route(self, src_ids: torch.Tensor):
|
| 246 |
+
enc_mask = (src_ids != self.config.pad_token_id).unsqueeze(1).unsqueeze(2)
|
| 247 |
+
x = self.embed_tokens(src_ids) * self.embed_scale
|
| 248 |
+
raw_states = [x]
|
| 249 |
+
enc_outputs = []
|
| 250 |
+
v0 = None
|
| 251 |
+
for i, layer in enumerate(self.encoder):
|
| 252 |
+
if i > 0:
|
| 253 |
+
w_dwa = F.softmax(self.enc_dwa_logits[i - 1], dim=0).to(dtype=x.dtype)
|
| 254 |
+
x_in = torch.einsum("j,jbsd->bsd", w_dwa, torch.stack(raw_states, dim=0))
|
| 255 |
+
else:
|
| 256 |
+
x_in = x
|
| 257 |
+
x_out, v_raw = layer(x_in, self.rope, enc_mask, v0=v0)
|
| 258 |
+
if i == 0:
|
| 259 |
+
v0 = v_raw
|
| 260 |
+
raw_states.append(x_out)
|
| 261 |
+
enc_outputs.append(x_out)
|
| 262 |
+
stacked_enc = torch.stack(enc_outputs, dim=0)
|
| 263 |
+
router_w = F.softmax(self.enc_router_logits, dim=-1).to(dtype=stacked_enc.dtype)
|
| 264 |
+
cross_kv = [
|
| 265 |
+
self.cross_kv_projs[p](torch.einsum("k,kbsd->bsd", router_w[p], stacked_enc))
|
| 266 |
+
for p in range(self.n_cross_pairs)
|
| 267 |
+
]
|
| 268 |
+
return cross_kv, enc_mask
|
| 269 |
+
|
| 270 |
+
def decode_seq(self, tgt_in: torch.Tensor, cross_kv: list, enc_mask: torch.Tensor):
|
| 271 |
+
x = self.embed_tokens(tgt_in) * self.embed_scale
|
| 272 |
+
v0 = None
|
| 273 |
+
for i, layer in enumerate(self.decoder):
|
| 274 |
+
kc, vc = cross_kv[i // 2]
|
| 275 |
+
x, v_raw = layer(x, kc, vc, self.rope, enc_mask, v0=v0)
|
| 276 |
+
if i == 0:
|
| 277 |
+
v0 = v_raw
|
| 278 |
+
return self.dec_norm(x)
|
| 279 |
+
|
| 280 |
+
def forward(self, input_ids=None, decoder_input_ids=None, labels=None, **kwargs):
|
| 281 |
+
if decoder_input_ids is None:
|
| 282 |
+
decoder_input_ids = torch.full((input_ids.size(0), 1), BOS_ID, dtype=torch.long, device=input_ids.device)
|
| 283 |
+
cross_kv, enc_mask = self.encode_and_route(input_ids)
|
| 284 |
+
h_dec = self.decode_seq(decoder_input_ids, cross_kv, enc_mask)
|
| 285 |
+
logits = self.lm_head(h_dec)
|
| 286 |
+
loss = None
|
| 287 |
+
if labels is not None:
|
| 288 |
+
loss = F.cross_entropy(logits.view(-1, logits.size(-1)), labels.view(-1), ignore_index=-100)
|
| 289 |
+
return Seq2SeqLMOutput(loss=loss, logits=logits)
|
| 290 |
+
|
| 291 |
+
@torch.inference_mode()
|
| 292 |
+
def translate(
|
| 293 |
+
self,
|
| 294 |
+
text: str,
|
| 295 |
+
sp_proc,
|
| 296 |
+
direction: str = "en2my",
|
| 297 |
+
style: str = "auto",
|
| 298 |
+
num_beams: int = 4,
|
| 299 |
+
max_new_tokens: int = 192,
|
| 300 |
+
length_penalty: float = 0.85,
|
| 301 |
+
rep_penalty: float = 1.12,
|
| 302 |
+
) -> str:
|
| 303 |
+
self.eval()
|
| 304 |
+
dev = self.embed_tokens.weight.device
|
| 305 |
+
dt = self.embed_tokens.weight.dtype
|
| 306 |
+
pref = [TOK_2MY if direction == "en2my" else TOK_2EN]
|
| 307 |
+
if style == "formal":
|
| 308 |
+
pref.append(TOK_FORMAL)
|
| 309 |
+
elif style == "spoken":
|
| 310 |
+
pref.append(TOK_SPOKEN)
|
| 311 |
+
src_ids = pref + sp_proc.encode_as_ids(text.strip())[: self.config.max_seq_len - 8] + [EOS_ID]
|
| 312 |
+
src_t = torch.tensor([src_ids], dtype=torch.long, device=dev)
|
| 313 |
+
cross_kv, enc_mask = self.encode_and_route(src_t)
|
| 314 |
+
|
| 315 |
+
if num_beams <= 1:
|
| 316 |
+
k_cache = torch.empty((self.dec_layers, 1, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=dt, device=dev)
|
| 317 |
+
v_cache = torch.empty((self.dec_layers, 1, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=dt, device=dev)
|
| 318 |
+
cur_tok = torch.tensor([[BOS_ID]], dtype=torch.long, device=dev)
|
| 319 |
+
cos_all = self.rope.cos_cached.to(dtype=dt, device=dev)
|
| 320 |
+
sin_all = self.rope.sin_cached.to(dtype=dt, device=dev)
|
| 321 |
+
ys = []
|
| 322 |
+
for pos in range(max_new_tokens):
|
| 323 |
+
x_dec = self.embed_tokens(cur_tok) * self.embed_scale
|
| 324 |
+
v0 = None
|
| 325 |
+
cos_p, sin_p = cos_all[:, :, pos : pos + 1, :], sin_all[:, :, pos : pos + 1, :]
|
| 326 |
+
for l_idx, layer in enumerate(self.decoder):
|
| 327 |
+
kc, vc = cross_kv[l_idx // 2]
|
| 328 |
+
x_dec, v_raw = layer.step_kv(x_dec, cos_p, sin_p, pos, k_cache[l_idx], v_cache[l_idx], kc, vc, enc_mask, v0=v0)
|
| 329 |
+
if l_idx == 0:
|
| 330 |
+
v0 = v_raw
|
| 331 |
+
logits = self.lm_head(self.dec_norm(x_dec)[:, 0, :]).float()[0]
|
| 332 |
+
logits[[0, 1, 3, 4, 5, 6, 7, 8, 9, 10]] = -1e9
|
| 333 |
+
if rep_penalty != 1.0 and ys:
|
| 334 |
+
seen = torch.tensor(list(set(ys)), dtype=torch.long, device=dev)
|
| 335 |
+
sl = logits[seen]
|
| 336 |
+
logits[seen] = torch.where(sl > 0, sl / rep_penalty, sl * rep_penalty)
|
| 337 |
+
nxt = int(logits.argmax().item())
|
| 338 |
+
if nxt == EOS_ID:
|
| 339 |
+
break
|
| 340 |
+
ys.append(nxt)
|
| 341 |
+
cur_tok = torch.tensor([[nxt]], dtype=torch.long, device=dev)
|
| 342 |
+
return sp_proc.decode_ids([x for x in ys if x >= 11]).strip()
|
| 343 |
+
|
| 344 |
+
beams = [(0.0, [BOS_ID], False)]
|
| 345 |
+
for _ in range(max_new_tokens):
|
| 346 |
+
active = [(sc, seq) for sc, seq, done in beams if not done]
|
| 347 |
+
if not active:
|
| 348 |
+
break
|
| 349 |
+
B_act = len(active)
|
| 350 |
+
tgt_t = torch.tensor([seq for _, seq in active], dtype=torch.long, device=dev)
|
| 351 |
+
ckv_exp = [(kc.expand(B_act, -1, -1, -1), vc.expand(B_act, -1, -1, -1)) for kc, vc in cross_kv]
|
| 352 |
+
h_dec = self.decode_seq(tgt_t, ckv_exp, enc_mask.expand(B_act, -1, -1, -1))
|
| 353 |
+
logits = self.lm_head(h_dec[:, -1, :]).float()
|
| 354 |
+
logits[:, [0, 1, 3, 4, 5, 6, 7, 8, 9, 10]] = -1e9
|
| 355 |
+
for b_i, (_, seq) in enumerate(active):
|
| 356 |
+
if rep_penalty != 1.0 and len(seq) > 1:
|
| 357 |
+
seen = torch.tensor(list(set(seq[1:])), dtype=torch.long, device=dev)
|
| 358 |
+
sl = logits[b_i, seen]
|
| 359 |
+
logits[b_i, seen] = torch.where(sl > 0, sl / rep_penalty, sl * rep_penalty)
|
| 360 |
+
lp = F.log_softmax(logits, dim=-1)
|
| 361 |
+
topk_lp, topk_ids = torch.topk(lp, num_beams * 2, dim=-1)
|
| 362 |
+
cands = [(sc, seq, True) for sc, seq, done in beams if done]
|
| 363 |
+
for b_i, (sc, seq) in enumerate(active):
|
| 364 |
+
for k in range(num_beams * 2):
|
| 365 |
+
tid = int(topk_ids[b_i, k].item())
|
| 366 |
+
n_sc = sc + float(topk_lp[b_i, k].item())
|
| 367 |
+
cands.append((n_sc, seq if tid == EOS_ID else seq + [tid], tid == EOS_ID))
|
| 368 |
+
cands.sort(key=lambda x: x[0] / (max(1, len(x[1]) - 1) ** length_penalty), reverse=True)
|
| 369 |
+
beams = cands[:num_beams]
|
| 370 |
+
if all(done for _, _, done in beams):
|
| 371 |
+
break
|
| 372 |
+
return sp_proc.decode_ids([x for x in beams[0][1][1:] if x >= 11]).strip()
|
special_tokens_map.json
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"unk_token": "<unk>",
|
| 3 |
+
"bos_token": "<s>",
|
| 4 |
+
"eos_token": "</s>",
|
| 5 |
+
"pad_token": "<pad>",
|
| 6 |
+
"additional_special_tokens": [
|
| 7 |
+
"<2my>",
|
| 8 |
+
"<2en>",
|
| 9 |
+
"<formal>",
|
| 10 |
+
"<spoken>",
|
| 11 |
+
"<bt>",
|
| 12 |
+
"<denoise>",
|
| 13 |
+
"<mask>"
|
| 14 |
+
]
|
| 15 |
+
}
|
tokenizer.model
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9aaeeaf36272c5c158d11ed0987ca5930686f1e02713292558003909bb147bc1
|
| 3 |
+
size 1840017
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_max_length": 1024,
|
| 3 |
+
"unk_token": "<unk>",
|
| 4 |
+
"bos_token": "<s>",
|
| 5 |
+
"eos_token": "</s>",
|
| 6 |
+
"pad_token": "<pad>",
|
| 7 |
+
"additional_special_tokens": [
|
| 8 |
+
"<2my>",
|
| 9 |
+
"<2en>",
|
| 10 |
+
"<formal>",
|
| 11 |
+
"<spoken>",
|
| 12 |
+
"<bt>",
|
| 13 |
+
"<denoise>",
|
| 14 |
+
"<mask>"
|
| 15 |
+
],
|
| 16 |
+
"special_tokens_map": {
|
| 17 |
+
"2my": 4,
|
| 18 |
+
"2en": 5,
|
| 19 |
+
"formal": 6,
|
| 20 |
+
"spoken": 7,
|
| 21 |
+
"bt": 8,
|
| 22 |
+
"denoise": 9,
|
| 23 |
+
"mask": 10
|
| 24 |
+
}
|
| 25 |
+
}
|
training_codes.txt
ADDED
|
@@ -0,0 +1,1214 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import os
|
| 2 |
+
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"
|
| 3 |
+
|
| 4 |
+
import argparse, json, math, time
|
| 5 |
+
from datetime import timedelta
|
| 6 |
+
import numpy as np
|
| 7 |
+
import sentencepiece as spm
|
| 8 |
+
import torch
|
| 9 |
+
import torch.distributed as dist
|
| 10 |
+
import torch.nn as nn
|
| 11 |
+
import torch.nn.functional as F
|
| 12 |
+
from torch.utils.checkpoint import checkpoint as grad_ckpt
|
| 13 |
+
from torch.utils.data import Dataset, DataLoader
|
| 14 |
+
|
| 15 |
+
torch.backends.cuda.matmul.allow_tf32 = True
|
| 16 |
+
torch.backends.cudnn.allow_tf32 = True
|
| 17 |
+
torch._dynamo.config.cache_size_limit = 64
|
| 18 |
+
|
| 19 |
+
DATA_DIR = "/workspace/mytrans_dl/tokenized_65k"
|
| 20 |
+
CKPT_DIR = "/workspace/ckpt_300m"
|
| 21 |
+
|
| 22 |
+
VOCAB_SIZE = 65536
|
| 23 |
+
UNK_ID, BOS_ID, EOS_ID, PAD_ID = 0, 1, 2, 3
|
| 24 |
+
TOK_2MY, TOK_2EN, TOK_FORMAL, TOK_SPOKEN, TOK_BT, TOK_DENOISE, TOK_MASK = 4, 5, 6, 7, 8, 9, 10
|
| 25 |
+
FLAG_BT, FLAG_FORMAL, FLAG_SPOKEN = 1, 2, 4
|
| 26 |
+
|
| 27 |
+
BULK_SUBSETS = {
|
| 28 |
+
"MyanmarTranslator_works/myanmar_spoken_corpus",
|
| 29 |
+
"MyanmarTranslator_works/myanmar-cc100-dataset",
|
| 30 |
+
"MyanmarTranslator_works/myanmar-written-corpus",
|
| 31 |
+
}
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def load_and_verify_contracts(data_dir: str):
|
| 35 |
+
with open(os.path.join(data_dir, "meta.json"), "r") as f:
|
| 36 |
+
meta = json.load(f)
|
| 37 |
+
sp = spm.SentencePieceProcessor()
|
| 38 |
+
sp.load(os.path.join(data_dir, "tokenizer.model"))
|
| 39 |
+
|
| 40 |
+
assert sp.get_piece_size() == VOCAB_SIZE
|
| 41 |
+
assert sp.unk_id() == UNK_ID and sp.bos_id() == BOS_ID and sp.eos_id() == EOS_ID and sp.pad_id() == PAD_ID
|
| 42 |
+
expected = {4: "<2my>", 5: "<2en>", 6: "<formal>", 7: "<spoken>", 8: "<bt>", 9: "<denoise>", 10: "<mask>"}
|
| 43 |
+
for tid, piece in expected.items():
|
| 44 |
+
assert sp.id_to_piece(tid) == piece, f"Token {tid} mismatch"
|
| 45 |
+
if "special_tokens" in meta:
|
| 46 |
+
st = meta["special_tokens"]
|
| 47 |
+
assert st["2my"] == TOK_2MY and st["2en"] == TOK_2EN
|
| 48 |
+
assert st["formal"] == TOK_FORMAL and st["spoken"] == TOK_SPOKEN
|
| 49 |
+
assert st["bt"] == TOK_BT and st["denoise"] == TOK_DENOISE and st["mask"] == TOK_MASK
|
| 50 |
+
return meta, sp
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
# ==============================================================================
|
| 54 |
+
# 1. MyTranslation-300M Architecture
|
| 55 |
+
# ==============================================================================
|
| 56 |
+
class RMSNorm(nn.Module):
|
| 57 |
+
def __init__(self, dim: int, eps: float = 1e-6):
|
| 58 |
+
super().__init__()
|
| 59 |
+
self.dim = (dim,)
|
| 60 |
+
self.eps = eps
|
| 61 |
+
self.weight = nn.Parameter(torch.ones(dim))
|
| 62 |
+
|
| 63 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 64 |
+
return F.rms_norm(x, self.dim, self.weight, self.eps)
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
class RotaryEmbedding(nn.Module):
|
| 68 |
+
def __init__(self, dim: int, max_seq_len: int = 2048, theta: float = 50000.0):
|
| 69 |
+
super().__init__()
|
| 70 |
+
inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))
|
| 71 |
+
t = torch.arange(max_seq_len, dtype=torch.float32)
|
| 72 |
+
freqs = torch.outer(t, inv_freq)
|
| 73 |
+
self.register_buffer("cos_cached", freqs.cos()[None, None, :, :], persistent=False)
|
| 74 |
+
self.register_buffer("sin_cached", freqs.sin()[None, None, :, :], persistent=False)
|
| 75 |
+
|
| 76 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 77 |
+
L = x.size(2)
|
| 78 |
+
cos = self.cos_cached[:, :, :L, :].to(dtype=x.dtype)
|
| 79 |
+
sin = self.sin_cached[:, :, :L, :].to(dtype=x.dtype)
|
| 80 |
+
x1, x2 = x[..., ::2], x[..., 1::2]
|
| 81 |
+
return torch.cat((x1 * cos - x2 * sin, x1 * sin + x2 * cos), dim=-1)
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
class SwiGLU(nn.Module):
|
| 85 |
+
def __init__(self, d_model: int, ffn_dim: int):
|
| 86 |
+
super().__init__()
|
| 87 |
+
self.gate_up = nn.Linear(d_model, 2 * ffn_dim, bias=False)
|
| 88 |
+
self.down = nn.Linear(ffn_dim, d_model, bias=False)
|
| 89 |
+
|
| 90 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 91 |
+
gate, up = self.gate_up(x).chunk(2, dim=-1)
|
| 92 |
+
return self.down(F.silu(gate) * up)
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
class GatedGQASelfAttention(nn.Module):
|
| 96 |
+
def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, is_causal: bool = False, has_v_res: bool = False):
|
| 97 |
+
super().__init__()
|
| 98 |
+
self.n_heads = n_heads
|
| 99 |
+
self.n_kv_heads = n_kv_heads
|
| 100 |
+
self.head_dim = d_model // n_heads
|
| 101 |
+
self.is_causal = is_causal
|
| 102 |
+
self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
|
| 103 |
+
self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * self.head_dim, bias=False)
|
| 104 |
+
self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
|
| 105 |
+
self.out = nn.Linear(d_model, d_model, bias=False)
|
| 106 |
+
self.q_norm = RMSNorm(self.head_dim)
|
| 107 |
+
self.k_norm = RMSNorm(self.head_dim)
|
| 108 |
+
self.v_res_logit = nn.Parameter(torch.tensor(-2.0)) if has_v_res else None
|
| 109 |
+
|
| 110 |
+
def forward(self, x: torch.Tensor, rope: RotaryEmbedding, attn_mask: torch.Tensor = None, v0: torch.Tensor = None):
|
| 111 |
+
B, L, D = x.shape
|
| 112 |
+
q = self.q_norm(self.q_proj(x).view(B, L, self.n_heads, self.head_dim).transpose(1, 2))
|
| 113 |
+
k, v = self.kv_proj(x).chunk(2, dim=-1)
|
| 114 |
+
k = self.k_norm(k.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2))
|
| 115 |
+
v = v.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
| 116 |
+
v_raw = v
|
| 117 |
+
if self.v_res_logit is not None and v0 is not None:
|
| 118 |
+
alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
|
| 119 |
+
v = (1.0 - alpha) * v + alpha * v0
|
| 120 |
+
q, k = rope(q), rope(k)
|
| 121 |
+
y = F.scaled_dot_product_attention(
|
| 122 |
+
q, k, v,
|
| 123 |
+
attn_mask=None if self.is_causal else attn_mask,
|
| 124 |
+
is_causal=self.is_causal,
|
| 125 |
+
enable_gqa=True,
|
| 126 |
+
)
|
| 127 |
+
gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
|
| 128 |
+
y = (y * gate).transpose(1, 2).contiguous().view(B, L, D)
|
| 129 |
+
return self.out(y), v_raw
|
| 130 |
+
|
| 131 |
+
def step_kv(self, x_step, cos_p, sin_p, pos, k_cache, v_cache, v0=None):
|
| 132 |
+
B, _, D = x_step.shape
|
| 133 |
+
q = self.q_norm(self.q_proj(x_step).view(B, 1, self.n_heads, self.head_dim).transpose(1, 2))
|
| 134 |
+
k, v = self.kv_proj(x_step).chunk(2, dim=-1)
|
| 135 |
+
k = self.k_norm(k.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2))
|
| 136 |
+
v = v.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
| 137 |
+
v_raw = v
|
| 138 |
+
if self.v_res_logit is not None and v0 is not None:
|
| 139 |
+
alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
|
| 140 |
+
v = (1.0 - alpha) * v + alpha * v0
|
| 141 |
+
q1, q2 = q[..., ::2], q[..., 1::2]
|
| 142 |
+
k1, k2 = k[..., ::2], k[..., 1::2]
|
| 143 |
+
q = torch.cat((q1 * cos_p - q2 * sin_p, q1 * sin_p + q2 * cos_p), dim=-1)
|
| 144 |
+
k = torch.cat((k1 * cos_p - k2 * sin_p, k1 * sin_p + k2 * cos_p), dim=-1)
|
| 145 |
+
k_cache[:, :, pos : pos + 1, :] = k
|
| 146 |
+
v_cache[:, :, pos : pos + 1, :] = v
|
| 147 |
+
y = F.scaled_dot_product_attention(
|
| 148 |
+
q, k_cache[:, :, : pos + 1, :], v_cache[:, :, : pos + 1, :],
|
| 149 |
+
is_causal=False, enable_gqa=True,
|
| 150 |
+
)
|
| 151 |
+
gate = torch.sigmoid(self.gate_proj(x_step)).transpose(1, 2).unsqueeze(-1)
|
| 152 |
+
y = (y * gate).transpose(1, 2).contiguous().view(B, 1, D)
|
| 153 |
+
return self.out(y), v_raw
|
| 154 |
+
|
| 155 |
+
|
| 156 |
+
class SharedCrossKVProj(nn.Module):
|
| 157 |
+
def __init__(self, d_model: int, n_kv_heads: int, head_dim: int):
|
| 158 |
+
super().__init__()
|
| 159 |
+
self.n_kv_heads = n_kv_heads
|
| 160 |
+
self.head_dim = head_dim
|
| 161 |
+
self.norm = RMSNorm(d_model)
|
| 162 |
+
self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * head_dim, bias=False)
|
| 163 |
+
self.k_norm = RMSNorm(head_dim)
|
| 164 |
+
|
| 165 |
+
def forward(self, enc_mix: torch.Tensor):
|
| 166 |
+
B, S, _ = enc_mix.shape
|
| 167 |
+
k, v = self.kv_proj(self.norm(enc_mix)).chunk(2, dim=-1)
|
| 168 |
+
k = self.k_norm(k.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2))
|
| 169 |
+
v = v.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
| 170 |
+
return k, v
|
| 171 |
+
|
| 172 |
+
|
| 173 |
+
class GatedGQACrossAttention(nn.Module):
|
| 174 |
+
def __init__(self, d_model: int, n_heads: int):
|
| 175 |
+
super().__init__()
|
| 176 |
+
self.n_heads = n_heads
|
| 177 |
+
self.head_dim = d_model // n_heads
|
| 178 |
+
self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
|
| 179 |
+
self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
|
| 180 |
+
self.out = nn.Linear(d_model, d_model, bias=False)
|
| 181 |
+
self.q_norm = RMSNorm(self.head_dim)
|
| 182 |
+
|
| 183 |
+
def forward(self, x: torch.Tensor, k_cross: torch.Tensor, v_cross: torch.Tensor, enc_mask: torch.Tensor):
|
| 184 |
+
B, T, D = x.shape
|
| 185 |
+
q = self.q_norm(self.q_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2))
|
| 186 |
+
y = F.scaled_dot_product_attention(q, k_cross, v_cross, attn_mask=enc_mask, is_causal=False, enable_gqa=True)
|
| 187 |
+
gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
|
| 188 |
+
y = (y * gate).transpose(1, 2).contiguous().view(B, T, D)
|
| 189 |
+
return self.out(y)
|
| 190 |
+
|
| 191 |
+
|
| 192 |
+
class EncoderLayer(nn.Module):
|
| 193 |
+
def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
|
| 194 |
+
super().__init__()
|
| 195 |
+
self.norm1 = RMSNorm(d_model)
|
| 196 |
+
self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=False, has_v_res=has_v_res)
|
| 197 |
+
self.norm2 = RMSNorm(d_model)
|
| 198 |
+
self.mlp = SwiGLU(d_model, ffn_dim)
|
| 199 |
+
|
| 200 |
+
def forward(self, x, rope, enc_mask, v0=None):
|
| 201 |
+
attn_out, v_raw = self.self_attn(self.norm1(x), rope, enc_mask, v0=v0)
|
| 202 |
+
x = x + attn_out
|
| 203 |
+
x = x + self.mlp(self.norm2(x))
|
| 204 |
+
return x, v_raw
|
| 205 |
+
|
| 206 |
+
|
| 207 |
+
class DecoderLayer(nn.Module):
|
| 208 |
+
def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
|
| 209 |
+
super().__init__()
|
| 210 |
+
self.norm1 = RMSNorm(d_model)
|
| 211 |
+
self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=True, has_v_res=has_v_res)
|
| 212 |
+
self.norm2 = RMSNorm(d_model)
|
| 213 |
+
self.cross_attn = GatedGQACrossAttention(d_model, n_heads)
|
| 214 |
+
self.norm3 = RMSNorm(d_model)
|
| 215 |
+
self.mlp = SwiGLU(d_model, ffn_dim)
|
| 216 |
+
|
| 217 |
+
def forward(self, x, k_cross, v_cross, rope, enc_mask, v0=None):
|
| 218 |
+
attn_out, v_raw = self.self_attn(self.norm1(x), rope, None, v0=v0)
|
| 219 |
+
x = x + attn_out
|
| 220 |
+
x = x + self.cross_attn(self.norm2(x), k_cross, v_cross, enc_mask)
|
| 221 |
+
x = x + self.mlp(self.norm3(x))
|
| 222 |
+
return x, v_raw
|
| 223 |
+
|
| 224 |
+
def step_kv(self, x_step, cos_p, sin_p, pos, k_self, v_self, k_cross, v_cross, enc_mask, v0=None):
|
| 225 |
+
attn_out, v_raw = self.self_attn.step_kv(self.norm1(x_step), cos_p, sin_p, pos, k_self, v_self, v0=v0)
|
| 226 |
+
x_step = x_step + attn_out
|
| 227 |
+
x_step = x_step + self.cross_attn(self.norm2(x_step), k_cross, v_cross, enc_mask)
|
| 228 |
+
x_step = x_step + self.mlp(self.norm3(x_step))
|
| 229 |
+
return x_step, v_raw
|
| 230 |
+
|
| 231 |
+
|
| 232 |
+
class MTPHeadBlock(nn.Module):
|
| 233 |
+
def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int):
|
| 234 |
+
super().__init__()
|
| 235 |
+
self.norm_h = RMSNorm(d_model)
|
| 236 |
+
self.norm_e = RMSNorm(d_model)
|
| 237 |
+
self.proj = nn.Linear(2 * d_model, d_model, bias=False)
|
| 238 |
+
self.norm1 = RMSNorm(d_model)
|
| 239 |
+
self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=True, has_v_res=False)
|
| 240 |
+
self.norm2 = RMSNorm(d_model)
|
| 241 |
+
self.mlp = SwiGLU(d_model, ffn_dim)
|
| 242 |
+
self.out_norm = RMSNorm(d_model)
|
| 243 |
+
|
| 244 |
+
def forward(self, h_main: torch.Tensor, next_tok_emb: torch.Tensor, rope: RotaryEmbedding):
|
| 245 |
+
x = self.proj(torch.cat([self.norm_h(h_main), self.norm_e(next_tok_emb)], dim=-1))
|
| 246 |
+
attn_out, _ = self.self_attn(self.norm1(x), rope, None)
|
| 247 |
+
x = x + attn_out
|
| 248 |
+
x = x + self.mlp(self.norm2(x))
|
| 249 |
+
return self.out_norm(x)
|
| 250 |
+
|
| 251 |
+
|
| 252 |
+
class MyTranslation300M(nn.Module):
|
| 253 |
+
def __init__(
|
| 254 |
+
self,
|
| 255 |
+
vocab_size=65536,
|
| 256 |
+
d_model=1024,
|
| 257 |
+
n_heads=16,
|
| 258 |
+
n_kv_heads=4,
|
| 259 |
+
enc_layers=12,
|
| 260 |
+
dec_layers=6,
|
| 261 |
+
ffn_dim=3072,
|
| 262 |
+
max_seq_len=2048,
|
| 263 |
+
):
|
| 264 |
+
super().__init__()
|
| 265 |
+
assert d_model % n_heads == 0 and n_heads % n_kv_heads == 0 and dec_layers % 2 == 0
|
| 266 |
+
self.config = dict(
|
| 267 |
+
vocab_size=vocab_size, d_model=d_model, n_heads=n_heads, n_kv_heads=n_kv_heads,
|
| 268 |
+
enc_layers=enc_layers, dec_layers=dec_layers, ffn_dim=ffn_dim, max_seq_len=max_seq_len,
|
| 269 |
+
)
|
| 270 |
+
self.d_model = d_model
|
| 271 |
+
self.n_heads = n_heads
|
| 272 |
+
self.n_kv_heads = n_kv_heads
|
| 273 |
+
self.head_dim = d_model // n_heads
|
| 274 |
+
self.enc_layers = enc_layers
|
| 275 |
+
self.dec_layers = dec_layers
|
| 276 |
+
self.n_cross_pairs = dec_layers // 2
|
| 277 |
+
self.embed_scale = math.sqrt(d_model)
|
| 278 |
+
|
| 279 |
+
self.embed_tokens = nn.Embedding(vocab_size, d_model, padding_idx=PAD_ID)
|
| 280 |
+
self.rope = RotaryEmbedding(self.head_dim, max_seq_len=max_seq_len, theta=50000.0)
|
| 281 |
+
|
| 282 |
+
self.encoder = nn.ModuleList([
|
| 283 |
+
EncoderLayer(d_model, n_heads, n_kv_heads, ffn_dim, has_v_res=(i > 0))
|
| 284 |
+
for i in range(enc_layers)
|
| 285 |
+
])
|
| 286 |
+
self.enc_dwa_logits = nn.ParameterList([
|
| 287 |
+
nn.Parameter(torch.tensor([-4.0] * i + [0.0], dtype=torch.float32))
|
| 288 |
+
for i in range(1, enc_layers)
|
| 289 |
+
])
|
| 290 |
+
|
| 291 |
+
self.enc_router_logits = nn.Parameter(torch.zeros(self.n_cross_pairs, enc_layers))
|
| 292 |
+
with torch.no_grad():
|
| 293 |
+
for p in range(self.n_cross_pairs):
|
| 294 |
+
center = (p + 0.5) * (enc_layers / self.n_cross_pairs) - 0.5
|
| 295 |
+
for k in range(enc_layers):
|
| 296 |
+
self.enc_router_logits[p, k] = -0.35 * ((k - center) ** 2)
|
| 297 |
+
|
| 298 |
+
self.cross_kv_projs = nn.ModuleList([
|
| 299 |
+
SharedCrossKVProj(d_model, n_kv_heads, self.head_dim)
|
| 300 |
+
for _ in range(self.n_cross_pairs)
|
| 301 |
+
])
|
| 302 |
+
|
| 303 |
+
self.decoder = nn.ModuleList([
|
| 304 |
+
DecoderLayer(d_model, n_heads, n_kv_heads, ffn_dim, has_v_res=(i > 0))
|
| 305 |
+
for i in range(dec_layers)
|
| 306 |
+
])
|
| 307 |
+
self.dec_norm = RMSNorm(d_model)
|
| 308 |
+
self.lm_head = nn.Linear(d_model, vocab_size, bias=False)
|
| 309 |
+
self.lm_head.weight = self.embed_tokens.weight
|
| 310 |
+
|
| 311 |
+
self.mtp_head = MTPHeadBlock(d_model, n_heads, n_kv_heads, ffn_dim)
|
| 312 |
+
self._init_weights(enc_layers + dec_layers)
|
| 313 |
+
|
| 314 |
+
def _init_weights(self, total_layers: int):
|
| 315 |
+
res_std = 0.02 / math.sqrt(2 * total_layers)
|
| 316 |
+
for name, m in self.named_modules():
|
| 317 |
+
if isinstance(m, nn.Linear):
|
| 318 |
+
if name.endswith(".out") or name.endswith(".down"):
|
| 319 |
+
nn.init.normal_(m.weight, mean=0.0, std=res_std)
|
| 320 |
+
elif name.endswith(".gate_proj"):
|
| 321 |
+
nn.init.normal_(m.weight, mean=0.0, std=0.01)
|
| 322 |
+
else:
|
| 323 |
+
nn.init.normal_(m.weight, mean=0.0, std=0.02)
|
| 324 |
+
elif isinstance(m, nn.Embedding):
|
| 325 |
+
nn.init.normal_(m.weight, mean=0.0, std=0.02)
|
| 326 |
+
if m.padding_idx is not None:
|
| 327 |
+
m.weight.data[m.padding_idx].zero_()
|
| 328 |
+
|
| 329 |
+
def compile_blocks(self):
|
| 330 |
+
for i in range(len(self.encoder)):
|
| 331 |
+
self.encoder[i] = torch.compile(self.encoder[i], dynamic=True)
|
| 332 |
+
for i in range(len(self.cross_kv_projs)):
|
| 333 |
+
self.cross_kv_projs[i] = torch.compile(self.cross_kv_projs[i], dynamic=True)
|
| 334 |
+
for i in range(len(self.decoder)):
|
| 335 |
+
self.decoder[i] = torch.compile(self.decoder[i], dynamic=True)
|
| 336 |
+
self.mtp_head = torch.compile(self.mtp_head, dynamic=True)
|
| 337 |
+
|
| 338 |
+
def encode_and_route(self, src_ids: torch.Tensor, ckpt_mode: str = "none"):
|
| 339 |
+
enc_mask = (src_ids != PAD_ID).unsqueeze(1).unsqueeze(2)
|
| 340 |
+
x = self.embed_tokens(src_ids) * self.embed_scale
|
| 341 |
+
raw_states = [x]
|
| 342 |
+
enc_outputs = []
|
| 343 |
+
v0 = None
|
| 344 |
+
|
| 345 |
+
for i, layer in enumerate(self.encoder):
|
| 346 |
+
if i > 0:
|
| 347 |
+
w_dwa = F.softmax(self.enc_dwa_logits[i - 1], dim=0).to(dtype=x.dtype)
|
| 348 |
+
if self.training and ckpt_mode != "none":
|
| 349 |
+
x_in = grad_ckpt(
|
| 350 |
+
lambda w, *st: torch.einsum("j,jbsd->bsd", w, torch.stack(st, dim=0)),
|
| 351 |
+
w_dwa, *raw_states, use_reentrant=False
|
| 352 |
+
)
|
| 353 |
+
else:
|
| 354 |
+
x_in = torch.einsum("j,jbsd->bsd", w_dwa, torch.stack(raw_states, dim=0))
|
| 355 |
+
else:
|
| 356 |
+
x_in = x
|
| 357 |
+
|
| 358 |
+
do_ckpt = self.training and (
|
| 359 |
+
(ckpt_mode == "heavy" and (i % 3 != 0)) or
|
| 360 |
+
(ckpt_mode == "light" and (i in (3, 7, 11)))
|
| 361 |
+
)
|
| 362 |
+
if do_ckpt:
|
| 363 |
+
x_out, v_raw = grad_ckpt(layer, x_in, self.rope, enc_mask, v0, use_reentrant=False)
|
| 364 |
+
else:
|
| 365 |
+
x_out, v_raw = layer(x_in, self.rope, enc_mask, v0=v0)
|
| 366 |
+
|
| 367 |
+
if i == 0:
|
| 368 |
+
v0 = v_raw
|
| 369 |
+
raw_states.append(x_out)
|
| 370 |
+
enc_outputs.append(x_out)
|
| 371 |
+
|
| 372 |
+
stacked_enc = torch.stack(enc_outputs, dim=0)
|
| 373 |
+
router_w = F.softmax(self.enc_router_logits, dim=-1).to(dtype=stacked_enc.dtype)
|
| 374 |
+
cross_kv = []
|
| 375 |
+
for p in range(self.n_cross_pairs):
|
| 376 |
+
enc_mix = torch.einsum("k,kbsd->bsd", router_w[p], stacked_enc)
|
| 377 |
+
cross_kv.append(self.cross_kv_projs[p](enc_mix))
|
| 378 |
+
return cross_kv, enc_mask
|
| 379 |
+
|
| 380 |
+
def decode_seq(self, tgt_in: torch.Tensor, cross_kv: list, enc_mask: torch.Tensor, ckpt_mode: str = "none"):
|
| 381 |
+
x = self.embed_tokens(tgt_in) * self.embed_scale
|
| 382 |
+
v0 = None
|
| 383 |
+
for i, layer in enumerate(self.decoder):
|
| 384 |
+
kc, vc = cross_kv[i // 2]
|
| 385 |
+
do_ckpt = self.training and (
|
| 386 |
+
(ckpt_mode == "heavy" and (i % 2 == 1)) or
|
| 387 |
+
(ckpt_mode == "light" and (i == 3))
|
| 388 |
+
)
|
| 389 |
+
if do_ckpt:
|
| 390 |
+
x, v_raw = grad_ckpt(layer, x, kc, vc, self.rope, enc_mask, v0, use_reentrant=False)
|
| 391 |
+
else:
|
| 392 |
+
x, v_raw = layer(x, kc, vc, self.rope, enc_mask, v0=v0)
|
| 393 |
+
if i == 0:
|
| 394 |
+
v0 = v_raw
|
| 395 |
+
return self.dec_norm(x)
|
| 396 |
+
|
| 397 |
+
def forward_and_backward(
|
| 398 |
+
self,
|
| 399 |
+
src_ids: torch.Tensor,
|
| 400 |
+
tgt_in: torch.Tensor,
|
| 401 |
+
labels: torch.Tensor,
|
| 402 |
+
main_denom: float,
|
| 403 |
+
mtp_denom: float,
|
| 404 |
+
label_smoothing: float = 0.1,
|
| 405 |
+
z_loss_weight: float = 1e-4,
|
| 406 |
+
mtp_weight: float = 0.3,
|
| 407 |
+
):
|
| 408 |
+
tot_tok = src_ids.numel() + tgt_in.numel()
|
| 409 |
+
max_len = max(src_ids.size(1), tgt_in.size(1))
|
| 410 |
+
if tot_tok <= 14336:
|
| 411 |
+
ckpt_mode = "none"
|
| 412 |
+
elif max_len <= 256:
|
| 413 |
+
ckpt_mode = "light"
|
| 414 |
+
else:
|
| 415 |
+
ckpt_mode = "heavy"
|
| 416 |
+
|
| 417 |
+
with torch.amp.autocast("cuda", dtype=torch.bfloat16):
|
| 418 |
+
cross_kv, enc_mask = self.encode_and_route(src_ids, ckpt_mode=ckpt_mode)
|
| 419 |
+
h_main = self.decode_seq(tgt_in, cross_kv, enc_mask, ckpt_mode=ckpt_mode)
|
| 420 |
+
|
| 421 |
+
valid_mask = labels.ne(-100)
|
| 422 |
+
valid_y = labels[valid_mask]
|
| 423 |
+
n_valid = valid_y.numel()
|
| 424 |
+
valid_h_leaf = h_main[valid_mask].detach().requires_grad_(True)
|
| 425 |
+
|
| 426 |
+
tot_main_ce_t = torch.zeros((), dtype=torch.float64, device=h_main.device)
|
| 427 |
+
tot_z_loss_t = torch.zeros((), dtype=torch.float64, device=h_main.device)
|
| 428 |
+
tot_mtp_ce_t = torch.zeros((), dtype=torch.float64, device=h_main.device)
|
| 429 |
+
chunk_sz = 2048
|
| 430 |
+
|
| 431 |
+
for i in range(0, n_valid, chunk_sz):
|
| 432 |
+
hc = valid_h_leaf[i : i + chunk_sz]
|
| 433 |
+
yc = valid_y[i : i + chunk_sz]
|
| 434 |
+
with torch.amp.autocast("cuda", dtype=torch.bfloat16):
|
| 435 |
+
logits_c = self.lm_head(hc)
|
| 436 |
+
ce_c = F.cross_entropy(logits_c, yc, label_smoothing=label_smoothing, reduction="sum")
|
| 437 |
+
lse = torch.logsumexp(logits_c.float(), dim=-1)
|
| 438 |
+
z_c = (lse * lse).sum()
|
| 439 |
+
loss_c = (ce_c + z_loss_weight * z_c) / main_denom
|
| 440 |
+
loss_c.backward()
|
| 441 |
+
tot_main_ce_t.add_(ce_c.detach().to(torch.float64))
|
| 442 |
+
tot_z_loss_t.add_(z_c.detach().to(torch.float64))
|
| 443 |
+
del logits_c, ce_c, z_c, loss_c
|
| 444 |
+
|
| 445 |
+
grad_h_main = torch.zeros_like(h_main)
|
| 446 |
+
if valid_h_leaf.grad is not None:
|
| 447 |
+
grad_h_main[valid_mask] = valid_h_leaf.grad
|
| 448 |
+
|
| 449 |
+
n_mtp = 0
|
| 450 |
+
if mtp_weight > 0.0 and tgt_in.size(1) > 1:
|
| 451 |
+
mtp_mask = labels[:, 1:].ne(-100) & labels[:, :-1].ne(-100)
|
| 452 |
+
valid_ym = labels[:, 1:][mtp_mask]
|
| 453 |
+
n_mtp = valid_ym.numel()
|
| 454 |
+
if n_mtp > 0:
|
| 455 |
+
h_main_leaf = h_main.detach().requires_grad_(True)
|
| 456 |
+
with torch.amp.autocast("cuda", dtype=torch.bfloat16):
|
| 457 |
+
next_emb = self.embed_tokens(tgt_in[:, 1:]) * self.embed_scale
|
| 458 |
+
h_mtp = self.mtp_head(h_main_leaf[:, :-1, :], next_emb, self.rope)
|
| 459 |
+
valid_hm_leaf = h_mtp[mtp_mask].detach().requires_grad_(True)
|
| 460 |
+
|
| 461 |
+
for i in range(0, n_mtp, chunk_sz):
|
| 462 |
+
hmc = valid_hm_leaf[i : i + chunk_sz]
|
| 463 |
+
ymc = valid_ym[i : i + chunk_sz]
|
| 464 |
+
with torch.amp.autocast("cuda", dtype=torch.bfloat16):
|
| 465 |
+
logits_m = self.lm_head(hmc)
|
| 466 |
+
ce_m = F.cross_entropy(logits_m, ymc, label_smoothing=label_smoothing, reduction="sum")
|
| 467 |
+
loss_m = (mtp_weight * ce_m) / mtp_denom
|
| 468 |
+
loss_m.backward()
|
| 469 |
+
tot_mtp_ce_t.add_(ce_m.detach().to(torch.float64))
|
| 470 |
+
del logits_m, ce_m, loss_m
|
| 471 |
+
|
| 472 |
+
grad_h_mtp = torch.zeros_like(h_mtp)
|
| 473 |
+
if valid_hm_leaf.grad is not None:
|
| 474 |
+
grad_h_mtp[mtp_mask] = valid_hm_leaf.grad
|
| 475 |
+
h_mtp.backward(grad_h_mtp)
|
| 476 |
+
if h_main_leaf.grad is not None:
|
| 477 |
+
grad_h_main.add_(h_main_leaf.grad)
|
| 478 |
+
|
| 479 |
+
h_main.backward(grad_h_main)
|
| 480 |
+
ce_v, mtp_v, z_v = torch.stack([tot_main_ce_t, tot_mtp_ce_t, tot_z_loss_t]).tolist()
|
| 481 |
+
return ce_v, mtp_v, z_v, n_valid, n_mtp
|
| 482 |
+
|
| 483 |
+
@torch.no_grad()
|
| 484 |
+
def eval_nll(self, src_ids: torch.Tensor, tgt_in: torch.Tensor, labels: torch.Tensor):
|
| 485 |
+
with torch.amp.autocast("cuda", dtype=torch.bfloat16):
|
| 486 |
+
cross_kv, enc_mask = self.encode_and_route(src_ids, ckpt_mode="none")
|
| 487 |
+
h_main = self.decode_seq(tgt_in, cross_kv, enc_mask, ckpt_mode="none")
|
| 488 |
+
valid_mask = labels.ne(-100)
|
| 489 |
+
valid_h = h_main[valid_mask]
|
| 490 |
+
valid_y = labels[valid_mask]
|
| 491 |
+
n_valid = valid_y.numel()
|
| 492 |
+
tot_nll = 0.0
|
| 493 |
+
for i in range(0, n_valid, 4096):
|
| 494 |
+
logits_c = self.lm_head(valid_h[i : i + 4096])
|
| 495 |
+
tot_nll += float(F.cross_entropy(logits_c, valid_y[i : i + 4096], reduction="sum").item())
|
| 496 |
+
return tot_nll, n_valid
|
| 497 |
+
|
| 498 |
+
@torch.inference_mode()
|
| 499 |
+
def translate_kv(self, prompt_ids_list: list[list[int]], sp_proc, max_new_tokens: int = 160, rep_penalty: float = 1.12) -> list[str]:
|
| 500 |
+
assert len(prompt_ids_list) > 0 and all(len(s) > 0 for s in prompt_ids_list), "Empty prompt batch!"
|
| 501 |
+
max_rope = self.rope.cos_cached.size(2)
|
| 502 |
+
assert 0 < max_new_tokens <= max_rope, f"max_new_tokens={max_new_tokens} exceeds RoPE={max_rope}"
|
| 503 |
+
|
| 504 |
+
self.eval()
|
| 505 |
+
dev = self.embed_tokens.weight.device
|
| 506 |
+
B = len(prompt_ids_list)
|
| 507 |
+
trimmed = [s[:max_rope] for s in prompt_ids_list]
|
| 508 |
+
max_s = min(max_rope, ((max(len(s) for s in trimmed) + 7) // 8) * 8)
|
| 509 |
+
src_ids = torch.full((B, max_s), PAD_ID, dtype=torch.long, device=dev)
|
| 510 |
+
for i, ids in enumerate(trimmed):
|
| 511 |
+
src_ids[i, : len(ids)] = torch.tensor(ids, dtype=torch.long, device=dev)
|
| 512 |
+
|
| 513 |
+
with torch.amp.autocast("cuda", dtype=torch.bfloat16):
|
| 514 |
+
cross_kv, enc_mask = self.encode_and_route(src_ids, ckpt_mode="none")
|
| 515 |
+
k_cache = torch.empty((self.dec_layers, B, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=torch.bfloat16, device=dev)
|
| 516 |
+
v_cache = torch.empty((self.dec_layers, B, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=torch.bfloat16, device=dev)
|
| 517 |
+
|
| 518 |
+
cur_tok = torch.full((B, 1), BOS_ID, dtype=torch.long, device=dev)
|
| 519 |
+
gen_ids = torch.full((B, max_new_tokens), EOS_ID, dtype=torch.long, device=dev)
|
| 520 |
+
done = torch.zeros(B, dtype=torch.bool, device=dev)
|
| 521 |
+
cos_all, sin_all = self.rope.cos_cached.to(torch.bfloat16), self.rope.sin_cached.to(torch.bfloat16)
|
| 522 |
+
|
| 523 |
+
for pos in range(max_new_tokens):
|
| 524 |
+
cos_p, sin_p = cos_all[:, :, pos : pos + 1, :], sin_all[:, :, pos : pos + 1, :]
|
| 525 |
+
x_dec = self.embed_tokens(cur_tok) * self.embed_scale
|
| 526 |
+
v0 = None
|
| 527 |
+
for l_idx, layer in enumerate(self.decoder):
|
| 528 |
+
kc, vc = cross_kv[l_idx // 2]
|
| 529 |
+
x_dec, v_raw = layer.step_kv(x_dec, cos_p, sin_p, pos, k_cache[l_idx], v_cache[l_idx], kc, vc, enc_mask, v0=v0)
|
| 530 |
+
if l_idx == 0:
|
| 531 |
+
v0 = v_raw
|
| 532 |
+
|
| 533 |
+
logits = self.lm_head(self.dec_norm(x_dec)[:, 0, :]).float()
|
| 534 |
+
logits[:, [0, 1, 3, 4, 5, 6, 7, 8, 9, 10]] = -1e9
|
| 535 |
+
if rep_penalty != 1.0 and pos > 0:
|
| 536 |
+
prev = gen_ids[:, :pos]
|
| 537 |
+
s_l = torch.gather(logits, 1, prev)
|
| 538 |
+
s_l = torch.where(s_l > 0, s_l / rep_penalty, s_l * rep_penalty)
|
| 539 |
+
logits.scatter_(1, prev, s_l)
|
| 540 |
+
|
| 541 |
+
nxt = torch.where(done, torch.tensor(EOS_ID, device=dev), logits.argmax(dim=-1))
|
| 542 |
+
gen_ids[:, pos] = nxt
|
| 543 |
+
done = done | (nxt == EOS_ID)
|
| 544 |
+
cur_tok = nxt.unsqueeze(1)
|
| 545 |
+
if bool(done.all()):
|
| 546 |
+
break
|
| 547 |
+
|
| 548 |
+
out_texts = []
|
| 549 |
+
for row in gen_ids.cpu().tolist():
|
| 550 |
+
if EOS_ID in row:
|
| 551 |
+
row = row[: row.index(EOS_ID)]
|
| 552 |
+
out_texts.append(sp_proc.decode_ids([x for x in row if x >= 11]).strip())
|
| 553 |
+
return out_texts
|
| 554 |
+
|
| 555 |
+
|
| 556 |
+
# ==============================================================================
|
| 557 |
+
# 2. 100%-Coverage Plan + Tight-Padding Dynamic Packing + Distributed Sharding
|
| 558 |
+
# ==============================================================================
|
| 559 |
+
class CanonicalMemmapStore:
|
| 560 |
+
def __init__(self, data_dir: str, split: str):
|
| 561 |
+
self.my_path = os.path.join(data_dir, f"{split}_my.bin")
|
| 562 |
+
self.en_path = os.path.join(data_dir, f"{split}_en.bin")
|
| 563 |
+
self.index = np.load(os.path.join(data_dir, f"{split}_index.npy"), mmap_mode="r")
|
| 564 |
+
self.my_mm = None
|
| 565 |
+
self.en_mm = None
|
| 566 |
+
|
| 567 |
+
def ensure_open(self):
|
| 568 |
+
if self.my_mm is None:
|
| 569 |
+
self.my_mm = np.memmap(self.my_path, dtype=np.uint16, mode="r")
|
| 570 |
+
self.en_mm = np.memmap(self.en_path, dtype=np.uint16, mode="r")
|
| 571 |
+
|
| 572 |
+
def get_pair(self, idx: int):
|
| 573 |
+
self.ensure_open()
|
| 574 |
+
rec = self.index[idx]
|
| 575 |
+
m = self.my_mm[int(rec["my_off"]) : int(rec["my_off"]) + int(rec["my_len"])].astype(np.int64)
|
| 576 |
+
e = self.en_mm[int(rec["en_off"]) : int(rec["en_off"]) + int(rec["en_len"])].astype(np.int64)
|
| 577 |
+
return m, e, int(rec["flags"])
|
| 578 |
+
|
| 579 |
+
|
| 580 |
+
def build_or_load_epoch_plan(train_index: np.ndarray, meta: dict, stage: str, plan_path: str, is_main: bool, world_size: int, seed: int = 2026) -> np.ndarray:
|
| 581 |
+
if is_main and not os.path.exists(plan_path):
|
| 582 |
+
rng = np.random.RandomState(seed)
|
| 583 |
+
n_subsets = len(meta["subset_to_id"])
|
| 584 |
+
sids = np.asarray(train_index["subset_id"])
|
| 585 |
+
assert int(sids.max()) < n_subsets and int(sids.min()) >= 0
|
| 586 |
+
|
| 587 |
+
if stage == "stable":
|
| 588 |
+
base_rows = rng.permutation(len(train_index)).astype(np.uint32)
|
| 589 |
+
replay_n = 12_000_000
|
| 590 |
+
eligible_mask = np.ones(len(train_index), dtype=bool)
|
| 591 |
+
for k, name in meta["id_to_subset"].items():
|
| 592 |
+
if name in BULK_SUBSETS:
|
| 593 |
+
eligible_mask[sids == int(k)] = False
|
| 594 |
+
else:
|
| 595 |
+
eligible_mask = (np.asarray(train_index["flags"]) & FLAG_BT) == 0
|
| 596 |
+
base_rows = np.flatnonzero(eligible_mask).astype(np.uint32)
|
| 597 |
+
rng.shuffle(base_rows)
|
| 598 |
+
replay_n = 1_500_000
|
| 599 |
+
|
| 600 |
+
elig_pos = np.flatnonzero(eligible_mask).astype(np.uint32)
|
| 601 |
+
elig_sids = sids[elig_pos]
|
| 602 |
+
order = elig_pos[np.argsort(elig_sids, kind="mergesort")]
|
| 603 |
+
sorted_sids = sids[order]
|
| 604 |
+
counts = np.bincount(sorted_sids, minlength=n_subsets)
|
| 605 |
+
starts = np.zeros(n_subsets, dtype=np.int64)
|
| 606 |
+
starts[1:] = np.cumsum(counts[:-1])
|
| 607 |
+
|
| 608 |
+
weights = np.zeros(n_subsets, dtype=np.float64)
|
| 609 |
+
for k, name in meta["id_to_subset"].items():
|
| 610 |
+
sid = int(k)
|
| 611 |
+
if counts[sid] == 0:
|
| 612 |
+
continue
|
| 613 |
+
if stage == "stable" and name in BULK_SUBSETS:
|
| 614 |
+
continue
|
| 615 |
+
if stage == "cooldown" and name == "non_myanmartranslator/finetranslations":
|
| 616 |
+
w = float(counts[sid]) ** 0.35 * 0.25
|
| 617 |
+
else:
|
| 618 |
+
w = float(counts[sid]) ** 0.50
|
| 619 |
+
if stage == "stable" and name.startswith("non_myanmartranslator/"):
|
| 620 |
+
w *= 4.0
|
| 621 |
+
weights[sid] = w
|
| 622 |
+
|
| 623 |
+
assert weights.sum() > 0, "Empty replay weights!"
|
| 624 |
+
probs = weights / weights.sum()
|
| 625 |
+
|
| 626 |
+
chosen_sids = rng.choice(n_subsets, size=replay_n, p=probs)
|
| 627 |
+
replay_rows = np.empty(replay_n, dtype=np.uint32)
|
| 628 |
+
for sid in range(n_subsets):
|
| 629 |
+
m = (chosen_sids == sid)
|
| 630 |
+
npick = int(m.sum())
|
| 631 |
+
if npick > 0:
|
| 632 |
+
replay_rows[m] = order[starts[sid] + rng.randint(0, counts[sid], size=npick)]
|
| 633 |
+
|
| 634 |
+
full_plan = np.concatenate([base_rows, replay_rows])
|
| 635 |
+
rng.shuffle(full_plan)
|
| 636 |
+
tmp_plan = plan_path + ".tmp.npy"
|
| 637 |
+
np.save(tmp_plan, full_plan)
|
| 638 |
+
os.replace(tmp_plan, plan_path)
|
| 639 |
+
print(
|
| 640 |
+
f"📋 Built {stage} 100%-coverage plan: {len(base_rows):,} unique base rows (100% coverage) "
|
| 641 |
+
f"+ {len(replay_rows):,} gold replay = {len(full_plan):,} total rows"
|
| 642 |
+
)
|
| 643 |
+
|
| 644 |
+
if world_size > 1:
|
| 645 |
+
dist.barrier()
|
| 646 |
+
plan = np.load(plan_path, mmap_mode="r")
|
| 647 |
+
if is_main:
|
| 648 |
+
print(f"📋 Ready {stage} plan: {len(plan):,} rows")
|
| 649 |
+
return plan
|
| 650 |
+
|
| 651 |
+
|
| 652 |
+
def pack_chunk_batches(
|
| 653 |
+
row_slice: np.ndarray,
|
| 654 |
+
train_index: np.ndarray,
|
| 655 |
+
max_tokens_combined: int,
|
| 656 |
+
max_seq_len: int,
|
| 657 |
+
step_group: int,
|
| 658 |
+
seed: int,
|
| 659 |
+
):
|
| 660 |
+
rng = np.random.RandomState(seed)
|
| 661 |
+
my_lens = np.minimum(max_seq_len, train_index["my_len"][row_slice].astype(np.int32))
|
| 662 |
+
en_lens = np.minimum(max_seq_len, train_index["en_len"][row_slice].astype(np.int32))
|
| 663 |
+
max_side = np.maximum(my_lens, en_lens)
|
| 664 |
+
comb_lens = my_lens + en_lens
|
| 665 |
+
|
| 666 |
+
mega = 65536
|
| 667 |
+
batches_cost = []
|
| 668 |
+
for i in range(0, len(row_slice), mega):
|
| 669 |
+
sub_r = row_slice[i : i + mega]
|
| 670 |
+
sub_ms = max_side[i : i + mega]
|
| 671 |
+
sub_cl = comb_lens[i : i + mega]
|
| 672 |
+
sub_r = sub_r[np.lexsort((sub_cl, sub_ms))]
|
| 673 |
+
|
| 674 |
+
cur_b, cur_s_max, cur_t_max = [], 0, 0
|
| 675 |
+
for r in sub_r:
|
| 676 |
+
ri = int(r)
|
| 677 |
+
ml = min(max_seq_len, int(train_index["my_len"][ri]))
|
| 678 |
+
el = min(max_seq_len, int(train_index["en_len"][ri]))
|
| 679 |
+
s_est = max(el, ml) + 4
|
| 680 |
+
t_est = max(ml, el) + 2
|
| 681 |
+
new_s = max(cur_s_max, s_est)
|
| 682 |
+
new_t = max(cur_t_max, t_est)
|
| 683 |
+
if cur_b and (len(cur_b) >= 256 or (len(cur_b) + 1) * (new_s + new_t) > max_tokens_combined):
|
| 684 |
+
batches_cost.append(((max(cur_s_max, cur_t_max), len(cur_b) * (cur_s_max + cur_t_max)), cur_b))
|
| 685 |
+
cur_b, cur_s_max, cur_t_max = [ri], s_est, t_est
|
| 686 |
+
else:
|
| 687 |
+
cur_b.append(ri)
|
| 688 |
+
cur_s_max, cur_t_max = new_s, new_t
|
| 689 |
+
if cur_b:
|
| 690 |
+
batches_cost.append(((max(cur_s_max, cur_t_max), len(cur_b) * (cur_s_max + cur_t_max)), cur_b))
|
| 691 |
+
|
| 692 |
+
# Load-balanced step grouping by (max_seq, tok_cost) + safe small-chunk pad
|
| 693 |
+
batches_cost.sort(key=lambda x: x[0])
|
| 694 |
+
batches = [b for _, b in batches_cost]
|
| 695 |
+
rem = len(batches) % step_group
|
| 696 |
+
if rem != 0 and len(batches) > 0:
|
| 697 |
+
need = step_group - rem
|
| 698 |
+
batches.extend((batches * ((need // len(batches)) + 1))[:need])
|
| 699 |
+
groups = [batches[j : j + step_group] for j in range(0, len(batches), step_group)]
|
| 700 |
+
rng.shuffle(groups)
|
| 701 |
+
return [b for g in groups for b in g]
|
| 702 |
+
|
| 703 |
+
|
| 704 |
+
class DynamicStreamDataset(Dataset):
|
| 705 |
+
def __init__(self, store: CanonicalMemmapStore, batches: list[list[int]], stage: str, max_seq_len: int, seed: int):
|
| 706 |
+
self.store = store
|
| 707 |
+
self.batches = batches
|
| 708 |
+
self.stage = stage
|
| 709 |
+
self.max_seq_len = max_seq_len
|
| 710 |
+
self.base_seed = seed
|
| 711 |
+
self.rng = np.random.RandomState(seed)
|
| 712 |
+
|
| 713 |
+
def __len__(self):
|
| 714 |
+
return len(self.batches)
|
| 715 |
+
|
| 716 |
+
def _corrupt_span(self, toks: np.ndarray):
|
| 717 |
+
n = len(toks)
|
| 718 |
+
if n < 4:
|
| 719 |
+
return toks
|
| 720 |
+
out = []
|
| 721 |
+
i = 0
|
| 722 |
+
masked_any = False
|
| 723 |
+
while i < n:
|
| 724 |
+
if self.rng.rand() < 0.15:
|
| 725 |
+
span_len = min(n - i, max(1, int(self.rng.poisson(3.0))))
|
| 726 |
+
out.append(TOK_MASK)
|
| 727 |
+
i += span_len
|
| 728 |
+
masked_any = True
|
| 729 |
+
else:
|
| 730 |
+
out.append(int(toks[i]))
|
| 731 |
+
i += 1
|
| 732 |
+
if not masked_any:
|
| 733 |
+
idx = int(self.rng.randint(0, n))
|
| 734 |
+
out = list(toks[:idx]) + [TOK_MASK] + list(toks[idx + min(2, n - idx):])
|
| 735 |
+
return np.asarray(out, dtype=np.int64)
|
| 736 |
+
|
| 737 |
+
def __getitem__(self, b_idx: int):
|
| 738 |
+
row_ids = self.batches[b_idx]
|
| 739 |
+
srcs, tgts = [], []
|
| 740 |
+
for r_id in row_ids:
|
| 741 |
+
m, e, flg = self.store.get_pair(r_id)
|
| 742 |
+
m = m[: self.max_seq_len - 4]
|
| 743 |
+
e = e[: self.max_seq_len - 4]
|
| 744 |
+
is_bt = bool(flg & FLAG_BT)
|
| 745 |
+
is_formal = bool(flg & FLAG_FORMAL)
|
| 746 |
+
is_spoken = bool(flg & FLAG_SPOKEN)
|
| 747 |
+
|
| 748 |
+
if self.stage == "cooldown":
|
| 749 |
+
mode = "en2my" if self.rng.rand() < 0.50 else "my2en"
|
| 750 |
+
else:
|
| 751 |
+
p = self.rng.rand()
|
| 752 |
+
if is_bt:
|
| 753 |
+
mode = "en2my" if p < 0.72 else ("my2en_bt" if p < 0.90 else "denoise")
|
| 754 |
+
else:
|
| 755 |
+
mode = "en2my" if p < 0.47 else ("my2en" if p < 0.94 else "denoise")
|
| 756 |
+
|
| 757 |
+
style_tok = None
|
| 758 |
+
if is_formal and self.rng.rand() < 0.70:
|
| 759 |
+
style_tok = TOK_FORMAL
|
| 760 |
+
elif is_spoken and self.rng.rand() < 0.70:
|
| 761 |
+
style_tok = TOK_SPOKEN
|
| 762 |
+
|
| 763 |
+
if mode == "en2my":
|
| 764 |
+
pref = [TOK_2MY] + ([style_tok] if style_tok is not None else [])
|
| 765 |
+
src_arr = np.concatenate([np.asarray(pref, dtype=np.int64), e, [EOS_ID]])
|
| 766 |
+
tgt_arr = np.concatenate([[BOS_ID], m, [EOS_ID]])
|
| 767 |
+
elif mode in ("my2en", "my2en_bt"):
|
| 768 |
+
pref = [TOK_2EN] + ([TOK_BT] if mode == "my2en_bt" else []) + ([style_tok] if style_tok is not None else [])
|
| 769 |
+
src_arr = np.concatenate([np.asarray(pref, dtype=np.int64), m, [EOS_ID]])
|
| 770 |
+
tgt_arr = np.concatenate([[BOS_ID], e, [EOS_ID]])
|
| 771 |
+
else:
|
| 772 |
+
base = m if (is_bt or self.rng.rand() < 0.5) else e
|
| 773 |
+
noisy = self._corrupt_span(base)
|
| 774 |
+
src_arr = np.concatenate([[TOK_DENOISE], noisy, [EOS_ID]])
|
| 775 |
+
tgt_arr = np.concatenate([[BOS_ID], base, [EOS_ID]])
|
| 776 |
+
|
| 777 |
+
srcs.append(src_arr)
|
| 778 |
+
tgts.append(tgt_arr)
|
| 779 |
+
return srcs, tgts
|
| 780 |
+
|
| 781 |
+
|
| 782 |
+
def worker_init_fn(worker_id: int):
|
| 783 |
+
info = torch.utils.data.get_worker_info()
|
| 784 |
+
if info is not None and hasattr(info.dataset, "rng"):
|
| 785 |
+
info.dataset.rng = np.random.RandomState((info.dataset.base_seed + worker_id * 9973) % (2**31 - 1))
|
| 786 |
+
|
| 787 |
+
|
| 788 |
+
def collate_prebatched(batch_list):
|
| 789 |
+
srcs, tgts = batch_list[0]
|
| 790 |
+
B = len(srcs)
|
| 791 |
+
max_s = min(2048, ((max(len(x) for x in srcs) + 31) // 32) * 32)
|
| 792 |
+
max_t = min(2048, ((max(len(x) - 1 for x in tgts) + 31) // 32) * 32)
|
| 793 |
+
src_ids = torch.full((B, max_s), PAD_ID, dtype=torch.long)
|
| 794 |
+
tgt_in = torch.full((B, max_t), PAD_ID, dtype=torch.long)
|
| 795 |
+
labels = torch.full((B, max_t), -100, dtype=torch.long)
|
| 796 |
+
for i, (s, t) in enumerate(zip(srcs, tgts)):
|
| 797 |
+
sl = min(len(s), max_s)
|
| 798 |
+
tl = min(len(t) - 1, max_t)
|
| 799 |
+
src_ids[i, :sl] = torch.from_numpy(s[:sl])
|
| 800 |
+
tgt_in[i, :tl] = torch.from_numpy(t[:tl])
|
| 801 |
+
labels[i, :tl] = torch.from_numpy(t[1 : tl + 1])
|
| 802 |
+
return src_ids, tgt_in, labels
|
| 803 |
+
|
| 804 |
+
|
| 805 |
+
def clean_state_dict(model: nn.Module):
|
| 806 |
+
return {k.replace("._orig_mod.", "."): v.detach().cpu() for k, v in model.state_dict().items()}
|
| 807 |
+
|
| 808 |
+
|
| 809 |
+
def broadcast_model(model: nn.Module, world_size: int):
|
| 810 |
+
if world_size == 1:
|
| 811 |
+
return
|
| 812 |
+
for p in model.parameters():
|
| 813 |
+
dist.broadcast(p.data, src=0)
|
| 814 |
+
for b in model.buffers():
|
| 815 |
+
dist.broadcast(b.data, src=0)
|
| 816 |
+
|
| 817 |
+
|
| 818 |
+
def allreduce_grads(model: nn.Module, world_size: int, grad_scale: float = 1024.0):
|
| 819 |
+
if world_size == 1:
|
| 820 |
+
return
|
| 821 |
+
grads = []
|
| 822 |
+
for p in model.parameters():
|
| 823 |
+
if p.requires_grad:
|
| 824 |
+
if p.grad is None:
|
| 825 |
+
p.grad = torch.zeros_like(p.data)
|
| 826 |
+
grads.append(p.grad)
|
| 827 |
+
b_grads = [g.bfloat16() for g in grads]
|
| 828 |
+
flat = torch._utils._flatten_dense_tensors(b_grads)
|
| 829 |
+
dist.all_reduce(flat, op=dist.ReduceOp.SUM)
|
| 830 |
+
inv_scale = 1.0 / grad_scale
|
| 831 |
+
for orig, synced in zip(grads, torch._utils._unflatten_dense_tensors(flat, b_grads)):
|
| 832 |
+
orig.copy_(synced.float() * inv_scale)
|
| 833 |
+
|
| 834 |
+
|
| 835 |
+
# ==============================================================================
|
| 836 |
+
# 3. Main Multi-GPU Training Loop
|
| 837 |
+
# ==============================================================================
|
| 838 |
+
def main():
|
| 839 |
+
parser = argparse.ArgumentParser()
|
| 840 |
+
parser.add_argument("--stage", choices=["stable", "cooldown"], default="stable")
|
| 841 |
+
parser.add_argument("--verify-steps", type=int, default=0)
|
| 842 |
+
parser.add_argument("--grad-accum", type=int, default=1)
|
| 843 |
+
parser.add_argument("--max-tok-micro", type=int, default=32768)
|
| 844 |
+
parser.add_argument("--max-seq-len", type=int, default=1024)
|
| 845 |
+
parser.add_argument("--chunk-rows", type=int, default=4000000)
|
| 846 |
+
parser.add_argument("--peak-lr", type=float, default=5.0e-4)
|
| 847 |
+
parser.add_argument("--warmup-steps", type=int, default=900)
|
| 848 |
+
parser.add_argument("--eval-every", type=int, default=500)
|
| 849 |
+
parser.add_argument("--save-every", type=int, default=500)
|
| 850 |
+
parser.add_argument("--milestone-every", type=int, default=5000)
|
| 851 |
+
parser.add_argument("--compile", action="store_true")
|
| 852 |
+
parser.add_argument("--fresh", action="store_true")
|
| 853 |
+
parser.add_argument("--init-weights", type=str, default="")
|
| 854 |
+
args = parser.parse_args()
|
| 855 |
+
|
| 856 |
+
world_size = int(os.environ.get("WORLD_SIZE", 1))
|
| 857 |
+
rank = int(os.environ.get("RANK", 0))
|
| 858 |
+
local_rank = int(os.environ.get("LOCAL_RANK", 0))
|
| 859 |
+
is_main = (rank == 0)
|
| 860 |
+
|
| 861 |
+
if world_size > 1:
|
| 862 |
+
torch.cuda.set_device(local_rank)
|
| 863 |
+
dist.init_process_group(backend="nccl", timeout=timedelta(minutes=30), device_id=torch.device(f"cuda:{local_rank}"))
|
| 864 |
+
device = torch.device("cuda", local_rank)
|
| 865 |
+
else:
|
| 866 |
+
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 867 |
+
|
| 868 |
+
if is_main:
|
| 869 |
+
os.makedirs(CKPT_DIR, exist_ok=True)
|
| 870 |
+
if world_size > 1:
|
| 871 |
+
dist.barrier()
|
| 872 |
+
|
| 873 |
+
meta, sp = load_and_verify_contracts(DATA_DIR)
|
| 874 |
+
train_store = CanonicalMemmapStore(DATA_DIR, "train")
|
| 875 |
+
val_store = CanonicalMemmapStore(DATA_DIR, "val")
|
| 876 |
+
|
| 877 |
+
all_val_non_bt = np.flatnonzero((val_store.index["flags"] & FLAG_BT) == 0)
|
| 878 |
+
assert len(all_val_non_bt) > 0, "Validation set has zero non-BT rows!"
|
| 879 |
+
val_rows = all_val_non_bt[::max(1, len(all_val_non_bt) // 1024)][:1024]
|
| 880 |
+
val_batches = [val_rows[i : i + 64].tolist() for i in range(0, len(val_rows), 64)]
|
| 881 |
+
|
| 882 |
+
plan_path = os.path.join(CKPT_DIR, f"{args.stage}_plan.npy")
|
| 883 |
+
latest_path_early = os.path.join(CKPT_DIR, "latest.pt")
|
| 884 |
+
if args.fresh and is_main:
|
| 885 |
+
if os.path.exists(plan_path):
|
| 886 |
+
os.remove(plan_path)
|
| 887 |
+
if os.path.exists(latest_path_early):
|
| 888 |
+
os.replace(latest_path_early, latest_path_early + ".bak")
|
| 889 |
+
if world_size > 1:
|
| 890 |
+
dist.barrier()
|
| 891 |
+
full_plan = build_or_load_epoch_plan(train_store.index, meta, args.stage, plan_path, is_main, world_size, seed=2026)
|
| 892 |
+
num_chunks = (len(full_plan) + args.chunk_rows - 1) // args.chunk_rows
|
| 893 |
+
|
| 894 |
+
model = MyTranslation300M().to(device)
|
| 895 |
+
decay_p = [p for n, p in model.named_parameters() if p.requires_grad and p.ndim >= 2]
|
| 896 |
+
nodecay_p = [p for n, p in model.named_parameters() if p.requires_grad and p.ndim < 2]
|
| 897 |
+
opt = torch.optim.AdamW(
|
| 898 |
+
[{"params": decay_p, "weight_decay": 0.01}, {"params": nodecay_p, "weight_decay": 0.0}],
|
| 899 |
+
lr=args.peak_lr, betas=(0.9, 0.95), eps=1e-8, fused=True,
|
| 900 |
+
)
|
| 901 |
+
|
| 902 |
+
step = 0
|
| 903 |
+
chunk_idx = 0
|
| 904 |
+
batch_cursor = 0
|
| 905 |
+
rows_seen = 0
|
| 906 |
+
best_val_nll = float("inf")
|
| 907 |
+
latest_path = os.path.join(CKPT_DIR, "latest.pt")
|
| 908 |
+
|
| 909 |
+
if args.init_weights:
|
| 910 |
+
if is_main:
|
| 911 |
+
print(f"📥 Loading weights from {args.init_weights}")
|
| 912 |
+
model.load_state_dict(torch.load(args.init_weights, map_location="cpu"), strict=True)
|
| 913 |
+
elif (not args.fresh) and os.path.exists(latest_path):
|
| 914 |
+
if is_main:
|
| 915 |
+
print(f"🔄 Resuming from {latest_path}")
|
| 916 |
+
ckpt = torch.load(latest_path, map_location="cpu")
|
| 917 |
+
model.load_state_dict(ckpt["model"], strict=True)
|
| 918 |
+
if args.stage == ckpt.get("stage", "stable"):
|
| 919 |
+
opt.load_state_dict(ckpt["opt"])
|
| 920 |
+
step = int(ckpt["step"])
|
| 921 |
+
chunk_idx = int(ckpt.get("chunk_idx", 0))
|
| 922 |
+
batch_cursor = int(ckpt.get("batch_cursor", 0))
|
| 923 |
+
rows_seen = int(ckpt.get("rows_seen", 0))
|
| 924 |
+
best_val_nll = float(ckpt.get("best_val_nll", float("inf")))
|
| 925 |
+
elif is_main:
|
| 926 |
+
print(f"➡️ Stage transition {ckpt.get('stage')} -> {args.stage}")
|
| 927 |
+
|
| 928 |
+
broadcast_model(model, world_size)
|
| 929 |
+
|
| 930 |
+
if args.compile:
|
| 931 |
+
if is_main:
|
| 932 |
+
print("⚡ Compiling encoder/cross_kv/decoder/MTP blocks with torch.compile(dynamic=True)...")
|
| 933 |
+
model.compile_blocks()
|
| 934 |
+
|
| 935 |
+
if is_main:
|
| 936 |
+
tot_params = sum(p.numel() for p in model.parameters())
|
| 937 |
+
inf_params = tot_params - sum(p.numel() for p in model.mtp_head.parameters())
|
| 938 |
+
print(f"🏛️ MyTranslation-300M | GPUs: {world_size}x | Stage: {args.stage} | Inf: {inf_params/1e6:.2f}M | Train: {tot_params/1e6:.2f}M")
|
| 939 |
+
|
| 940 |
+
label_smoothing = 0.1 if args.stage == "stable" else 0.0
|
| 941 |
+
mtp_weight = 0.3 if args.stage == "stable" else 0.15
|
| 942 |
+
|
| 943 |
+
STYLE_EVAL_PROMPTS = [
|
| 944 |
+
("En->My Formal", [TOK_2MY, TOK_FORMAL], "The central bank raised interest rates to control inflation and stabilize the economy."),
|
| 945 |
+
("En->My Spoken", [TOK_2MY, TOK_SPOKEN], "Honestly, I didn't expect things to turn out like this, but don't worry—we'll figure it out together."),
|
| 946 |
+
("My->En Formal", [TOK_2EN, TOK_FORMAL], "ဗဟိုဘဏ်သည် ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးတည်ငြိမ်စေရန် အတိုးနှုန်းများကို မြှင့်တင်ခဲ့သည်။"),
|
| 947 |
+
("My->En Spoken", [TOK_2EN, TOK_SPOKEN], "ဒီနေ့ ရန်ကုန်မှာ မိုးတွေ အရမ်းရွာနေလို့ အိမ်ပြန်ရတာ နည်းနည်းနောက်ကျမယ်နော်။"),
|
| 948 |
+
]
|
| 949 |
+
|
| 950 |
+
@torch.no_grad()
|
| 951 |
+
def eval_val_nll():
|
| 952 |
+
model.eval()
|
| 953 |
+
en2my_nll, en2my_tok = 0.0, 0
|
| 954 |
+
my2en_nll, my2en_tok = 0.0, 0
|
| 955 |
+
for b in val_batches:
|
| 956 |
+
for direction in ("en2my", "my2en"):
|
| 957 |
+
srcs, tgts = [], []
|
| 958 |
+
for r_id in b:
|
| 959 |
+
m, e, _ = val_store.get_pair(int(r_id))
|
| 960 |
+
m, e = m[: args.max_seq_len - 4], e[: args.max_seq_len - 4]
|
| 961 |
+
if direction == "en2my":
|
| 962 |
+
srcs.append(np.concatenate([[TOK_2MY], e, [EOS_ID]]))
|
| 963 |
+
tgts.append(np.concatenate([[BOS_ID], m, [EOS_ID]]))
|
| 964 |
+
else:
|
| 965 |
+
srcs.append(np.concatenate([[TOK_2EN], m, [EOS_ID]]))
|
| 966 |
+
tgts.append(np.concatenate([[BOS_ID], e, [EOS_ID]]))
|
| 967 |
+
src_ids, tgt_in, labels = collate_prebatched([(srcs, tgts)])
|
| 968 |
+
nll, nv = model.eval_nll(src_ids.to(device), tgt_in.to(device), labels.to(device))
|
| 969 |
+
if direction == "en2my":
|
| 970 |
+
en2my_nll += nll
|
| 971 |
+
en2my_tok += nv
|
| 972 |
+
else:
|
| 973 |
+
my2en_nll += nll
|
| 974 |
+
my2en_tok += nv
|
| 975 |
+
e2m = en2my_nll / max(1, en2my_tok)
|
| 976 |
+
m2e = my2en_nll / max(1, my2en_tok)
|
| 977 |
+
comb = (en2my_nll + my2en_nll) / max(1, en2my_tok + my2en_tok)
|
| 978 |
+
return comb, e2m, m2e
|
| 979 |
+
|
| 980 |
+
def run_style_generations(step_num: int):
|
| 981 |
+
prompts = [pref + sp.encode_as_ids(txt)[:500] + [EOS_ID] for _, pref, txt in STYLE_EVAL_PROMPTS]
|
| 982 |
+
preds = model.translate_kv(prompts, sp, max_new_tokens=96)
|
| 983 |
+
print(f" 🗣️ Style Generations @ Step {step_num}:")
|
| 984 |
+
for (tag, _, _), out in zip(STYLE_EVAL_PROMPTS, preds):
|
| 985 |
+
print(f" [{tag}]: {out[:180]}")
|
| 986 |
+
|
| 987 |
+
def make_chunk_loader(c_idx: int, skip_steps: int = 0):
|
| 988 |
+
s_off = c_idx * args.chunk_rows
|
| 989 |
+
e_off = min(len(full_plan), s_off + args.chunk_rows)
|
| 990 |
+
row_slice = np.asarray(full_plan[s_off:e_off])
|
| 991 |
+
step_group = world_size * args.grad_accum
|
| 992 |
+
all_batches = pack_chunk_batches(
|
| 993 |
+
row_slice, train_store.index,
|
| 994 |
+
max_tokens_combined=args.max_tok_micro,
|
| 995 |
+
max_seq_len=args.max_seq_len,
|
| 996 |
+
step_group=step_group,
|
| 997 |
+
seed=2026 + c_idx * 17,
|
| 998 |
+
)
|
| 999 |
+
rank_batches = all_batches[rank::world_size]
|
| 1000 |
+
if skip_steps > 0:
|
| 1001 |
+
rank_batches = rank_batches[skip_steps * args.grad_accum :]
|
| 1002 |
+
n_windows = len(rank_batches) // args.grad_accum
|
| 1003 |
+
rank_batches = rank_batches[: n_windows * args.grad_accum]
|
| 1004 |
+
ds = DynamicStreamDataset(
|
| 1005 |
+
train_store, rank_batches, args.stage, args.max_seq_len,
|
| 1006 |
+
seed=10000 + c_idx * 997 + rank * 131 + skip_steps,
|
| 1007 |
+
)
|
| 1008 |
+
loader = DataLoader(
|
| 1009 |
+
ds, batch_size=1, shuffle=False, collate_fn=collate_prebatched,
|
| 1010 |
+
num_workers=4, pin_memory=True, worker_init_fn=worker_init_fn, persistent_workers=False,
|
| 1011 |
+
)
|
| 1012 |
+
return loader, n_windows
|
| 1013 |
+
|
| 1014 |
+
if is_main:
|
| 1015 |
+
comb, e2m, m2e = eval_val_nll()
|
| 1016 |
+
print(f"🔎 Step {step:6d} | Val Comb: {comb:.4f} | En->My: {e2m:.4f} | My->En: {m2e:.4f}")
|
| 1017 |
+
if world_size > 1:
|
| 1018 |
+
dist.barrier()
|
| 1019 |
+
|
| 1020 |
+
model.train()
|
| 1021 |
+
opt.zero_grad(set_to_none=True)
|
| 1022 |
+
acc_ce = acc_mtp = acc_z = 0.0
|
| 1023 |
+
acc_valid_tok = acc_mtp_tok = acc_tot_tok = 0
|
| 1024 |
+
t0 = time.time()
|
| 1025 |
+
|
| 1026 |
+
while chunk_idx < num_chunks:
|
| 1027 |
+
train_loader, n_windows = make_chunk_loader(chunk_idx, skip_steps=batch_cursor)
|
| 1028 |
+
train_iter = iter(train_loader)
|
| 1029 |
+
done_flag = False
|
| 1030 |
+
|
| 1031 |
+
for _ in range(n_windows):
|
| 1032 |
+
window = [next(train_iter) for _ in range(args.grad_accum)]
|
| 1033 |
+
batch_cursor += 1
|
| 1034 |
+
|
| 1035 |
+
loc_valid = sum(int((lb != -100).sum().item()) for _, _, lb in window)
|
| 1036 |
+
loc_mtp = sum(
|
| 1037 |
+
int(((lb[:, 1:] != -100) & (lb[:, :-1] != -100)).sum().item()) if lb.size(1) > 1 else 0
|
| 1038 |
+
for _, _, lb in window
|
| 1039 |
+
)
|
| 1040 |
+
loc_rows = sum(s.size(0) for s, _, _ in window)
|
| 1041 |
+
|
| 1042 |
+
if world_size > 1:
|
| 1043 |
+
counts_t = torch.tensor([loc_valid, loc_mtp, loc_rows], dtype=torch.long, device=device)
|
| 1044 |
+
dist.all_reduce(counts_t, op=dist.ReduceOp.SUM)
|
| 1045 |
+
glob_valid, glob_mtp, glob_rows = int(counts_t[0].item()), int(counts_t[1].item()), int(counts_t[2].item())
|
| 1046 |
+
else:
|
| 1047 |
+
glob_valid, glob_mtp, glob_rows = loc_valid, loc_mtp, loc_rows
|
| 1048 |
+
|
| 1049 |
+
g_scale = 1024.0 if world_size > 1 else 1.0
|
| 1050 |
+
main_denom = float(max(1, glob_valid)) / g_scale
|
| 1051 |
+
mtp_denom = float(max(1, glob_mtp)) / g_scale
|
| 1052 |
+
|
| 1053 |
+
loc_ce = loc_mtp_ce = loc_z = 0.0
|
| 1054 |
+
loc_tok = sum(int((s != PAD_ID).sum().item()) for s, _, _ in window)
|
| 1055 |
+
for src_ids, tgt_in, labels in window:
|
| 1056 |
+
src_ids = src_ids.to(device, non_blocking=True)
|
| 1057 |
+
tgt_in = tgt_in.to(device, non_blocking=True)
|
| 1058 |
+
labels = labels.to(device, non_blocking=True)
|
| 1059 |
+
ce_s, mtp_s, z_s, nv, nm = model.forward_and_backward(
|
| 1060 |
+
src_ids, tgt_in, labels,
|
| 1061 |
+
main_denom=main_denom, mtp_denom=mtp_denom,
|
| 1062 |
+
label_smoothing=label_smoothing, mtp_weight=mtp_weight,
|
| 1063 |
+
)
|
| 1064 |
+
loc_ce += ce_s
|
| 1065 |
+
loc_mtp_ce += mtp_s
|
| 1066 |
+
loc_z += z_s
|
| 1067 |
+
loc_tok += nv
|
| 1068 |
+
|
| 1069 |
+
allreduce_grads(model, world_size)
|
| 1070 |
+
rows_seen += glob_rows
|
| 1071 |
+
|
| 1072 |
+
if args.stage == "stable":
|
| 1073 |
+
lr = args.peak_lr * min(1.0, (step + 1) / max(1, args.warmup_steps))
|
| 1074 |
+
else:
|
| 1075 |
+
if step < 100:
|
| 1076 |
+
lr = args.peak_lr * (step + 1) / 100.0
|
| 1077 |
+
else:
|
| 1078 |
+
progress = min(1.0, rows_seen / max(1, len(full_plan)))
|
| 1079 |
+
lr = 1.0e-5 + 0.5 * (args.peak_lr - 1.0e-5) * (1.0 + math.cos(math.pi * progress))
|
| 1080 |
+
|
| 1081 |
+
for pg in opt.param_groups:
|
| 1082 |
+
pg["lr"] = lr
|
| 1083 |
+
grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
|
| 1084 |
+
opt.step()
|
| 1085 |
+
opt.zero_grad(set_to_none=True)
|
| 1086 |
+
step += 1
|
| 1087 |
+
|
| 1088 |
+
if world_size > 1:
|
| 1089 |
+
stats_t = torch.tensor([loc_ce, loc_mtp_ce, loc_z, float(loc_tok)], dtype=torch.float64, device=device)
|
| 1090 |
+
dist.all_reduce(stats_t, op=dist.ReduceOp.SUM)
|
| 1091 |
+
acc_ce += float(stats_t[0].item())
|
| 1092 |
+
acc_mtp += float(stats_t[1].item())
|
| 1093 |
+
acc_z += float(stats_t[2].item())
|
| 1094 |
+
acc_tot_tok += int(stats_t[3].item())
|
| 1095 |
+
else:
|
| 1096 |
+
acc_ce += loc_ce
|
| 1097 |
+
acc_mtp += loc_mtp_ce
|
| 1098 |
+
acc_z += loc_z
|
| 1099 |
+
acc_tot_tok += loc_tok
|
| 1100 |
+
acc_valid_tok += glob_valid
|
| 1101 |
+
acc_mtp_tok += glob_mtp
|
| 1102 |
+
|
| 1103 |
+
log_int = 10 if args.verify_steps > 0 else 50
|
| 1104 |
+
if step % log_int == 0:
|
| 1105 |
+
if is_main:
|
| 1106 |
+
dt = time.time() - t0
|
| 1107 |
+
tok_s = acc_tot_tok / max(1e-3, dt)
|
| 1108 |
+
vram_g = torch.cuda.max_memory_allocated(device) / 1e9
|
| 1109 |
+
pct = 100.0 * min(rows_seen, len(full_plan)) / len(full_plan)
|
| 1110 |
+
print(
|
| 1111 |
+
f"Step {step:6d} ({pct:5.1f}% | Chunk {chunk_idx+1}/{num_chunks}) | "
|
| 1112 |
+
f"Main: {acc_ce/max(1,acc_valid_tok):.4f} | MTP2: {acc_mtp/max(1,acc_mtp_tok):.4f} | "
|
| 1113 |
+
f"Z: {acc_z/max(1,acc_valid_tok):.2f} | LR: {lr:.2e} | Grad: {grad_norm:.2f} | "
|
| 1114 |
+
f"{tok_s/1e3:.1f}k tok/s | VRAM: {vram_g:.1f}G"
|
| 1115 |
+
)
|
| 1116 |
+
acc_ce = acc_mtp = acc_z = 0.0
|
| 1117 |
+
acc_valid_tok = acc_mtp_tok = acc_tot_tok = 0
|
| 1118 |
+
t0 = time.time()
|
| 1119 |
+
|
| 1120 |
+
if args.verify_steps > 0 and step >= args.verify_steps:
|
| 1121 |
+
done_flag = True
|
| 1122 |
+
break
|
| 1123 |
+
|
| 1124 |
+
if step % args.eval_every == 0:
|
| 1125 |
+
if is_main:
|
| 1126 |
+
comb, e2m, m2e = eval_val_nll()
|
| 1127 |
+
print("-" * 96)
|
| 1128 |
+
print(f"🔎 EVAL @ Step {step:6d} | Comb NLL: {comb:.4f} (PPL {math.exp(min(comb, 30.0)):.2f}) | En->My: {e2m:.4f} | My->En: {m2e:.4f}")
|
| 1129 |
+
run_style_generations(step)
|
| 1130 |
+
if comb < best_val_nll:
|
| 1131 |
+
best_val_nll = comb
|
| 1132 |
+
best_sd = {k: v.to(torch.bfloat16) for k, v in clean_state_dict(model).items()}
|
| 1133 |
+
torch.save(best_sd, os.path.join(CKPT_DIR, "best_model.pt"))
|
| 1134 |
+
print(f" ⭐ New Best Model saved! Comb NLL: {best_val_nll:.4f}")
|
| 1135 |
+
print("-" * 96)
|
| 1136 |
+
model.train()
|
| 1137 |
+
if world_size > 1:
|
| 1138 |
+
dist.barrier()
|
| 1139 |
+
t0 = time.time()
|
| 1140 |
+
|
| 1141 |
+
if step % args.save_every == 0:
|
| 1142 |
+
if is_main:
|
| 1143 |
+
tmp_p = latest_path + ".tmp"
|
| 1144 |
+
torch.save({
|
| 1145 |
+
"model": clean_state_dict(model),
|
| 1146 |
+
"opt": opt.state_dict(),
|
| 1147 |
+
"step": step,
|
| 1148 |
+
"stage": args.stage,
|
| 1149 |
+
"chunk_idx": chunk_idx,
|
| 1150 |
+
"batch_cursor": batch_cursor,
|
| 1151 |
+
"rows_seen": rows_seen,
|
| 1152 |
+
"best_val_nll": best_val_nll,
|
| 1153 |
+
}, tmp_p)
|
| 1154 |
+
os.replace(tmp_p, latest_path)
|
| 1155 |
+
if world_size > 1:
|
| 1156 |
+
dist.barrier()
|
| 1157 |
+
|
| 1158 |
+
if step % args.milestone_every == 0:
|
| 1159 |
+
if is_main:
|
| 1160 |
+
ms_sd = {k: v.to(torch.bfloat16) for k, v in clean_state_dict(model).items()}
|
| 1161 |
+
torch.save(ms_sd, os.path.join(CKPT_DIR, f"{args.stage}_step_{step:06d}.pt"))
|
| 1162 |
+
if world_size > 1:
|
| 1163 |
+
dist.barrier()
|
| 1164 |
+
|
| 1165 |
+
if done_flag:
|
| 1166 |
+
break
|
| 1167 |
+
chunk_idx += 1
|
| 1168 |
+
batch_cursor = 0
|
| 1169 |
+
|
| 1170 |
+
if args.verify_steps > 0:
|
| 1171 |
+
if is_main:
|
| 1172 |
+
print(f"✅ {world_size}-GPU VERIFY PASSED ({step} steps)! Ready for launch.")
|
| 1173 |
+
if world_size > 1:
|
| 1174 |
+
dist.barrier()
|
| 1175 |
+
dist.destroy_process_group()
|
| 1176 |
+
return
|
| 1177 |
+
|
| 1178 |
+
if is_main:
|
| 1179 |
+
comb, e2m, m2e = eval_val_nll()
|
| 1180 |
+
print("-" * 96)
|
| 1181 |
+
print(f"🏁 FINAL EVAL @ Step {step:6d} | Comb NLL: {comb:.4f} (PPL {math.exp(min(comb, 30.0)):.2f}) | En->My: {e2m:.4f} | My->En: {m2e:.4f}")
|
| 1182 |
+
run_style_generations(step)
|
| 1183 |
+
if comb < best_val_nll:
|
| 1184 |
+
best_val_nll = comb
|
| 1185 |
+
best_sd = {k: v.to(torch.bfloat16) for k, v in clean_state_dict(model).items()}
|
| 1186 |
+
torch.save(best_sd, os.path.join(CKPT_DIR, "best_model.pt"))
|
| 1187 |
+
tmp_p = latest_path + ".tmp"
|
| 1188 |
+
torch.save({
|
| 1189 |
+
"model": clean_state_dict(model),
|
| 1190 |
+
"opt": opt.state_dict(),
|
| 1191 |
+
"step": step,
|
| 1192 |
+
"stage": args.stage,
|
| 1193 |
+
"chunk_idx": chunk_idx,
|
| 1194 |
+
"batch_cursor": batch_cursor,
|
| 1195 |
+
"rows_seen": rows_seen,
|
| 1196 |
+
"best_val_nll": best_val_nll,
|
| 1197 |
+
}, tmp_p)
|
| 1198 |
+
os.replace(tmp_p, latest_path)
|
| 1199 |
+
|
| 1200 |
+
with torch.no_grad():
|
| 1201 |
+
rw = F.softmax(model.enc_router_logits, dim=-1).cpu().numpy()
|
| 1202 |
+
for p in range(model.n_cross_pairs):
|
| 1203 |
+
top_layers = np.argsort(rw[p])[::-1][:4].tolist()
|
| 1204 |
+
print(f" • Cross-KV Pair {p} top encoder mixture weights: {top_layers} ({[round(float(rw[p][k]), 3) for k in top_layers]})")
|
| 1205 |
+
|
| 1206 |
+
print(f"🎉 Stage '{args.stage}' 100% COMPLETE ({rows_seen:,} rows trained across {step:,} steps)!")
|
| 1207 |
+
|
| 1208 |
+
if world_size > 1:
|
| 1209 |
+
dist.barrier()
|
| 1210 |
+
dist.destroy_process_group()
|
| 1211 |
+
|
| 1212 |
+
|
| 1213 |
+
if __name__ == "__main__":
|
| 1214 |
+
main()
|
trainingcelloutputwitheval.txt
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|