freococo commited on
Commit
3bcdbab
·
verified ·
1 Parent(s): 6711729

Release MyTranslation_300M_Base (298.76M inf / 9x RTX 5090, Val PPL 3.20 + custom_code, code & logs)

Browse files
README.md ADDED
@@ -0,0 +1,135 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - my
4
+ - en
5
+ license: apache-2.0
6
+ library_name: transformers
7
+ tags:
8
+ - Translation
9
+ - Safetensors
10
+ - Burmese
11
+ - English
12
+ - myanmartranslator
13
+ - MyanmarLLM
14
+ - custom_code
15
+ - myanmar
16
+ ---
17
+
18
+ # MyTranslation_300M_Base
19
+
20
+ **MyTranslation_300M_Base** is a **`298.76M`-parameter** (inference; `312.94M` training parameters including the Multi-Token Prediction head) bidirectional **English ↔ Burmese (`my`)** Neural Machine Translation encoder-decoder model trained from scratch on **~10B+ training tokens** using **9× NVIDIA GeForce RTX 5090 GPUs**.
21
+
22
+ > **Status:** Base checkpoint (`Stage 1 Pretraining + Stage 2 Gold Cooldown`), published prior to optional spoken/idiom Stage 3 polishing.
23
+
24
+ ---
25
+
26
+ ## 1. Architecture Highlights
27
+
28
+ | Component | Specification |
29
+ |---|---|
30
+ | **Inference Parameters** | `298,763,201` (`298.76M`, BF16 `model.safetensors` ~598 MB) |
31
+ | **Training Parameters** | `312,940,000` (`312.94M`, includes `MTPHeadBlock` dropped at export) |
32
+ | **Encoder** | `12` bidirectional layers, `d_model=1024`, `n_heads=16`, `n_kv_heads=4` (GQA, `head_dim=64`), `ffn_dim=3072` SwiGLU |
33
+ | **Decoder** | `6` causal layers, `d_model=1024`, `n_heads=16`, `n_kv_heads=4` (GQA), `ffn_dim=3072` SwiGLU |
34
+ | **Positional Encoding** | RoPE (`theta=50000.0`, `max_seq_len=2048` / trained at `1024`) |
35
+ | **Attention & Normalization** | `RMSNorm`, QK-Norm, Gated Attention (`sigmoid(gate_proj)`), Value Residual (`v0` mixing) |
36
+ | **Encoder-Decoder Routing** | Dense Layer-Wise Attention (`enc_dwa_logits`) + `3` learned `SharedCrossKVProj` mixtures (`enc_router_logits`) shared by decoder pairs `(0,1)`, `(2,3)`, `(4,5)` |
37
+ | **Tokenizer & Vocab** | `65,536`-piece SentencePiece (`UNK=0`, `BOS=1`, `EOS=2`, `PAD=3`, `<2my>=4`, `<2en>=5`, `<formal>=6`, `<spoken>=7`, `<bt>=8`, `<denoise>=9`, `<mask>=10`), tied embeddings (`embed_scale=32.0`) |
38
+
39
+ ### Learned Cross-KV Encoder Mixture Weights (Post-Stage 2)
40
+ * **Cross-KV Pair 0 (Decoder 0–1):** Top encoder layers `[7, 8, 6, 2]` (`[0.226, 0.200, 0.187, 0.102]`)
41
+ * **Cross-KV Pair 1 (Decoder 2–3):** Top encoder layers `[9, 10, 8, 11]` (`[0.339, 0.206, 0.187, 0.124]`)
42
+ * **Cross-KV Pair 2 (Decoder 4–5):** Top encoder layers `[11, 9, 10, 8]` (`[0.340, 0.335, 0.278, 0.041]`)
43
+
44
+ ---
45
+
46
+ ## 2. Training Hardware, Data & Schedule
47
+
48
+ ### Cluster & Precision Setup
49
+ * **GPUs:** **9× NVIDIA GeForce RTX 5090** (`32,607 MiB` VRAM each, `torchrun --nproc_per_node=9`, NCCL)
50
+ * **Precision & Sync:** `bfloat16` autocast, TF32 matmul/cuDNN, custom BF16 gradient `AllReduce` with `1024x` loss-scale protection, chunked cross-entropy (`chunk_sz=2048`) with Z-loss (`1e-4`), dynamic length-packed microbatches (`32,768` tokens/GPU, `~295k` budgeted tokens/step), load-balanced step grouping.
51
+ * **Throughput:** `~375k–400k tok/s` in Stage 1 (`98%–100%` utilization across all 9 GPUs), `~405k–429k tok/s` in Stage 2.
52
+
53
+ ### Dataset Coverage
54
+ * **Raw Dataset (`tokenized_65k`):** `48,329,808` unique parallel/monolingual pairs (`train_en.bin` 7.9 GiB + `train_my.bin` 7.0 GiB ≈ `~8.0B` raw tokens) + `22,329` held-out validation pairs.
55
+ * **Tasks & Register Control:** Bidirectional `En -> My` (`<2my>`) and `My -> En` (`<2en>`), explicit `<formal>` / `<spoken>` style conditioning (`70%` dropout-free application on tagged subsets), `<bt>` marker for backtranslated `My -> En`, and span-corruption `<denoise>` (`Poisson(3.0)`, `<mask>`).
56
+
57
+ ### Stage 1 (`stable` — 100% Coverage + Gold Replay)
58
+ * **Plan:** `48,329,808` unique base rows (`100%` coverage) + `12,000,000` gold replay = `60,329,808` planned rows (`60,331,713` rows trained across `58,916` steps, ~10B+ tokens).
59
+ * **Hyperparameters:** `AdamW` (`lr=5.0e-4` after 900-step warmup, `betas=(0.9, 0.95)`, `weight_decay=0.01`), `label_smoothing=0.1`, `MTP weight=0.3`, `grad_clip=1.0`.
60
+ * **Stage 1 Validation Curve (`eval_val_nll` on 1,024 uniform non-BT held-out pairs):**
61
+ * Step `0`: `Comb NLL 20.1282`
62
+ * Step `1,500`: `Comb NLL 3.0945` (**PPL `22.08`**)
63
+ * Step `8,000`: `Comb NLL 1.7760` (**PPL `5.91`**)
64
+ * Step `19,000`: `Comb NLL 1.5803` (**PPL `4.86`**)
65
+ * Step `39,000`: `Comb NLL 1.4648` (**PPL `4.33`**)
66
+ * Step `58,000`: `Comb NLL 1.4298` (**PPL `4.18`**)
67
+ * **Stage 1 Final (Step `58,916`):** `Comb NLL 1.4316` (**PPL `4.19`** | `En->My: 1.7539` | `My->En: 1.1548`)
68
+
69
+ ### Stage 2 (`cooldown` — Clean Gold Only, Unsmoothed)
70
+ * **Plan:** `2,706,748` unique non-BT base rows + `1,500,000` gold replay = `4,206,748` planned rows (`4,210,076` rows trained across `6,793` steps).
71
+ * **Hyperparameters:** `100`-step warmup to `3.0e-4` → cosine decay to `1.0e-5`, `label_smoothing=0.0`, `MTP weight=0.15`, `50/50 En↔My`.
72
+ * **Stage 2 Validation Curve:**
73
+ * Step `250`: `Comb NLL 1.2945` (**PPL `3.65`**)
74
+ * Step `1,500`: `Comb NLL 1.2403` (**PPL `3.46`**)
75
+ * Step `3,500`: `Comb NLL 1.1941` (**PPL `3.30`**)
76
+ * Step `5,500`: `Comb NLL 1.1687` (**PPL `3.22`**)
77
+ * **Stage 2 Final (Step `6,793`):** **`Comb NLL 1.1640` (PPL `3.20`)** | `En->My: 1.4829` | `My->En: 0.8901` (`PPL 2.44`)
78
+
79
+ *(Note: PPL values are measured on the `tokenized_65k/val` split and are not directly comparable 1:1 against models evaluated on older splits/vocabularies.)*
80
+
81
+ ---
82
+
83
+ ## 3. Beam=4 Benchmark Outputs (Unseen + Audit Prompts)
84
+
85
+ ```text
86
+ 🏆 MyTranslation-300M Final Beam=4 Benchmark:
87
+ [En->My Formal]: ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးကို တည်ငြိမ်စေရန် ဗဟိုဘဏ်သည် အတိုးနှုန်းကို မြှင့်တင်ခဲ့သည်။
88
+ [En->My Spoken]: ရိုးရိုးသားသားပြောရရင် ဒီလိုဖြစ်လာမယ်လို့ ကျွန်တော် မမျှော်လင့်ထားပါဘူး၊ ဒါပေမယ့် စိတ်မပူပါနဲ့၊ အတူတူ အဖြေရှာကြည့်ပါမယ်။
89
+ [My->En Formal]: The central bank raised interest rates to control inflation and stabilize the economy.
90
+ [My->En Spoken]: It is a little late to go home because it is raining heavily in Yangon today
91
+ [My->En Idiom]: Love, I have been deeply in love with you and myself, I know God, I know the sugar, I know the spirit, and I'm convinced by the envious deniers, so you're going crazy.
92
+ [En->My News]: မုတ်သုံမိုးသည်းထန်စွာ ရွာသွန်းမှုကြောင့် မြို့နယ်အချို့တွင် ရေကြီးရေလျှံမှုများ ဆိုးရွားစွာ ဖြစ်ပေါ်ခဲ့ပြီး ထောင်နှင့်ချီသော ဒေသခံများသည် ယာယီခိုလှုံရေးစခန်းများသို့ ရွှေ့ပြောင်းခဲ့ရသည်။
93
+ [En->My Health]: ရာသီတုပ်ကွေးကနေ မြန်မြန်သက်သာဖို့ သန့်ရှင်းတဲ့ရေ များများသောက်ပြီး လုံလုံလောက်လောက် အနားယူဖို့ ဆရာဝန်တွေက အကြံပြုထားပါတယ်။
94
+ [En->My Spoken2]: ကျွန်တော် ဒီမနက် တစ်နာရီ ကျော်ကျော်လောက် ကားကြပ်နေလို့ ကျွန်တော် မပါဘဲ အစည်းအဝေးကို စတင်ပါ ။
95
+ [My->En News]: Border trade checkpoints are being accelerated to improve international trade.
96
+ [My->En Tech]: Artificial intelligence technology can help make processes more efficient in education, health, and business.
97
+ [My->En Spoken2]: I have a need to go early tomorrow morning so I will go to bed early tonight so I will call me tomorrow
98
+ ```
99
+
100
+ ---
101
+
102
+ ## 4. Usage Example (PyTorch + Transformers `trust_remote_code=True`)
103
+
104
+ ```python
105
+ import torch, sentencepiece as spm
106
+ from huggingface_hub import hf_hub_download
107
+ from transformers import AutoModelForSeq2SeqLM
108
+
109
+ repo_id = "freococo/MyTranslation_300M_Base"
110
+ dtype = torch.float16 # Use torch.bfloat16 on Ampere/Hopper/Blackwell, torch.float16 on T4
111
+
112
+ model = AutoModelForSeq2SeqLM.from_pretrained(repo_id, trust_remote_code=True, dtype=dtype).cuda().eval()
113
+ sp = spm.SentencePieceProcessor()
114
+ sp.load(hf_hub_download(repo_id, "tokenizer.model"))
115
+
116
+ print("En->My Formal:", model.translate(
117
+ "The central bank raised interest rates to control inflation and stabilize the economy.",
118
+ sp, direction="en2my", style="formal", num_beams=4
119
+ ))
120
+ print("En->My Spoken:", model.translate(
121
+ "Honestly, I didn't expect things to turn out like this, but don't worry—we'll figure it out together.",
122
+ sp, direction="en2my", style="spoken", num_beams=4
123
+ ))
124
+ print("My->En Formal:", model.translate(
125
+ "ဗဟိုဘဏ်သည် ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးတည်ငြိမ်စေရန် အတိုးနှုန်းများကို မြှင့်တင်ခဲ့သည်။",
126
+ sp, direction="my2en", style="formal", num_beams=4
127
+ ))
128
+ ```
129
+
130
+ ---
131
+
132
+ ## 5. Reproducibility Artifacts Included
133
+ * `training_codes.txt` — exact multi-GPU PyTorch training script (`train_300m.py`)
134
+ * `trainingcelloutputwitheval.txt` — complete Stage 1 + Stage 2 training logs & Beam=4 benchmark output
135
+ * `checkpoint-stage2-step-6793/` — FP32 `model.safetensors` + `training_state.pt` (`best_model.pt`)
checkpoint-stage2-step-6793/config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "MyTranslationForConditionalGeneration"
4
+ ],
5
+ "auto_map": {
6
+ "AutoConfig": "configuration_mytranslation.MyTranslationConfig",
7
+ "AutoModelForSeq2SeqLM": "modeling_mytranslation.MyTranslationForConditionalGeneration"
8
+ },
9
+ "model_type": "mytranslation",
10
+ "is_encoder_decoder": true,
11
+ "tie_word_embeddings": true,
12
+ "vocab_size": 65536,
13
+ "d_model": 1024,
14
+ "n_heads": 16,
15
+ "n_kv_heads": 4,
16
+ "enc_layers": 12,
17
+ "dec_layers": 6,
18
+ "ffn_dim": 3072,
19
+ "max_seq_len": 2048,
20
+ "rope_theta": 50000.0,
21
+ "unk_token_id": 0,
22
+ "bos_token_id": 1,
23
+ "eos_token_id": 2,
24
+ "pad_token_id": 3,
25
+ "stage1_final_val_nll": 1.4316,
26
+ "stage1_final_val_ppl": 4.19,
27
+ "val_nll": 1.164,
28
+ "val_ppl": 3.2,
29
+ "en2my_val_nll": 1.4829,
30
+ "my2en_val_nll": 0.8901,
31
+ "torch_dtype": "float32"
32
+ }
checkpoint-stage2-step-6793/configuration_mytranslation.py ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from transformers import PretrainedConfig
2
+
3
+ class MyTranslationConfig(PretrainedConfig):
4
+ model_type = "mytranslation"
5
+
6
+ def __init__(
7
+ self,
8
+ vocab_size=65536,
9
+ d_model=1024,
10
+ n_heads=16,
11
+ n_kv_heads=4,
12
+ enc_layers=12,
13
+ dec_layers=6,
14
+ ffn_dim=3072,
15
+ max_seq_len=2048,
16
+ rope_theta=50000.0,
17
+ unk_token_id=0,
18
+ bos_token_id=1,
19
+ eos_token_id=2,
20
+ pad_token_id=3,
21
+ tie_word_embeddings=True,
22
+ is_encoder_decoder=True,
23
+ **kwargs,
24
+ ):
25
+ self.vocab_size = vocab_size
26
+ self.d_model = d_model
27
+ self.n_heads = n_heads
28
+ self.n_kv_heads = n_kv_heads
29
+ self.enc_layers = enc_layers
30
+ self.dec_layers = dec_layers
31
+ self.ffn_dim = ffn_dim
32
+ self.max_seq_len = max_seq_len
33
+ self.rope_theta = rope_theta
34
+ super().__init__(
35
+ unk_token_id=unk_token_id,
36
+ bos_token_id=bos_token_id,
37
+ eos_token_id=eos_token_id,
38
+ pad_token_id=pad_token_id,
39
+ tie_word_embeddings=tie_word_embeddings,
40
+ is_encoder_decoder=is_encoder_decoder,
41
+ **kwargs,
42
+ )
checkpoint-stage2-step-6793/generation_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 1,
3
+ "eos_token_id": 2,
4
+ "pad_token_id": 3,
5
+ "decoder_start_token_id": 1,
6
+ "max_new_tokens": 192,
7
+ "num_beams": 4,
8
+ "length_penalty": 0.85,
9
+ "repetition_penalty": 1.12
10
+ }
checkpoint-stage2-step-6793/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:90bd4db2e504e2c0ea42f3b5fb2c9fbfa6e657e84fa21fbe560faa947644c20e
3
+ size 1195078172
checkpoint-stage2-step-6793/modeling_mytranslation.py ADDED
@@ -0,0 +1,372 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import math
2
+ import torch
3
+ import torch.nn as nn
4
+ import torch.nn.functional as F
5
+ import transformers
6
+ from transformers import PreTrainedModel
7
+ from transformers.modeling_outputs import Seq2SeqLMOutput
8
+ from .configuration_mytranslation import MyTranslationConfig
9
+
10
+ PAD_ID = 3
11
+ BOS_ID = 1
12
+ EOS_ID = 2
13
+ TOK_2MY, TOK_2EN, TOK_FORMAL, TOK_SPOKEN = 4, 5, 6, 7
14
+ _IS_TF5 = int(transformers.__version__.split(".")[0]) >= 5
15
+
16
+ def _sdpa(q, k, v, attn_mask=None, is_causal=False):
17
+ if q.size(1) != k.size(1):
18
+ try:
19
+ return F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask, is_causal=is_causal, enable_gqa=True)
20
+ except TypeError:
21
+ rep = q.size(1) // k.size(1)
22
+ k = k.repeat_interleave(rep, dim=1)
23
+ v = v.repeat_interleave(rep, dim=1)
24
+ return F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask, is_causal=is_causal)
25
+
26
+ class RMSNorm(nn.Module):
27
+ def __init__(self, dim: int, eps: float = 1e-6):
28
+ super().__init__()
29
+ self.dim = (dim,)
30
+ self.eps = eps
31
+ self.weight = nn.Parameter(torch.ones(dim))
32
+
33
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
34
+ if hasattr(F, "rms_norm"):
35
+ return F.rms_norm(x, self.dim, self.weight, self.eps)
36
+ xf = x.float()
37
+ y = xf * torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + self.eps)
38
+ return (y * self.weight.float()).to(dtype=x.dtype)
39
+
40
+ class RotaryEmbedding(nn.Module):
41
+ def __init__(self, dim: int, max_seq_len: int = 2048, theta: float = 50000.0):
42
+ super().__init__()
43
+ inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))
44
+ t = torch.arange(max_seq_len, dtype=torch.float32)
45
+ freqs = torch.outer(t, inv_freq)
46
+ self.register_buffer("cos_cached", freqs.cos()[None, None, :, :], persistent=False)
47
+ self.register_buffer("sin_cached", freqs.sin()[None, None, :, :], persistent=False)
48
+
49
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
50
+ L = x.size(2)
51
+ cos = self.cos_cached[:, :, :L, :].to(dtype=x.dtype)
52
+ sin = self.sin_cached[:, :, :L, :].to(dtype=x.dtype)
53
+ x1, x2 = x[..., ::2], x[..., 1::2]
54
+ return torch.cat((x1 * cos - x2 * sin, x1 * sin + x2 * cos), dim=-1)
55
+
56
+ class SwiGLU(nn.Module):
57
+ def __init__(self, d_model: int, ffn_dim: int):
58
+ super().__init__()
59
+ self.gate_up = nn.Linear(d_model, 2 * ffn_dim, bias=False)
60
+ self.down = nn.Linear(ffn_dim, d_model, bias=False)
61
+
62
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
63
+ gate, up = self.gate_up(x).chunk(2, dim=-1)
64
+ return self.down(F.silu(gate) * up)
65
+
66
+ class GatedGQASelfAttention(nn.Module):
67
+ def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, is_causal: bool = False, has_v_res: bool = False):
68
+ super().__init__()
69
+ self.n_heads = n_heads
70
+ self.n_kv_heads = n_kv_heads
71
+ self.head_dim = d_model // n_heads
72
+ self.is_causal = is_causal
73
+ self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
74
+ self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * self.head_dim, bias=False)
75
+ self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
76
+ self.out = nn.Linear(d_model, d_model, bias=False)
77
+ self.q_norm = RMSNorm(self.head_dim)
78
+ self.k_norm = RMSNorm(self.head_dim)
79
+ self.v_res_logit = nn.Parameter(torch.tensor(-2.0)) if has_v_res else None
80
+
81
+ def forward(self, x: torch.Tensor, rope: RotaryEmbedding, attn_mask: torch.Tensor = None, v0: torch.Tensor = None):
82
+ B, L, D = x.shape
83
+ q = self.q_norm(self.q_proj(x).view(B, L, self.n_heads, self.head_dim).transpose(1, 2))
84
+ k, v = self.kv_proj(x).chunk(2, dim=-1)
85
+ k = self.k_norm(k.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2))
86
+ v = v.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2)
87
+ v_raw = v
88
+ if self.v_res_logit is not None and v0 is not None:
89
+ alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
90
+ v = (1.0 - alpha) * v + alpha * v0
91
+ q, k = rope(q), rope(k)
92
+ y = _sdpa(q, k, v, attn_mask=None if self.is_causal else attn_mask, is_causal=self.is_causal)
93
+ gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
94
+ y = (y * gate).transpose(1, 2).contiguous().view(B, L, D)
95
+ return self.out(y), v_raw
96
+
97
+ def step_kv(self, x_step, cos_p, sin_p, pos, k_cache, v_cache, v0=None):
98
+ B, _, D = x_step.shape
99
+ q = self.q_norm(self.q_proj(x_step).view(B, 1, self.n_heads, self.head_dim).transpose(1, 2))
100
+ k, v = self.kv_proj(x_step).chunk(2, dim=-1)
101
+ k = self.k_norm(k.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2))
102
+ v = v.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2)
103
+ v_raw = v
104
+ if self.v_res_logit is not None and v0 is not None:
105
+ alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
106
+ v = (1.0 - alpha) * v + alpha * v0
107
+ q1, q2 = q[..., ::2], q[..., 1::2]
108
+ k1, k2 = k[..., ::2], k[..., 1::2]
109
+ q = torch.cat((q1 * cos_p - q2 * sin_p, q1 * sin_p + q2 * cos_p), dim=-1)
110
+ k = torch.cat((k1 * cos_p - k2 * sin_p, k1 * sin_p + k2 * cos_p), dim=-1)
111
+ k_cache[:, :, pos : pos + 1, :] = k
112
+ v_cache[:, :, pos : pos + 1, :] = v
113
+ y = _sdpa(q, k_cache[:, :, : pos + 1, :], v_cache[:, :, : pos + 1, :], is_causal=False)
114
+ gate = torch.sigmoid(self.gate_proj(x_step)).transpose(1, 2).unsqueeze(-1)
115
+ y = (y * gate).transpose(1, 2).contiguous().view(B, 1, D)
116
+ return self.out(y), v_raw
117
+
118
+ class SharedCrossKVProj(nn.Module):
119
+ def __init__(self, d_model: int, n_kv_heads: int, head_dim: int):
120
+ super().__init__()
121
+ self.n_kv_heads = n_kv_heads
122
+ self.head_dim = head_dim
123
+ self.norm = RMSNorm(d_model)
124
+ self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * head_dim, bias=False)
125
+ self.k_norm = RMSNorm(head_dim)
126
+
127
+ def forward(self, enc_mix: torch.Tensor):
128
+ B, S, _ = enc_mix.shape
129
+ k, v = self.kv_proj(self.norm(enc_mix)).chunk(2, dim=-1)
130
+ k = self.k_norm(k.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2))
131
+ v = v.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2)
132
+ return k, v
133
+
134
+ class GatedGQACrossAttention(nn.Module):
135
+ def __init__(self, d_model: int, n_heads: int):
136
+ super().__init__()
137
+ self.n_heads = n_heads
138
+ self.head_dim = d_model // n_heads
139
+ self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
140
+ self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
141
+ self.out = nn.Linear(d_model, d_model, bias=False)
142
+ self.q_norm = RMSNorm(self.head_dim)
143
+
144
+ def forward(self, x: torch.Tensor, k_cross: torch.Tensor, v_cross: torch.Tensor, enc_mask: torch.Tensor):
145
+ B, T, D = x.shape
146
+ q = self.q_norm(self.q_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2))
147
+ y = _sdpa(q, k_cross, v_cross, attn_mask=enc_mask, is_causal=False)
148
+ gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
149
+ y = (y * gate).transpose(1, 2).contiguous().view(B, T, D)
150
+ return self.out(y)
151
+
152
+ class EncoderLayer(nn.Module):
153
+ def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
154
+ super().__init__()
155
+ self.norm1 = RMSNorm(d_model)
156
+ self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=False, has_v_res=has_v_res)
157
+ self.norm2 = RMSNorm(d_model)
158
+ self.mlp = SwiGLU(d_model, ffn_dim)
159
+
160
+ def forward(self, x, rope, enc_mask, v0=None):
161
+ attn_out, v_raw = self.self_attn(self.norm1(x), rope, enc_mask, v0=v0)
162
+ x = x + attn_out
163
+ x = x + self.mlp(self.norm2(x))
164
+ return x, v_raw
165
+
166
+ class DecoderLayer(nn.Module):
167
+ def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
168
+ super().__init__()
169
+ self.norm1 = RMSNorm(d_model)
170
+ self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=True, has_v_res=has_v_res)
171
+ self.norm2 = RMSNorm(d_model)
172
+ self.cross_attn = GatedGQACrossAttention(d_model, n_heads)
173
+ self.norm3 = RMSNorm(d_model)
174
+ self.mlp = SwiGLU(d_model, ffn_dim)
175
+
176
+ def forward(self, x, k_cross, v_cross, rope, enc_mask, v0=None):
177
+ attn_out, v_raw = self.self_attn(self.norm1(x), rope, None, v0=v0)
178
+ x = x + attn_out
179
+ x = x + self.cross_attn(self.norm2(x), k_cross, v_cross, enc_mask)
180
+ x = x + self.mlp(self.norm3(x))
181
+ return x, v_raw
182
+
183
+ def step_kv(self, x_step, cos_p, sin_p, pos, k_self, v_self, k_cross, v_cross, enc_mask, v0=None):
184
+ attn_out, v_raw = self.self_attn.step_kv(self.norm1(x_step), cos_p, sin_p, pos, k_self, v_self, v0=v0)
185
+ x_step = x_step + attn_out
186
+ x_step = x_step + self.cross_attn(self.norm2(x_step), k_cross, v_cross, enc_mask)
187
+ x_step = x_step + self.mlp(self.norm3(x_step))
188
+ return x_step, v_raw
189
+
190
+ class MyTranslationForConditionalGeneration(PreTrainedModel):
191
+ config_class = MyTranslationConfig
192
+ base_model_prefix = ""
193
+ _tied_weights_keys = {"lm_head.weight": "embed_tokens.weight"} if hasattr(PreTrainedModel, "get_expanded_tied_weights_keys") else ["lm_head.weight"]
194
+
195
+ def __init__(self, config: MyTranslationConfig):
196
+ super().__init__(config)
197
+ self.d_model = config.d_model
198
+ self.n_heads = config.n_heads
199
+ self.n_kv_heads = config.n_kv_heads
200
+ self.head_dim = config.d_model // config.n_heads
201
+ self.enc_layers = config.enc_layers
202
+ self.dec_layers = config.dec_layers
203
+ self.n_cross_pairs = config.dec_layers // 2
204
+ self.embed_scale = math.sqrt(config.d_model)
205
+
206
+ self.embed_tokens = nn.Embedding(config.vocab_size, config.d_model, padding_idx=config.pad_token_id)
207
+ self.rope = RotaryEmbedding(self.head_dim, max_seq_len=config.max_seq_len, theta=config.rope_theta)
208
+ self.encoder = nn.ModuleList([
209
+ EncoderLayer(config.d_model, config.n_heads, config.n_kv_heads, config.ffn_dim, has_v_res=(i > 0))
210
+ for i in range(config.enc_layers)
211
+ ])
212
+ self.enc_dwa_logits = nn.ParameterList([
213
+ nn.Parameter(torch.tensor([-4.0] * i + [0.0], dtype=torch.float32))
214
+ for i in range(1, config.enc_layers)
215
+ ])
216
+ self.enc_router_logits = nn.Parameter(torch.zeros(self.n_cross_pairs, config.enc_layers))
217
+ self.cross_kv_projs = nn.ModuleList([
218
+ SharedCrossKVProj(config.d_model, config.n_kv_heads, self.head_dim)
219
+ for _ in range(self.n_cross_pairs)
220
+ ])
221
+ self.decoder = nn.ModuleList([
222
+ DecoderLayer(config.d_model, config.n_heads, config.n_kv_heads, config.ffn_dim, has_v_res=(i > 0))
223
+ for i in range(config.dec_layers)
224
+ ])
225
+ self.dec_norm = RMSNorm(config.d_model)
226
+ self.lm_head = nn.Linear(config.d_model, config.vocab_size, bias=False)
227
+ self.post_init()
228
+ self.tie_weights()
229
+
230
+ def get_input_embeddings(self):
231
+ return self.embed_tokens
232
+
233
+ def set_input_embeddings(self, value):
234
+ self.embed_tokens = value
235
+
236
+ def get_output_embeddings(self):
237
+ return self.lm_head
238
+
239
+ def set_output_embeddings(self, new_embeddings):
240
+ self.lm_head = new_embeddings
241
+
242
+ def tie_weights(self, **kwargs):
243
+ self.lm_head.weight = self.embed_tokens.weight
244
+
245
+ def encode_and_route(self, src_ids: torch.Tensor):
246
+ enc_mask = (src_ids != self.config.pad_token_id).unsqueeze(1).unsqueeze(2)
247
+ x = self.embed_tokens(src_ids) * self.embed_scale
248
+ raw_states = [x]
249
+ enc_outputs = []
250
+ v0 = None
251
+ for i, layer in enumerate(self.encoder):
252
+ if i > 0:
253
+ w_dwa = F.softmax(self.enc_dwa_logits[i - 1], dim=0).to(dtype=x.dtype)
254
+ x_in = torch.einsum("j,jbsd->bsd", w_dwa, torch.stack(raw_states, dim=0))
255
+ else:
256
+ x_in = x
257
+ x_out, v_raw = layer(x_in, self.rope, enc_mask, v0=v0)
258
+ if i == 0:
259
+ v0 = v_raw
260
+ raw_states.append(x_out)
261
+ enc_outputs.append(x_out)
262
+ stacked_enc = torch.stack(enc_outputs, dim=0)
263
+ router_w = F.softmax(self.enc_router_logits, dim=-1).to(dtype=stacked_enc.dtype)
264
+ cross_kv = [
265
+ self.cross_kv_projs[p](torch.einsum("k,kbsd->bsd", router_w[p], stacked_enc))
266
+ for p in range(self.n_cross_pairs)
267
+ ]
268
+ return cross_kv, enc_mask
269
+
270
+ def decode_seq(self, tgt_in: torch.Tensor, cross_kv: list, enc_mask: torch.Tensor):
271
+ x = self.embed_tokens(tgt_in) * self.embed_scale
272
+ v0 = None
273
+ for i, layer in enumerate(self.decoder):
274
+ kc, vc = cross_kv[i // 2]
275
+ x, v_raw = layer(x, kc, vc, self.rope, enc_mask, v0=v0)
276
+ if i == 0:
277
+ v0 = v_raw
278
+ return self.dec_norm(x)
279
+
280
+ def forward(self, input_ids=None, decoder_input_ids=None, labels=None, **kwargs):
281
+ if decoder_input_ids is None:
282
+ decoder_input_ids = torch.full((input_ids.size(0), 1), BOS_ID, dtype=torch.long, device=input_ids.device)
283
+ cross_kv, enc_mask = self.encode_and_route(input_ids)
284
+ h_dec = self.decode_seq(decoder_input_ids, cross_kv, enc_mask)
285
+ logits = self.lm_head(h_dec)
286
+ loss = None
287
+ if labels is not None:
288
+ loss = F.cross_entropy(logits.view(-1, logits.size(-1)), labels.view(-1), ignore_index=-100)
289
+ return Seq2SeqLMOutput(loss=loss, logits=logits)
290
+
291
+ @torch.inference_mode()
292
+ def translate(
293
+ self,
294
+ text: str,
295
+ sp_proc,
296
+ direction: str = "en2my",
297
+ style: str = "auto",
298
+ num_beams: int = 4,
299
+ max_new_tokens: int = 192,
300
+ length_penalty: float = 0.85,
301
+ rep_penalty: float = 1.12,
302
+ ) -> str:
303
+ self.eval()
304
+ dev = self.embed_tokens.weight.device
305
+ dt = self.embed_tokens.weight.dtype
306
+ pref = [TOK_2MY if direction == "en2my" else TOK_2EN]
307
+ if style == "formal":
308
+ pref.append(TOK_FORMAL)
309
+ elif style == "spoken":
310
+ pref.append(TOK_SPOKEN)
311
+ src_ids = pref + sp_proc.encode_as_ids(text.strip())[: self.config.max_seq_len - 8] + [EOS_ID]
312
+ src_t = torch.tensor([src_ids], dtype=torch.long, device=dev)
313
+ cross_kv, enc_mask = self.encode_and_route(src_t)
314
+
315
+ if num_beams <= 1:
316
+ k_cache = torch.empty((self.dec_layers, 1, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=dt, device=dev)
317
+ v_cache = torch.empty((self.dec_layers, 1, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=dt, device=dev)
318
+ cur_tok = torch.tensor([[BOS_ID]], dtype=torch.long, device=dev)
319
+ cos_all = self.rope.cos_cached.to(dtype=dt, device=dev)
320
+ sin_all = self.rope.sin_cached.to(dtype=dt, device=dev)
321
+ ys = []
322
+ for pos in range(max_new_tokens):
323
+ x_dec = self.embed_tokens(cur_tok) * self.embed_scale
324
+ v0 = None
325
+ cos_p, sin_p = cos_all[:, :, pos : pos + 1, :], sin_all[:, :, pos : pos + 1, :]
326
+ for l_idx, layer in enumerate(self.decoder):
327
+ kc, vc = cross_kv[l_idx // 2]
328
+ x_dec, v_raw = layer.step_kv(x_dec, cos_p, sin_p, pos, k_cache[l_idx], v_cache[l_idx], kc, vc, enc_mask, v0=v0)
329
+ if l_idx == 0:
330
+ v0 = v_raw
331
+ logits = self.lm_head(self.dec_norm(x_dec)[:, 0, :]).float()[0]
332
+ logits[[0, 1, 3, 4, 5, 6, 7, 8, 9, 10]] = -1e9
333
+ if rep_penalty != 1.0 and ys:
334
+ seen = torch.tensor(list(set(ys)), dtype=torch.long, device=dev)
335
+ sl = logits[seen]
336
+ logits[seen] = torch.where(sl > 0, sl / rep_penalty, sl * rep_penalty)
337
+ nxt = int(logits.argmax().item())
338
+ if nxt == EOS_ID:
339
+ break
340
+ ys.append(nxt)
341
+ cur_tok = torch.tensor([[nxt]], dtype=torch.long, device=dev)
342
+ return sp_proc.decode_ids([x for x in ys if x >= 11]).strip()
343
+
344
+ beams = [(0.0, [BOS_ID], False)]
345
+ for _ in range(max_new_tokens):
346
+ active = [(sc, seq) for sc, seq, done in beams if not done]
347
+ if not active:
348
+ break
349
+ B_act = len(active)
350
+ tgt_t = torch.tensor([seq for _, seq in active], dtype=torch.long, device=dev)
351
+ ckv_exp = [(kc.expand(B_act, -1, -1, -1), vc.expand(B_act, -1, -1, -1)) for kc, vc in cross_kv]
352
+ h_dec = self.decode_seq(tgt_t, ckv_exp, enc_mask.expand(B_act, -1, -1, -1))
353
+ logits = self.lm_head(h_dec[:, -1, :]).float()
354
+ logits[:, [0, 1, 3, 4, 5, 6, 7, 8, 9, 10]] = -1e9
355
+ for b_i, (_, seq) in enumerate(active):
356
+ if rep_penalty != 1.0 and len(seq) > 1:
357
+ seen = torch.tensor(list(set(seq[1:])), dtype=torch.long, device=dev)
358
+ sl = logits[b_i, seen]
359
+ logits[b_i, seen] = torch.where(sl > 0, sl / rep_penalty, sl * rep_penalty)
360
+ lp = F.log_softmax(logits, dim=-1)
361
+ topk_lp, topk_ids = torch.topk(lp, num_beams * 2, dim=-1)
362
+ cands = [(sc, seq, True) for sc, seq, done in beams if done]
363
+ for b_i, (sc, seq) in enumerate(active):
364
+ for k in range(num_beams * 2):
365
+ tid = int(topk_ids[b_i, k].item())
366
+ n_sc = sc + float(topk_lp[b_i, k].item())
367
+ cands.append((n_sc, seq if tid == EOS_ID else seq + [tid], tid == EOS_ID))
368
+ cands.sort(key=lambda x: x[0] / (max(1, len(x[1]) - 1) ** length_penalty), reverse=True)
369
+ beams = cands[:num_beams]
370
+ if all(done for _, _, done in beams):
371
+ break
372
+ return sp_proc.decode_ids([x for x in beams[0][1][1:] if x >= 11]).strip()
checkpoint-stage2-step-6793/special_tokens_map.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "unk_token": "<unk>",
3
+ "bos_token": "<s>",
4
+ "eos_token": "</s>",
5
+ "pad_token": "<pad>",
6
+ "additional_special_tokens": [
7
+ "<2my>",
8
+ "<2en>",
9
+ "<formal>",
10
+ "<spoken>",
11
+ "<bt>",
12
+ "<denoise>",
13
+ "<mask>"
14
+ ]
15
+ }
checkpoint-stage2-step-6793/tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9aaeeaf36272c5c158d11ed0987ca5930686f1e02713292558003909bb147bc1
3
+ size 1840017
checkpoint-stage2-step-6793/tokenizer_config.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_max_length": 1024,
3
+ "unk_token": "<unk>",
4
+ "bos_token": "<s>",
5
+ "eos_token": "</s>",
6
+ "pad_token": "<pad>",
7
+ "additional_special_tokens": [
8
+ "<2my>",
9
+ "<2en>",
10
+ "<formal>",
11
+ "<spoken>",
12
+ "<bt>",
13
+ "<denoise>",
14
+ "<mask>"
15
+ ],
16
+ "special_tokens_map": {
17
+ "2my": 4,
18
+ "2en": 5,
19
+ "formal": 6,
20
+ "spoken": 7,
21
+ "bt": 8,
22
+ "denoise": 9,
23
+ "mask": 10
24
+ }
25
+ }
checkpoint-stage2-step-6793/training_codes.txt ADDED
@@ -0,0 +1,1214 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"
3
+
4
+ import argparse, json, math, time
5
+ from datetime import timedelta
6
+ import numpy as np
7
+ import sentencepiece as spm
8
+ import torch
9
+ import torch.distributed as dist
10
+ import torch.nn as nn
11
+ import torch.nn.functional as F
12
+ from torch.utils.checkpoint import checkpoint as grad_ckpt
13
+ from torch.utils.data import Dataset, DataLoader
14
+
15
+ torch.backends.cuda.matmul.allow_tf32 = True
16
+ torch.backends.cudnn.allow_tf32 = True
17
+ torch._dynamo.config.cache_size_limit = 64
18
+
19
+ DATA_DIR = "/workspace/mytrans_dl/tokenized_65k"
20
+ CKPT_DIR = "/workspace/ckpt_300m"
21
+
22
+ VOCAB_SIZE = 65536
23
+ UNK_ID, BOS_ID, EOS_ID, PAD_ID = 0, 1, 2, 3
24
+ TOK_2MY, TOK_2EN, TOK_FORMAL, TOK_SPOKEN, TOK_BT, TOK_DENOISE, TOK_MASK = 4, 5, 6, 7, 8, 9, 10
25
+ FLAG_BT, FLAG_FORMAL, FLAG_SPOKEN = 1, 2, 4
26
+
27
+ BULK_SUBSETS = {
28
+ "MyanmarTranslator_works/myanmar_spoken_corpus",
29
+ "MyanmarTranslator_works/myanmar-cc100-dataset",
30
+ "MyanmarTranslator_works/myanmar-written-corpus",
31
+ }
32
+
33
+
34
+ def load_and_verify_contracts(data_dir: str):
35
+ with open(os.path.join(data_dir, "meta.json"), "r") as f:
36
+ meta = json.load(f)
37
+ sp = spm.SentencePieceProcessor()
38
+ sp.load(os.path.join(data_dir, "tokenizer.model"))
39
+
40
+ assert sp.get_piece_size() == VOCAB_SIZE
41
+ assert sp.unk_id() == UNK_ID and sp.bos_id() == BOS_ID and sp.eos_id() == EOS_ID and sp.pad_id() == PAD_ID
42
+ expected = {4: "<2my>", 5: "<2en>", 6: "<formal>", 7: "<spoken>", 8: "<bt>", 9: "<denoise>", 10: "<mask>"}
43
+ for tid, piece in expected.items():
44
+ assert sp.id_to_piece(tid) == piece, f"Token {tid} mismatch"
45
+ if "special_tokens" in meta:
46
+ st = meta["special_tokens"]
47
+ assert st["2my"] == TOK_2MY and st["2en"] == TOK_2EN
48
+ assert st["formal"] == TOK_FORMAL and st["spoken"] == TOK_SPOKEN
49
+ assert st["bt"] == TOK_BT and st["denoise"] == TOK_DENOISE and st["mask"] == TOK_MASK
50
+ return meta, sp
51
+
52
+
53
+ # ==============================================================================
54
+ # 1. MyTranslation-300M Architecture
55
+ # ==============================================================================
56
+ class RMSNorm(nn.Module):
57
+ def __init__(self, dim: int, eps: float = 1e-6):
58
+ super().__init__()
59
+ self.dim = (dim,)
60
+ self.eps = eps
61
+ self.weight = nn.Parameter(torch.ones(dim))
62
+
63
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
64
+ return F.rms_norm(x, self.dim, self.weight, self.eps)
65
+
66
+
67
+ class RotaryEmbedding(nn.Module):
68
+ def __init__(self, dim: int, max_seq_len: int = 2048, theta: float = 50000.0):
69
+ super().__init__()
70
+ inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))
71
+ t = torch.arange(max_seq_len, dtype=torch.float32)
72
+ freqs = torch.outer(t, inv_freq)
73
+ self.register_buffer("cos_cached", freqs.cos()[None, None, :, :], persistent=False)
74
+ self.register_buffer("sin_cached", freqs.sin()[None, None, :, :], persistent=False)
75
+
76
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
77
+ L = x.size(2)
78
+ cos = self.cos_cached[:, :, :L, :].to(dtype=x.dtype)
79
+ sin = self.sin_cached[:, :, :L, :].to(dtype=x.dtype)
80
+ x1, x2 = x[..., ::2], x[..., 1::2]
81
+ return torch.cat((x1 * cos - x2 * sin, x1 * sin + x2 * cos), dim=-1)
82
+
83
+
84
+ class SwiGLU(nn.Module):
85
+ def __init__(self, d_model: int, ffn_dim: int):
86
+ super().__init__()
87
+ self.gate_up = nn.Linear(d_model, 2 * ffn_dim, bias=False)
88
+ self.down = nn.Linear(ffn_dim, d_model, bias=False)
89
+
90
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
91
+ gate, up = self.gate_up(x).chunk(2, dim=-1)
92
+ return self.down(F.silu(gate) * up)
93
+
94
+
95
+ class GatedGQASelfAttention(nn.Module):
96
+ def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, is_causal: bool = False, has_v_res: bool = False):
97
+ super().__init__()
98
+ self.n_heads = n_heads
99
+ self.n_kv_heads = n_kv_heads
100
+ self.head_dim = d_model // n_heads
101
+ self.is_causal = is_causal
102
+ self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
103
+ self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * self.head_dim, bias=False)
104
+ self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
105
+ self.out = nn.Linear(d_model, d_model, bias=False)
106
+ self.q_norm = RMSNorm(self.head_dim)
107
+ self.k_norm = RMSNorm(self.head_dim)
108
+ self.v_res_logit = nn.Parameter(torch.tensor(-2.0)) if has_v_res else None
109
+
110
+ def forward(self, x: torch.Tensor, rope: RotaryEmbedding, attn_mask: torch.Tensor = None, v0: torch.Tensor = None):
111
+ B, L, D = x.shape
112
+ q = self.q_norm(self.q_proj(x).view(B, L, self.n_heads, self.head_dim).transpose(1, 2))
113
+ k, v = self.kv_proj(x).chunk(2, dim=-1)
114
+ k = self.k_norm(k.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2))
115
+ v = v.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2)
116
+ v_raw = v
117
+ if self.v_res_logit is not None and v0 is not None:
118
+ alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
119
+ v = (1.0 - alpha) * v + alpha * v0
120
+ q, k = rope(q), rope(k)
121
+ y = F.scaled_dot_product_attention(
122
+ q, k, v,
123
+ attn_mask=None if self.is_causal else attn_mask,
124
+ is_causal=self.is_causal,
125
+ enable_gqa=True,
126
+ )
127
+ gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
128
+ y = (y * gate).transpose(1, 2).contiguous().view(B, L, D)
129
+ return self.out(y), v_raw
130
+
131
+ def step_kv(self, x_step, cos_p, sin_p, pos, k_cache, v_cache, v0=None):
132
+ B, _, D = x_step.shape
133
+ q = self.q_norm(self.q_proj(x_step).view(B, 1, self.n_heads, self.head_dim).transpose(1, 2))
134
+ k, v = self.kv_proj(x_step).chunk(2, dim=-1)
135
+ k = self.k_norm(k.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2))
136
+ v = v.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2)
137
+ v_raw = v
138
+ if self.v_res_logit is not None and v0 is not None:
139
+ alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
140
+ v = (1.0 - alpha) * v + alpha * v0
141
+ q1, q2 = q[..., ::2], q[..., 1::2]
142
+ k1, k2 = k[..., ::2], k[..., 1::2]
143
+ q = torch.cat((q1 * cos_p - q2 * sin_p, q1 * sin_p + q2 * cos_p), dim=-1)
144
+ k = torch.cat((k1 * cos_p - k2 * sin_p, k1 * sin_p + k2 * cos_p), dim=-1)
145
+ k_cache[:, :, pos : pos + 1, :] = k
146
+ v_cache[:, :, pos : pos + 1, :] = v
147
+ y = F.scaled_dot_product_attention(
148
+ q, k_cache[:, :, : pos + 1, :], v_cache[:, :, : pos + 1, :],
149
+ is_causal=False, enable_gqa=True,
150
+ )
151
+ gate = torch.sigmoid(self.gate_proj(x_step)).transpose(1, 2).unsqueeze(-1)
152
+ y = (y * gate).transpose(1, 2).contiguous().view(B, 1, D)
153
+ return self.out(y), v_raw
154
+
155
+
156
+ class SharedCrossKVProj(nn.Module):
157
+ def __init__(self, d_model: int, n_kv_heads: int, head_dim: int):
158
+ super().__init__()
159
+ self.n_kv_heads = n_kv_heads
160
+ self.head_dim = head_dim
161
+ self.norm = RMSNorm(d_model)
162
+ self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * head_dim, bias=False)
163
+ self.k_norm = RMSNorm(head_dim)
164
+
165
+ def forward(self, enc_mix: torch.Tensor):
166
+ B, S, _ = enc_mix.shape
167
+ k, v = self.kv_proj(self.norm(enc_mix)).chunk(2, dim=-1)
168
+ k = self.k_norm(k.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2))
169
+ v = v.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2)
170
+ return k, v
171
+
172
+
173
+ class GatedGQACrossAttention(nn.Module):
174
+ def __init__(self, d_model: int, n_heads: int):
175
+ super().__init__()
176
+ self.n_heads = n_heads
177
+ self.head_dim = d_model // n_heads
178
+ self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
179
+ self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
180
+ self.out = nn.Linear(d_model, d_model, bias=False)
181
+ self.q_norm = RMSNorm(self.head_dim)
182
+
183
+ def forward(self, x: torch.Tensor, k_cross: torch.Tensor, v_cross: torch.Tensor, enc_mask: torch.Tensor):
184
+ B, T, D = x.shape
185
+ q = self.q_norm(self.q_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2))
186
+ y = F.scaled_dot_product_attention(q, k_cross, v_cross, attn_mask=enc_mask, is_causal=False, enable_gqa=True)
187
+ gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
188
+ y = (y * gate).transpose(1, 2).contiguous().view(B, T, D)
189
+ return self.out(y)
190
+
191
+
192
+ class EncoderLayer(nn.Module):
193
+ def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
194
+ super().__init__()
195
+ self.norm1 = RMSNorm(d_model)
196
+ self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=False, has_v_res=has_v_res)
197
+ self.norm2 = RMSNorm(d_model)
198
+ self.mlp = SwiGLU(d_model, ffn_dim)
199
+
200
+ def forward(self, x, rope, enc_mask, v0=None):
201
+ attn_out, v_raw = self.self_attn(self.norm1(x), rope, enc_mask, v0=v0)
202
+ x = x + attn_out
203
+ x = x + self.mlp(self.norm2(x))
204
+ return x, v_raw
205
+
206
+
207
+ class DecoderLayer(nn.Module):
208
+ def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
209
+ super().__init__()
210
+ self.norm1 = RMSNorm(d_model)
211
+ self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=True, has_v_res=has_v_res)
212
+ self.norm2 = RMSNorm(d_model)
213
+ self.cross_attn = GatedGQACrossAttention(d_model, n_heads)
214
+ self.norm3 = RMSNorm(d_model)
215
+ self.mlp = SwiGLU(d_model, ffn_dim)
216
+
217
+ def forward(self, x, k_cross, v_cross, rope, enc_mask, v0=None):
218
+ attn_out, v_raw = self.self_attn(self.norm1(x), rope, None, v0=v0)
219
+ x = x + attn_out
220
+ x = x + self.cross_attn(self.norm2(x), k_cross, v_cross, enc_mask)
221
+ x = x + self.mlp(self.norm3(x))
222
+ return x, v_raw
223
+
224
+ def step_kv(self, x_step, cos_p, sin_p, pos, k_self, v_self, k_cross, v_cross, enc_mask, v0=None):
225
+ attn_out, v_raw = self.self_attn.step_kv(self.norm1(x_step), cos_p, sin_p, pos, k_self, v_self, v0=v0)
226
+ x_step = x_step + attn_out
227
+ x_step = x_step + self.cross_attn(self.norm2(x_step), k_cross, v_cross, enc_mask)
228
+ x_step = x_step + self.mlp(self.norm3(x_step))
229
+ return x_step, v_raw
230
+
231
+
232
+ class MTPHeadBlock(nn.Module):
233
+ def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int):
234
+ super().__init__()
235
+ self.norm_h = RMSNorm(d_model)
236
+ self.norm_e = RMSNorm(d_model)
237
+ self.proj = nn.Linear(2 * d_model, d_model, bias=False)
238
+ self.norm1 = RMSNorm(d_model)
239
+ self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=True, has_v_res=False)
240
+ self.norm2 = RMSNorm(d_model)
241
+ self.mlp = SwiGLU(d_model, ffn_dim)
242
+ self.out_norm = RMSNorm(d_model)
243
+
244
+ def forward(self, h_main: torch.Tensor, next_tok_emb: torch.Tensor, rope: RotaryEmbedding):
245
+ x = self.proj(torch.cat([self.norm_h(h_main), self.norm_e(next_tok_emb)], dim=-1))
246
+ attn_out, _ = self.self_attn(self.norm1(x), rope, None)
247
+ x = x + attn_out
248
+ x = x + self.mlp(self.norm2(x))
249
+ return self.out_norm(x)
250
+
251
+
252
+ class MyTranslation300M(nn.Module):
253
+ def __init__(
254
+ self,
255
+ vocab_size=65536,
256
+ d_model=1024,
257
+ n_heads=16,
258
+ n_kv_heads=4,
259
+ enc_layers=12,
260
+ dec_layers=6,
261
+ ffn_dim=3072,
262
+ max_seq_len=2048,
263
+ ):
264
+ super().__init__()
265
+ assert d_model % n_heads == 0 and n_heads % n_kv_heads == 0 and dec_layers % 2 == 0
266
+ self.config = dict(
267
+ vocab_size=vocab_size, d_model=d_model, n_heads=n_heads, n_kv_heads=n_kv_heads,
268
+ enc_layers=enc_layers, dec_layers=dec_layers, ffn_dim=ffn_dim, max_seq_len=max_seq_len,
269
+ )
270
+ self.d_model = d_model
271
+ self.n_heads = n_heads
272
+ self.n_kv_heads = n_kv_heads
273
+ self.head_dim = d_model // n_heads
274
+ self.enc_layers = enc_layers
275
+ self.dec_layers = dec_layers
276
+ self.n_cross_pairs = dec_layers // 2
277
+ self.embed_scale = math.sqrt(d_model)
278
+
279
+ self.embed_tokens = nn.Embedding(vocab_size, d_model, padding_idx=PAD_ID)
280
+ self.rope = RotaryEmbedding(self.head_dim, max_seq_len=max_seq_len, theta=50000.0)
281
+
282
+ self.encoder = nn.ModuleList([
283
+ EncoderLayer(d_model, n_heads, n_kv_heads, ffn_dim, has_v_res=(i > 0))
284
+ for i in range(enc_layers)
285
+ ])
286
+ self.enc_dwa_logits = nn.ParameterList([
287
+ nn.Parameter(torch.tensor([-4.0] * i + [0.0], dtype=torch.float32))
288
+ for i in range(1, enc_layers)
289
+ ])
290
+
291
+ self.enc_router_logits = nn.Parameter(torch.zeros(self.n_cross_pairs, enc_layers))
292
+ with torch.no_grad():
293
+ for p in range(self.n_cross_pairs):
294
+ center = (p + 0.5) * (enc_layers / self.n_cross_pairs) - 0.5
295
+ for k in range(enc_layers):
296
+ self.enc_router_logits[p, k] = -0.35 * ((k - center) ** 2)
297
+
298
+ self.cross_kv_projs = nn.ModuleList([
299
+ SharedCrossKVProj(d_model, n_kv_heads, self.head_dim)
300
+ for _ in range(self.n_cross_pairs)
301
+ ])
302
+
303
+ self.decoder = nn.ModuleList([
304
+ DecoderLayer(d_model, n_heads, n_kv_heads, ffn_dim, has_v_res=(i > 0))
305
+ for i in range(dec_layers)
306
+ ])
307
+ self.dec_norm = RMSNorm(d_model)
308
+ self.lm_head = nn.Linear(d_model, vocab_size, bias=False)
309
+ self.lm_head.weight = self.embed_tokens.weight
310
+
311
+ self.mtp_head = MTPHeadBlock(d_model, n_heads, n_kv_heads, ffn_dim)
312
+ self._init_weights(enc_layers + dec_layers)
313
+
314
+ def _init_weights(self, total_layers: int):
315
+ res_std = 0.02 / math.sqrt(2 * total_layers)
316
+ for name, m in self.named_modules():
317
+ if isinstance(m, nn.Linear):
318
+ if name.endswith(".out") or name.endswith(".down"):
319
+ nn.init.normal_(m.weight, mean=0.0, std=res_std)
320
+ elif name.endswith(".gate_proj"):
321
+ nn.init.normal_(m.weight, mean=0.0, std=0.01)
322
+ else:
323
+ nn.init.normal_(m.weight, mean=0.0, std=0.02)
324
+ elif isinstance(m, nn.Embedding):
325
+ nn.init.normal_(m.weight, mean=0.0, std=0.02)
326
+ if m.padding_idx is not None:
327
+ m.weight.data[m.padding_idx].zero_()
328
+
329
+ def compile_blocks(self):
330
+ for i in range(len(self.encoder)):
331
+ self.encoder[i] = torch.compile(self.encoder[i], dynamic=True)
332
+ for i in range(len(self.cross_kv_projs)):
333
+ self.cross_kv_projs[i] = torch.compile(self.cross_kv_projs[i], dynamic=True)
334
+ for i in range(len(self.decoder)):
335
+ self.decoder[i] = torch.compile(self.decoder[i], dynamic=True)
336
+ self.mtp_head = torch.compile(self.mtp_head, dynamic=True)
337
+
338
+ def encode_and_route(self, src_ids: torch.Tensor, ckpt_mode: str = "none"):
339
+ enc_mask = (src_ids != PAD_ID).unsqueeze(1).unsqueeze(2)
340
+ x = self.embed_tokens(src_ids) * self.embed_scale
341
+ raw_states = [x]
342
+ enc_outputs = []
343
+ v0 = None
344
+
345
+ for i, layer in enumerate(self.encoder):
346
+ if i > 0:
347
+ w_dwa = F.softmax(self.enc_dwa_logits[i - 1], dim=0).to(dtype=x.dtype)
348
+ if self.training and ckpt_mode != "none":
349
+ x_in = grad_ckpt(
350
+ lambda w, *st: torch.einsum("j,jbsd->bsd", w, torch.stack(st, dim=0)),
351
+ w_dwa, *raw_states, use_reentrant=False
352
+ )
353
+ else:
354
+ x_in = torch.einsum("j,jbsd->bsd", w_dwa, torch.stack(raw_states, dim=0))
355
+ else:
356
+ x_in = x
357
+
358
+ do_ckpt = self.training and (
359
+ (ckpt_mode == "heavy" and (i % 3 != 0)) or
360
+ (ckpt_mode == "light" and (i in (3, 7, 11)))
361
+ )
362
+ if do_ckpt:
363
+ x_out, v_raw = grad_ckpt(layer, x_in, self.rope, enc_mask, v0, use_reentrant=False)
364
+ else:
365
+ x_out, v_raw = layer(x_in, self.rope, enc_mask, v0=v0)
366
+
367
+ if i == 0:
368
+ v0 = v_raw
369
+ raw_states.append(x_out)
370
+ enc_outputs.append(x_out)
371
+
372
+ stacked_enc = torch.stack(enc_outputs, dim=0)
373
+ router_w = F.softmax(self.enc_router_logits, dim=-1).to(dtype=stacked_enc.dtype)
374
+ cross_kv = []
375
+ for p in range(self.n_cross_pairs):
376
+ enc_mix = torch.einsum("k,kbsd->bsd", router_w[p], stacked_enc)
377
+ cross_kv.append(self.cross_kv_projs[p](enc_mix))
378
+ return cross_kv, enc_mask
379
+
380
+ def decode_seq(self, tgt_in: torch.Tensor, cross_kv: list, enc_mask: torch.Tensor, ckpt_mode: str = "none"):
381
+ x = self.embed_tokens(tgt_in) * self.embed_scale
382
+ v0 = None
383
+ for i, layer in enumerate(self.decoder):
384
+ kc, vc = cross_kv[i // 2]
385
+ do_ckpt = self.training and (
386
+ (ckpt_mode == "heavy" and (i % 2 == 1)) or
387
+ (ckpt_mode == "light" and (i == 3))
388
+ )
389
+ if do_ckpt:
390
+ x, v_raw = grad_ckpt(layer, x, kc, vc, self.rope, enc_mask, v0, use_reentrant=False)
391
+ else:
392
+ x, v_raw = layer(x, kc, vc, self.rope, enc_mask, v0=v0)
393
+ if i == 0:
394
+ v0 = v_raw
395
+ return self.dec_norm(x)
396
+
397
+ def forward_and_backward(
398
+ self,
399
+ src_ids: torch.Tensor,
400
+ tgt_in: torch.Tensor,
401
+ labels: torch.Tensor,
402
+ main_denom: float,
403
+ mtp_denom: float,
404
+ label_smoothing: float = 0.1,
405
+ z_loss_weight: float = 1e-4,
406
+ mtp_weight: float = 0.3,
407
+ ):
408
+ tot_tok = src_ids.numel() + tgt_in.numel()
409
+ max_len = max(src_ids.size(1), tgt_in.size(1))
410
+ if tot_tok <= 14336:
411
+ ckpt_mode = "none"
412
+ elif max_len <= 256:
413
+ ckpt_mode = "light"
414
+ else:
415
+ ckpt_mode = "heavy"
416
+
417
+ with torch.amp.autocast("cuda", dtype=torch.bfloat16):
418
+ cross_kv, enc_mask = self.encode_and_route(src_ids, ckpt_mode=ckpt_mode)
419
+ h_main = self.decode_seq(tgt_in, cross_kv, enc_mask, ckpt_mode=ckpt_mode)
420
+
421
+ valid_mask = labels.ne(-100)
422
+ valid_y = labels[valid_mask]
423
+ n_valid = valid_y.numel()
424
+ valid_h_leaf = h_main[valid_mask].detach().requires_grad_(True)
425
+
426
+ tot_main_ce_t = torch.zeros((), dtype=torch.float64, device=h_main.device)
427
+ tot_z_loss_t = torch.zeros((), dtype=torch.float64, device=h_main.device)
428
+ tot_mtp_ce_t = torch.zeros((), dtype=torch.float64, device=h_main.device)
429
+ chunk_sz = 2048
430
+
431
+ for i in range(0, n_valid, chunk_sz):
432
+ hc = valid_h_leaf[i : i + chunk_sz]
433
+ yc = valid_y[i : i + chunk_sz]
434
+ with torch.amp.autocast("cuda", dtype=torch.bfloat16):
435
+ logits_c = self.lm_head(hc)
436
+ ce_c = F.cross_entropy(logits_c, yc, label_smoothing=label_smoothing, reduction="sum")
437
+ lse = torch.logsumexp(logits_c.float(), dim=-1)
438
+ z_c = (lse * lse).sum()
439
+ loss_c = (ce_c + z_loss_weight * z_c) / main_denom
440
+ loss_c.backward()
441
+ tot_main_ce_t.add_(ce_c.detach().to(torch.float64))
442
+ tot_z_loss_t.add_(z_c.detach().to(torch.float64))
443
+ del logits_c, ce_c, z_c, loss_c
444
+
445
+ grad_h_main = torch.zeros_like(h_main)
446
+ if valid_h_leaf.grad is not None:
447
+ grad_h_main[valid_mask] = valid_h_leaf.grad
448
+
449
+ n_mtp = 0
450
+ if mtp_weight > 0.0 and tgt_in.size(1) > 1:
451
+ mtp_mask = labels[:, 1:].ne(-100) & labels[:, :-1].ne(-100)
452
+ valid_ym = labels[:, 1:][mtp_mask]
453
+ n_mtp = valid_ym.numel()
454
+ if n_mtp > 0:
455
+ h_main_leaf = h_main.detach().requires_grad_(True)
456
+ with torch.amp.autocast("cuda", dtype=torch.bfloat16):
457
+ next_emb = self.embed_tokens(tgt_in[:, 1:]) * self.embed_scale
458
+ h_mtp = self.mtp_head(h_main_leaf[:, :-1, :], next_emb, self.rope)
459
+ valid_hm_leaf = h_mtp[mtp_mask].detach().requires_grad_(True)
460
+
461
+ for i in range(0, n_mtp, chunk_sz):
462
+ hmc = valid_hm_leaf[i : i + chunk_sz]
463
+ ymc = valid_ym[i : i + chunk_sz]
464
+ with torch.amp.autocast("cuda", dtype=torch.bfloat16):
465
+ logits_m = self.lm_head(hmc)
466
+ ce_m = F.cross_entropy(logits_m, ymc, label_smoothing=label_smoothing, reduction="sum")
467
+ loss_m = (mtp_weight * ce_m) / mtp_denom
468
+ loss_m.backward()
469
+ tot_mtp_ce_t.add_(ce_m.detach().to(torch.float64))
470
+ del logits_m, ce_m, loss_m
471
+
472
+ grad_h_mtp = torch.zeros_like(h_mtp)
473
+ if valid_hm_leaf.grad is not None:
474
+ grad_h_mtp[mtp_mask] = valid_hm_leaf.grad
475
+ h_mtp.backward(grad_h_mtp)
476
+ if h_main_leaf.grad is not None:
477
+ grad_h_main.add_(h_main_leaf.grad)
478
+
479
+ h_main.backward(grad_h_main)
480
+ ce_v, mtp_v, z_v = torch.stack([tot_main_ce_t, tot_mtp_ce_t, tot_z_loss_t]).tolist()
481
+ return ce_v, mtp_v, z_v, n_valid, n_mtp
482
+
483
+ @torch.no_grad()
484
+ def eval_nll(self, src_ids: torch.Tensor, tgt_in: torch.Tensor, labels: torch.Tensor):
485
+ with torch.amp.autocast("cuda", dtype=torch.bfloat16):
486
+ cross_kv, enc_mask = self.encode_and_route(src_ids, ckpt_mode="none")
487
+ h_main = self.decode_seq(tgt_in, cross_kv, enc_mask, ckpt_mode="none")
488
+ valid_mask = labels.ne(-100)
489
+ valid_h = h_main[valid_mask]
490
+ valid_y = labels[valid_mask]
491
+ n_valid = valid_y.numel()
492
+ tot_nll = 0.0
493
+ for i in range(0, n_valid, 4096):
494
+ logits_c = self.lm_head(valid_h[i : i + 4096])
495
+ tot_nll += float(F.cross_entropy(logits_c, valid_y[i : i + 4096], reduction="sum").item())
496
+ return tot_nll, n_valid
497
+
498
+ @torch.inference_mode()
499
+ def translate_kv(self, prompt_ids_list: list[list[int]], sp_proc, max_new_tokens: int = 160, rep_penalty: float = 1.12) -> list[str]:
500
+ assert len(prompt_ids_list) > 0 and all(len(s) > 0 for s in prompt_ids_list), "Empty prompt batch!"
501
+ max_rope = self.rope.cos_cached.size(2)
502
+ assert 0 < max_new_tokens <= max_rope, f"max_new_tokens={max_new_tokens} exceeds RoPE={max_rope}"
503
+
504
+ self.eval()
505
+ dev = self.embed_tokens.weight.device
506
+ B = len(prompt_ids_list)
507
+ trimmed = [s[:max_rope] for s in prompt_ids_list]
508
+ max_s = min(max_rope, ((max(len(s) for s in trimmed) + 7) // 8) * 8)
509
+ src_ids = torch.full((B, max_s), PAD_ID, dtype=torch.long, device=dev)
510
+ for i, ids in enumerate(trimmed):
511
+ src_ids[i, : len(ids)] = torch.tensor(ids, dtype=torch.long, device=dev)
512
+
513
+ with torch.amp.autocast("cuda", dtype=torch.bfloat16):
514
+ cross_kv, enc_mask = self.encode_and_route(src_ids, ckpt_mode="none")
515
+ k_cache = torch.empty((self.dec_layers, B, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=torch.bfloat16, device=dev)
516
+ v_cache = torch.empty((self.dec_layers, B, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=torch.bfloat16, device=dev)
517
+
518
+ cur_tok = torch.full((B, 1), BOS_ID, dtype=torch.long, device=dev)
519
+ gen_ids = torch.full((B, max_new_tokens), EOS_ID, dtype=torch.long, device=dev)
520
+ done = torch.zeros(B, dtype=torch.bool, device=dev)
521
+ cos_all, sin_all = self.rope.cos_cached.to(torch.bfloat16), self.rope.sin_cached.to(torch.bfloat16)
522
+
523
+ for pos in range(max_new_tokens):
524
+ cos_p, sin_p = cos_all[:, :, pos : pos + 1, :], sin_all[:, :, pos : pos + 1, :]
525
+ x_dec = self.embed_tokens(cur_tok) * self.embed_scale
526
+ v0 = None
527
+ for l_idx, layer in enumerate(self.decoder):
528
+ kc, vc = cross_kv[l_idx // 2]
529
+ x_dec, v_raw = layer.step_kv(x_dec, cos_p, sin_p, pos, k_cache[l_idx], v_cache[l_idx], kc, vc, enc_mask, v0=v0)
530
+ if l_idx == 0:
531
+ v0 = v_raw
532
+
533
+ logits = self.lm_head(self.dec_norm(x_dec)[:, 0, :]).float()
534
+ logits[:, [0, 1, 3, 4, 5, 6, 7, 8, 9, 10]] = -1e9
535
+ if rep_penalty != 1.0 and pos > 0:
536
+ prev = gen_ids[:, :pos]
537
+ s_l = torch.gather(logits, 1, prev)
538
+ s_l = torch.where(s_l > 0, s_l / rep_penalty, s_l * rep_penalty)
539
+ logits.scatter_(1, prev, s_l)
540
+
541
+ nxt = torch.where(done, torch.tensor(EOS_ID, device=dev), logits.argmax(dim=-1))
542
+ gen_ids[:, pos] = nxt
543
+ done = done | (nxt == EOS_ID)
544
+ cur_tok = nxt.unsqueeze(1)
545
+ if bool(done.all()):
546
+ break
547
+
548
+ out_texts = []
549
+ for row in gen_ids.cpu().tolist():
550
+ if EOS_ID in row:
551
+ row = row[: row.index(EOS_ID)]
552
+ out_texts.append(sp_proc.decode_ids([x for x in row if x >= 11]).strip())
553
+ return out_texts
554
+
555
+
556
+ # ==============================================================================
557
+ # 2. 100%-Coverage Plan + Tight-Padding Dynamic Packing + Distributed Sharding
558
+ # ==============================================================================
559
+ class CanonicalMemmapStore:
560
+ def __init__(self, data_dir: str, split: str):
561
+ self.my_path = os.path.join(data_dir, f"{split}_my.bin")
562
+ self.en_path = os.path.join(data_dir, f"{split}_en.bin")
563
+ self.index = np.load(os.path.join(data_dir, f"{split}_index.npy"), mmap_mode="r")
564
+ self.my_mm = None
565
+ self.en_mm = None
566
+
567
+ def ensure_open(self):
568
+ if self.my_mm is None:
569
+ self.my_mm = np.memmap(self.my_path, dtype=np.uint16, mode="r")
570
+ self.en_mm = np.memmap(self.en_path, dtype=np.uint16, mode="r")
571
+
572
+ def get_pair(self, idx: int):
573
+ self.ensure_open()
574
+ rec = self.index[idx]
575
+ m = self.my_mm[int(rec["my_off"]) : int(rec["my_off"]) + int(rec["my_len"])].astype(np.int64)
576
+ e = self.en_mm[int(rec["en_off"]) : int(rec["en_off"]) + int(rec["en_len"])].astype(np.int64)
577
+ return m, e, int(rec["flags"])
578
+
579
+
580
+ def build_or_load_epoch_plan(train_index: np.ndarray, meta: dict, stage: str, plan_path: str, is_main: bool, world_size: int, seed: int = 2026) -> np.ndarray:
581
+ if is_main and not os.path.exists(plan_path):
582
+ rng = np.random.RandomState(seed)
583
+ n_subsets = len(meta["subset_to_id"])
584
+ sids = np.asarray(train_index["subset_id"])
585
+ assert int(sids.max()) < n_subsets and int(sids.min()) >= 0
586
+
587
+ if stage == "stable":
588
+ base_rows = rng.permutation(len(train_index)).astype(np.uint32)
589
+ replay_n = 12_000_000
590
+ eligible_mask = np.ones(len(train_index), dtype=bool)
591
+ for k, name in meta["id_to_subset"].items():
592
+ if name in BULK_SUBSETS:
593
+ eligible_mask[sids == int(k)] = False
594
+ else:
595
+ eligible_mask = (np.asarray(train_index["flags"]) & FLAG_BT) == 0
596
+ base_rows = np.flatnonzero(eligible_mask).astype(np.uint32)
597
+ rng.shuffle(base_rows)
598
+ replay_n = 1_500_000
599
+
600
+ elig_pos = np.flatnonzero(eligible_mask).astype(np.uint32)
601
+ elig_sids = sids[elig_pos]
602
+ order = elig_pos[np.argsort(elig_sids, kind="mergesort")]
603
+ sorted_sids = sids[order]
604
+ counts = np.bincount(sorted_sids, minlength=n_subsets)
605
+ starts = np.zeros(n_subsets, dtype=np.int64)
606
+ starts[1:] = np.cumsum(counts[:-1])
607
+
608
+ weights = np.zeros(n_subsets, dtype=np.float64)
609
+ for k, name in meta["id_to_subset"].items():
610
+ sid = int(k)
611
+ if counts[sid] == 0:
612
+ continue
613
+ if stage == "stable" and name in BULK_SUBSETS:
614
+ continue
615
+ if stage == "cooldown" and name == "non_myanmartranslator/finetranslations":
616
+ w = float(counts[sid]) ** 0.35 * 0.25
617
+ else:
618
+ w = float(counts[sid]) ** 0.50
619
+ if stage == "stable" and name.startswith("non_myanmartranslator/"):
620
+ w *= 4.0
621
+ weights[sid] = w
622
+
623
+ assert weights.sum() > 0, "Empty replay weights!"
624
+ probs = weights / weights.sum()
625
+
626
+ chosen_sids = rng.choice(n_subsets, size=replay_n, p=probs)
627
+ replay_rows = np.empty(replay_n, dtype=np.uint32)
628
+ for sid in range(n_subsets):
629
+ m = (chosen_sids == sid)
630
+ npick = int(m.sum())
631
+ if npick > 0:
632
+ replay_rows[m] = order[starts[sid] + rng.randint(0, counts[sid], size=npick)]
633
+
634
+ full_plan = np.concatenate([base_rows, replay_rows])
635
+ rng.shuffle(full_plan)
636
+ tmp_plan = plan_path + ".tmp.npy"
637
+ np.save(tmp_plan, full_plan)
638
+ os.replace(tmp_plan, plan_path)
639
+ print(
640
+ f"📋 Built {stage} 100%-coverage plan: {len(base_rows):,} unique base rows (100% coverage) "
641
+ f"+ {len(replay_rows):,} gold replay = {len(full_plan):,} total rows"
642
+ )
643
+
644
+ if world_size > 1:
645
+ dist.barrier()
646
+ plan = np.load(plan_path, mmap_mode="r")
647
+ if is_main:
648
+ print(f"📋 Ready {stage} plan: {len(plan):,} rows")
649
+ return plan
650
+
651
+
652
+ def pack_chunk_batches(
653
+ row_slice: np.ndarray,
654
+ train_index: np.ndarray,
655
+ max_tokens_combined: int,
656
+ max_seq_len: int,
657
+ step_group: int,
658
+ seed: int,
659
+ ):
660
+ rng = np.random.RandomState(seed)
661
+ my_lens = np.minimum(max_seq_len, train_index["my_len"][row_slice].astype(np.int32))
662
+ en_lens = np.minimum(max_seq_len, train_index["en_len"][row_slice].astype(np.int32))
663
+ max_side = np.maximum(my_lens, en_lens)
664
+ comb_lens = my_lens + en_lens
665
+
666
+ mega = 65536
667
+ batches_cost = []
668
+ for i in range(0, len(row_slice), mega):
669
+ sub_r = row_slice[i : i + mega]
670
+ sub_ms = max_side[i : i + mega]
671
+ sub_cl = comb_lens[i : i + mega]
672
+ sub_r = sub_r[np.lexsort((sub_cl, sub_ms))]
673
+
674
+ cur_b, cur_s_max, cur_t_max = [], 0, 0
675
+ for r in sub_r:
676
+ ri = int(r)
677
+ ml = min(max_seq_len, int(train_index["my_len"][ri]))
678
+ el = min(max_seq_len, int(train_index["en_len"][ri]))
679
+ s_est = max(el, ml) + 4
680
+ t_est = max(ml, el) + 2
681
+ new_s = max(cur_s_max, s_est)
682
+ new_t = max(cur_t_max, t_est)
683
+ if cur_b and (len(cur_b) >= 256 or (len(cur_b) + 1) * (new_s + new_t) > max_tokens_combined):
684
+ batches_cost.append(((max(cur_s_max, cur_t_max), len(cur_b) * (cur_s_max + cur_t_max)), cur_b))
685
+ cur_b, cur_s_max, cur_t_max = [ri], s_est, t_est
686
+ else:
687
+ cur_b.append(ri)
688
+ cur_s_max, cur_t_max = new_s, new_t
689
+ if cur_b:
690
+ batches_cost.append(((max(cur_s_max, cur_t_max), len(cur_b) * (cur_s_max + cur_t_max)), cur_b))
691
+
692
+ # Load-balanced step grouping by (max_seq, tok_cost) + safe small-chunk pad
693
+ batches_cost.sort(key=lambda x: x[0])
694
+ batches = [b for _, b in batches_cost]
695
+ rem = len(batches) % step_group
696
+ if rem != 0 and len(batches) > 0:
697
+ need = step_group - rem
698
+ batches.extend((batches * ((need // len(batches)) + 1))[:need])
699
+ groups = [batches[j : j + step_group] for j in range(0, len(batches), step_group)]
700
+ rng.shuffle(groups)
701
+ return [b for g in groups for b in g]
702
+
703
+
704
+ class DynamicStreamDataset(Dataset):
705
+ def __init__(self, store: CanonicalMemmapStore, batches: list[list[int]], stage: str, max_seq_len: int, seed: int):
706
+ self.store = store
707
+ self.batches = batches
708
+ self.stage = stage
709
+ self.max_seq_len = max_seq_len
710
+ self.base_seed = seed
711
+ self.rng = np.random.RandomState(seed)
712
+
713
+ def __len__(self):
714
+ return len(self.batches)
715
+
716
+ def _corrupt_span(self, toks: np.ndarray):
717
+ n = len(toks)
718
+ if n < 4:
719
+ return toks
720
+ out = []
721
+ i = 0
722
+ masked_any = False
723
+ while i < n:
724
+ if self.rng.rand() < 0.15:
725
+ span_len = min(n - i, max(1, int(self.rng.poisson(3.0))))
726
+ out.append(TOK_MASK)
727
+ i += span_len
728
+ masked_any = True
729
+ else:
730
+ out.append(int(toks[i]))
731
+ i += 1
732
+ if not masked_any:
733
+ idx = int(self.rng.randint(0, n))
734
+ out = list(toks[:idx]) + [TOK_MASK] + list(toks[idx + min(2, n - idx):])
735
+ return np.asarray(out, dtype=np.int64)
736
+
737
+ def __getitem__(self, b_idx: int):
738
+ row_ids = self.batches[b_idx]
739
+ srcs, tgts = [], []
740
+ for r_id in row_ids:
741
+ m, e, flg = self.store.get_pair(r_id)
742
+ m = m[: self.max_seq_len - 4]
743
+ e = e[: self.max_seq_len - 4]
744
+ is_bt = bool(flg & FLAG_BT)
745
+ is_formal = bool(flg & FLAG_FORMAL)
746
+ is_spoken = bool(flg & FLAG_SPOKEN)
747
+
748
+ if self.stage == "cooldown":
749
+ mode = "en2my" if self.rng.rand() < 0.50 else "my2en"
750
+ else:
751
+ p = self.rng.rand()
752
+ if is_bt:
753
+ mode = "en2my" if p < 0.72 else ("my2en_bt" if p < 0.90 else "denoise")
754
+ else:
755
+ mode = "en2my" if p < 0.47 else ("my2en" if p < 0.94 else "denoise")
756
+
757
+ style_tok = None
758
+ if is_formal and self.rng.rand() < 0.70:
759
+ style_tok = TOK_FORMAL
760
+ elif is_spoken and self.rng.rand() < 0.70:
761
+ style_tok = TOK_SPOKEN
762
+
763
+ if mode == "en2my":
764
+ pref = [TOK_2MY] + ([style_tok] if style_tok is not None else [])
765
+ src_arr = np.concatenate([np.asarray(pref, dtype=np.int64), e, [EOS_ID]])
766
+ tgt_arr = np.concatenate([[BOS_ID], m, [EOS_ID]])
767
+ elif mode in ("my2en", "my2en_bt"):
768
+ pref = [TOK_2EN] + ([TOK_BT] if mode == "my2en_bt" else []) + ([style_tok] if style_tok is not None else [])
769
+ src_arr = np.concatenate([np.asarray(pref, dtype=np.int64), m, [EOS_ID]])
770
+ tgt_arr = np.concatenate([[BOS_ID], e, [EOS_ID]])
771
+ else:
772
+ base = m if (is_bt or self.rng.rand() < 0.5) else e
773
+ noisy = self._corrupt_span(base)
774
+ src_arr = np.concatenate([[TOK_DENOISE], noisy, [EOS_ID]])
775
+ tgt_arr = np.concatenate([[BOS_ID], base, [EOS_ID]])
776
+
777
+ srcs.append(src_arr)
778
+ tgts.append(tgt_arr)
779
+ return srcs, tgts
780
+
781
+
782
+ def worker_init_fn(worker_id: int):
783
+ info = torch.utils.data.get_worker_info()
784
+ if info is not None and hasattr(info.dataset, "rng"):
785
+ info.dataset.rng = np.random.RandomState((info.dataset.base_seed + worker_id * 9973) % (2**31 - 1))
786
+
787
+
788
+ def collate_prebatched(batch_list):
789
+ srcs, tgts = batch_list[0]
790
+ B = len(srcs)
791
+ max_s = min(2048, ((max(len(x) for x in srcs) + 31) // 32) * 32)
792
+ max_t = min(2048, ((max(len(x) - 1 for x in tgts) + 31) // 32) * 32)
793
+ src_ids = torch.full((B, max_s), PAD_ID, dtype=torch.long)
794
+ tgt_in = torch.full((B, max_t), PAD_ID, dtype=torch.long)
795
+ labels = torch.full((B, max_t), -100, dtype=torch.long)
796
+ for i, (s, t) in enumerate(zip(srcs, tgts)):
797
+ sl = min(len(s), max_s)
798
+ tl = min(len(t) - 1, max_t)
799
+ src_ids[i, :sl] = torch.from_numpy(s[:sl])
800
+ tgt_in[i, :tl] = torch.from_numpy(t[:tl])
801
+ labels[i, :tl] = torch.from_numpy(t[1 : tl + 1])
802
+ return src_ids, tgt_in, labels
803
+
804
+
805
+ def clean_state_dict(model: nn.Module):
806
+ return {k.replace("._orig_mod.", "."): v.detach().cpu() for k, v in model.state_dict().items()}
807
+
808
+
809
+ def broadcast_model(model: nn.Module, world_size: int):
810
+ if world_size == 1:
811
+ return
812
+ for p in model.parameters():
813
+ dist.broadcast(p.data, src=0)
814
+ for b in model.buffers():
815
+ dist.broadcast(b.data, src=0)
816
+
817
+
818
+ def allreduce_grads(model: nn.Module, world_size: int, grad_scale: float = 1024.0):
819
+ if world_size == 1:
820
+ return
821
+ grads = []
822
+ for p in model.parameters():
823
+ if p.requires_grad:
824
+ if p.grad is None:
825
+ p.grad = torch.zeros_like(p.data)
826
+ grads.append(p.grad)
827
+ b_grads = [g.bfloat16() for g in grads]
828
+ flat = torch._utils._flatten_dense_tensors(b_grads)
829
+ dist.all_reduce(flat, op=dist.ReduceOp.SUM)
830
+ inv_scale = 1.0 / grad_scale
831
+ for orig, synced in zip(grads, torch._utils._unflatten_dense_tensors(flat, b_grads)):
832
+ orig.copy_(synced.float() * inv_scale)
833
+
834
+
835
+ # ==============================================================================
836
+ # 3. Main Multi-GPU Training Loop
837
+ # ==============================================================================
838
+ def main():
839
+ parser = argparse.ArgumentParser()
840
+ parser.add_argument("--stage", choices=["stable", "cooldown"], default="stable")
841
+ parser.add_argument("--verify-steps", type=int, default=0)
842
+ parser.add_argument("--grad-accum", type=int, default=1)
843
+ parser.add_argument("--max-tok-micro", type=int, default=32768)
844
+ parser.add_argument("--max-seq-len", type=int, default=1024)
845
+ parser.add_argument("--chunk-rows", type=int, default=4000000)
846
+ parser.add_argument("--peak-lr", type=float, default=5.0e-4)
847
+ parser.add_argument("--warmup-steps", type=int, default=900)
848
+ parser.add_argument("--eval-every", type=int, default=500)
849
+ parser.add_argument("--save-every", type=int, default=500)
850
+ parser.add_argument("--milestone-every", type=int, default=5000)
851
+ parser.add_argument("--compile", action="store_true")
852
+ parser.add_argument("--fresh", action="store_true")
853
+ parser.add_argument("--init-weights", type=str, default="")
854
+ args = parser.parse_args()
855
+
856
+ world_size = int(os.environ.get("WORLD_SIZE", 1))
857
+ rank = int(os.environ.get("RANK", 0))
858
+ local_rank = int(os.environ.get("LOCAL_RANK", 0))
859
+ is_main = (rank == 0)
860
+
861
+ if world_size > 1:
862
+ torch.cuda.set_device(local_rank)
863
+ dist.init_process_group(backend="nccl", timeout=timedelta(minutes=30), device_id=torch.device(f"cuda:{local_rank}"))
864
+ device = torch.device("cuda", local_rank)
865
+ else:
866
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
867
+
868
+ if is_main:
869
+ os.makedirs(CKPT_DIR, exist_ok=True)
870
+ if world_size > 1:
871
+ dist.barrier()
872
+
873
+ meta, sp = load_and_verify_contracts(DATA_DIR)
874
+ train_store = CanonicalMemmapStore(DATA_DIR, "train")
875
+ val_store = CanonicalMemmapStore(DATA_DIR, "val")
876
+
877
+ all_val_non_bt = np.flatnonzero((val_store.index["flags"] & FLAG_BT) == 0)
878
+ assert len(all_val_non_bt) > 0, "Validation set has zero non-BT rows!"
879
+ val_rows = all_val_non_bt[::max(1, len(all_val_non_bt) // 1024)][:1024]
880
+ val_batches = [val_rows[i : i + 64].tolist() for i in range(0, len(val_rows), 64)]
881
+
882
+ plan_path = os.path.join(CKPT_DIR, f"{args.stage}_plan.npy")
883
+ latest_path_early = os.path.join(CKPT_DIR, "latest.pt")
884
+ if args.fresh and is_main:
885
+ if os.path.exists(plan_path):
886
+ os.remove(plan_path)
887
+ if os.path.exists(latest_path_early):
888
+ os.replace(latest_path_early, latest_path_early + ".bak")
889
+ if world_size > 1:
890
+ dist.barrier()
891
+ full_plan = build_or_load_epoch_plan(train_store.index, meta, args.stage, plan_path, is_main, world_size, seed=2026)
892
+ num_chunks = (len(full_plan) + args.chunk_rows - 1) // args.chunk_rows
893
+
894
+ model = MyTranslation300M().to(device)
895
+ decay_p = [p for n, p in model.named_parameters() if p.requires_grad and p.ndim >= 2]
896
+ nodecay_p = [p for n, p in model.named_parameters() if p.requires_grad and p.ndim < 2]
897
+ opt = torch.optim.AdamW(
898
+ [{"params": decay_p, "weight_decay": 0.01}, {"params": nodecay_p, "weight_decay": 0.0}],
899
+ lr=args.peak_lr, betas=(0.9, 0.95), eps=1e-8, fused=True,
900
+ )
901
+
902
+ step = 0
903
+ chunk_idx = 0
904
+ batch_cursor = 0
905
+ rows_seen = 0
906
+ best_val_nll = float("inf")
907
+ latest_path = os.path.join(CKPT_DIR, "latest.pt")
908
+
909
+ if args.init_weights:
910
+ if is_main:
911
+ print(f"📥 Loading weights from {args.init_weights}")
912
+ model.load_state_dict(torch.load(args.init_weights, map_location="cpu"), strict=True)
913
+ elif (not args.fresh) and os.path.exists(latest_path):
914
+ if is_main:
915
+ print(f"🔄 Resuming from {latest_path}")
916
+ ckpt = torch.load(latest_path, map_location="cpu")
917
+ model.load_state_dict(ckpt["model"], strict=True)
918
+ if args.stage == ckpt.get("stage", "stable"):
919
+ opt.load_state_dict(ckpt["opt"])
920
+ step = int(ckpt["step"])
921
+ chunk_idx = int(ckpt.get("chunk_idx", 0))
922
+ batch_cursor = int(ckpt.get("batch_cursor", 0))
923
+ rows_seen = int(ckpt.get("rows_seen", 0))
924
+ best_val_nll = float(ckpt.get("best_val_nll", float("inf")))
925
+ elif is_main:
926
+ print(f"➡️ Stage transition {ckpt.get('stage')} -> {args.stage}")
927
+
928
+ broadcast_model(model, world_size)
929
+
930
+ if args.compile:
931
+ if is_main:
932
+ print("⚡ Compiling encoder/cross_kv/decoder/MTP blocks with torch.compile(dynamic=True)...")
933
+ model.compile_blocks()
934
+
935
+ if is_main:
936
+ tot_params = sum(p.numel() for p in model.parameters())
937
+ inf_params = tot_params - sum(p.numel() for p in model.mtp_head.parameters())
938
+ print(f"🏛️ MyTranslation-300M | GPUs: {world_size}x | Stage: {args.stage} | Inf: {inf_params/1e6:.2f}M | Train: {tot_params/1e6:.2f}M")
939
+
940
+ label_smoothing = 0.1 if args.stage == "stable" else 0.0
941
+ mtp_weight = 0.3 if args.stage == "stable" else 0.15
942
+
943
+ STYLE_EVAL_PROMPTS = [
944
+ ("En->My Formal", [TOK_2MY, TOK_FORMAL], "The central bank raised interest rates to control inflation and stabilize the economy."),
945
+ ("En->My Spoken", [TOK_2MY, TOK_SPOKEN], "Honestly, I didn't expect things to turn out like this, but don't worry—we'll figure it out together."),
946
+ ("My->En Formal", [TOK_2EN, TOK_FORMAL], "ဗဟိုဘဏ်သည် ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးတည်ငြိမ်စေရန် အတိုးနှုန်းများကို မြှင့်တင်ခဲ့သည်။"),
947
+ ("My->En Spoken", [TOK_2EN, TOK_SPOKEN], "ဒီနေ့ ရန်ကုန်မှာ မိုးတွေ အရမ်းရွာနေလို့ အိမ်ပြန်ရတာ နည်းနည်းနောက်ကျမယ်နော်။"),
948
+ ]
949
+
950
+ @torch.no_grad()
951
+ def eval_val_nll():
952
+ model.eval()
953
+ en2my_nll, en2my_tok = 0.0, 0
954
+ my2en_nll, my2en_tok = 0.0, 0
955
+ for b in val_batches:
956
+ for direction in ("en2my", "my2en"):
957
+ srcs, tgts = [], []
958
+ for r_id in b:
959
+ m, e, _ = val_store.get_pair(int(r_id))
960
+ m, e = m[: args.max_seq_len - 4], e[: args.max_seq_len - 4]
961
+ if direction == "en2my":
962
+ srcs.append(np.concatenate([[TOK_2MY], e, [EOS_ID]]))
963
+ tgts.append(np.concatenate([[BOS_ID], m, [EOS_ID]]))
964
+ else:
965
+ srcs.append(np.concatenate([[TOK_2EN], m, [EOS_ID]]))
966
+ tgts.append(np.concatenate([[BOS_ID], e, [EOS_ID]]))
967
+ src_ids, tgt_in, labels = collate_prebatched([(srcs, tgts)])
968
+ nll, nv = model.eval_nll(src_ids.to(device), tgt_in.to(device), labels.to(device))
969
+ if direction == "en2my":
970
+ en2my_nll += nll
971
+ en2my_tok += nv
972
+ else:
973
+ my2en_nll += nll
974
+ my2en_tok += nv
975
+ e2m = en2my_nll / max(1, en2my_tok)
976
+ m2e = my2en_nll / max(1, my2en_tok)
977
+ comb = (en2my_nll + my2en_nll) / max(1, en2my_tok + my2en_tok)
978
+ return comb, e2m, m2e
979
+
980
+ def run_style_generations(step_num: int):
981
+ prompts = [pref + sp.encode_as_ids(txt)[:500] + [EOS_ID] for _, pref, txt in STYLE_EVAL_PROMPTS]
982
+ preds = model.translate_kv(prompts, sp, max_new_tokens=96)
983
+ print(f" 🗣️ Style Generations @ Step {step_num}:")
984
+ for (tag, _, _), out in zip(STYLE_EVAL_PROMPTS, preds):
985
+ print(f" [{tag}]: {out[:180]}")
986
+
987
+ def make_chunk_loader(c_idx: int, skip_steps: int = 0):
988
+ s_off = c_idx * args.chunk_rows
989
+ e_off = min(len(full_plan), s_off + args.chunk_rows)
990
+ row_slice = np.asarray(full_plan[s_off:e_off])
991
+ step_group = world_size * args.grad_accum
992
+ all_batches = pack_chunk_batches(
993
+ row_slice, train_store.index,
994
+ max_tokens_combined=args.max_tok_micro,
995
+ max_seq_len=args.max_seq_len,
996
+ step_group=step_group,
997
+ seed=2026 + c_idx * 17,
998
+ )
999
+ rank_batches = all_batches[rank::world_size]
1000
+ if skip_steps > 0:
1001
+ rank_batches = rank_batches[skip_steps * args.grad_accum :]
1002
+ n_windows = len(rank_batches) // args.grad_accum
1003
+ rank_batches = rank_batches[: n_windows * args.grad_accum]
1004
+ ds = DynamicStreamDataset(
1005
+ train_store, rank_batches, args.stage, args.max_seq_len,
1006
+ seed=10000 + c_idx * 997 + rank * 131 + skip_steps,
1007
+ )
1008
+ loader = DataLoader(
1009
+ ds, batch_size=1, shuffle=False, collate_fn=collate_prebatched,
1010
+ num_workers=4, pin_memory=True, worker_init_fn=worker_init_fn, persistent_workers=False,
1011
+ )
1012
+ return loader, n_windows
1013
+
1014
+ if is_main:
1015
+ comb, e2m, m2e = eval_val_nll()
1016
+ print(f"🔎 Step {step:6d} | Val Comb: {comb:.4f} | En->My: {e2m:.4f} | My->En: {m2e:.4f}")
1017
+ if world_size > 1:
1018
+ dist.barrier()
1019
+
1020
+ model.train()
1021
+ opt.zero_grad(set_to_none=True)
1022
+ acc_ce = acc_mtp = acc_z = 0.0
1023
+ acc_valid_tok = acc_mtp_tok = acc_tot_tok = 0
1024
+ t0 = time.time()
1025
+
1026
+ while chunk_idx < num_chunks:
1027
+ train_loader, n_windows = make_chunk_loader(chunk_idx, skip_steps=batch_cursor)
1028
+ train_iter = iter(train_loader)
1029
+ done_flag = False
1030
+
1031
+ for _ in range(n_windows):
1032
+ window = [next(train_iter) for _ in range(args.grad_accum)]
1033
+ batch_cursor += 1
1034
+
1035
+ loc_valid = sum(int((lb != -100).sum().item()) for _, _, lb in window)
1036
+ loc_mtp = sum(
1037
+ int(((lb[:, 1:] != -100) & (lb[:, :-1] != -100)).sum().item()) if lb.size(1) > 1 else 0
1038
+ for _, _, lb in window
1039
+ )
1040
+ loc_rows = sum(s.size(0) for s, _, _ in window)
1041
+
1042
+ if world_size > 1:
1043
+ counts_t = torch.tensor([loc_valid, loc_mtp, loc_rows], dtype=torch.long, device=device)
1044
+ dist.all_reduce(counts_t, op=dist.ReduceOp.SUM)
1045
+ glob_valid, glob_mtp, glob_rows = int(counts_t[0].item()), int(counts_t[1].item()), int(counts_t[2].item())
1046
+ else:
1047
+ glob_valid, glob_mtp, glob_rows = loc_valid, loc_mtp, loc_rows
1048
+
1049
+ g_scale = 1024.0 if world_size > 1 else 1.0
1050
+ main_denom = float(max(1, glob_valid)) / g_scale
1051
+ mtp_denom = float(max(1, glob_mtp)) / g_scale
1052
+
1053
+ loc_ce = loc_mtp_ce = loc_z = 0.0
1054
+ loc_tok = sum(int((s != PAD_ID).sum().item()) for s, _, _ in window)
1055
+ for src_ids, tgt_in, labels in window:
1056
+ src_ids = src_ids.to(device, non_blocking=True)
1057
+ tgt_in = tgt_in.to(device, non_blocking=True)
1058
+ labels = labels.to(device, non_blocking=True)
1059
+ ce_s, mtp_s, z_s, nv, nm = model.forward_and_backward(
1060
+ src_ids, tgt_in, labels,
1061
+ main_denom=main_denom, mtp_denom=mtp_denom,
1062
+ label_smoothing=label_smoothing, mtp_weight=mtp_weight,
1063
+ )
1064
+ loc_ce += ce_s
1065
+ loc_mtp_ce += mtp_s
1066
+ loc_z += z_s
1067
+ loc_tok += nv
1068
+
1069
+ allreduce_grads(model, world_size)
1070
+ rows_seen += glob_rows
1071
+
1072
+ if args.stage == "stable":
1073
+ lr = args.peak_lr * min(1.0, (step + 1) / max(1, args.warmup_steps))
1074
+ else:
1075
+ if step < 100:
1076
+ lr = args.peak_lr * (step + 1) / 100.0
1077
+ else:
1078
+ progress = min(1.0, rows_seen / max(1, len(full_plan)))
1079
+ lr = 1.0e-5 + 0.5 * (args.peak_lr - 1.0e-5) * (1.0 + math.cos(math.pi * progress))
1080
+
1081
+ for pg in opt.param_groups:
1082
+ pg["lr"] = lr
1083
+ grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
1084
+ opt.step()
1085
+ opt.zero_grad(set_to_none=True)
1086
+ step += 1
1087
+
1088
+ if world_size > 1:
1089
+ stats_t = torch.tensor([loc_ce, loc_mtp_ce, loc_z, float(loc_tok)], dtype=torch.float64, device=device)
1090
+ dist.all_reduce(stats_t, op=dist.ReduceOp.SUM)
1091
+ acc_ce += float(stats_t[0].item())
1092
+ acc_mtp += float(stats_t[1].item())
1093
+ acc_z += float(stats_t[2].item())
1094
+ acc_tot_tok += int(stats_t[3].item())
1095
+ else:
1096
+ acc_ce += loc_ce
1097
+ acc_mtp += loc_mtp_ce
1098
+ acc_z += loc_z
1099
+ acc_tot_tok += loc_tok
1100
+ acc_valid_tok += glob_valid
1101
+ acc_mtp_tok += glob_mtp
1102
+
1103
+ log_int = 10 if args.verify_steps > 0 else 50
1104
+ if step % log_int == 0:
1105
+ if is_main:
1106
+ dt = time.time() - t0
1107
+ tok_s = acc_tot_tok / max(1e-3, dt)
1108
+ vram_g = torch.cuda.max_memory_allocated(device) / 1e9
1109
+ pct = 100.0 * min(rows_seen, len(full_plan)) / len(full_plan)
1110
+ print(
1111
+ f"Step {step:6d} ({pct:5.1f}% | Chunk {chunk_idx+1}/{num_chunks}) | "
1112
+ f"Main: {acc_ce/max(1,acc_valid_tok):.4f} | MTP2: {acc_mtp/max(1,acc_mtp_tok):.4f} | "
1113
+ f"Z: {acc_z/max(1,acc_valid_tok):.2f} | LR: {lr:.2e} | Grad: {grad_norm:.2f} | "
1114
+ f"{tok_s/1e3:.1f}k tok/s | VRAM: {vram_g:.1f}G"
1115
+ )
1116
+ acc_ce = acc_mtp = acc_z = 0.0
1117
+ acc_valid_tok = acc_mtp_tok = acc_tot_tok = 0
1118
+ t0 = time.time()
1119
+
1120
+ if args.verify_steps > 0 and step >= args.verify_steps:
1121
+ done_flag = True
1122
+ break
1123
+
1124
+ if step % args.eval_every == 0:
1125
+ if is_main:
1126
+ comb, e2m, m2e = eval_val_nll()
1127
+ print("-" * 96)
1128
+ print(f"🔎 EVAL @ Step {step:6d} | Comb NLL: {comb:.4f} (PPL {math.exp(min(comb, 30.0)):.2f}) | En->My: {e2m:.4f} | My->En: {m2e:.4f}")
1129
+ run_style_generations(step)
1130
+ if comb < best_val_nll:
1131
+ best_val_nll = comb
1132
+ best_sd = {k: v.to(torch.bfloat16) for k, v in clean_state_dict(model).items()}
1133
+ torch.save(best_sd, os.path.join(CKPT_DIR, "best_model.pt"))
1134
+ print(f" ⭐ New Best Model saved! Comb NLL: {best_val_nll:.4f}")
1135
+ print("-" * 96)
1136
+ model.train()
1137
+ if world_size > 1:
1138
+ dist.barrier()
1139
+ t0 = time.time()
1140
+
1141
+ if step % args.save_every == 0:
1142
+ if is_main:
1143
+ tmp_p = latest_path + ".tmp"
1144
+ torch.save({
1145
+ "model": clean_state_dict(model),
1146
+ "opt": opt.state_dict(),
1147
+ "step": step,
1148
+ "stage": args.stage,
1149
+ "chunk_idx": chunk_idx,
1150
+ "batch_cursor": batch_cursor,
1151
+ "rows_seen": rows_seen,
1152
+ "best_val_nll": best_val_nll,
1153
+ }, tmp_p)
1154
+ os.replace(tmp_p, latest_path)
1155
+ if world_size > 1:
1156
+ dist.barrier()
1157
+
1158
+ if step % args.milestone_every == 0:
1159
+ if is_main:
1160
+ ms_sd = {k: v.to(torch.bfloat16) for k, v in clean_state_dict(model).items()}
1161
+ torch.save(ms_sd, os.path.join(CKPT_DIR, f"{args.stage}_step_{step:06d}.pt"))
1162
+ if world_size > 1:
1163
+ dist.barrier()
1164
+
1165
+ if done_flag:
1166
+ break
1167
+ chunk_idx += 1
1168
+ batch_cursor = 0
1169
+
1170
+ if args.verify_steps > 0:
1171
+ if is_main:
1172
+ print(f"✅ {world_size}-GPU VERIFY PASSED ({step} steps)! Ready for launch.")
1173
+ if world_size > 1:
1174
+ dist.barrier()
1175
+ dist.destroy_process_group()
1176
+ return
1177
+
1178
+ if is_main:
1179
+ comb, e2m, m2e = eval_val_nll()
1180
+ print("-" * 96)
1181
+ print(f"🏁 FINAL EVAL @ Step {step:6d} | Comb NLL: {comb:.4f} (PPL {math.exp(min(comb, 30.0)):.2f}) | En->My: {e2m:.4f} | My->En: {m2e:.4f}")
1182
+ run_style_generations(step)
1183
+ if comb < best_val_nll:
1184
+ best_val_nll = comb
1185
+ best_sd = {k: v.to(torch.bfloat16) for k, v in clean_state_dict(model).items()}
1186
+ torch.save(best_sd, os.path.join(CKPT_DIR, "best_model.pt"))
1187
+ tmp_p = latest_path + ".tmp"
1188
+ torch.save({
1189
+ "model": clean_state_dict(model),
1190
+ "opt": opt.state_dict(),
1191
+ "step": step,
1192
+ "stage": args.stage,
1193
+ "chunk_idx": chunk_idx,
1194
+ "batch_cursor": batch_cursor,
1195
+ "rows_seen": rows_seen,
1196
+ "best_val_nll": best_val_nll,
1197
+ }, tmp_p)
1198
+ os.replace(tmp_p, latest_path)
1199
+
1200
+ with torch.no_grad():
1201
+ rw = F.softmax(model.enc_router_logits, dim=-1).cpu().numpy()
1202
+ for p in range(model.n_cross_pairs):
1203
+ top_layers = np.argsort(rw[p])[::-1][:4].tolist()
1204
+ print(f" • Cross-KV Pair {p} top encoder mixture weights: {top_layers} ({[round(float(rw[p][k]), 3) for k in top_layers]})")
1205
+
1206
+ print(f"🎉 Stage '{args.stage}' 100% COMPLETE ({rows_seen:,} rows trained across {step:,} steps)!")
1207
+
1208
+ if world_size > 1:
1209
+ dist.barrier()
1210
+ dist.destroy_process_group()
1211
+
1212
+
1213
+ if __name__ == "__main__":
1214
+ main()
checkpoint-stage2-step-6793/training_state.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4508a7fcc2c1bf57e6d1ed1bfe89eba8826af3cb740665f9a8d0a6bf0ca2a22b
3
+ size 760180633
checkpoint-stage2-step-6793/trainingcelloutputwitheval.txt ADDED
The diff for this file is too large to render. See raw diff
 
config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "MyTranslationForConditionalGeneration"
4
+ ],
5
+ "auto_map": {
6
+ "AutoConfig": "configuration_mytranslation.MyTranslationConfig",
7
+ "AutoModelForSeq2SeqLM": "modeling_mytranslation.MyTranslationForConditionalGeneration"
8
+ },
9
+ "model_type": "mytranslation",
10
+ "is_encoder_decoder": true,
11
+ "tie_word_embeddings": true,
12
+ "vocab_size": 65536,
13
+ "d_model": 1024,
14
+ "n_heads": 16,
15
+ "n_kv_heads": 4,
16
+ "enc_layers": 12,
17
+ "dec_layers": 6,
18
+ "ffn_dim": 3072,
19
+ "max_seq_len": 2048,
20
+ "rope_theta": 50000.0,
21
+ "unk_token_id": 0,
22
+ "bos_token_id": 1,
23
+ "eos_token_id": 2,
24
+ "pad_token_id": 3,
25
+ "stage1_final_val_nll": 1.4316,
26
+ "stage1_final_val_ppl": 4.19,
27
+ "val_nll": 1.164,
28
+ "val_ppl": 3.2,
29
+ "en2my_val_nll": 1.4829,
30
+ "my2en_val_nll": 0.8901,
31
+ "torch_dtype": "bfloat16"
32
+ }
configuration_mytranslation.py ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from transformers import PretrainedConfig
2
+
3
+ class MyTranslationConfig(PretrainedConfig):
4
+ model_type = "mytranslation"
5
+
6
+ def __init__(
7
+ self,
8
+ vocab_size=65536,
9
+ d_model=1024,
10
+ n_heads=16,
11
+ n_kv_heads=4,
12
+ enc_layers=12,
13
+ dec_layers=6,
14
+ ffn_dim=3072,
15
+ max_seq_len=2048,
16
+ rope_theta=50000.0,
17
+ unk_token_id=0,
18
+ bos_token_id=1,
19
+ eos_token_id=2,
20
+ pad_token_id=3,
21
+ tie_word_embeddings=True,
22
+ is_encoder_decoder=True,
23
+ **kwargs,
24
+ ):
25
+ self.vocab_size = vocab_size
26
+ self.d_model = d_model
27
+ self.n_heads = n_heads
28
+ self.n_kv_heads = n_kv_heads
29
+ self.enc_layers = enc_layers
30
+ self.dec_layers = dec_layers
31
+ self.ffn_dim = ffn_dim
32
+ self.max_seq_len = max_seq_len
33
+ self.rope_theta = rope_theta
34
+ super().__init__(
35
+ unk_token_id=unk_token_id,
36
+ bos_token_id=bos_token_id,
37
+ eos_token_id=eos_token_id,
38
+ pad_token_id=pad_token_id,
39
+ tie_word_embeddings=tie_word_embeddings,
40
+ is_encoder_decoder=is_encoder_decoder,
41
+ **kwargs,
42
+ )
generation_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 1,
3
+ "eos_token_id": 2,
4
+ "pad_token_id": 3,
5
+ "decoder_start_token_id": 1,
6
+ "max_new_tokens": 192,
7
+ "num_beams": 4,
8
+ "length_penalty": 0.85,
9
+ "repetition_penalty": 1.12
10
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:58cf818fcf629f0c9ab4d5981a7a3ceadfbcb3a28af5141e8859ebd3102b4da3
3
+ size 597551858
modeling_mytranslation.py ADDED
@@ -0,0 +1,372 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import math
2
+ import torch
3
+ import torch.nn as nn
4
+ import torch.nn.functional as F
5
+ import transformers
6
+ from transformers import PreTrainedModel
7
+ from transformers.modeling_outputs import Seq2SeqLMOutput
8
+ from .configuration_mytranslation import MyTranslationConfig
9
+
10
+ PAD_ID = 3
11
+ BOS_ID = 1
12
+ EOS_ID = 2
13
+ TOK_2MY, TOK_2EN, TOK_FORMAL, TOK_SPOKEN = 4, 5, 6, 7
14
+ _IS_TF5 = int(transformers.__version__.split(".")[0]) >= 5
15
+
16
+ def _sdpa(q, k, v, attn_mask=None, is_causal=False):
17
+ if q.size(1) != k.size(1):
18
+ try:
19
+ return F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask, is_causal=is_causal, enable_gqa=True)
20
+ except TypeError:
21
+ rep = q.size(1) // k.size(1)
22
+ k = k.repeat_interleave(rep, dim=1)
23
+ v = v.repeat_interleave(rep, dim=1)
24
+ return F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask, is_causal=is_causal)
25
+
26
+ class RMSNorm(nn.Module):
27
+ def __init__(self, dim: int, eps: float = 1e-6):
28
+ super().__init__()
29
+ self.dim = (dim,)
30
+ self.eps = eps
31
+ self.weight = nn.Parameter(torch.ones(dim))
32
+
33
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
34
+ if hasattr(F, "rms_norm"):
35
+ return F.rms_norm(x, self.dim, self.weight, self.eps)
36
+ xf = x.float()
37
+ y = xf * torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + self.eps)
38
+ return (y * self.weight.float()).to(dtype=x.dtype)
39
+
40
+ class RotaryEmbedding(nn.Module):
41
+ def __init__(self, dim: int, max_seq_len: int = 2048, theta: float = 50000.0):
42
+ super().__init__()
43
+ inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))
44
+ t = torch.arange(max_seq_len, dtype=torch.float32)
45
+ freqs = torch.outer(t, inv_freq)
46
+ self.register_buffer("cos_cached", freqs.cos()[None, None, :, :], persistent=False)
47
+ self.register_buffer("sin_cached", freqs.sin()[None, None, :, :], persistent=False)
48
+
49
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
50
+ L = x.size(2)
51
+ cos = self.cos_cached[:, :, :L, :].to(dtype=x.dtype)
52
+ sin = self.sin_cached[:, :, :L, :].to(dtype=x.dtype)
53
+ x1, x2 = x[..., ::2], x[..., 1::2]
54
+ return torch.cat((x1 * cos - x2 * sin, x1 * sin + x2 * cos), dim=-1)
55
+
56
+ class SwiGLU(nn.Module):
57
+ def __init__(self, d_model: int, ffn_dim: int):
58
+ super().__init__()
59
+ self.gate_up = nn.Linear(d_model, 2 * ffn_dim, bias=False)
60
+ self.down = nn.Linear(ffn_dim, d_model, bias=False)
61
+
62
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
63
+ gate, up = self.gate_up(x).chunk(2, dim=-1)
64
+ return self.down(F.silu(gate) * up)
65
+
66
+ class GatedGQASelfAttention(nn.Module):
67
+ def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, is_causal: bool = False, has_v_res: bool = False):
68
+ super().__init__()
69
+ self.n_heads = n_heads
70
+ self.n_kv_heads = n_kv_heads
71
+ self.head_dim = d_model // n_heads
72
+ self.is_causal = is_causal
73
+ self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
74
+ self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * self.head_dim, bias=False)
75
+ self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
76
+ self.out = nn.Linear(d_model, d_model, bias=False)
77
+ self.q_norm = RMSNorm(self.head_dim)
78
+ self.k_norm = RMSNorm(self.head_dim)
79
+ self.v_res_logit = nn.Parameter(torch.tensor(-2.0)) if has_v_res else None
80
+
81
+ def forward(self, x: torch.Tensor, rope: RotaryEmbedding, attn_mask: torch.Tensor = None, v0: torch.Tensor = None):
82
+ B, L, D = x.shape
83
+ q = self.q_norm(self.q_proj(x).view(B, L, self.n_heads, self.head_dim).transpose(1, 2))
84
+ k, v = self.kv_proj(x).chunk(2, dim=-1)
85
+ k = self.k_norm(k.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2))
86
+ v = v.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2)
87
+ v_raw = v
88
+ if self.v_res_logit is not None and v0 is not None:
89
+ alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
90
+ v = (1.0 - alpha) * v + alpha * v0
91
+ q, k = rope(q), rope(k)
92
+ y = _sdpa(q, k, v, attn_mask=None if self.is_causal else attn_mask, is_causal=self.is_causal)
93
+ gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
94
+ y = (y * gate).transpose(1, 2).contiguous().view(B, L, D)
95
+ return self.out(y), v_raw
96
+
97
+ def step_kv(self, x_step, cos_p, sin_p, pos, k_cache, v_cache, v0=None):
98
+ B, _, D = x_step.shape
99
+ q = self.q_norm(self.q_proj(x_step).view(B, 1, self.n_heads, self.head_dim).transpose(1, 2))
100
+ k, v = self.kv_proj(x_step).chunk(2, dim=-1)
101
+ k = self.k_norm(k.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2))
102
+ v = v.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2)
103
+ v_raw = v
104
+ if self.v_res_logit is not None and v0 is not None:
105
+ alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
106
+ v = (1.0 - alpha) * v + alpha * v0
107
+ q1, q2 = q[..., ::2], q[..., 1::2]
108
+ k1, k2 = k[..., ::2], k[..., 1::2]
109
+ q = torch.cat((q1 * cos_p - q2 * sin_p, q1 * sin_p + q2 * cos_p), dim=-1)
110
+ k = torch.cat((k1 * cos_p - k2 * sin_p, k1 * sin_p + k2 * cos_p), dim=-1)
111
+ k_cache[:, :, pos : pos + 1, :] = k
112
+ v_cache[:, :, pos : pos + 1, :] = v
113
+ y = _sdpa(q, k_cache[:, :, : pos + 1, :], v_cache[:, :, : pos + 1, :], is_causal=False)
114
+ gate = torch.sigmoid(self.gate_proj(x_step)).transpose(1, 2).unsqueeze(-1)
115
+ y = (y * gate).transpose(1, 2).contiguous().view(B, 1, D)
116
+ return self.out(y), v_raw
117
+
118
+ class SharedCrossKVProj(nn.Module):
119
+ def __init__(self, d_model: int, n_kv_heads: int, head_dim: int):
120
+ super().__init__()
121
+ self.n_kv_heads = n_kv_heads
122
+ self.head_dim = head_dim
123
+ self.norm = RMSNorm(d_model)
124
+ self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * head_dim, bias=False)
125
+ self.k_norm = RMSNorm(head_dim)
126
+
127
+ def forward(self, enc_mix: torch.Tensor):
128
+ B, S, _ = enc_mix.shape
129
+ k, v = self.kv_proj(self.norm(enc_mix)).chunk(2, dim=-1)
130
+ k = self.k_norm(k.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2))
131
+ v = v.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2)
132
+ return k, v
133
+
134
+ class GatedGQACrossAttention(nn.Module):
135
+ def __init__(self, d_model: int, n_heads: int):
136
+ super().__init__()
137
+ self.n_heads = n_heads
138
+ self.head_dim = d_model // n_heads
139
+ self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
140
+ self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
141
+ self.out = nn.Linear(d_model, d_model, bias=False)
142
+ self.q_norm = RMSNorm(self.head_dim)
143
+
144
+ def forward(self, x: torch.Tensor, k_cross: torch.Tensor, v_cross: torch.Tensor, enc_mask: torch.Tensor):
145
+ B, T, D = x.shape
146
+ q = self.q_norm(self.q_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2))
147
+ y = _sdpa(q, k_cross, v_cross, attn_mask=enc_mask, is_causal=False)
148
+ gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
149
+ y = (y * gate).transpose(1, 2).contiguous().view(B, T, D)
150
+ return self.out(y)
151
+
152
+ class EncoderLayer(nn.Module):
153
+ def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
154
+ super().__init__()
155
+ self.norm1 = RMSNorm(d_model)
156
+ self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=False, has_v_res=has_v_res)
157
+ self.norm2 = RMSNorm(d_model)
158
+ self.mlp = SwiGLU(d_model, ffn_dim)
159
+
160
+ def forward(self, x, rope, enc_mask, v0=None):
161
+ attn_out, v_raw = self.self_attn(self.norm1(x), rope, enc_mask, v0=v0)
162
+ x = x + attn_out
163
+ x = x + self.mlp(self.norm2(x))
164
+ return x, v_raw
165
+
166
+ class DecoderLayer(nn.Module):
167
+ def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
168
+ super().__init__()
169
+ self.norm1 = RMSNorm(d_model)
170
+ self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=True, has_v_res=has_v_res)
171
+ self.norm2 = RMSNorm(d_model)
172
+ self.cross_attn = GatedGQACrossAttention(d_model, n_heads)
173
+ self.norm3 = RMSNorm(d_model)
174
+ self.mlp = SwiGLU(d_model, ffn_dim)
175
+
176
+ def forward(self, x, k_cross, v_cross, rope, enc_mask, v0=None):
177
+ attn_out, v_raw = self.self_attn(self.norm1(x), rope, None, v0=v0)
178
+ x = x + attn_out
179
+ x = x + self.cross_attn(self.norm2(x), k_cross, v_cross, enc_mask)
180
+ x = x + self.mlp(self.norm3(x))
181
+ return x, v_raw
182
+
183
+ def step_kv(self, x_step, cos_p, sin_p, pos, k_self, v_self, k_cross, v_cross, enc_mask, v0=None):
184
+ attn_out, v_raw = self.self_attn.step_kv(self.norm1(x_step), cos_p, sin_p, pos, k_self, v_self, v0=v0)
185
+ x_step = x_step + attn_out
186
+ x_step = x_step + self.cross_attn(self.norm2(x_step), k_cross, v_cross, enc_mask)
187
+ x_step = x_step + self.mlp(self.norm3(x_step))
188
+ return x_step, v_raw
189
+
190
+ class MyTranslationForConditionalGeneration(PreTrainedModel):
191
+ config_class = MyTranslationConfig
192
+ base_model_prefix = ""
193
+ _tied_weights_keys = {"lm_head.weight": "embed_tokens.weight"} if hasattr(PreTrainedModel, "get_expanded_tied_weights_keys") else ["lm_head.weight"]
194
+
195
+ def __init__(self, config: MyTranslationConfig):
196
+ super().__init__(config)
197
+ self.d_model = config.d_model
198
+ self.n_heads = config.n_heads
199
+ self.n_kv_heads = config.n_kv_heads
200
+ self.head_dim = config.d_model // config.n_heads
201
+ self.enc_layers = config.enc_layers
202
+ self.dec_layers = config.dec_layers
203
+ self.n_cross_pairs = config.dec_layers // 2
204
+ self.embed_scale = math.sqrt(config.d_model)
205
+
206
+ self.embed_tokens = nn.Embedding(config.vocab_size, config.d_model, padding_idx=config.pad_token_id)
207
+ self.rope = RotaryEmbedding(self.head_dim, max_seq_len=config.max_seq_len, theta=config.rope_theta)
208
+ self.encoder = nn.ModuleList([
209
+ EncoderLayer(config.d_model, config.n_heads, config.n_kv_heads, config.ffn_dim, has_v_res=(i > 0))
210
+ for i in range(config.enc_layers)
211
+ ])
212
+ self.enc_dwa_logits = nn.ParameterList([
213
+ nn.Parameter(torch.tensor([-4.0] * i + [0.0], dtype=torch.float32))
214
+ for i in range(1, config.enc_layers)
215
+ ])
216
+ self.enc_router_logits = nn.Parameter(torch.zeros(self.n_cross_pairs, config.enc_layers))
217
+ self.cross_kv_projs = nn.ModuleList([
218
+ SharedCrossKVProj(config.d_model, config.n_kv_heads, self.head_dim)
219
+ for _ in range(self.n_cross_pairs)
220
+ ])
221
+ self.decoder = nn.ModuleList([
222
+ DecoderLayer(config.d_model, config.n_heads, config.n_kv_heads, config.ffn_dim, has_v_res=(i > 0))
223
+ for i in range(config.dec_layers)
224
+ ])
225
+ self.dec_norm = RMSNorm(config.d_model)
226
+ self.lm_head = nn.Linear(config.d_model, config.vocab_size, bias=False)
227
+ self.post_init()
228
+ self.tie_weights()
229
+
230
+ def get_input_embeddings(self):
231
+ return self.embed_tokens
232
+
233
+ def set_input_embeddings(self, value):
234
+ self.embed_tokens = value
235
+
236
+ def get_output_embeddings(self):
237
+ return self.lm_head
238
+
239
+ def set_output_embeddings(self, new_embeddings):
240
+ self.lm_head = new_embeddings
241
+
242
+ def tie_weights(self, **kwargs):
243
+ self.lm_head.weight = self.embed_tokens.weight
244
+
245
+ def encode_and_route(self, src_ids: torch.Tensor):
246
+ enc_mask = (src_ids != self.config.pad_token_id).unsqueeze(1).unsqueeze(2)
247
+ x = self.embed_tokens(src_ids) * self.embed_scale
248
+ raw_states = [x]
249
+ enc_outputs = []
250
+ v0 = None
251
+ for i, layer in enumerate(self.encoder):
252
+ if i > 0:
253
+ w_dwa = F.softmax(self.enc_dwa_logits[i - 1], dim=0).to(dtype=x.dtype)
254
+ x_in = torch.einsum("j,jbsd->bsd", w_dwa, torch.stack(raw_states, dim=0))
255
+ else:
256
+ x_in = x
257
+ x_out, v_raw = layer(x_in, self.rope, enc_mask, v0=v0)
258
+ if i == 0:
259
+ v0 = v_raw
260
+ raw_states.append(x_out)
261
+ enc_outputs.append(x_out)
262
+ stacked_enc = torch.stack(enc_outputs, dim=0)
263
+ router_w = F.softmax(self.enc_router_logits, dim=-1).to(dtype=stacked_enc.dtype)
264
+ cross_kv = [
265
+ self.cross_kv_projs[p](torch.einsum("k,kbsd->bsd", router_w[p], stacked_enc))
266
+ for p in range(self.n_cross_pairs)
267
+ ]
268
+ return cross_kv, enc_mask
269
+
270
+ def decode_seq(self, tgt_in: torch.Tensor, cross_kv: list, enc_mask: torch.Tensor):
271
+ x = self.embed_tokens(tgt_in) * self.embed_scale
272
+ v0 = None
273
+ for i, layer in enumerate(self.decoder):
274
+ kc, vc = cross_kv[i // 2]
275
+ x, v_raw = layer(x, kc, vc, self.rope, enc_mask, v0=v0)
276
+ if i == 0:
277
+ v0 = v_raw
278
+ return self.dec_norm(x)
279
+
280
+ def forward(self, input_ids=None, decoder_input_ids=None, labels=None, **kwargs):
281
+ if decoder_input_ids is None:
282
+ decoder_input_ids = torch.full((input_ids.size(0), 1), BOS_ID, dtype=torch.long, device=input_ids.device)
283
+ cross_kv, enc_mask = self.encode_and_route(input_ids)
284
+ h_dec = self.decode_seq(decoder_input_ids, cross_kv, enc_mask)
285
+ logits = self.lm_head(h_dec)
286
+ loss = None
287
+ if labels is not None:
288
+ loss = F.cross_entropy(logits.view(-1, logits.size(-1)), labels.view(-1), ignore_index=-100)
289
+ return Seq2SeqLMOutput(loss=loss, logits=logits)
290
+
291
+ @torch.inference_mode()
292
+ def translate(
293
+ self,
294
+ text: str,
295
+ sp_proc,
296
+ direction: str = "en2my",
297
+ style: str = "auto",
298
+ num_beams: int = 4,
299
+ max_new_tokens: int = 192,
300
+ length_penalty: float = 0.85,
301
+ rep_penalty: float = 1.12,
302
+ ) -> str:
303
+ self.eval()
304
+ dev = self.embed_tokens.weight.device
305
+ dt = self.embed_tokens.weight.dtype
306
+ pref = [TOK_2MY if direction == "en2my" else TOK_2EN]
307
+ if style == "formal":
308
+ pref.append(TOK_FORMAL)
309
+ elif style == "spoken":
310
+ pref.append(TOK_SPOKEN)
311
+ src_ids = pref + sp_proc.encode_as_ids(text.strip())[: self.config.max_seq_len - 8] + [EOS_ID]
312
+ src_t = torch.tensor([src_ids], dtype=torch.long, device=dev)
313
+ cross_kv, enc_mask = self.encode_and_route(src_t)
314
+
315
+ if num_beams <= 1:
316
+ k_cache = torch.empty((self.dec_layers, 1, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=dt, device=dev)
317
+ v_cache = torch.empty((self.dec_layers, 1, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=dt, device=dev)
318
+ cur_tok = torch.tensor([[BOS_ID]], dtype=torch.long, device=dev)
319
+ cos_all = self.rope.cos_cached.to(dtype=dt, device=dev)
320
+ sin_all = self.rope.sin_cached.to(dtype=dt, device=dev)
321
+ ys = []
322
+ for pos in range(max_new_tokens):
323
+ x_dec = self.embed_tokens(cur_tok) * self.embed_scale
324
+ v0 = None
325
+ cos_p, sin_p = cos_all[:, :, pos : pos + 1, :], sin_all[:, :, pos : pos + 1, :]
326
+ for l_idx, layer in enumerate(self.decoder):
327
+ kc, vc = cross_kv[l_idx // 2]
328
+ x_dec, v_raw = layer.step_kv(x_dec, cos_p, sin_p, pos, k_cache[l_idx], v_cache[l_idx], kc, vc, enc_mask, v0=v0)
329
+ if l_idx == 0:
330
+ v0 = v_raw
331
+ logits = self.lm_head(self.dec_norm(x_dec)[:, 0, :]).float()[0]
332
+ logits[[0, 1, 3, 4, 5, 6, 7, 8, 9, 10]] = -1e9
333
+ if rep_penalty != 1.0 and ys:
334
+ seen = torch.tensor(list(set(ys)), dtype=torch.long, device=dev)
335
+ sl = logits[seen]
336
+ logits[seen] = torch.where(sl > 0, sl / rep_penalty, sl * rep_penalty)
337
+ nxt = int(logits.argmax().item())
338
+ if nxt == EOS_ID:
339
+ break
340
+ ys.append(nxt)
341
+ cur_tok = torch.tensor([[nxt]], dtype=torch.long, device=dev)
342
+ return sp_proc.decode_ids([x for x in ys if x >= 11]).strip()
343
+
344
+ beams = [(0.0, [BOS_ID], False)]
345
+ for _ in range(max_new_tokens):
346
+ active = [(sc, seq) for sc, seq, done in beams if not done]
347
+ if not active:
348
+ break
349
+ B_act = len(active)
350
+ tgt_t = torch.tensor([seq for _, seq in active], dtype=torch.long, device=dev)
351
+ ckv_exp = [(kc.expand(B_act, -1, -1, -1), vc.expand(B_act, -1, -1, -1)) for kc, vc in cross_kv]
352
+ h_dec = self.decode_seq(tgt_t, ckv_exp, enc_mask.expand(B_act, -1, -1, -1))
353
+ logits = self.lm_head(h_dec[:, -1, :]).float()
354
+ logits[:, [0, 1, 3, 4, 5, 6, 7, 8, 9, 10]] = -1e9
355
+ for b_i, (_, seq) in enumerate(active):
356
+ if rep_penalty != 1.0 and len(seq) > 1:
357
+ seen = torch.tensor(list(set(seq[1:])), dtype=torch.long, device=dev)
358
+ sl = logits[b_i, seen]
359
+ logits[b_i, seen] = torch.where(sl > 0, sl / rep_penalty, sl * rep_penalty)
360
+ lp = F.log_softmax(logits, dim=-1)
361
+ topk_lp, topk_ids = torch.topk(lp, num_beams * 2, dim=-1)
362
+ cands = [(sc, seq, True) for sc, seq, done in beams if done]
363
+ for b_i, (sc, seq) in enumerate(active):
364
+ for k in range(num_beams * 2):
365
+ tid = int(topk_ids[b_i, k].item())
366
+ n_sc = sc + float(topk_lp[b_i, k].item())
367
+ cands.append((n_sc, seq if tid == EOS_ID else seq + [tid], tid == EOS_ID))
368
+ cands.sort(key=lambda x: x[0] / (max(1, len(x[1]) - 1) ** length_penalty), reverse=True)
369
+ beams = cands[:num_beams]
370
+ if all(done for _, _, done in beams):
371
+ break
372
+ return sp_proc.decode_ids([x for x in beams[0][1][1:] if x >= 11]).strip()
special_tokens_map.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "unk_token": "<unk>",
3
+ "bos_token": "<s>",
4
+ "eos_token": "</s>",
5
+ "pad_token": "<pad>",
6
+ "additional_special_tokens": [
7
+ "<2my>",
8
+ "<2en>",
9
+ "<formal>",
10
+ "<spoken>",
11
+ "<bt>",
12
+ "<denoise>",
13
+ "<mask>"
14
+ ]
15
+ }
tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9aaeeaf36272c5c158d11ed0987ca5930686f1e02713292558003909bb147bc1
3
+ size 1840017
tokenizer_config.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_max_length": 1024,
3
+ "unk_token": "<unk>",
4
+ "bos_token": "<s>",
5
+ "eos_token": "</s>",
6
+ "pad_token": "<pad>",
7
+ "additional_special_tokens": [
8
+ "<2my>",
9
+ "<2en>",
10
+ "<formal>",
11
+ "<spoken>",
12
+ "<bt>",
13
+ "<denoise>",
14
+ "<mask>"
15
+ ],
16
+ "special_tokens_map": {
17
+ "2my": 4,
18
+ "2en": 5,
19
+ "formal": 6,
20
+ "spoken": 7,
21
+ "bt": 8,
22
+ "denoise": 9,
23
+ "mask": 10
24
+ }
25
+ }
training_codes.txt ADDED
@@ -0,0 +1,1214 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"
3
+
4
+ import argparse, json, math, time
5
+ from datetime import timedelta
6
+ import numpy as np
7
+ import sentencepiece as spm
8
+ import torch
9
+ import torch.distributed as dist
10
+ import torch.nn as nn
11
+ import torch.nn.functional as F
12
+ from torch.utils.checkpoint import checkpoint as grad_ckpt
13
+ from torch.utils.data import Dataset, DataLoader
14
+
15
+ torch.backends.cuda.matmul.allow_tf32 = True
16
+ torch.backends.cudnn.allow_tf32 = True
17
+ torch._dynamo.config.cache_size_limit = 64
18
+
19
+ DATA_DIR = "/workspace/mytrans_dl/tokenized_65k"
20
+ CKPT_DIR = "/workspace/ckpt_300m"
21
+
22
+ VOCAB_SIZE = 65536
23
+ UNK_ID, BOS_ID, EOS_ID, PAD_ID = 0, 1, 2, 3
24
+ TOK_2MY, TOK_2EN, TOK_FORMAL, TOK_SPOKEN, TOK_BT, TOK_DENOISE, TOK_MASK = 4, 5, 6, 7, 8, 9, 10
25
+ FLAG_BT, FLAG_FORMAL, FLAG_SPOKEN = 1, 2, 4
26
+
27
+ BULK_SUBSETS = {
28
+ "MyanmarTranslator_works/myanmar_spoken_corpus",
29
+ "MyanmarTranslator_works/myanmar-cc100-dataset",
30
+ "MyanmarTranslator_works/myanmar-written-corpus",
31
+ }
32
+
33
+
34
+ def load_and_verify_contracts(data_dir: str):
35
+ with open(os.path.join(data_dir, "meta.json"), "r") as f:
36
+ meta = json.load(f)
37
+ sp = spm.SentencePieceProcessor()
38
+ sp.load(os.path.join(data_dir, "tokenizer.model"))
39
+
40
+ assert sp.get_piece_size() == VOCAB_SIZE
41
+ assert sp.unk_id() == UNK_ID and sp.bos_id() == BOS_ID and sp.eos_id() == EOS_ID and sp.pad_id() == PAD_ID
42
+ expected = {4: "<2my>", 5: "<2en>", 6: "<formal>", 7: "<spoken>", 8: "<bt>", 9: "<denoise>", 10: "<mask>"}
43
+ for tid, piece in expected.items():
44
+ assert sp.id_to_piece(tid) == piece, f"Token {tid} mismatch"
45
+ if "special_tokens" in meta:
46
+ st = meta["special_tokens"]
47
+ assert st["2my"] == TOK_2MY and st["2en"] == TOK_2EN
48
+ assert st["formal"] == TOK_FORMAL and st["spoken"] == TOK_SPOKEN
49
+ assert st["bt"] == TOK_BT and st["denoise"] == TOK_DENOISE and st["mask"] == TOK_MASK
50
+ return meta, sp
51
+
52
+
53
+ # ==============================================================================
54
+ # 1. MyTranslation-300M Architecture
55
+ # ==============================================================================
56
+ class RMSNorm(nn.Module):
57
+ def __init__(self, dim: int, eps: float = 1e-6):
58
+ super().__init__()
59
+ self.dim = (dim,)
60
+ self.eps = eps
61
+ self.weight = nn.Parameter(torch.ones(dim))
62
+
63
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
64
+ return F.rms_norm(x, self.dim, self.weight, self.eps)
65
+
66
+
67
+ class RotaryEmbedding(nn.Module):
68
+ def __init__(self, dim: int, max_seq_len: int = 2048, theta: float = 50000.0):
69
+ super().__init__()
70
+ inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))
71
+ t = torch.arange(max_seq_len, dtype=torch.float32)
72
+ freqs = torch.outer(t, inv_freq)
73
+ self.register_buffer("cos_cached", freqs.cos()[None, None, :, :], persistent=False)
74
+ self.register_buffer("sin_cached", freqs.sin()[None, None, :, :], persistent=False)
75
+
76
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
77
+ L = x.size(2)
78
+ cos = self.cos_cached[:, :, :L, :].to(dtype=x.dtype)
79
+ sin = self.sin_cached[:, :, :L, :].to(dtype=x.dtype)
80
+ x1, x2 = x[..., ::2], x[..., 1::2]
81
+ return torch.cat((x1 * cos - x2 * sin, x1 * sin + x2 * cos), dim=-1)
82
+
83
+
84
+ class SwiGLU(nn.Module):
85
+ def __init__(self, d_model: int, ffn_dim: int):
86
+ super().__init__()
87
+ self.gate_up = nn.Linear(d_model, 2 * ffn_dim, bias=False)
88
+ self.down = nn.Linear(ffn_dim, d_model, bias=False)
89
+
90
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
91
+ gate, up = self.gate_up(x).chunk(2, dim=-1)
92
+ return self.down(F.silu(gate) * up)
93
+
94
+
95
+ class GatedGQASelfAttention(nn.Module):
96
+ def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, is_causal: bool = False, has_v_res: bool = False):
97
+ super().__init__()
98
+ self.n_heads = n_heads
99
+ self.n_kv_heads = n_kv_heads
100
+ self.head_dim = d_model // n_heads
101
+ self.is_causal = is_causal
102
+ self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
103
+ self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * self.head_dim, bias=False)
104
+ self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
105
+ self.out = nn.Linear(d_model, d_model, bias=False)
106
+ self.q_norm = RMSNorm(self.head_dim)
107
+ self.k_norm = RMSNorm(self.head_dim)
108
+ self.v_res_logit = nn.Parameter(torch.tensor(-2.0)) if has_v_res else None
109
+
110
+ def forward(self, x: torch.Tensor, rope: RotaryEmbedding, attn_mask: torch.Tensor = None, v0: torch.Tensor = None):
111
+ B, L, D = x.shape
112
+ q = self.q_norm(self.q_proj(x).view(B, L, self.n_heads, self.head_dim).transpose(1, 2))
113
+ k, v = self.kv_proj(x).chunk(2, dim=-1)
114
+ k = self.k_norm(k.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2))
115
+ v = v.view(B, L, self.n_kv_heads, self.head_dim).transpose(1, 2)
116
+ v_raw = v
117
+ if self.v_res_logit is not None and v0 is not None:
118
+ alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
119
+ v = (1.0 - alpha) * v + alpha * v0
120
+ q, k = rope(q), rope(k)
121
+ y = F.scaled_dot_product_attention(
122
+ q, k, v,
123
+ attn_mask=None if self.is_causal else attn_mask,
124
+ is_causal=self.is_causal,
125
+ enable_gqa=True,
126
+ )
127
+ gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
128
+ y = (y * gate).transpose(1, 2).contiguous().view(B, L, D)
129
+ return self.out(y), v_raw
130
+
131
+ def step_kv(self, x_step, cos_p, sin_p, pos, k_cache, v_cache, v0=None):
132
+ B, _, D = x_step.shape
133
+ q = self.q_norm(self.q_proj(x_step).view(B, 1, self.n_heads, self.head_dim).transpose(1, 2))
134
+ k, v = self.kv_proj(x_step).chunk(2, dim=-1)
135
+ k = self.k_norm(k.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2))
136
+ v = v.view(B, 1, self.n_kv_heads, self.head_dim).transpose(1, 2)
137
+ v_raw = v
138
+ if self.v_res_logit is not None and v0 is not None:
139
+ alpha = torch.sigmoid(self.v_res_logit).to(dtype=v.dtype)
140
+ v = (1.0 - alpha) * v + alpha * v0
141
+ q1, q2 = q[..., ::2], q[..., 1::2]
142
+ k1, k2 = k[..., ::2], k[..., 1::2]
143
+ q = torch.cat((q1 * cos_p - q2 * sin_p, q1 * sin_p + q2 * cos_p), dim=-1)
144
+ k = torch.cat((k1 * cos_p - k2 * sin_p, k1 * sin_p + k2 * cos_p), dim=-1)
145
+ k_cache[:, :, pos : pos + 1, :] = k
146
+ v_cache[:, :, pos : pos + 1, :] = v
147
+ y = F.scaled_dot_product_attention(
148
+ q, k_cache[:, :, : pos + 1, :], v_cache[:, :, : pos + 1, :],
149
+ is_causal=False, enable_gqa=True,
150
+ )
151
+ gate = torch.sigmoid(self.gate_proj(x_step)).transpose(1, 2).unsqueeze(-1)
152
+ y = (y * gate).transpose(1, 2).contiguous().view(B, 1, D)
153
+ return self.out(y), v_raw
154
+
155
+
156
+ class SharedCrossKVProj(nn.Module):
157
+ def __init__(self, d_model: int, n_kv_heads: int, head_dim: int):
158
+ super().__init__()
159
+ self.n_kv_heads = n_kv_heads
160
+ self.head_dim = head_dim
161
+ self.norm = RMSNorm(d_model)
162
+ self.kv_proj = nn.Linear(d_model, 2 * n_kv_heads * head_dim, bias=False)
163
+ self.k_norm = RMSNorm(head_dim)
164
+
165
+ def forward(self, enc_mix: torch.Tensor):
166
+ B, S, _ = enc_mix.shape
167
+ k, v = self.kv_proj(self.norm(enc_mix)).chunk(2, dim=-1)
168
+ k = self.k_norm(k.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2))
169
+ v = v.view(B, S, self.n_kv_heads, self.head_dim).transpose(1, 2)
170
+ return k, v
171
+
172
+
173
+ class GatedGQACrossAttention(nn.Module):
174
+ def __init__(self, d_model: int, n_heads: int):
175
+ super().__init__()
176
+ self.n_heads = n_heads
177
+ self.head_dim = d_model // n_heads
178
+ self.q_proj = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
179
+ self.gate_proj = nn.Linear(d_model, n_heads, bias=False)
180
+ self.out = nn.Linear(d_model, d_model, bias=False)
181
+ self.q_norm = RMSNorm(self.head_dim)
182
+
183
+ def forward(self, x: torch.Tensor, k_cross: torch.Tensor, v_cross: torch.Tensor, enc_mask: torch.Tensor):
184
+ B, T, D = x.shape
185
+ q = self.q_norm(self.q_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2))
186
+ y = F.scaled_dot_product_attention(q, k_cross, v_cross, attn_mask=enc_mask, is_causal=False, enable_gqa=True)
187
+ gate = torch.sigmoid(self.gate_proj(x)).transpose(1, 2).unsqueeze(-1)
188
+ y = (y * gate).transpose(1, 2).contiguous().view(B, T, D)
189
+ return self.out(y)
190
+
191
+
192
+ class EncoderLayer(nn.Module):
193
+ def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
194
+ super().__init__()
195
+ self.norm1 = RMSNorm(d_model)
196
+ self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=False, has_v_res=has_v_res)
197
+ self.norm2 = RMSNorm(d_model)
198
+ self.mlp = SwiGLU(d_model, ffn_dim)
199
+
200
+ def forward(self, x, rope, enc_mask, v0=None):
201
+ attn_out, v_raw = self.self_attn(self.norm1(x), rope, enc_mask, v0=v0)
202
+ x = x + attn_out
203
+ x = x + self.mlp(self.norm2(x))
204
+ return x, v_raw
205
+
206
+
207
+ class DecoderLayer(nn.Module):
208
+ def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int, has_v_res: bool):
209
+ super().__init__()
210
+ self.norm1 = RMSNorm(d_model)
211
+ self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=True, has_v_res=has_v_res)
212
+ self.norm2 = RMSNorm(d_model)
213
+ self.cross_attn = GatedGQACrossAttention(d_model, n_heads)
214
+ self.norm3 = RMSNorm(d_model)
215
+ self.mlp = SwiGLU(d_model, ffn_dim)
216
+
217
+ def forward(self, x, k_cross, v_cross, rope, enc_mask, v0=None):
218
+ attn_out, v_raw = self.self_attn(self.norm1(x), rope, None, v0=v0)
219
+ x = x + attn_out
220
+ x = x + self.cross_attn(self.norm2(x), k_cross, v_cross, enc_mask)
221
+ x = x + self.mlp(self.norm3(x))
222
+ return x, v_raw
223
+
224
+ def step_kv(self, x_step, cos_p, sin_p, pos, k_self, v_self, k_cross, v_cross, enc_mask, v0=None):
225
+ attn_out, v_raw = self.self_attn.step_kv(self.norm1(x_step), cos_p, sin_p, pos, k_self, v_self, v0=v0)
226
+ x_step = x_step + attn_out
227
+ x_step = x_step + self.cross_attn(self.norm2(x_step), k_cross, v_cross, enc_mask)
228
+ x_step = x_step + self.mlp(self.norm3(x_step))
229
+ return x_step, v_raw
230
+
231
+
232
+ class MTPHeadBlock(nn.Module):
233
+ def __init__(self, d_model: int, n_heads: int, n_kv_heads: int, ffn_dim: int):
234
+ super().__init__()
235
+ self.norm_h = RMSNorm(d_model)
236
+ self.norm_e = RMSNorm(d_model)
237
+ self.proj = nn.Linear(2 * d_model, d_model, bias=False)
238
+ self.norm1 = RMSNorm(d_model)
239
+ self.self_attn = GatedGQASelfAttention(d_model, n_heads, n_kv_heads, is_causal=True, has_v_res=False)
240
+ self.norm2 = RMSNorm(d_model)
241
+ self.mlp = SwiGLU(d_model, ffn_dim)
242
+ self.out_norm = RMSNorm(d_model)
243
+
244
+ def forward(self, h_main: torch.Tensor, next_tok_emb: torch.Tensor, rope: RotaryEmbedding):
245
+ x = self.proj(torch.cat([self.norm_h(h_main), self.norm_e(next_tok_emb)], dim=-1))
246
+ attn_out, _ = self.self_attn(self.norm1(x), rope, None)
247
+ x = x + attn_out
248
+ x = x + self.mlp(self.norm2(x))
249
+ return self.out_norm(x)
250
+
251
+
252
+ class MyTranslation300M(nn.Module):
253
+ def __init__(
254
+ self,
255
+ vocab_size=65536,
256
+ d_model=1024,
257
+ n_heads=16,
258
+ n_kv_heads=4,
259
+ enc_layers=12,
260
+ dec_layers=6,
261
+ ffn_dim=3072,
262
+ max_seq_len=2048,
263
+ ):
264
+ super().__init__()
265
+ assert d_model % n_heads == 0 and n_heads % n_kv_heads == 0 and dec_layers % 2 == 0
266
+ self.config = dict(
267
+ vocab_size=vocab_size, d_model=d_model, n_heads=n_heads, n_kv_heads=n_kv_heads,
268
+ enc_layers=enc_layers, dec_layers=dec_layers, ffn_dim=ffn_dim, max_seq_len=max_seq_len,
269
+ )
270
+ self.d_model = d_model
271
+ self.n_heads = n_heads
272
+ self.n_kv_heads = n_kv_heads
273
+ self.head_dim = d_model // n_heads
274
+ self.enc_layers = enc_layers
275
+ self.dec_layers = dec_layers
276
+ self.n_cross_pairs = dec_layers // 2
277
+ self.embed_scale = math.sqrt(d_model)
278
+
279
+ self.embed_tokens = nn.Embedding(vocab_size, d_model, padding_idx=PAD_ID)
280
+ self.rope = RotaryEmbedding(self.head_dim, max_seq_len=max_seq_len, theta=50000.0)
281
+
282
+ self.encoder = nn.ModuleList([
283
+ EncoderLayer(d_model, n_heads, n_kv_heads, ffn_dim, has_v_res=(i > 0))
284
+ for i in range(enc_layers)
285
+ ])
286
+ self.enc_dwa_logits = nn.ParameterList([
287
+ nn.Parameter(torch.tensor([-4.0] * i + [0.0], dtype=torch.float32))
288
+ for i in range(1, enc_layers)
289
+ ])
290
+
291
+ self.enc_router_logits = nn.Parameter(torch.zeros(self.n_cross_pairs, enc_layers))
292
+ with torch.no_grad():
293
+ for p in range(self.n_cross_pairs):
294
+ center = (p + 0.5) * (enc_layers / self.n_cross_pairs) - 0.5
295
+ for k in range(enc_layers):
296
+ self.enc_router_logits[p, k] = -0.35 * ((k - center) ** 2)
297
+
298
+ self.cross_kv_projs = nn.ModuleList([
299
+ SharedCrossKVProj(d_model, n_kv_heads, self.head_dim)
300
+ for _ in range(self.n_cross_pairs)
301
+ ])
302
+
303
+ self.decoder = nn.ModuleList([
304
+ DecoderLayer(d_model, n_heads, n_kv_heads, ffn_dim, has_v_res=(i > 0))
305
+ for i in range(dec_layers)
306
+ ])
307
+ self.dec_norm = RMSNorm(d_model)
308
+ self.lm_head = nn.Linear(d_model, vocab_size, bias=False)
309
+ self.lm_head.weight = self.embed_tokens.weight
310
+
311
+ self.mtp_head = MTPHeadBlock(d_model, n_heads, n_kv_heads, ffn_dim)
312
+ self._init_weights(enc_layers + dec_layers)
313
+
314
+ def _init_weights(self, total_layers: int):
315
+ res_std = 0.02 / math.sqrt(2 * total_layers)
316
+ for name, m in self.named_modules():
317
+ if isinstance(m, nn.Linear):
318
+ if name.endswith(".out") or name.endswith(".down"):
319
+ nn.init.normal_(m.weight, mean=0.0, std=res_std)
320
+ elif name.endswith(".gate_proj"):
321
+ nn.init.normal_(m.weight, mean=0.0, std=0.01)
322
+ else:
323
+ nn.init.normal_(m.weight, mean=0.0, std=0.02)
324
+ elif isinstance(m, nn.Embedding):
325
+ nn.init.normal_(m.weight, mean=0.0, std=0.02)
326
+ if m.padding_idx is not None:
327
+ m.weight.data[m.padding_idx].zero_()
328
+
329
+ def compile_blocks(self):
330
+ for i in range(len(self.encoder)):
331
+ self.encoder[i] = torch.compile(self.encoder[i], dynamic=True)
332
+ for i in range(len(self.cross_kv_projs)):
333
+ self.cross_kv_projs[i] = torch.compile(self.cross_kv_projs[i], dynamic=True)
334
+ for i in range(len(self.decoder)):
335
+ self.decoder[i] = torch.compile(self.decoder[i], dynamic=True)
336
+ self.mtp_head = torch.compile(self.mtp_head, dynamic=True)
337
+
338
+ def encode_and_route(self, src_ids: torch.Tensor, ckpt_mode: str = "none"):
339
+ enc_mask = (src_ids != PAD_ID).unsqueeze(1).unsqueeze(2)
340
+ x = self.embed_tokens(src_ids) * self.embed_scale
341
+ raw_states = [x]
342
+ enc_outputs = []
343
+ v0 = None
344
+
345
+ for i, layer in enumerate(self.encoder):
346
+ if i > 0:
347
+ w_dwa = F.softmax(self.enc_dwa_logits[i - 1], dim=0).to(dtype=x.dtype)
348
+ if self.training and ckpt_mode != "none":
349
+ x_in = grad_ckpt(
350
+ lambda w, *st: torch.einsum("j,jbsd->bsd", w, torch.stack(st, dim=0)),
351
+ w_dwa, *raw_states, use_reentrant=False
352
+ )
353
+ else:
354
+ x_in = torch.einsum("j,jbsd->bsd", w_dwa, torch.stack(raw_states, dim=0))
355
+ else:
356
+ x_in = x
357
+
358
+ do_ckpt = self.training and (
359
+ (ckpt_mode == "heavy" and (i % 3 != 0)) or
360
+ (ckpt_mode == "light" and (i in (3, 7, 11)))
361
+ )
362
+ if do_ckpt:
363
+ x_out, v_raw = grad_ckpt(layer, x_in, self.rope, enc_mask, v0, use_reentrant=False)
364
+ else:
365
+ x_out, v_raw = layer(x_in, self.rope, enc_mask, v0=v0)
366
+
367
+ if i == 0:
368
+ v0 = v_raw
369
+ raw_states.append(x_out)
370
+ enc_outputs.append(x_out)
371
+
372
+ stacked_enc = torch.stack(enc_outputs, dim=0)
373
+ router_w = F.softmax(self.enc_router_logits, dim=-1).to(dtype=stacked_enc.dtype)
374
+ cross_kv = []
375
+ for p in range(self.n_cross_pairs):
376
+ enc_mix = torch.einsum("k,kbsd->bsd", router_w[p], stacked_enc)
377
+ cross_kv.append(self.cross_kv_projs[p](enc_mix))
378
+ return cross_kv, enc_mask
379
+
380
+ def decode_seq(self, tgt_in: torch.Tensor, cross_kv: list, enc_mask: torch.Tensor, ckpt_mode: str = "none"):
381
+ x = self.embed_tokens(tgt_in) * self.embed_scale
382
+ v0 = None
383
+ for i, layer in enumerate(self.decoder):
384
+ kc, vc = cross_kv[i // 2]
385
+ do_ckpt = self.training and (
386
+ (ckpt_mode == "heavy" and (i % 2 == 1)) or
387
+ (ckpt_mode == "light" and (i == 3))
388
+ )
389
+ if do_ckpt:
390
+ x, v_raw = grad_ckpt(layer, x, kc, vc, self.rope, enc_mask, v0, use_reentrant=False)
391
+ else:
392
+ x, v_raw = layer(x, kc, vc, self.rope, enc_mask, v0=v0)
393
+ if i == 0:
394
+ v0 = v_raw
395
+ return self.dec_norm(x)
396
+
397
+ def forward_and_backward(
398
+ self,
399
+ src_ids: torch.Tensor,
400
+ tgt_in: torch.Tensor,
401
+ labels: torch.Tensor,
402
+ main_denom: float,
403
+ mtp_denom: float,
404
+ label_smoothing: float = 0.1,
405
+ z_loss_weight: float = 1e-4,
406
+ mtp_weight: float = 0.3,
407
+ ):
408
+ tot_tok = src_ids.numel() + tgt_in.numel()
409
+ max_len = max(src_ids.size(1), tgt_in.size(1))
410
+ if tot_tok <= 14336:
411
+ ckpt_mode = "none"
412
+ elif max_len <= 256:
413
+ ckpt_mode = "light"
414
+ else:
415
+ ckpt_mode = "heavy"
416
+
417
+ with torch.amp.autocast("cuda", dtype=torch.bfloat16):
418
+ cross_kv, enc_mask = self.encode_and_route(src_ids, ckpt_mode=ckpt_mode)
419
+ h_main = self.decode_seq(tgt_in, cross_kv, enc_mask, ckpt_mode=ckpt_mode)
420
+
421
+ valid_mask = labels.ne(-100)
422
+ valid_y = labels[valid_mask]
423
+ n_valid = valid_y.numel()
424
+ valid_h_leaf = h_main[valid_mask].detach().requires_grad_(True)
425
+
426
+ tot_main_ce_t = torch.zeros((), dtype=torch.float64, device=h_main.device)
427
+ tot_z_loss_t = torch.zeros((), dtype=torch.float64, device=h_main.device)
428
+ tot_mtp_ce_t = torch.zeros((), dtype=torch.float64, device=h_main.device)
429
+ chunk_sz = 2048
430
+
431
+ for i in range(0, n_valid, chunk_sz):
432
+ hc = valid_h_leaf[i : i + chunk_sz]
433
+ yc = valid_y[i : i + chunk_sz]
434
+ with torch.amp.autocast("cuda", dtype=torch.bfloat16):
435
+ logits_c = self.lm_head(hc)
436
+ ce_c = F.cross_entropy(logits_c, yc, label_smoothing=label_smoothing, reduction="sum")
437
+ lse = torch.logsumexp(logits_c.float(), dim=-1)
438
+ z_c = (lse * lse).sum()
439
+ loss_c = (ce_c + z_loss_weight * z_c) / main_denom
440
+ loss_c.backward()
441
+ tot_main_ce_t.add_(ce_c.detach().to(torch.float64))
442
+ tot_z_loss_t.add_(z_c.detach().to(torch.float64))
443
+ del logits_c, ce_c, z_c, loss_c
444
+
445
+ grad_h_main = torch.zeros_like(h_main)
446
+ if valid_h_leaf.grad is not None:
447
+ grad_h_main[valid_mask] = valid_h_leaf.grad
448
+
449
+ n_mtp = 0
450
+ if mtp_weight > 0.0 and tgt_in.size(1) > 1:
451
+ mtp_mask = labels[:, 1:].ne(-100) & labels[:, :-1].ne(-100)
452
+ valid_ym = labels[:, 1:][mtp_mask]
453
+ n_mtp = valid_ym.numel()
454
+ if n_mtp > 0:
455
+ h_main_leaf = h_main.detach().requires_grad_(True)
456
+ with torch.amp.autocast("cuda", dtype=torch.bfloat16):
457
+ next_emb = self.embed_tokens(tgt_in[:, 1:]) * self.embed_scale
458
+ h_mtp = self.mtp_head(h_main_leaf[:, :-1, :], next_emb, self.rope)
459
+ valid_hm_leaf = h_mtp[mtp_mask].detach().requires_grad_(True)
460
+
461
+ for i in range(0, n_mtp, chunk_sz):
462
+ hmc = valid_hm_leaf[i : i + chunk_sz]
463
+ ymc = valid_ym[i : i + chunk_sz]
464
+ with torch.amp.autocast("cuda", dtype=torch.bfloat16):
465
+ logits_m = self.lm_head(hmc)
466
+ ce_m = F.cross_entropy(logits_m, ymc, label_smoothing=label_smoothing, reduction="sum")
467
+ loss_m = (mtp_weight * ce_m) / mtp_denom
468
+ loss_m.backward()
469
+ tot_mtp_ce_t.add_(ce_m.detach().to(torch.float64))
470
+ del logits_m, ce_m, loss_m
471
+
472
+ grad_h_mtp = torch.zeros_like(h_mtp)
473
+ if valid_hm_leaf.grad is not None:
474
+ grad_h_mtp[mtp_mask] = valid_hm_leaf.grad
475
+ h_mtp.backward(grad_h_mtp)
476
+ if h_main_leaf.grad is not None:
477
+ grad_h_main.add_(h_main_leaf.grad)
478
+
479
+ h_main.backward(grad_h_main)
480
+ ce_v, mtp_v, z_v = torch.stack([tot_main_ce_t, tot_mtp_ce_t, tot_z_loss_t]).tolist()
481
+ return ce_v, mtp_v, z_v, n_valid, n_mtp
482
+
483
+ @torch.no_grad()
484
+ def eval_nll(self, src_ids: torch.Tensor, tgt_in: torch.Tensor, labels: torch.Tensor):
485
+ with torch.amp.autocast("cuda", dtype=torch.bfloat16):
486
+ cross_kv, enc_mask = self.encode_and_route(src_ids, ckpt_mode="none")
487
+ h_main = self.decode_seq(tgt_in, cross_kv, enc_mask, ckpt_mode="none")
488
+ valid_mask = labels.ne(-100)
489
+ valid_h = h_main[valid_mask]
490
+ valid_y = labels[valid_mask]
491
+ n_valid = valid_y.numel()
492
+ tot_nll = 0.0
493
+ for i in range(0, n_valid, 4096):
494
+ logits_c = self.lm_head(valid_h[i : i + 4096])
495
+ tot_nll += float(F.cross_entropy(logits_c, valid_y[i : i + 4096], reduction="sum").item())
496
+ return tot_nll, n_valid
497
+
498
+ @torch.inference_mode()
499
+ def translate_kv(self, prompt_ids_list: list[list[int]], sp_proc, max_new_tokens: int = 160, rep_penalty: float = 1.12) -> list[str]:
500
+ assert len(prompt_ids_list) > 0 and all(len(s) > 0 for s in prompt_ids_list), "Empty prompt batch!"
501
+ max_rope = self.rope.cos_cached.size(2)
502
+ assert 0 < max_new_tokens <= max_rope, f"max_new_tokens={max_new_tokens} exceeds RoPE={max_rope}"
503
+
504
+ self.eval()
505
+ dev = self.embed_tokens.weight.device
506
+ B = len(prompt_ids_list)
507
+ trimmed = [s[:max_rope] for s in prompt_ids_list]
508
+ max_s = min(max_rope, ((max(len(s) for s in trimmed) + 7) // 8) * 8)
509
+ src_ids = torch.full((B, max_s), PAD_ID, dtype=torch.long, device=dev)
510
+ for i, ids in enumerate(trimmed):
511
+ src_ids[i, : len(ids)] = torch.tensor(ids, dtype=torch.long, device=dev)
512
+
513
+ with torch.amp.autocast("cuda", dtype=torch.bfloat16):
514
+ cross_kv, enc_mask = self.encode_and_route(src_ids, ckpt_mode="none")
515
+ k_cache = torch.empty((self.dec_layers, B, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=torch.bfloat16, device=dev)
516
+ v_cache = torch.empty((self.dec_layers, B, self.n_kv_heads, max_new_tokens, self.head_dim), dtype=torch.bfloat16, device=dev)
517
+
518
+ cur_tok = torch.full((B, 1), BOS_ID, dtype=torch.long, device=dev)
519
+ gen_ids = torch.full((B, max_new_tokens), EOS_ID, dtype=torch.long, device=dev)
520
+ done = torch.zeros(B, dtype=torch.bool, device=dev)
521
+ cos_all, sin_all = self.rope.cos_cached.to(torch.bfloat16), self.rope.sin_cached.to(torch.bfloat16)
522
+
523
+ for pos in range(max_new_tokens):
524
+ cos_p, sin_p = cos_all[:, :, pos : pos + 1, :], sin_all[:, :, pos : pos + 1, :]
525
+ x_dec = self.embed_tokens(cur_tok) * self.embed_scale
526
+ v0 = None
527
+ for l_idx, layer in enumerate(self.decoder):
528
+ kc, vc = cross_kv[l_idx // 2]
529
+ x_dec, v_raw = layer.step_kv(x_dec, cos_p, sin_p, pos, k_cache[l_idx], v_cache[l_idx], kc, vc, enc_mask, v0=v0)
530
+ if l_idx == 0:
531
+ v0 = v_raw
532
+
533
+ logits = self.lm_head(self.dec_norm(x_dec)[:, 0, :]).float()
534
+ logits[:, [0, 1, 3, 4, 5, 6, 7, 8, 9, 10]] = -1e9
535
+ if rep_penalty != 1.0 and pos > 0:
536
+ prev = gen_ids[:, :pos]
537
+ s_l = torch.gather(logits, 1, prev)
538
+ s_l = torch.where(s_l > 0, s_l / rep_penalty, s_l * rep_penalty)
539
+ logits.scatter_(1, prev, s_l)
540
+
541
+ nxt = torch.where(done, torch.tensor(EOS_ID, device=dev), logits.argmax(dim=-1))
542
+ gen_ids[:, pos] = nxt
543
+ done = done | (nxt == EOS_ID)
544
+ cur_tok = nxt.unsqueeze(1)
545
+ if bool(done.all()):
546
+ break
547
+
548
+ out_texts = []
549
+ for row in gen_ids.cpu().tolist():
550
+ if EOS_ID in row:
551
+ row = row[: row.index(EOS_ID)]
552
+ out_texts.append(sp_proc.decode_ids([x for x in row if x >= 11]).strip())
553
+ return out_texts
554
+
555
+
556
+ # ==============================================================================
557
+ # 2. 100%-Coverage Plan + Tight-Padding Dynamic Packing + Distributed Sharding
558
+ # ==============================================================================
559
+ class CanonicalMemmapStore:
560
+ def __init__(self, data_dir: str, split: str):
561
+ self.my_path = os.path.join(data_dir, f"{split}_my.bin")
562
+ self.en_path = os.path.join(data_dir, f"{split}_en.bin")
563
+ self.index = np.load(os.path.join(data_dir, f"{split}_index.npy"), mmap_mode="r")
564
+ self.my_mm = None
565
+ self.en_mm = None
566
+
567
+ def ensure_open(self):
568
+ if self.my_mm is None:
569
+ self.my_mm = np.memmap(self.my_path, dtype=np.uint16, mode="r")
570
+ self.en_mm = np.memmap(self.en_path, dtype=np.uint16, mode="r")
571
+
572
+ def get_pair(self, idx: int):
573
+ self.ensure_open()
574
+ rec = self.index[idx]
575
+ m = self.my_mm[int(rec["my_off"]) : int(rec["my_off"]) + int(rec["my_len"])].astype(np.int64)
576
+ e = self.en_mm[int(rec["en_off"]) : int(rec["en_off"]) + int(rec["en_len"])].astype(np.int64)
577
+ return m, e, int(rec["flags"])
578
+
579
+
580
+ def build_or_load_epoch_plan(train_index: np.ndarray, meta: dict, stage: str, plan_path: str, is_main: bool, world_size: int, seed: int = 2026) -> np.ndarray:
581
+ if is_main and not os.path.exists(plan_path):
582
+ rng = np.random.RandomState(seed)
583
+ n_subsets = len(meta["subset_to_id"])
584
+ sids = np.asarray(train_index["subset_id"])
585
+ assert int(sids.max()) < n_subsets and int(sids.min()) >= 0
586
+
587
+ if stage == "stable":
588
+ base_rows = rng.permutation(len(train_index)).astype(np.uint32)
589
+ replay_n = 12_000_000
590
+ eligible_mask = np.ones(len(train_index), dtype=bool)
591
+ for k, name in meta["id_to_subset"].items():
592
+ if name in BULK_SUBSETS:
593
+ eligible_mask[sids == int(k)] = False
594
+ else:
595
+ eligible_mask = (np.asarray(train_index["flags"]) & FLAG_BT) == 0
596
+ base_rows = np.flatnonzero(eligible_mask).astype(np.uint32)
597
+ rng.shuffle(base_rows)
598
+ replay_n = 1_500_000
599
+
600
+ elig_pos = np.flatnonzero(eligible_mask).astype(np.uint32)
601
+ elig_sids = sids[elig_pos]
602
+ order = elig_pos[np.argsort(elig_sids, kind="mergesort")]
603
+ sorted_sids = sids[order]
604
+ counts = np.bincount(sorted_sids, minlength=n_subsets)
605
+ starts = np.zeros(n_subsets, dtype=np.int64)
606
+ starts[1:] = np.cumsum(counts[:-1])
607
+
608
+ weights = np.zeros(n_subsets, dtype=np.float64)
609
+ for k, name in meta["id_to_subset"].items():
610
+ sid = int(k)
611
+ if counts[sid] == 0:
612
+ continue
613
+ if stage == "stable" and name in BULK_SUBSETS:
614
+ continue
615
+ if stage == "cooldown" and name == "non_myanmartranslator/finetranslations":
616
+ w = float(counts[sid]) ** 0.35 * 0.25
617
+ else:
618
+ w = float(counts[sid]) ** 0.50
619
+ if stage == "stable" and name.startswith("non_myanmartranslator/"):
620
+ w *= 4.0
621
+ weights[sid] = w
622
+
623
+ assert weights.sum() > 0, "Empty replay weights!"
624
+ probs = weights / weights.sum()
625
+
626
+ chosen_sids = rng.choice(n_subsets, size=replay_n, p=probs)
627
+ replay_rows = np.empty(replay_n, dtype=np.uint32)
628
+ for sid in range(n_subsets):
629
+ m = (chosen_sids == sid)
630
+ npick = int(m.sum())
631
+ if npick > 0:
632
+ replay_rows[m] = order[starts[sid] + rng.randint(0, counts[sid], size=npick)]
633
+
634
+ full_plan = np.concatenate([base_rows, replay_rows])
635
+ rng.shuffle(full_plan)
636
+ tmp_plan = plan_path + ".tmp.npy"
637
+ np.save(tmp_plan, full_plan)
638
+ os.replace(tmp_plan, plan_path)
639
+ print(
640
+ f"📋 Built {stage} 100%-coverage plan: {len(base_rows):,} unique base rows (100% coverage) "
641
+ f"+ {len(replay_rows):,} gold replay = {len(full_plan):,} total rows"
642
+ )
643
+
644
+ if world_size > 1:
645
+ dist.barrier()
646
+ plan = np.load(plan_path, mmap_mode="r")
647
+ if is_main:
648
+ print(f"📋 Ready {stage} plan: {len(plan):,} rows")
649
+ return plan
650
+
651
+
652
+ def pack_chunk_batches(
653
+ row_slice: np.ndarray,
654
+ train_index: np.ndarray,
655
+ max_tokens_combined: int,
656
+ max_seq_len: int,
657
+ step_group: int,
658
+ seed: int,
659
+ ):
660
+ rng = np.random.RandomState(seed)
661
+ my_lens = np.minimum(max_seq_len, train_index["my_len"][row_slice].astype(np.int32))
662
+ en_lens = np.minimum(max_seq_len, train_index["en_len"][row_slice].astype(np.int32))
663
+ max_side = np.maximum(my_lens, en_lens)
664
+ comb_lens = my_lens + en_lens
665
+
666
+ mega = 65536
667
+ batches_cost = []
668
+ for i in range(0, len(row_slice), mega):
669
+ sub_r = row_slice[i : i + mega]
670
+ sub_ms = max_side[i : i + mega]
671
+ sub_cl = comb_lens[i : i + mega]
672
+ sub_r = sub_r[np.lexsort((sub_cl, sub_ms))]
673
+
674
+ cur_b, cur_s_max, cur_t_max = [], 0, 0
675
+ for r in sub_r:
676
+ ri = int(r)
677
+ ml = min(max_seq_len, int(train_index["my_len"][ri]))
678
+ el = min(max_seq_len, int(train_index["en_len"][ri]))
679
+ s_est = max(el, ml) + 4
680
+ t_est = max(ml, el) + 2
681
+ new_s = max(cur_s_max, s_est)
682
+ new_t = max(cur_t_max, t_est)
683
+ if cur_b and (len(cur_b) >= 256 or (len(cur_b) + 1) * (new_s + new_t) > max_tokens_combined):
684
+ batches_cost.append(((max(cur_s_max, cur_t_max), len(cur_b) * (cur_s_max + cur_t_max)), cur_b))
685
+ cur_b, cur_s_max, cur_t_max = [ri], s_est, t_est
686
+ else:
687
+ cur_b.append(ri)
688
+ cur_s_max, cur_t_max = new_s, new_t
689
+ if cur_b:
690
+ batches_cost.append(((max(cur_s_max, cur_t_max), len(cur_b) * (cur_s_max + cur_t_max)), cur_b))
691
+
692
+ # Load-balanced step grouping by (max_seq, tok_cost) + safe small-chunk pad
693
+ batches_cost.sort(key=lambda x: x[0])
694
+ batches = [b for _, b in batches_cost]
695
+ rem = len(batches) % step_group
696
+ if rem != 0 and len(batches) > 0:
697
+ need = step_group - rem
698
+ batches.extend((batches * ((need // len(batches)) + 1))[:need])
699
+ groups = [batches[j : j + step_group] for j in range(0, len(batches), step_group)]
700
+ rng.shuffle(groups)
701
+ return [b for g in groups for b in g]
702
+
703
+
704
+ class DynamicStreamDataset(Dataset):
705
+ def __init__(self, store: CanonicalMemmapStore, batches: list[list[int]], stage: str, max_seq_len: int, seed: int):
706
+ self.store = store
707
+ self.batches = batches
708
+ self.stage = stage
709
+ self.max_seq_len = max_seq_len
710
+ self.base_seed = seed
711
+ self.rng = np.random.RandomState(seed)
712
+
713
+ def __len__(self):
714
+ return len(self.batches)
715
+
716
+ def _corrupt_span(self, toks: np.ndarray):
717
+ n = len(toks)
718
+ if n < 4:
719
+ return toks
720
+ out = []
721
+ i = 0
722
+ masked_any = False
723
+ while i < n:
724
+ if self.rng.rand() < 0.15:
725
+ span_len = min(n - i, max(1, int(self.rng.poisson(3.0))))
726
+ out.append(TOK_MASK)
727
+ i += span_len
728
+ masked_any = True
729
+ else:
730
+ out.append(int(toks[i]))
731
+ i += 1
732
+ if not masked_any:
733
+ idx = int(self.rng.randint(0, n))
734
+ out = list(toks[:idx]) + [TOK_MASK] + list(toks[idx + min(2, n - idx):])
735
+ return np.asarray(out, dtype=np.int64)
736
+
737
+ def __getitem__(self, b_idx: int):
738
+ row_ids = self.batches[b_idx]
739
+ srcs, tgts = [], []
740
+ for r_id in row_ids:
741
+ m, e, flg = self.store.get_pair(r_id)
742
+ m = m[: self.max_seq_len - 4]
743
+ e = e[: self.max_seq_len - 4]
744
+ is_bt = bool(flg & FLAG_BT)
745
+ is_formal = bool(flg & FLAG_FORMAL)
746
+ is_spoken = bool(flg & FLAG_SPOKEN)
747
+
748
+ if self.stage == "cooldown":
749
+ mode = "en2my" if self.rng.rand() < 0.50 else "my2en"
750
+ else:
751
+ p = self.rng.rand()
752
+ if is_bt:
753
+ mode = "en2my" if p < 0.72 else ("my2en_bt" if p < 0.90 else "denoise")
754
+ else:
755
+ mode = "en2my" if p < 0.47 else ("my2en" if p < 0.94 else "denoise")
756
+
757
+ style_tok = None
758
+ if is_formal and self.rng.rand() < 0.70:
759
+ style_tok = TOK_FORMAL
760
+ elif is_spoken and self.rng.rand() < 0.70:
761
+ style_tok = TOK_SPOKEN
762
+
763
+ if mode == "en2my":
764
+ pref = [TOK_2MY] + ([style_tok] if style_tok is not None else [])
765
+ src_arr = np.concatenate([np.asarray(pref, dtype=np.int64), e, [EOS_ID]])
766
+ tgt_arr = np.concatenate([[BOS_ID], m, [EOS_ID]])
767
+ elif mode in ("my2en", "my2en_bt"):
768
+ pref = [TOK_2EN] + ([TOK_BT] if mode == "my2en_bt" else []) + ([style_tok] if style_tok is not None else [])
769
+ src_arr = np.concatenate([np.asarray(pref, dtype=np.int64), m, [EOS_ID]])
770
+ tgt_arr = np.concatenate([[BOS_ID], e, [EOS_ID]])
771
+ else:
772
+ base = m if (is_bt or self.rng.rand() < 0.5) else e
773
+ noisy = self._corrupt_span(base)
774
+ src_arr = np.concatenate([[TOK_DENOISE], noisy, [EOS_ID]])
775
+ tgt_arr = np.concatenate([[BOS_ID], base, [EOS_ID]])
776
+
777
+ srcs.append(src_arr)
778
+ tgts.append(tgt_arr)
779
+ return srcs, tgts
780
+
781
+
782
+ def worker_init_fn(worker_id: int):
783
+ info = torch.utils.data.get_worker_info()
784
+ if info is not None and hasattr(info.dataset, "rng"):
785
+ info.dataset.rng = np.random.RandomState((info.dataset.base_seed + worker_id * 9973) % (2**31 - 1))
786
+
787
+
788
+ def collate_prebatched(batch_list):
789
+ srcs, tgts = batch_list[0]
790
+ B = len(srcs)
791
+ max_s = min(2048, ((max(len(x) for x in srcs) + 31) // 32) * 32)
792
+ max_t = min(2048, ((max(len(x) - 1 for x in tgts) + 31) // 32) * 32)
793
+ src_ids = torch.full((B, max_s), PAD_ID, dtype=torch.long)
794
+ tgt_in = torch.full((B, max_t), PAD_ID, dtype=torch.long)
795
+ labels = torch.full((B, max_t), -100, dtype=torch.long)
796
+ for i, (s, t) in enumerate(zip(srcs, tgts)):
797
+ sl = min(len(s), max_s)
798
+ tl = min(len(t) - 1, max_t)
799
+ src_ids[i, :sl] = torch.from_numpy(s[:sl])
800
+ tgt_in[i, :tl] = torch.from_numpy(t[:tl])
801
+ labels[i, :tl] = torch.from_numpy(t[1 : tl + 1])
802
+ return src_ids, tgt_in, labels
803
+
804
+
805
+ def clean_state_dict(model: nn.Module):
806
+ return {k.replace("._orig_mod.", "."): v.detach().cpu() for k, v in model.state_dict().items()}
807
+
808
+
809
+ def broadcast_model(model: nn.Module, world_size: int):
810
+ if world_size == 1:
811
+ return
812
+ for p in model.parameters():
813
+ dist.broadcast(p.data, src=0)
814
+ for b in model.buffers():
815
+ dist.broadcast(b.data, src=0)
816
+
817
+
818
+ def allreduce_grads(model: nn.Module, world_size: int, grad_scale: float = 1024.0):
819
+ if world_size == 1:
820
+ return
821
+ grads = []
822
+ for p in model.parameters():
823
+ if p.requires_grad:
824
+ if p.grad is None:
825
+ p.grad = torch.zeros_like(p.data)
826
+ grads.append(p.grad)
827
+ b_grads = [g.bfloat16() for g in grads]
828
+ flat = torch._utils._flatten_dense_tensors(b_grads)
829
+ dist.all_reduce(flat, op=dist.ReduceOp.SUM)
830
+ inv_scale = 1.0 / grad_scale
831
+ for orig, synced in zip(grads, torch._utils._unflatten_dense_tensors(flat, b_grads)):
832
+ orig.copy_(synced.float() * inv_scale)
833
+
834
+
835
+ # ==============================================================================
836
+ # 3. Main Multi-GPU Training Loop
837
+ # ==============================================================================
838
+ def main():
839
+ parser = argparse.ArgumentParser()
840
+ parser.add_argument("--stage", choices=["stable", "cooldown"], default="stable")
841
+ parser.add_argument("--verify-steps", type=int, default=0)
842
+ parser.add_argument("--grad-accum", type=int, default=1)
843
+ parser.add_argument("--max-tok-micro", type=int, default=32768)
844
+ parser.add_argument("--max-seq-len", type=int, default=1024)
845
+ parser.add_argument("--chunk-rows", type=int, default=4000000)
846
+ parser.add_argument("--peak-lr", type=float, default=5.0e-4)
847
+ parser.add_argument("--warmup-steps", type=int, default=900)
848
+ parser.add_argument("--eval-every", type=int, default=500)
849
+ parser.add_argument("--save-every", type=int, default=500)
850
+ parser.add_argument("--milestone-every", type=int, default=5000)
851
+ parser.add_argument("--compile", action="store_true")
852
+ parser.add_argument("--fresh", action="store_true")
853
+ parser.add_argument("--init-weights", type=str, default="")
854
+ args = parser.parse_args()
855
+
856
+ world_size = int(os.environ.get("WORLD_SIZE", 1))
857
+ rank = int(os.environ.get("RANK", 0))
858
+ local_rank = int(os.environ.get("LOCAL_RANK", 0))
859
+ is_main = (rank == 0)
860
+
861
+ if world_size > 1:
862
+ torch.cuda.set_device(local_rank)
863
+ dist.init_process_group(backend="nccl", timeout=timedelta(minutes=30), device_id=torch.device(f"cuda:{local_rank}"))
864
+ device = torch.device("cuda", local_rank)
865
+ else:
866
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
867
+
868
+ if is_main:
869
+ os.makedirs(CKPT_DIR, exist_ok=True)
870
+ if world_size > 1:
871
+ dist.barrier()
872
+
873
+ meta, sp = load_and_verify_contracts(DATA_DIR)
874
+ train_store = CanonicalMemmapStore(DATA_DIR, "train")
875
+ val_store = CanonicalMemmapStore(DATA_DIR, "val")
876
+
877
+ all_val_non_bt = np.flatnonzero((val_store.index["flags"] & FLAG_BT) == 0)
878
+ assert len(all_val_non_bt) > 0, "Validation set has zero non-BT rows!"
879
+ val_rows = all_val_non_bt[::max(1, len(all_val_non_bt) // 1024)][:1024]
880
+ val_batches = [val_rows[i : i + 64].tolist() for i in range(0, len(val_rows), 64)]
881
+
882
+ plan_path = os.path.join(CKPT_DIR, f"{args.stage}_plan.npy")
883
+ latest_path_early = os.path.join(CKPT_DIR, "latest.pt")
884
+ if args.fresh and is_main:
885
+ if os.path.exists(plan_path):
886
+ os.remove(plan_path)
887
+ if os.path.exists(latest_path_early):
888
+ os.replace(latest_path_early, latest_path_early + ".bak")
889
+ if world_size > 1:
890
+ dist.barrier()
891
+ full_plan = build_or_load_epoch_plan(train_store.index, meta, args.stage, plan_path, is_main, world_size, seed=2026)
892
+ num_chunks = (len(full_plan) + args.chunk_rows - 1) // args.chunk_rows
893
+
894
+ model = MyTranslation300M().to(device)
895
+ decay_p = [p for n, p in model.named_parameters() if p.requires_grad and p.ndim >= 2]
896
+ nodecay_p = [p for n, p in model.named_parameters() if p.requires_grad and p.ndim < 2]
897
+ opt = torch.optim.AdamW(
898
+ [{"params": decay_p, "weight_decay": 0.01}, {"params": nodecay_p, "weight_decay": 0.0}],
899
+ lr=args.peak_lr, betas=(0.9, 0.95), eps=1e-8, fused=True,
900
+ )
901
+
902
+ step = 0
903
+ chunk_idx = 0
904
+ batch_cursor = 0
905
+ rows_seen = 0
906
+ best_val_nll = float("inf")
907
+ latest_path = os.path.join(CKPT_DIR, "latest.pt")
908
+
909
+ if args.init_weights:
910
+ if is_main:
911
+ print(f"📥 Loading weights from {args.init_weights}")
912
+ model.load_state_dict(torch.load(args.init_weights, map_location="cpu"), strict=True)
913
+ elif (not args.fresh) and os.path.exists(latest_path):
914
+ if is_main:
915
+ print(f"🔄 Resuming from {latest_path}")
916
+ ckpt = torch.load(latest_path, map_location="cpu")
917
+ model.load_state_dict(ckpt["model"], strict=True)
918
+ if args.stage == ckpt.get("stage", "stable"):
919
+ opt.load_state_dict(ckpt["opt"])
920
+ step = int(ckpt["step"])
921
+ chunk_idx = int(ckpt.get("chunk_idx", 0))
922
+ batch_cursor = int(ckpt.get("batch_cursor", 0))
923
+ rows_seen = int(ckpt.get("rows_seen", 0))
924
+ best_val_nll = float(ckpt.get("best_val_nll", float("inf")))
925
+ elif is_main:
926
+ print(f"➡️ Stage transition {ckpt.get('stage')} -> {args.stage}")
927
+
928
+ broadcast_model(model, world_size)
929
+
930
+ if args.compile:
931
+ if is_main:
932
+ print("⚡ Compiling encoder/cross_kv/decoder/MTP blocks with torch.compile(dynamic=True)...")
933
+ model.compile_blocks()
934
+
935
+ if is_main:
936
+ tot_params = sum(p.numel() for p in model.parameters())
937
+ inf_params = tot_params - sum(p.numel() for p in model.mtp_head.parameters())
938
+ print(f"🏛️ MyTranslation-300M | GPUs: {world_size}x | Stage: {args.stage} | Inf: {inf_params/1e6:.2f}M | Train: {tot_params/1e6:.2f}M")
939
+
940
+ label_smoothing = 0.1 if args.stage == "stable" else 0.0
941
+ mtp_weight = 0.3 if args.stage == "stable" else 0.15
942
+
943
+ STYLE_EVAL_PROMPTS = [
944
+ ("En->My Formal", [TOK_2MY, TOK_FORMAL], "The central bank raised interest rates to control inflation and stabilize the economy."),
945
+ ("En->My Spoken", [TOK_2MY, TOK_SPOKEN], "Honestly, I didn't expect things to turn out like this, but don't worry—we'll figure it out together."),
946
+ ("My->En Formal", [TOK_2EN, TOK_FORMAL], "ဗဟိုဘဏ်သည် ငွေကြေးဖောင်းပွမှုကို ထိန်းချုပ်ရန်နှင့် စီးပွားရေးတည်ငြိမ်စေရန် အတိုးနှုန်းများကို မြှင့်တင်ခဲ့သည်။"),
947
+ ("My->En Spoken", [TOK_2EN, TOK_SPOKEN], "ဒီနေ့ ရန်ကုန်မှာ မိုးတွေ အရမ်းရွာနေလို့ အိမ်ပြန်ရတာ နည်းနည်းနောက်ကျမယ်နော်။"),
948
+ ]
949
+
950
+ @torch.no_grad()
951
+ def eval_val_nll():
952
+ model.eval()
953
+ en2my_nll, en2my_tok = 0.0, 0
954
+ my2en_nll, my2en_tok = 0.0, 0
955
+ for b in val_batches:
956
+ for direction in ("en2my", "my2en"):
957
+ srcs, tgts = [], []
958
+ for r_id in b:
959
+ m, e, _ = val_store.get_pair(int(r_id))
960
+ m, e = m[: args.max_seq_len - 4], e[: args.max_seq_len - 4]
961
+ if direction == "en2my":
962
+ srcs.append(np.concatenate([[TOK_2MY], e, [EOS_ID]]))
963
+ tgts.append(np.concatenate([[BOS_ID], m, [EOS_ID]]))
964
+ else:
965
+ srcs.append(np.concatenate([[TOK_2EN], m, [EOS_ID]]))
966
+ tgts.append(np.concatenate([[BOS_ID], e, [EOS_ID]]))
967
+ src_ids, tgt_in, labels = collate_prebatched([(srcs, tgts)])
968
+ nll, nv = model.eval_nll(src_ids.to(device), tgt_in.to(device), labels.to(device))
969
+ if direction == "en2my":
970
+ en2my_nll += nll
971
+ en2my_tok += nv
972
+ else:
973
+ my2en_nll += nll
974
+ my2en_tok += nv
975
+ e2m = en2my_nll / max(1, en2my_tok)
976
+ m2e = my2en_nll / max(1, my2en_tok)
977
+ comb = (en2my_nll + my2en_nll) / max(1, en2my_tok + my2en_tok)
978
+ return comb, e2m, m2e
979
+
980
+ def run_style_generations(step_num: int):
981
+ prompts = [pref + sp.encode_as_ids(txt)[:500] + [EOS_ID] for _, pref, txt in STYLE_EVAL_PROMPTS]
982
+ preds = model.translate_kv(prompts, sp, max_new_tokens=96)
983
+ print(f" 🗣️ Style Generations @ Step {step_num}:")
984
+ for (tag, _, _), out in zip(STYLE_EVAL_PROMPTS, preds):
985
+ print(f" [{tag}]: {out[:180]}")
986
+
987
+ def make_chunk_loader(c_idx: int, skip_steps: int = 0):
988
+ s_off = c_idx * args.chunk_rows
989
+ e_off = min(len(full_plan), s_off + args.chunk_rows)
990
+ row_slice = np.asarray(full_plan[s_off:e_off])
991
+ step_group = world_size * args.grad_accum
992
+ all_batches = pack_chunk_batches(
993
+ row_slice, train_store.index,
994
+ max_tokens_combined=args.max_tok_micro,
995
+ max_seq_len=args.max_seq_len,
996
+ step_group=step_group,
997
+ seed=2026 + c_idx * 17,
998
+ )
999
+ rank_batches = all_batches[rank::world_size]
1000
+ if skip_steps > 0:
1001
+ rank_batches = rank_batches[skip_steps * args.grad_accum :]
1002
+ n_windows = len(rank_batches) // args.grad_accum
1003
+ rank_batches = rank_batches[: n_windows * args.grad_accum]
1004
+ ds = DynamicStreamDataset(
1005
+ train_store, rank_batches, args.stage, args.max_seq_len,
1006
+ seed=10000 + c_idx * 997 + rank * 131 + skip_steps,
1007
+ )
1008
+ loader = DataLoader(
1009
+ ds, batch_size=1, shuffle=False, collate_fn=collate_prebatched,
1010
+ num_workers=4, pin_memory=True, worker_init_fn=worker_init_fn, persistent_workers=False,
1011
+ )
1012
+ return loader, n_windows
1013
+
1014
+ if is_main:
1015
+ comb, e2m, m2e = eval_val_nll()
1016
+ print(f"🔎 Step {step:6d} | Val Comb: {comb:.4f} | En->My: {e2m:.4f} | My->En: {m2e:.4f}")
1017
+ if world_size > 1:
1018
+ dist.barrier()
1019
+
1020
+ model.train()
1021
+ opt.zero_grad(set_to_none=True)
1022
+ acc_ce = acc_mtp = acc_z = 0.0
1023
+ acc_valid_tok = acc_mtp_tok = acc_tot_tok = 0
1024
+ t0 = time.time()
1025
+
1026
+ while chunk_idx < num_chunks:
1027
+ train_loader, n_windows = make_chunk_loader(chunk_idx, skip_steps=batch_cursor)
1028
+ train_iter = iter(train_loader)
1029
+ done_flag = False
1030
+
1031
+ for _ in range(n_windows):
1032
+ window = [next(train_iter) for _ in range(args.grad_accum)]
1033
+ batch_cursor += 1
1034
+
1035
+ loc_valid = sum(int((lb != -100).sum().item()) for _, _, lb in window)
1036
+ loc_mtp = sum(
1037
+ int(((lb[:, 1:] != -100) & (lb[:, :-1] != -100)).sum().item()) if lb.size(1) > 1 else 0
1038
+ for _, _, lb in window
1039
+ )
1040
+ loc_rows = sum(s.size(0) for s, _, _ in window)
1041
+
1042
+ if world_size > 1:
1043
+ counts_t = torch.tensor([loc_valid, loc_mtp, loc_rows], dtype=torch.long, device=device)
1044
+ dist.all_reduce(counts_t, op=dist.ReduceOp.SUM)
1045
+ glob_valid, glob_mtp, glob_rows = int(counts_t[0].item()), int(counts_t[1].item()), int(counts_t[2].item())
1046
+ else:
1047
+ glob_valid, glob_mtp, glob_rows = loc_valid, loc_mtp, loc_rows
1048
+
1049
+ g_scale = 1024.0 if world_size > 1 else 1.0
1050
+ main_denom = float(max(1, glob_valid)) / g_scale
1051
+ mtp_denom = float(max(1, glob_mtp)) / g_scale
1052
+
1053
+ loc_ce = loc_mtp_ce = loc_z = 0.0
1054
+ loc_tok = sum(int((s != PAD_ID).sum().item()) for s, _, _ in window)
1055
+ for src_ids, tgt_in, labels in window:
1056
+ src_ids = src_ids.to(device, non_blocking=True)
1057
+ tgt_in = tgt_in.to(device, non_blocking=True)
1058
+ labels = labels.to(device, non_blocking=True)
1059
+ ce_s, mtp_s, z_s, nv, nm = model.forward_and_backward(
1060
+ src_ids, tgt_in, labels,
1061
+ main_denom=main_denom, mtp_denom=mtp_denom,
1062
+ label_smoothing=label_smoothing, mtp_weight=mtp_weight,
1063
+ )
1064
+ loc_ce += ce_s
1065
+ loc_mtp_ce += mtp_s
1066
+ loc_z += z_s
1067
+ loc_tok += nv
1068
+
1069
+ allreduce_grads(model, world_size)
1070
+ rows_seen += glob_rows
1071
+
1072
+ if args.stage == "stable":
1073
+ lr = args.peak_lr * min(1.0, (step + 1) / max(1, args.warmup_steps))
1074
+ else:
1075
+ if step < 100:
1076
+ lr = args.peak_lr * (step + 1) / 100.0
1077
+ else:
1078
+ progress = min(1.0, rows_seen / max(1, len(full_plan)))
1079
+ lr = 1.0e-5 + 0.5 * (args.peak_lr - 1.0e-5) * (1.0 + math.cos(math.pi * progress))
1080
+
1081
+ for pg in opt.param_groups:
1082
+ pg["lr"] = lr
1083
+ grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
1084
+ opt.step()
1085
+ opt.zero_grad(set_to_none=True)
1086
+ step += 1
1087
+
1088
+ if world_size > 1:
1089
+ stats_t = torch.tensor([loc_ce, loc_mtp_ce, loc_z, float(loc_tok)], dtype=torch.float64, device=device)
1090
+ dist.all_reduce(stats_t, op=dist.ReduceOp.SUM)
1091
+ acc_ce += float(stats_t[0].item())
1092
+ acc_mtp += float(stats_t[1].item())
1093
+ acc_z += float(stats_t[2].item())
1094
+ acc_tot_tok += int(stats_t[3].item())
1095
+ else:
1096
+ acc_ce += loc_ce
1097
+ acc_mtp += loc_mtp_ce
1098
+ acc_z += loc_z
1099
+ acc_tot_tok += loc_tok
1100
+ acc_valid_tok += glob_valid
1101
+ acc_mtp_tok += glob_mtp
1102
+
1103
+ log_int = 10 if args.verify_steps > 0 else 50
1104
+ if step % log_int == 0:
1105
+ if is_main:
1106
+ dt = time.time() - t0
1107
+ tok_s = acc_tot_tok / max(1e-3, dt)
1108
+ vram_g = torch.cuda.max_memory_allocated(device) / 1e9
1109
+ pct = 100.0 * min(rows_seen, len(full_plan)) / len(full_plan)
1110
+ print(
1111
+ f"Step {step:6d} ({pct:5.1f}% | Chunk {chunk_idx+1}/{num_chunks}) | "
1112
+ f"Main: {acc_ce/max(1,acc_valid_tok):.4f} | MTP2: {acc_mtp/max(1,acc_mtp_tok):.4f} | "
1113
+ f"Z: {acc_z/max(1,acc_valid_tok):.2f} | LR: {lr:.2e} | Grad: {grad_norm:.2f} | "
1114
+ f"{tok_s/1e3:.1f}k tok/s | VRAM: {vram_g:.1f}G"
1115
+ )
1116
+ acc_ce = acc_mtp = acc_z = 0.0
1117
+ acc_valid_tok = acc_mtp_tok = acc_tot_tok = 0
1118
+ t0 = time.time()
1119
+
1120
+ if args.verify_steps > 0 and step >= args.verify_steps:
1121
+ done_flag = True
1122
+ break
1123
+
1124
+ if step % args.eval_every == 0:
1125
+ if is_main:
1126
+ comb, e2m, m2e = eval_val_nll()
1127
+ print("-" * 96)
1128
+ print(f"🔎 EVAL @ Step {step:6d} | Comb NLL: {comb:.4f} (PPL {math.exp(min(comb, 30.0)):.2f}) | En->My: {e2m:.4f} | My->En: {m2e:.4f}")
1129
+ run_style_generations(step)
1130
+ if comb < best_val_nll:
1131
+ best_val_nll = comb
1132
+ best_sd = {k: v.to(torch.bfloat16) for k, v in clean_state_dict(model).items()}
1133
+ torch.save(best_sd, os.path.join(CKPT_DIR, "best_model.pt"))
1134
+ print(f" ⭐ New Best Model saved! Comb NLL: {best_val_nll:.4f}")
1135
+ print("-" * 96)
1136
+ model.train()
1137
+ if world_size > 1:
1138
+ dist.barrier()
1139
+ t0 = time.time()
1140
+
1141
+ if step % args.save_every == 0:
1142
+ if is_main:
1143
+ tmp_p = latest_path + ".tmp"
1144
+ torch.save({
1145
+ "model": clean_state_dict(model),
1146
+ "opt": opt.state_dict(),
1147
+ "step": step,
1148
+ "stage": args.stage,
1149
+ "chunk_idx": chunk_idx,
1150
+ "batch_cursor": batch_cursor,
1151
+ "rows_seen": rows_seen,
1152
+ "best_val_nll": best_val_nll,
1153
+ }, tmp_p)
1154
+ os.replace(tmp_p, latest_path)
1155
+ if world_size > 1:
1156
+ dist.barrier()
1157
+
1158
+ if step % args.milestone_every == 0:
1159
+ if is_main:
1160
+ ms_sd = {k: v.to(torch.bfloat16) for k, v in clean_state_dict(model).items()}
1161
+ torch.save(ms_sd, os.path.join(CKPT_DIR, f"{args.stage}_step_{step:06d}.pt"))
1162
+ if world_size > 1:
1163
+ dist.barrier()
1164
+
1165
+ if done_flag:
1166
+ break
1167
+ chunk_idx += 1
1168
+ batch_cursor = 0
1169
+
1170
+ if args.verify_steps > 0:
1171
+ if is_main:
1172
+ print(f"✅ {world_size}-GPU VERIFY PASSED ({step} steps)! Ready for launch.")
1173
+ if world_size > 1:
1174
+ dist.barrier()
1175
+ dist.destroy_process_group()
1176
+ return
1177
+
1178
+ if is_main:
1179
+ comb, e2m, m2e = eval_val_nll()
1180
+ print("-" * 96)
1181
+ print(f"🏁 FINAL EVAL @ Step {step:6d} | Comb NLL: {comb:.4f} (PPL {math.exp(min(comb, 30.0)):.2f}) | En->My: {e2m:.4f} | My->En: {m2e:.4f}")
1182
+ run_style_generations(step)
1183
+ if comb < best_val_nll:
1184
+ best_val_nll = comb
1185
+ best_sd = {k: v.to(torch.bfloat16) for k, v in clean_state_dict(model).items()}
1186
+ torch.save(best_sd, os.path.join(CKPT_DIR, "best_model.pt"))
1187
+ tmp_p = latest_path + ".tmp"
1188
+ torch.save({
1189
+ "model": clean_state_dict(model),
1190
+ "opt": opt.state_dict(),
1191
+ "step": step,
1192
+ "stage": args.stage,
1193
+ "chunk_idx": chunk_idx,
1194
+ "batch_cursor": batch_cursor,
1195
+ "rows_seen": rows_seen,
1196
+ "best_val_nll": best_val_nll,
1197
+ }, tmp_p)
1198
+ os.replace(tmp_p, latest_path)
1199
+
1200
+ with torch.no_grad():
1201
+ rw = F.softmax(model.enc_router_logits, dim=-1).cpu().numpy()
1202
+ for p in range(model.n_cross_pairs):
1203
+ top_layers = np.argsort(rw[p])[::-1][:4].tolist()
1204
+ print(f" • Cross-KV Pair {p} top encoder mixture weights: {top_layers} ({[round(float(rw[p][k]), 3) for k in top_layers]})")
1205
+
1206
+ print(f"🎉 Stage '{args.stage}' 100% COMPLETE ({rows_seen:,} rows trained across {step:,} steps)!")
1207
+
1208
+ if world_size > 1:
1209
+ dist.barrier()
1210
+ dist.destroy_process_group()
1211
+
1212
+
1213
+ if __name__ == "__main__":
1214
+ main()
trainingcelloutputwitheval.txt ADDED
The diff for this file is too large to render. See raw diff