CKTN-ELECTRA (ablation: no linear λ schedule)

This is an ablation checkpoint from the paper Echoes Across Vietnam's Highlands, Delta, and Coast: A Multilingual Corpus for Cham, Khmer, and Tay-Nung. It is released so that the ablation study can be reproduced. For downstream use, take the full model, nhatkhangdtp/CKTN-ELECTRA.

It was trained exactly like CKTN-ELECTRA, except that the discriminator loss weight is fixed at λ = 50 from the first update instead of following the linear 0 → 50 schedule. All other settings are the same: 8-layer generator with shared embeddings, top-64 candidates at temperature 1.25, similarity band [0.15, 0.95], AdamW at lr 2e-5, 6 epochs, 15% masking and sequence length 512.

The weights are the discriminator from the epoch with the best development replaced-token-detection F1: epoch 1, with RTD F1 0.937 and accuracy 0.984. It is a plain RemBertModel with no task head.

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("nhatkhangdtp/CKTN-ELECTRA-no-linear-lambda", use_fast=False)
model = AutoModel.from_pretrained("nhatkhangdtp/CKTN-ELECTRA-no-linear-lambda")

Code and training configuration: github.com/TracDucAnh/CKTN-ELECTRA, config configs/pretrain/no_linear_lambda.json.

Citation

@misc{dinh2026echoes,
  title         = {Echoes Across Vietnam's Highlands, Delta, and Coast: A Multilingual Corpus for Cham, Khmer, and Tay-Nung},
  author        = {Dinh, Anh Trac Duc and Vo, Khang Nhat Hoang and Doan, Vinh Cong and Ta, Tai Tien and Lam, Khoa Duc Anh},
  year          = {2026},
  eprint        = {2607.08362},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2607.08362}
}