DoL-1.5-Flash

Traditional Mongolian and Chinese base language model, continued from the language core of DoL-1.2-OCR.

Homoglyphs. In the traditional Mongolian script, O (U+1823) and U (U+1824) share one written form, as do OE (U+1825) and UE (U+1826). The Mongolian training text uses these code points inconsistently, so the tokenizer and the model contain several encodings of the same written word: 943 written forms have more than one token in the vocabulary, for example Mongol as tokens 4794, 7788 and 5827. Generated text may mix these encodings. Text that displays correctly can differ from the standard spelling at the code-point level, which affects search, deduplication, string-match evaluation and rule-based rewards.

We do not recommend using this model as a base model for fine-tuning or continued pretraining, or running reinforcement learning on it directly, in research or production.

• Parameters: 1,906,794,619 (the 1.0B DoL-1.2-OCR language core with each SwiGLU FFN widened from 8,192 to 20,480 hidden units; the widening is output-preserving).
• Architecture: two-stage Recurrent Depth Transformer, hidden size 2048, 3 prelude and 3 coda layers around a recurrent core of 5 Mamba layers and 1 MLA attention layer run 4 times.
• Tokenizer: the DoL native MorphBPE bundle under tokenizer/, byte-identical to DoL-1.2-OCR.
• Checkpoint: optimizer step 10720. Trained to 262,144-token context with YaRN position scaling after the 8K stage.
• Training data: 1.16B continuation tokens, half traditional Mongolian (cleaned v3 corpus) and half Chinese (web, Wikipedia, news and six IndustryCorpus2 domains).
• Vision weights: the OMVT tower inherited from DoL-1.2-OCR is stored in model.pt but was not trained with this language core; OCR use requires re-alignment.

Source code

The RDT model implementation and tokenizer runtime are included in this repository under Model/ and Tokenizer/. The source bundle matches the configuration layout in config.json. RDT training and post-training code, corpus data, tests, optimizer state, and model training checkpoints are not part of this bundle. See SOURCE_PROVENANCE.md for its scope.

Evaluation

Mean token loss (nats) on 32 Mongolian and 32 Chinese 8,192-token blocks the checkpoint never trained on (blocks 450,000+ of the continuation data), and on four long Mongolian sequences, each four consecutive blocks with block padding removed (22,517 to 29,697 tokens), scored on their first and last 8,192 positions.

Checkpoint	Mongolian 8K	Chinese 8K	Long Mongolian, first 8K	Long Mongolian, last 8K
step 10228 (8K)	0.7903	4.4231	0.6415	1.0448
step 10720 (262K)	0.7939	4.4058	0.6433	1.0373

Selection rule, fixed before the run: the lowest mean of Mongolian 8K, Chinese 8K and long Mongolian last-8K loss.

Limitations

• Base model only: no instruction tuning or alignment.
• Chinese ability is limited by 0.58B training tokens; greedy decoding repeats and factual answers are unreliable.
• Long-context ability has not been measured with retrieval tasks; long-context training used packed ordinary documents.

Files

• model.pt: state dict for Model.model.RDTForCausalLM (FP32).
• meta.pt: training metadata including rdt_config and omvt_config.
• config.json: the same configurations as JSON.
• tokenizer/: tokenizer bundle.
• Model/ and Tokenizer/: model and tokenizer implementation source.
• SOURCE_PROVENANCE.md: source-bundle scope and provenance.
• SHA256SUMS: SHA256 checksums for every repository file except the manifest itself.