mengsay's picture
Initial private draft of the Samdei Type release
7a7ad8f verified
Raw History Blame Contribute Delete
1.16 kB
Samdei Type Base
Copyright 2026 The Samdei Type Authors
Licensed under the Apache License, Version 2.0 (see LICENSE). This covers every file in this
repository: the model weights, the ONNX graphs, onnx/vocab.json, the channel model, rerank.json
and the language models.
Language models (lm/char5_nowiki.npz, lm/wordfull_nowiki.json.gz)
Character and word n-gram counts of Khmer text, built without Wikipedia, from:
- HPLT v2 (HPLT/HPLT2.0_cleaned, khm_Khmr): CC0 packaging; the Common Crawl Terms of Use
apply to the text.
- FineWeb-2 (HuggingFaceFW/fineweb-2, khm_Khmr): ODC-By 1.0; the Common Crawl Terms of Use
apply to the text.
- Asian Language Treebank (ALT), Khmer side: CC BY 4.0.
- Samdei Type chat sentences (samdei/samdei-type-resources): CC BY 4.0.
Training data
The model and the channel model were trained on samdei/samdei-type-pretraining-data and
samdei/samdei-type-finetuning-data (CC BY-SA 4.0); their cards list every source. Part of the
fine-tuning pairs were spelled with tables derived from the Chhunneng romanized Khmer
dataset, MIT License, Copyright (c) 2025 Chrea Chanchhunneng, and from Wikidata (CC0).