High quality corpus for Taiwanese culture and Traditional Chinese

Taiwan Text Excellence (TTE)

• Contains high quality news and articles in Traditional Chinese.
• The data processing pipeline is optimized for LLM performance.
• Is de-duplicated and cleaned using both rule-based and learning-based filters.
  • E.g., urls/emails/html tags/abnormal characters are cleaned, and numbers (full-width or half-width) are normalized.
• Contains ~2 billion tokens, measured using BPE tokenizer trained on this corpus.

Dataset Details

Dataset source:

• benchang1110/technewstw
• benchang1110/newstw
• benchang1110/humantw
• benchang1110/sciencetw
• AWeirdDev/zh-tw-pts-articles-sm
• AWeirdDev/zh-tw-articles-2k
• AWeirdDev/zh-tw-articles-6k
• erhwenkuo/wikinews-zhtw
• erhwenkuo/wikipedia-zhtw
• The World Wide Web