File size: 5,447 Bytes
f211247 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 | # TLHdig 0.2 — Triple-Parallel Corpus Exploitation
**Zenodo 15459134** (TLHdig Beta 0.2) — 315K Hittite transliteration line.
**Fotoğraf yok AMA cuneiform Unicode + transliteration + phonetic** üçlü paralel corpus.
## Keşif
TLHdig 0.2 dataset'i sadece transliteration değil, **üçlü-paralel** corpus:
```
Cuneiform Unicode: 𒈨𒈾𒄴𒄩𒀭𒁕𒆠𒅖𒊭
Transliteration: me-na-aḫ-ḫa-an-da ki-iš-ša-
Phonetic: menaḫḫanta kišša~
```
Bu 3 modalite arasında **doğrudan supervised training** yapılabilir — fotoğraf **gerekmeden**.
## Üretilen corpora (`datasets/processed/tlhdig_corpora/`)
| Corpus | Kayıt | Kullanım |
|---|---:|---|
| `cuneiform_to_transliteration.jsonl` | 315,108 | Seq2seq: Unicode → ABZ transliteration |
| `hittite_parallel.jsonl` | **280,624** | ⭐ Hitit-only parallel (gold) |
| `transliteration_to_phonetic.jsonl` | 279,168 | Transliteration → Pronunciation |
| `cuneiform_lm.jsonl` | 347,015 | Cuneiform Unicode LM |
| `transliteration_lm.jsonl` | 341,409 | KenLM 5-gram input |
| `phonetic_lm.jsonl` | 279,168 | Phonetic LM |
| `language_id.jsonl` | 341,409 | Hit/Akk/Hur/Hat/Luw/Sum classification |
| `sign_abz_pairs.jsonl` | 2,742,928 | Cuneiform char frequency + ABZ discovery |
**Hittite cuneiform unique chars**: 337 (ABZ sign repertoire doğrulanabilir).
Top-10 Hitit sign'ı:
```
𒀭 (AN/DINGIR) 176,832 occurrence
𒀀 (A) 116,937
𒀸 (AŠ) 76,579
𒈾 (NA) 76,167
𒍣 (ZI) 73,829
𒈠 (MA) 60,485
𒉡 (NU) 52,674
𒄿 (I) 51,676
𒊭 (ŠA) 47,490
𒅀 (IA) 46,857
```
## 5 Training Signal
### 1. Cuneiform Unicode → Transliteration (seq2seq)
- **Input**: Cuneiform character sequence `𒈨𒈾𒄴...`
- **Output**: Transliteration `me-na-aḫ-...`
- **Train size**: 280,624 Hitit pairs
- **Model**: ByT5 char-to-char (no image required!)
- **Kullanım**:
- Seq2seq decoder initialization
- Image-to-text seq2seq pipeline'ın text-to-text pretraining
- Downstream: image tanıma yapılamıyorsa text yoluyla hatasız
### 2. Transliteration → Phonetic
- **Input**: `me-na-aḫ-ḫa-an-da`
- **Output**: `menaḫḫanta`
- **Train size**: 279,168
- **Kullanım**: Morphological analysis, lemmatization
### 3. Cuneiform LM (char-level)
- **Size**: 315K line × avg 20 char = 6M+ char
- **ByT5 / GPT-2 small** ön-eğitim
- **Kullanım**: Sign sequence priors (bigram/trigram probabilities)
### 4. Transliteration LM (KenLM 5-gram)
- **Input data**: 341K line transliteration
- **Output**: `hitit_kenlm_5gram.binary`
- **Kullanım**: Viterbi lattice rescoring (Stage 4, v3/v4)
### 5. Sign ABZ Discovery
- 337 Hitit cuneiform char → ABZ code mapping
- Unicode U+12000-U+1254F range
- Frequency-weighted: top 100 sign = %95+ coverage
## Entegrasyon — pipeline v4'te nerede
### Stage 2 (Classification) desteği
- Cuneiform char frequency → class priors
- ABZ class label mapping (unified_label) verify
### Stage 4 (Transliteration) ana motor
- ByT5 model initialization: Hittite LM pretrain
- KenLM 5-gram ready
- Phonetic reading output (bonus)
### Seq2seq pipeline (BEYOND_95)
- **Pretraining**: 280K cuneiform↔transliteration (text-only)
- **Fine-tuning**: 83 Hitit tablet image-text pair (from build_pairs.py)
- **Inference**: image → transliteration direct
## Faydalar (realistic gain)
| Kullanım | Beklenen gain | Maliyet |
|---|---|---|
| ByT5 Hitit LM pretrain (char-level) | +2-4% seq2seq | 1 GPU hafta |
| KenLM 5-gram Viterbi rescoring | +1-3% (v4 config'de hedef) | 1 CPU gün |
| Cuneiform char frequency prior | +0.3-0.5% | 0 |
| Phonetic reading side-task | +0.3-0.5% multi-task | marjinal |
| Language ID auxiliary | filtering only | 0 |
**Toplam realistic**: +3-5% downstream (seq2seq paradigm + LM rescoring dahil).
## Özel değer
- **CC-BY-4.0** (free for research + commercial with attribution)
- **280K parallel pair** — cuneiform OCR için rekor
- **Foto gerekmiyor** — hemen başlatılabilir
- **Würzburg authoritative** (Schwemer grubu)
- **98%+ published Hittite fragments** coverage
## Hemen yapılabilecek (veri bizde)
```bash
# 1. KenLM 5-gram Hitit LM
python3 -c "
import json
with open('datasets/processed/tlhdig_corpora/transliteration_lm.jsonl') as f, \
open('/tmp/hitit_translit.txt', 'w') as g:
for line in f:
r = json.loads(line)
if r.get('lang') == 'Hit':
g.write(r['text'] + '\n')
"
# Pip install via GitHub: pip install https://github.com/kpu/kenlm/archive/master.zip
# kenlm/bin/lmplz -o 5 < /tmp/hitit_translit.txt > hitit.arpa
# kenlm/bin/build_binary hitit.arpa hitit_kenlm_5gram.binary
# 2. ByT5 pretrain — 280K cuneiform→translit
# Training script: hitit_ocr/src/seq2seq/train_text_only.py (TODO)
# 3. Cuneiform char embedding (supervised)
# Each of 337 unique cuneiform chars → embedding via usage context
```
## Status
- ✅ TLHdig 0.2 zaten indirildi (datasets/sources/tlhdig/)
- ✅ 8 corpus türetildi (datasets/processed/tlhdig_corpora/)
- ✅ Hitit parallel 280K kayıt hazır
- ⏳ ByT5 pretraining script TODO
- ⏳ KenLM binary generation TODO (pip install kenlm)
## Referans
- [Zenodo TLHdig 0.2](https://zenodo.org/records/15459134)
- License: CC-BY-4.0
- Authors: Müller, Prechel, Rieken, Schwemer (Würzburg/Mainz/Marburg)
- Web: https://www.hethport.uni-wuerzburg.de/TLHdig/
|