transybao1393 commited on
Commit
b6cfc7b
·
1 Parent(s): d9bc945

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +21 -1
README.md CHANGED
@@ -20,9 +20,29 @@ This Space provides an interactive viewer for the **300K English-Vietnamese-Japa
20
 
21
  - **Languages:** English, Vietnamese, Japanese
22
  - **Size:** 300,000 sentence-level aligned triplets
23
- - **Source:** Curated from OPUS parallel corpora (CCMatrix, OpenSubtitles, JParaCrawl, WikiMatrix)
24
  - **Format:** JSONL with `en`, `vi`, `ja` keys
25
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
  ## Usage
27
 
28
  Click the "Sample random triplet" button to view random examples from the dataset.
 
20
 
21
  - **Languages:** English, Vietnamese, Japanese
22
  - **Size:** 300,000 sentence-level aligned triplets
 
23
  - **Format:** JSONL with `en`, `vi`, `ja` keys
24
 
25
+ ## Data Sources
26
+
27
+ This dataset is curated from [OPUS](https://opus.nlpl.eu/) parallel corpora:
28
+
29
+ | Corpus | Language Pair | Description |
30
+ |--------|---------------|-------------|
31
+ | CCMatrix v1 | EN-VI, EN-JA | Web-crawled parallel sentences |
32
+ | OpenSubtitles v2024 | EN-VI | Movie and TV subtitle translations |
33
+ | JParaCrawl v3.0 | EN-JA | Japanese web parallel corpus |
34
+ | WikiMatrix v1 | EN-JA | Wikipedia-extracted translations |
35
+
36
+ ## Data Pipeline
37
+
38
+ 1. **Collection** - Downloaded parallel sentence pairs from OPUS
39
+ 2. **Cleaning** - Removed noise, special characters, and malformed text
40
+ 3. **Filtering** - Kept sentences with appropriate length and quality
41
+ 4. **Language Verification** - Validated correct languages in each pair
42
+ 5. **Deduplication** - Removed duplicate and near-duplicate sentences
43
+ 6. **Triplet Creation** - Joined EN-VI and EN-JA pairs using English as pivot
44
+ 7. **Final Sampling** - Selected 300K high-quality triplets
45
+
46
  ## Usage
47
 
48
  Click the "Sample random triplet" button to view random examples from the dataset.