transybao1393 commited on
Commit ·
b6cfc7b
1
Parent(s): d9bc945
Update README.md
Browse files
README.md
CHANGED
|
@@ -20,9 +20,29 @@ This Space provides an interactive viewer for the **300K English-Vietnamese-Japa
|
|
| 20 |
|
| 21 |
- **Languages:** English, Vietnamese, Japanese
|
| 22 |
- **Size:** 300,000 sentence-level aligned triplets
|
| 23 |
-
- **Source:** Curated from OPUS parallel corpora (CCMatrix, OpenSubtitles, JParaCrawl, WikiMatrix)
|
| 24 |
- **Format:** JSONL with `en`, `vi`, `ja` keys
|
| 25 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
## Usage
|
| 27 |
|
| 28 |
Click the "Sample random triplet" button to view random examples from the dataset.
|
|
|
|
| 20 |
|
| 21 |
- **Languages:** English, Vietnamese, Japanese
|
| 22 |
- **Size:** 300,000 sentence-level aligned triplets
|
|
|
|
| 23 |
- **Format:** JSONL with `en`, `vi`, `ja` keys
|
| 24 |
|
| 25 |
+
## Data Sources
|
| 26 |
+
|
| 27 |
+
This dataset is curated from [OPUS](https://opus.nlpl.eu/) parallel corpora:
|
| 28 |
+
|
| 29 |
+
| Corpus | Language Pair | Description |
|
| 30 |
+
|--------|---------------|-------------|
|
| 31 |
+
| CCMatrix v1 | EN-VI, EN-JA | Web-crawled parallel sentences |
|
| 32 |
+
| OpenSubtitles v2024 | EN-VI | Movie and TV subtitle translations |
|
| 33 |
+
| JParaCrawl v3.0 | EN-JA | Japanese web parallel corpus |
|
| 34 |
+
| WikiMatrix v1 | EN-JA | Wikipedia-extracted translations |
|
| 35 |
+
|
| 36 |
+
## Data Pipeline
|
| 37 |
+
|
| 38 |
+
1. **Collection** - Downloaded parallel sentence pairs from OPUS
|
| 39 |
+
2. **Cleaning** - Removed noise, special characters, and malformed text
|
| 40 |
+
3. **Filtering** - Kept sentences with appropriate length and quality
|
| 41 |
+
4. **Language Verification** - Validated correct languages in each pair
|
| 42 |
+
5. **Deduplication** - Removed duplicate and near-duplicate sentences
|
| 43 |
+
6. **Triplet Creation** - Joined EN-VI and EN-JA pairs using English as pivot
|
| 44 |
+
7. **Final Sampling** - Selected 300K high-quality triplets
|
| 45 |
+
|
| 46 |
## Usage
|
| 47 |
|
| 48 |
Click the "Sample random triplet" button to view random examples from the dataset.
|