RohanMuralidharan commited on
Commit
3fa8f3e
ยท
verified ยท
1 Parent(s): ea13bfe

Upload 10 files

Browse files
.gitattributes ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ pytorch_model.bin filter=lfs diff=lfs merge=lfs -text
2
+ sentencepiece.bpe.model filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Rohan Muralidharan
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
README.md ADDED
@@ -0,0 +1,383 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - asm
4
+ - ben
5
+ - brx
6
+ - doi
7
+ - gom
8
+ - guj
9
+ - hin
10
+ - kan
11
+ - kas
12
+ - mai
13
+ - mal
14
+ - mar
15
+ - mni
16
+ - npi
17
+ - ory
18
+ - pan
19
+ - san
20
+ - sat
21
+ - snd
22
+ - tam
23
+ - tel
24
+ - urd
25
+ - eng
26
+ license: mit
27
+ library_name: transformers
28
+ pipeline_tag: text2text-generation
29
+ tags:
30
+ - translation
31
+ - multilingual
32
+ - indic
33
+ - mbart
34
+ - seq2seq
35
+ - huggingface
36
+ ---
37
+
38
+ # Transync
39
+
40
+ [![Hugging Face Hub](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue)](https://huggingface.co/RohanMuralidharan/transync)
41
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
42
+ [![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
43
+ [![PyTorch](https://img.shields.io/badge/PyTorch-%23EE4C2C.svg?style=flat&logo=pytorch&logoColor=white)](https://pytorch.org/)
44
+
45
+ ## Model Overview
46
+
47
+ Transync is a multilingual translation model based on the MBart architecture, designed for offline neural machine translation. This repository provides inference capabilities for translating between 50+ languages including all major Indian languages (Hindi, Telugu, Tamil, Bengali, Gujarati, Marathi, Punjabi, Urdu, and more).
48
+
49
+ ## Features
50
+
51
+ - โœ… **Multilingual Support**: 50+ languages including all major Indian languages
52
+ - โœ… **High-Quality Translations**: Based on the MBart architecture
53
+ - โœ… **Offline Operation**: Works without internet connection
54
+ - โœ… **Efficient Inference**: Optimized for both CPU and GPU
55
+ - โœ… **Batch Processing**: Supports batch translation for efficiency
56
+ - โœ… **CLI Interface**: Command-line interface for easy usage
57
+
58
+ ## Supported Languages
59
+
60
+ The model supports 50+ languages including:
61
+
62
+ ### Indian Languages
63
+ - Assamese (asm)
64
+ - Bengali (ben)
65
+ - Bodo (brx)
66
+ - Dogri (doi)
67
+ - Konkani (gom)
68
+ - Gujarati (guj)
69
+ - Hindi (hin)
70
+ - Kannada (kan)
71
+ - Kashmiri (kas)
72
+ - Maithili (mai)
73
+ - Malayalam (mal)
74
+ - Marathi (mar)
75
+ - Manipuri (mni)
76
+ - Nepali (npi)
77
+ - Odia (ory)
78
+ - Punjabi (pan)
79
+ - Sanskrit (san)
80
+ - Santali (sat)
81
+ - Sindhi (snd)
82
+ - Tamil (tam)
83
+ - Telugu (tel)
84
+ - Urdu (urd)
85
+
86
+ ### Other Languages
87
+ - Arabic (ara)
88
+ - Czech (ces)
89
+ - German (deu)
90
+ - Spanish (spa)
91
+ - Estonian (est)
92
+ - Finnish (fin)
93
+ - French (fra)
94
+ - Hebrew (heb)
95
+ - Croatian (hrv)
96
+ - Indonesian (ind)
97
+ - Italian (ita)
98
+ - Japanese (jpn)
99
+ - Georgian (kat)
100
+ - Kazakh (kaz)
101
+ - Khmer (khm)
102
+ - Korean (kor)
103
+ - Lithuanian (lit)
104
+ - Latvian (lav)
105
+ - Macedonian (mkd)
106
+ - Mongolian (mon)
107
+ - Burmese (mya)
108
+ - Dutch (nld)
109
+ - Polish (pol)
110
+ - Pashto (pus)
111
+ - Portuguese (por)
112
+ - Romanian (ron)
113
+ - Russian (rus)
114
+ - Sinhala (sin)
115
+ - Slovak (slk)
116
+ - Swedish (swe)
117
+ - Swahili (swa)
118
+ - Thai (tha)
119
+ - Tagalog (tgl)
120
+ - Turkish (tur)
121
+ - Ukrainian (ukr)
122
+ - Vietnamese (vie)
123
+ - Xhosa (xho)
124
+ - Chinese (zho)
125
+ - Azerbaijani (aze)
126
+ - Persian (fas)
127
+ - Galician (glg)
128
+ - Afrikaans (afr)
129
+
130
+ ## Installation
131
+
132
+ Install the required dependencies:
133
+
134
+ ```bash
135
+ pip install -r requirements.txt
136
+ ```
137
+
138
+ ## Quick Start
139
+
140
+ ```python
141
+ from transync_inference import translate_onemt
142
+
143
+ # Translate English to Hindi
144
+ result = translate_onemt("Hello, how are you?", "eng", "hin")
145
+ print(result) # เคจเคฎเคธเฅเคคเฅ‡, เค†เคช เค•เฅˆเคธเฅ‡ เคนเฅˆเค‚?
146
+ ```
147
+
148
+ ## Python Example
149
+
150
+ ```python
151
+ from transync_inference import translate_onemt, translate_batch
152
+
153
+ # Single translation
154
+ translation = translate_onemt(
155
+ text="Good morning!",
156
+ source_lang="eng",
157
+ target_lang="hin"
158
+ )
159
+ print(translation)
160
+
161
+ # Batch translation
162
+ texts = ["Hello", "How are you?", "Goodbye"]
163
+ results = translate_batch(
164
+ texts=texts,
165
+ source_lang="eng",
166
+ target_lang="hin"
167
+ )
168
+ print(results)
169
+ ```
170
+
171
+ ## Transformers Example
172
+
173
+ ```python
174
+ from transformers import MBartForConditionalGeneration, MBart50Tokenizer
175
+
176
+ # Load model and tokenizer
177
+ model = MBartForConditionalGeneration.from_pretrained("RohanMuralidharan/transync")
178
+ tokenizer = MBart50Tokenizer.from_pretrained("RohanMuralidharan/transync")
179
+
180
+ # Set source and target languages
181
+ tokenizer.src_lang = "en_XX"
182
+ target_lang = "hi_IN"
183
+
184
+ # Encode input text
185
+ input_ids = tokenizer("Hello, how are you?", return_tensors="pt").input_ids
186
+
187
+ # Generate translation
188
+ with torch.no_grad():
189
+ outputs = model.generate(
190
+ input_ids,
191
+ forced_bos_token_id=tokenizer.lang_code_to_id[target_lang]
192
+ )
193
+
194
+ # Decode translation
195
+ translated = tokenizer.decode(outputs[0], skip_special_tokens=True)
196
+ print(translated)
197
+ ```
198
+
199
+ ## CLI Example
200
+
201
+ ```bash
202
+ # Translate single text
203
+ python transync_inference.py eng hin "Hello, how are you?"
204
+
205
+ # Batch translation from file
206
+ python transync_inference.py --batch eng hin -f input.txt -o output.txt
207
+ ```
208
+
209
+ ## Repository Structure
210
+
211
+ ```
212
+ transync/
213
+ โ”œโ”€โ”€ config.json # Model configuration
214
+ โ”œโ”€โ”€ generation_config.json # Generation configuration
215
+ โ”œโ”€โ”€ pytorch_model.bin # Model weights
216
+ โ”œโ”€โ”€ sentencepiece.bpe.model # SentencePiece tokenizer model
217
+ โ”œโ”€โ”€ tokenizer_config.json # Tokenizer configuration
218
+ โ”œโ”€โ”€ special_tokens_map.json # Special tokens mapping
219
+ โ”œโ”€โ”€ README.md # This file
220
+ โ”œโ”€โ”€ LICENSE # MIT License
221
+ โ”œโ”€โ”€ requirements.txt # Dependencies
222
+ โ””โ”€โ”€ transync_inference.py # Inference code
223
+ ```
224
+
225
+ ## Model Architecture
226
+
227
+ This model is built on the MBart (Multilingual BART) architecture, which is a sequence-to-sequence model pre-trained on multilingual data. It leverages the power of BART's bidirectional encoder and autoregressive decoder for translation tasks.
228
+
229
+ ## Tokenizer
230
+
231
+ The model uses SentencePiece tokenizer for subword tokenization. The tokenizer is compatible with the MBart50 tokenizer format and supports 50+ languages.
232
+
233
+ ## Intended Uses
234
+
235
+ - Offline multilingual translation
236
+ - Research and educational purposes
237
+ - Integration into translation applications
238
+ - Batch processing of text translations
239
+
240
+ ## Out-of-Scope Uses
241
+
242
+ - Training new models from scratch (this is a pre-trained model)
243
+ - Commercial use without proper licensing (please check license terms)
244
+ - Use in production systems without proper testing and validation
245
+
246
+ ## Limitations
247
+
248
+ - Translation quality depends on training data coverage
249
+ - May struggle with domain-specific terminology
250
+ - Performance varies between languages
251
+ - Requires sufficient computational resources for optimal performance
252
+
253
+ ## Ethical Considerations
254
+
255
+ This model is intended for educational and research purposes. Users should be aware of the following considerations:
256
+
257
+ - Translation accuracy may vary depending on language pair and domain
258
+ - The model should not be used to generate misleading or harmful content
259
+ - Users should respect copyright and intellectual property rights when using translations
260
+ - The model's training data sources and limitations should be understood
261
+
262
+ ## Hardware Requirements
263
+
264
+ - CPU: Minimum 4GB RAM, Recommended 8GB+
265
+ - GPU: NVIDIA GPU with CUDA support (recommended for faster inference)
266
+ - Storage: At least 2.4GB of storage space for model weights
267
+
268
+ ## Performance Notes
269
+
270
+ - CPU inference is suitable for small-scale tasks
271
+ - GPU acceleration significantly improves translation speed
272
+ - Batch processing is more efficient than individual translations
273
+ - Translation quality may vary between language pairs
274
+
275
+ ## License
276
+
277
+ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
278
+
279
+ This model is based on the MBart architecture and uses pre-trained weights from the Hugging Face ecosystem. The model weights are not included in this repository and must be downloaded separately from the Hugging Face Hub.
280
+ **Option 1: Direct Download**
281
+ 1. Go to: https://huggingface.co/RohanMuralidharan/Transync
282
+ 2. Download `pytorch_model.bin` (2.44 GB)
283
+ 3. Place it in the project root directory
284
+
285
+ **Option 2: Using Python**
286
+ ```python
287
+ from huggingface_hub import hf_hub_download
288
+ hf_hub_download(
289
+ repo_id='RohanMuralidharan/Transync',
290
+ filename='pytorch_model.bin',
291
+ local_dir='.'
292
+ )
293
+ ```
294
+
295
+ **Option 3: Using huggingface-cli**
296
+ ```bash
297
+ huggingface-cli download RohanMuralidharan/Transync pytorch_model.bin
298
+ ```
299
+
300
+ ---
301
+
302
+ ### Command Line Usage
303
+
304
+ ```bash
305
+ python transync_inference.py eng hin "Hello, how are you?"
306
+ ```
307
+
308
+ ---
309
+
310
+ ## Supported Languages
311
+
312
+ | Short Code | Language | Script | MBart Code |
313
+ |---|---|---|---|
314
+ | `eng` | English | Latin | `en_XX` |
315
+ | `hin` | Hindi | Devanagari | `hi_IN` |
316
+ | `tel` | Telugu | Telugu | `te_IN` |
317
+ | `tam` | Tamil | Tamil | `ta_IN` |
318
+ | `mal` | Malayalam | Malayalam | `ml_IN` |
319
+ | `kan` | Kannada | Kannada | `kn_IN` |
320
+ | `ben` | Bengali | Bengali | `bn_IN` |
321
+ | `guj` | Gujarati | Gujarati | `gu_IN` |
322
+ | `mar` | Marathi | Devanagari | `mr_IN` |
323
+ | `pan` | Punjabi | Gurmukhi | `pa_IN` |
324
+ | `urd` | Urdu | Arabic | `ur_PK` |
325
+ | `asm` | Assamese | Bengali | `as_IN` |
326
+ | `npi` | Nepali | Devanagari | `ne_NP` |
327
+ | `ory` | Odia | Odia | `or_IN` |
328
+ | `san` | Sanskrit | Devanagari | `sa_IN` |
329
+ | `mai` | Maithili | Devanagari | `mai_IN` |
330
+ | `brx` | Bodo | Devanagari | `brx_IN` |
331
+ | `doi` | Dogri | Devanagari | `doi_IN` |
332
+ | `gom` | Konkani | Devanagari | `gom_IN` |
333
+ | `mni` | Meitei | Bengali | `mni_IN` |
334
+ | `sat` | Santali | Ol Chiki | `sat_IN` |
335
+ | `kas` | Kashmiri | Arabic | `ks_IN` |
336
+ | `snd` | Sindhi | Arabic | `sd_IN` |
337
+
338
+ ---
339
+
340
+ ## How It Works
341
+
342
+ Transync uses standard MBart tokenization with SentencePiece:
343
+
344
+ ```
345
+ source text
346
+ โ†’ Add language tag: [en_XX] Hello, how are you?
347
+ โ†’ SentencePiece encoding โ†’ subword pieces
348
+ โ†’ MBart encoder
349
+ ```
350
+
351
+ The model uses the standard MBart50 tokenizer with language codes for high-quality multilingual translation.
352
+
353
+ ---
354
+
355
+ ## Repository Files
356
+
357
+ | File | Description |
358
+ |---|---|
359
+ | `pytorch_model.bin` | Model weights (~2.4 GB) |
360
+ | `config.json` | Model architecture config |
361
+ | `generation_config.json` | Generation parameters |
362
+ | `sentencepiece.bpe.model` | SentencePiece tokenizer |
363
+ | `tokenizer_config.json` | Tokenizer config |
364
+ | `special_tokens_map.json` | Special tokens mapping |
365
+ | `transync_inference.py` | Main inference script |
366
+
367
+ ---
368
+
369
+ ## Verification
370
+
371
+ Test the model with:
372
+ ```bash
373
+ python transync_inference.py eng hin "Hello, how are you?"
374
+ ```
375
+
376
+ Expected output: `เคจเคฎเคธเฅเคคเฅ‡, เค†เคช เค•เฅˆเคธเฅ‡ เคนเฅˆเค‚?`
377
+
378
+ ---
379
+
380
+ ## License
381
+
382
+ MIT
383
+
config.json ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "facebook/mbart-large-50-one-to-many-mmt",
3
+ "_num_labels": 3,
4
+ "activation_dropout": 0.0,
5
+ "activation_function": "relu",
6
+ "add_bias_logits": false,
7
+ "add_final_layer_norm": true,
8
+ "architectures": [
9
+ "MBartForConditionalGeneration"
10
+ ],
11
+ "attention_dropout": 0.0,
12
+ "bos_token_id": 0,
13
+ "classif_dropout": 0.0,
14
+ "classifier_dropout": 0.0,
15
+ "d_model": 1024,
16
+ "decoder_attention_heads": 16,
17
+ "decoder_ffn_dim": 4096,
18
+ "decoder_layerdrop": 0.0,
19
+ "decoder_layers": 12,
20
+ "decoder_start_token_id": 2,
21
+ "dropout": 0.1,
22
+ "early_stopping": true,
23
+ "encoder_attention_heads": 16,
24
+ "encoder_ffn_dim": 4096,
25
+ "encoder_layerdrop": 0.0,
26
+ "encoder_layers": 12,
27
+ "eos_token_id": 2,
28
+ "forced_eos_token_id": 2,
29
+ "gradient_checkpointing": false,
30
+ "id2label": {
31
+ "0": "LABEL_0",
32
+ "1": "LABEL_1",
33
+ "2": "LABEL_2"
34
+ },
35
+ "init_std": 0.02,
36
+ "is_encoder_decoder": true,
37
+ "label2id": {
38
+ "LABEL_0": 0,
39
+ "LABEL_1": 1,
40
+ "LABEL_2": 2
41
+ },
42
+ "max_length": 200,
43
+ "max_position_embeddings": 1024,
44
+ "model_type": "mbart",
45
+ "normalize_before": true,
46
+ "normalize_embedding": true,
47
+ "num_beams": 5,
48
+ "num_hidden_layers": 12,
49
+ "output_past": true,
50
+ "pad_token_id": 1,
51
+ "scale_embedding": true,
52
+ "static_position_embeddings": false,
53
+ "tokenizer_class": "MBart50Tokenizer",
54
+ "transformers_version": "4.44.0",
55
+ "use_cache": true,
56
+ "vocab_size": 250054
57
+ }
generation_config.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 0,
4
+ "decoder_start_token_id": 2,
5
+ "early_stopping": true,
6
+ "eos_token_id": 2,
7
+ "forced_eos_token_id": 2,
8
+ "max_length": 200,
9
+ "num_beams": 5,
10
+ "pad_token_id": 1,
11
+ "transformers_version": "4.44.0"
12
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:51fbf92efa747400b06d368d5c86b79e4a67799c754bf2c2a81f0bf210efe4c5
3
+ size 2444714899
requirements.txt ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ torch>=2.3.1
2
+ transformers>=4.57.6
3
+ sentencepiece>=0.2.0
4
+ accelerate>=1.12.0
5
+ safetensors>=0.7.0
6
+ tokenizers>=0.20.3
7
+ pyarrow>=23.0.0
sentencepiece.bpe.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cfc8146abe2a0488e9e2a0c56de7952f7c11ab059eca145a0a727afce0db2865
3
+ size 5069051
special_tokens_map.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "eos_token": "</s>",
3
+ "unk_token": "<unk>",
4
+ "sep_token": "</s>",
5
+ "pad_token": "<pad>",
6
+ "cls_token": "<s>",
7
+ "mask_token": {
8
+ "content": "<mask>",
9
+ "single_word": false,
10
+ "lstrip": true,
11
+ "rstrip": false,
12
+ "normalized": true
13
+ },
14
+ "additional_special_tokens": [
15
+ "af_ZA", "ar_AR", "az_AZ", "bn_IN", "cs_CZ", "de_DE", "en_XX", "es_XX",
16
+ "et_EE", "fa_IR", "fi_FI", "fr_XX", "gl_ES", "gu_IN", "he_IL", "hi_IN",
17
+ "hr_HR", "id_ID", "it_IT", "ja_XX", "ka_GE", "kk_KZ", "km_KH", "ko_KR",
18
+ "lt_LT", "lv_LV", "mk_MK", "ml_IN", "mn_MN", "mr_IN", "my_MM", "ne_NP",
19
+ "nl_XX", "pl_PL", "ps_AF", "pt_XX", "ro_RO", "ru_RU", "si_LK", "sl_SI",
20
+ "sv_SE", "sw_KE", "ta_IN", "te_IN", "th_TH", "tl_XX", "tr_TR", "uk_UA",
21
+ "ur_PK", "vi_VN", "xh_ZA", "zh_CN"
22
+ ]
23
+ }
tokenizer_config.json ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "src_lang": null,
3
+ "tgt_lang": null,
4
+ "eos_token": "</s>",
5
+ "unk_token": "<unk>",
6
+ "sep_token": "</s>",
7
+ "cls_token": "<s>",
8
+ "pad_token": "<pad>",
9
+ "mask_token": {
10
+ "content": "<mask>",
11
+ "single_word": false,
12
+ "lstrip": true,
13
+ "rstrip": false,
14
+ "normalized": true,
15
+ "__type": "AddedToken"
16
+ },
17
+ "bos_token": "<s>",
18
+ "tokenizer_file": null,
19
+ "language_codes": "ML50",
20
+ "special_tokens_map_file": "special_tokens_map.json",
21
+ "name_or_path": "facebook/mbart-large-50-one-to-many-mmt"
22
+ }
transync_inference.py ADDED
@@ -0,0 +1,509 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ""
2
+ Transync - Indic Multilingual Translation Inference
3
+ Uses facebook/mbart-large-50-one-to-many-mmt model
4
+ Supports 50+ languages including all major Indian languages
5
+ """
6
+
7
+ import sys
8
+ import io
9
+ import torch
10
+ from transformers import MBartForConditionalGeneration, MBart50Tokenizer
11
+
12
+ # Fix Windows console encoding
13
+ if sys.platform == 'win32':
14
+ sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8')
15
+
16
+ # Language code mapping (short code to MBart format)
17
+ LANG_CODES = {
18
+ 'eng': 'en_XX',
19
+ 'hin': 'hi_IN',
20
+ 'tel': 'te_IN',
21
+ 'tam': 'ta_IN',
22
+ 'mal': 'ml_IN',
23
+ 'kan': 'kn_IN',
24
+ 'ben': 'bn_IN',
25
+ 'guj': 'gu_IN',
26
+ 'mar': 'mr_IN',
27
+ 'pan': 'pa_IN',
28
+ 'urd': 'ur_PK',
29
+ 'asm': 'as_IN',
30
+ 'npi': 'ne_NP',
31
+ 'ory': 'or_IN',
32
+ 'san': 'sa_IN',
33
+ 'mai': 'mai_IN',
34
+ 'brx': 'brx_IN',
35
+ 'doi': 'doi_IN',
36
+ 'gom': 'gom_IN',
37
+ 'mni': 'mni_IN',
38
+ 'sat': 'sat_IN',
39
+ 'kas': 'ks_IN',
40
+ 'snd': 'sd_IN',
41
+ # Additional ML50 languages
42
+ 'ara': 'ar_AR',
43
+ 'ces': 'cs_CZ',
44
+ 'deu': 'de_DE',
45
+ 'spa': 'es_XX',
46
+ 'est': 'et_EE',
47
+ 'fin': 'fi_FI',
48
+ 'fra': 'fr_XX',
49
+ 'heb': 'he_IL',
50
+ 'hrv': 'hr_HR',
51
+ 'ind': 'id_ID',
52
+ 'ita': 'it_IT',
53
+ 'jpn': 'ja_XX',
54
+ 'kat': 'ka_GE',
55
+ 'kaz': 'kk_KZ',
56
+ 'khm': 'km_KH',
57
+ 'kor': 'ko_KR',
58
+ 'lit': 'lt_LT',
59
+ 'lav': 'lv_LV',
60
+ 'mkd': 'mk_MK',
61
+ 'mon': 'mn_MN',
62
+ 'mya': 'my_MM',
63
+ 'nld': 'nl_XX',
64
+ 'pol': 'pl_PL',
65
+ 'pus': 'ps_AF',
66
+ 'por': 'pt_XX',
67
+ 'ron': 'ro_RO',
68
+ 'rus': 'ru_RU',
69
+ 'sin': 'si_LK',
70
+ 'slk': 'sl_SI',
71
+ 'swe': 'sv_SE',
72
+ 'swa': 'sw_KE',
73
+ 'tha': 'th_TH',
74
+ 'tgl': 'tl_XX',
75
+ 'tur': 'tr_TR',
76
+ 'ukr': 'uk_UA',
77
+ 'vie': 'vi_VN',
78
+ 'xho': 'xh_ZA',
79
+ 'zho': 'zh_CN',
80
+ 'aze': 'az_AZ',
81
+ 'fas': 'fa_IR',
82
+ 'glg': 'gl_ES',
83
+ 'afr': 'af_ZA',
84
+ }
85
+
86
+ # Reverse mapping for display
87
+ CODE_TO_LANG = {
88
+ 'eng': 'English', 'hin': 'Hindi', 'tel': 'Telugu', 'tam': 'Tamil',
89
+ 'mal': 'Malayalam', 'kan': 'Kannada', 'ben': 'Bengali', 'guj': 'Gujarati',
90
+ 'mar': 'Marathi', 'pan': 'Punjabi', 'urd': 'Urdu', 'asm': 'Assamese',
91
+ 'npi': 'Nepali', 'ory': 'Odia', 'san': 'Sanskrit', 'mai': 'Maithili',
92
+ 'brx': 'Bodo', 'doi': 'Dogri', 'gom': 'Konkani', 'mni': 'Manipuri',
93
+ 'sat': 'Santali', 'kas': 'Kashmiri', 'snd': 'Sindhi',
94
+ 'ara': 'Arabic', 'ces': 'Czech', 'deu': 'German', 'spa': 'Spanish',
95
+ 'est': 'Estonian', 'fin': 'Finnish', 'fra': 'French', 'heb': 'Hebrew',
96
+ 'hrv': 'Croatian', 'ind': 'Indonesian', 'ita': 'Italian', 'jpn': 'Japanese',
97
+ 'kat': 'Georgian', 'kaz': 'Kazakh', 'khm': 'Khmer', 'kor': 'Korean',
98
+ 'lit': 'Lithuanian', 'lav': 'Latvian', 'mkd': 'Macedonian', 'mon': 'Mongolian',
99
+ 'mya': 'Burmese', 'nld': 'Dutch', 'pol': 'Polish', 'pus': 'Pashto',
100
+ 'por': 'Portuguese', 'ron': 'Romanian', 'rus': 'Russian', 'sin': 'Sinhala',
101
+ 'slk': 'Slovak', 'swe': 'Swedish', 'swa': 'Swahili', 'tha': 'Thai',
102
+ 'tgl': 'Tagalog', 'tur': 'Turkish', 'ukr': 'Ukrainian', 'vie': 'Vietnamese',
103
+ 'xho': 'Xhosa', 'zho': 'Chinese', 'aze': 'Azerbaijani', 'fas': 'Persian',
104
+ 'glg': 'Galician', 'afr': 'Afrikaans',
105
+ }
106
+
107
+ # Load model and tokenizer (cached after first load)
108
+ _model = None
109
+ _tokenizer = None
110
+ _device = None
111
+
112
+
113
+ def _get_device() -> str:
114
+ """Detect and return the best available device (CUDA/CPU)."""
115
+ global _device
116
+ if _device is None:
117
+ if torch.cuda.is_available():
118
+ _device = "cuda"
119
+ print(f"โœ“ Using GPU: {torch.cuda.get_device_name(0)}")
120
+ else:
121
+ _device = "cpu"
122
+ print("โ„น Using CPU (CUDA not available)")
123
+ return _device
124
+
125
+
126
+ def _load_model():
127
+ """Lazy load model and tokenizer with device optimization."""
128
+ global _model, _tokenizer
129
+ if _model is None:
130
+ device = _get_device()
131
+ print("Loading Transync model...")
132
+ _model = MBartForConditionalGeneration.from_pretrained('.').to(device)
133
+ _tokenizer = MBart50Tokenizer.from_pretrained('.')
134
+ if device == "cuda":
135
+ _model = _model.half() # Use FP16 for faster inference on GPU
136
+ print("โœ“ Model ready")
137
+ return _model, _tokenizer
138
+
139
+
140
+ def translate_onemt(
141
+ text: str,
142
+ source_lang: str,
143
+ target_lang: str,
144
+ max_length: int = 256,
145
+ num_beams: int = 5,
146
+ temperature: float = 1.0,
147
+ repetition_penalty: float = 1.3,
148
+ no_repeat_ngram_size: int = 3,
149
+ ) -> str:
150
+ """
151
+ Translate text from source language to target language.
152
+
153
+ Args:
154
+ text: Input text to translate.
155
+ source_lang: Source language code (e.g., 'eng', 'hin', 'tel').
156
+ target_lang: Target language code (e.g., 'eng', 'hin', 'tel').
157
+ max_length: Maximum length of generated translation.
158
+ num_beams: Number of beams for beam search.
159
+ temperature: Sampling temperature (higher = more diverse).
160
+ repetition_penalty: Penalty for repeating tokens.
161
+ no_repeat_ngram_size: Size of n-grams to avoid repeating.
162
+
163
+ Returns:
164
+ Translated text.
165
+
166
+ Raises:
167
+ ValueError: If an unsupported language code is provided.
168
+
169
+ Example:
170
+ >>> translate_onemt("Hello, how are you?", "eng", "hin")
171
+ 'เคจเคฎเคธเฅเคคเฅ‡, เค†เคช เค•เฅˆเคธเฅ‡ เคนเฅˆเค‚?'
172
+ """
173
+ if not text or not text.strip():
174
+ return ""
175
+
176
+ model, tokenizer = _load_model()
177
+
178
+ # Get MBart language codes
179
+ src_code = LANG_CODES.get(source_lang, source_lang)
180
+ tgt_code = LANG_CODES.get(target_lang, target_lang)
181
+
182
+ # Validate source language
183
+ if src_code not in tokenizer.lang_code_to_id:
184
+ valid_codes = sorted(LANG_CODES.keys())
185
+ raise ValueError(
186
+ f"Unsupported source language: '{source_lang}'. "
187
+ f"Supported codes: {', '.join(valid_codes)}"
188
+ )
189
+
190
+ # Validate target language
191
+ tgt_token_id = tokenizer.lang_code_to_id.get(tgt_code)
192
+ if tgt_token_id is None:
193
+ valid_codes = sorted(LANG_CODES.keys())
194
+ raise ValueError(
195
+ f"Unsupported target language: '{target_lang}'. "
196
+ f"Supported codes: {', '.join(valid_codes)}"
197
+ )
198
+
199
+ # Set source language and tokenize
200
+ tokenizer.src_lang = src_code
201
+ inputs = tokenizer(
202
+ text,
203
+ return_tensors="pt",
204
+ truncation=True,
205
+ max_length=max_length,
206
+ padding=True,
207
+ ).to(_device)
208
+
209
+ # Generate translation
210
+ with torch.no_grad():
211
+ outputs = model.generate(
212
+ **inputs,
213
+ forced_bos_token_id=tgt_token_id,
214
+ max_length=max_length,
215
+ num_beams=num_beams,
216
+ no_repeat_ngram_size=no_repeat_ngram_size,
217
+ repetition_penalty=repetition_penalty,
218
+ temperature=temperature,
219
+ early_stopping=True,
220
+ )
221
+
222
+ # Decode
223
+ translated = tokenizer.decode(outputs[0], skip_special_tokens=True)
224
+ return translated
225
+
226
+
227
+ def translate_batch(
228
+ texts: list,
229
+ source_lang: str,
230
+ target_lang: str,
231
+ batch_size: int = 32,
232
+ max_length: int = 256,
233
+ num_beams: int = 5,
234
+ show_progress: bool = True,
235
+ ) -> list:
236
+ """
237
+ Translate a batch of texts efficiently using optimized batching.
238
+
239
+ Args:
240
+ texts: List of input texts to translate.
241
+ source_lang: Source language code.
242
+ target_lang: Target language code.
243
+ batch_size: Number of texts to process at once (default: 32).
244
+ max_length: Maximum length of generated translation.
245
+ num_beams: Number of beams for beam search.
246
+ show_progress: Whether to show a progress bar.
247
+
248
+ Returns:
249
+ List of translated texts.
250
+
251
+ Example:
252
+ >>> translate_batch(["Hello", "How are you?"], "eng", "hin")
253
+ ['เคจเคฎเคธเฅเคคเฅ‡', 'เค†เคช เค•เฅˆเคธเฅ‡ เคนเฅˆเค‚?']
254
+ """
255
+ if not texts:
256
+ return []
257
+
258
+ model, tokenizer = _load_model()
259
+ tgt_code = LANG_CODES.get(target_lang, target_lang)
260
+ tgt_token_id = tokenizer.lang_code_to_id.get(tgt_code)
261
+
262
+ if tgt_token_id is None:
263
+ raise ValueError(f"Unsupported target language: {target_lang}")
264
+
265
+ results = []
266
+ total_batches = (len(texts) + batch_size - 1) // batch_size
267
+
268
+ if show_progress:
269
+ try:
270
+ from tqdm import tqdm
271
+ iterator = tqdm(
272
+ range(0, len(texts), batch_size),
273
+ desc="Translating",
274
+ unit="batch",
275
+ total=total_batches,
276
+ )
277
+ except ImportError:
278
+ iterator = range(0, len(texts), batch_size)
279
+ print(f"Translating {len(texts)} texts in {total_batches} batches...")
280
+ else:
281
+ iterator = range(0, len(texts), batch_size)
282
+
283
+ tokenizer.src_lang = LANG_CODES.get(source_lang, source_lang)
284
+
285
+ for i in iterator:
286
+ batch_texts = texts[i:i + batch_size]
287
+
288
+ # Tokenize batch
289
+ inputs = tokenizer(
290
+ batch_texts,
291
+ return_tensors="pt",
292
+ truncation=True,
293
+ max_length=max_length,
294
+ padding=True,
295
+ ).to(_device)
296
+
297
+ # Generate batch
298
+ with torch.no_grad():
299
+ outputs = model.generate(
300
+ **inputs,
301
+ forced_bos_token_id=tgt_token_id,
302
+ max_length=max_length,
303
+ num_beams=num_beams,
304
+ no_repeat_ngram_size=3,
305
+ repetition_penalty=1.3,
306
+ early_stopping=True,
307
+ )
308
+
309
+ # Decode batch results
310
+ batch_results = tokenizer.batch_decode(outputs, skip_special_tokens=True)
311
+ results.extend(batch_results)
312
+
313
+ return results
314
+
315
+
316
+ def list_languages(category: str = "all") -> None:
317
+ """
318
+ Print available languages and their codes.
319
+
320
+ Args:
321
+ category: Filter by category ('all', 'indic', 'other').
322
+ """
323
+ indic_langs = {
324
+ 'asm': 'Assamese', 'ben': 'Bengali', 'brx': 'Bodo', 'doi': 'Dogri',
325
+ 'gom': 'Konkani', 'guj': 'Gujarati', 'hin': 'Hindi', 'kan': 'Kannada',
326
+ 'kas': 'Kashmiri', 'mai': 'Maithili', 'mal': 'Malayalam', 'mar': 'Marathi',
327
+ 'mni': 'Manipuri', 'npi': 'Nepali', 'ory': 'Odia', 'pan': 'Punjabi',
328
+ 'san': 'Sanskrit', 'sat': 'Santali', 'snd': 'Sindhi', 'tam': 'Tamil',
329
+ 'tel': 'Telugu', 'urd': 'Urdu',
330
+ }
331
+
332
+ print("\nโ•”โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•—")
333
+ print("โ•‘ Transync - Supported Languages โ•‘")
334
+ print("โ•šโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•")
335
+
336
+ if category in ("all", "indic"):
337
+ print(f"\n๐Ÿ“š Indian Languages ({len(indic_langs)}):")
338
+ print("โ”€" * 45)
339
+ for code in sorted(indic_langs):
340
+ print(f" {code:6s} โ†’ {indic_langs[code]}")
341
+
342
+ if category in ("all", "other"):
343
+ other = {k: v for k, v in sorted(CODE_TO_LANG.items()) if k not in indic_langs}
344
+ print(f"\n๐ŸŒ Other Languages ({len(other)}):")
345
+ print("โ”€" * 45)
346
+ for code, name in other.items():
347
+ print(f" {code:6s} โ†’ {name}")
348
+
349
+ print()
350
+
351
+
352
+ # CLI interface
353
+ if __name__ == "__main__":
354
+ import argparse
355
+
356
+ parser = argparse.ArgumentParser(
357
+ description="Transync - Indic Multilingual Translation Tool",
358
+ formatter_class=argparse.RawDescriptionHelpFormatter,
359
+ epilog="""
360
+ Examples:
361
+ python transync_inference.py eng hin "Hello, how are you?"
362
+ python transync_inference.py hin tel "เคจเคฎเคธเฅเคคเฅ‡, เค†เคช เค•เฅˆเคธเฅ‡ เคนเฅˆเค‚?" --beams 3
363
+ python transync_inference.py --batch eng hin -f input.txt -o output.txt
364
+ python transync_inference.py --list-langs
365
+ """,
366
+ )
367
+
368
+ parser.add_argument(
369
+ "source_lang",
370
+ nargs="?",
371
+ help="Source language code (e.g., 'eng', 'hin', 'tel')",
372
+ )
373
+ parser.add_argument(
374
+ "target_lang",
375
+ nargs="?",
376
+ help="Target language code (e.g., 'eng', 'hin', 'tel')",
377
+ )
378
+ parser.add_argument(
379
+ "text",
380
+ nargs="*",
381
+ help="Text to translate",
382
+ )
383
+ parser.add_argument(
384
+ "--beams",
385
+ type=int,
386
+ default=5,
387
+ help="Number of beams for beam search (default: 5)",
388
+ )
389
+ parser.add_argument(
390
+ "--max-length",
391
+ type=int,
392
+ default=256,
393
+ help="Maximum translation length (default: 256)",
394
+ )
395
+ parser.add_argument(
396
+ "--temperature",
397
+ type=float,
398
+ default=1.0,
399
+ help="Sampling temperature (default: 1.0)",
400
+ )
401
+ parser.add_argument(
402
+ "--list-langs",
403
+ action="store_true",
404
+ help="List all supported languages and exit",
405
+ )
406
+ parser.add_argument(
407
+ "--batch",
408
+ action="store_true",
409
+ help="Batch translation mode (requires --file)",
410
+ )
411
+ parser.add_argument(
412
+ "-f", "--file",
413
+ type=str,
414
+ help="Input file path for batch translation",
415
+ )
416
+ parser.add_argument(
417
+ "-o", "--output",
418
+ type=str,
419
+ help="Output file path for batch translation",
420
+ )
421
+ parser.add_argument(
422
+ "--batch-size",
423
+ type=int,
424
+ default=32,
425
+ help="Batch size for batch translation (default: 32)",
426
+ )
427
+ parser.add_argument(
428
+ "--no-progress",
429
+ action="store_true",
430
+ help="Hide progress bar during batch translation",
431
+ )
432
+
433
+ args = parser.parse_args()
434
+
435
+ # List languages mode
436
+ if args.list_langs:
437
+ list_languages()
438
+ sys.exit(0)
439
+
440
+ # Validate required arguments
441
+ if not args.source_lang or not args.target_lang:
442
+ parser.print_help()
443
+ print("\nโŒ Error: source_lang and target_lang are required.")
444
+ print(" Use --list-langs to see all supported language codes.")
445
+ sys.exit(1)
446
+
447
+ # Batch translation from file
448
+ if args.batch or args.file:
449
+ if not args.file:
450
+ print("โŒ Error: --file is required for batch translation mode.")
451
+ sys.exit(1)
452
+
453
+ try:
454
+ with open(args.file, "r", encoding="utf-8") as f:
455
+ texts = [line.strip() for line in f if line.strip()]
456
+ except FileNotFoundError:
457
+ print(f"โŒ Error: File not found: {args.file}")
458
+ sys.exit(1)
459
+
460
+ if not texts:
461
+ print("โŒ Error: Input file is empty.")
462
+ sys.exit(1)
463
+
464
+ print(f"๐Ÿ“– Loaded {len(texts)} texts from {args.file}")
465
+ results = translate_batch(
466
+ texts,
467
+ args.source_lang,
468
+ args.target_lang,
469
+ batch_size=args.batch_size,
470
+ max_length=args.max_length,
471
+ num_beams=args.beams,
472
+ show_progress=not args.no_progress,
473
+ )
474
+
475
+ if args.output:
476
+ with open(args.output, "w", encoding="utf-8") as f:
477
+ for result in results:
478
+ f.write(result + "\n")
479
+ print(f"โœ“ Results written to {args.output}")
480
+ else:
481
+ for i, (orig, trans) in enumerate(zip(texts, results), 1):
482
+ print(f"\n[{i}]")
483
+ print(f" Input: {orig}")
484
+ print(f" Output: {trans}")
485
+
486
+ # Single translation
487
+ elif args.text:
488
+ text = " ".join(args.text)
489
+ src_name = CODE_TO_LANG.get(args.source_lang, args.source_lang)
490
+ tgt_name = CODE_TO_LANG.get(args.target_lang, args.target_lang)
491
+
492
+ print(f"\n๐Ÿ”ค {src_name} โ†’ {tgt_name}")
493
+ print(f" Input: {text}")
494
+
495
+ result = translate_onemt(
496
+ text,
497
+ args.source_lang,
498
+ args.target_lang,
499
+ max_length=args.max_length,
500
+ num_beams=args.beams,
501
+ temperature=args.temperature,
502
+ )
503
+
504
+ print(f" Output: {result}")
505
+
506
+ else:
507
+ print("โŒ Error: No text provided for translation.")
508
+ print(" Usage: python transync_inference.py <source> <target> <text>")
509
+ sys.exit(1)