rishik1111 commited on
Commit
1f1f823
Β·
1 Parent(s): f29226d

docs: upgrade Multilingual Language Matrix with professional enterprise terminology and STT endpoints

Browse files
Files changed (1) hide show
  1. README.md +22 -22
README.md CHANGED
@@ -122,30 +122,30 @@ graph LR
122
 
123
  ---
124
 
125
- ## 🌐 Language Extensibility Matrix
126
-
127
- VECTOR treats `config.LANGUAGES` as the single source of truth for active runtime languages. Adding or removing languages from active memory requires changing only this configuration list. The underlying processed dataset contains **~743,000 deduplicated passages** across all 15 languages:
128
-
129
- | Code | Language | Script Family | Active Status | MS MARCO Dataset Source | Deduplicated Passages |
130
- | :--- | :--- | :--- | :---: | :--- | :--- |
131
- | **`en`** | English | Latin (`Latn`) | 🟒 **Active Loaded** | MS MARCO English Stream | 49,507 |
132
- | **`hi`** | Hindi | Devanagari (`Deva`) | 🟒 **Active Loaded** | `train/hintrain.parquet` & `val` | 49,509 |
133
- | **`mr`** | Marathi | Devanagari (`Deva`) | 🟒 **Active Loaded** | `train/martrain.parquet` | 49,529 |
134
- | **`as`** | Assamese | Bengali/Assamese (`Beng`) | 🟑 Zero-Code Extensible | `train/asmtrain.parquet` | 49,550 |
135
- | **`bn`** | Bengali | Bengali (`Beng`) | 🟑 Zero-Code Extensible | `train/bentrain.parquet` | 49,531 |
136
- | **`gu`** | Gujarati | Gujarati (`Gujr`) | 🟑 Zero-Code Extensible | `train/gujtrain.parquet` | 49,550 |
137
- | **`kn`** | Kannada | Kannada (`Knda`) | 🟑 Zero-Code Extensible | `train/kantrain.parquet` | 49,545 |
138
- | **`ml`** | Malayalam | Malayalam (`Mlym`) | 🟑 Zero-Code Extensible | `train/maltrain.parquet` | 49,542 |
139
- | **`ne`** | Nepali | Devanagari (`Deva`) | 🟑 Zero-Code Extensible | `train/neptrain.parquet` | 49,520 |
140
- | **`or`** | Odia | Odia (`Orya`) | 🟑 Zero-Code Extensible | `train/oritrain.parquet` | 49,560 |
141
- | **`pa`** | Punjabi | Gurmukhi (`Guru`) | 🟑 Zero-Code Extensible | `train/pantrain.parquet` | 49,534 |
142
- | **`sa`** | Sanskrit | Devanagari (`Deva`) | 🟑 Zero-Code Extensible | `validation/sanval.parquet` | 49,633 |
143
- | **`ta`** | Tamil | Tamil (`Taml`) | 🟑 Zero-Code Extensible | `train/tamtrain.parquet` & `val` | 49,581 |
144
- | **`te`** | Telugu | Telugu (`Telu`) | 🟑 Zero-Code Extensible | `validation/telval.parquet` | 49,604 |
145
- | **`ur`** | Urdu | Perso-Arabic (`Arab`) | 🟑 Zero-Code Extensible | `validation/urdval.parquet` | 49,576 |
146
 
147
  > [!NOTE]
148
- > Active Loaded In-Memory Vectors: **148,545 Native Passages** + **309 LongDoc Chunks** = **148,854 In-Memory Vectors**. Total available corpus across all 15 languages: **~743,000 Passages**.
149
 
150
  ---
151
 
 
122
 
123
  ---
124
 
125
+ ## 🌐 Multilingual Indic Language Capability & Provisioning Matrix
126
+
127
+ VECTOR employs dynamic runtime configuration via `config.LANGUAGES` as the single source of truth for language federation. The engine provides zero-code hot-swappable expansion across **14 Indic languages** and **English** (~743,000 deduplicated passage records).
128
+
129
+ | ISO Code | Language Target | Script Family | STT Engine Endpoint | Runtime Provisioning Status | Corpus Benchmark Source | Deduplicated Passages |
130
+ | :---: | :--- | :--- | :---: | :---: | :--- | :---: |
131
+ | **`en`** | English | Latin (`Latn`) | `en-IN` | ⚑ **Active In-Memory Index** | MS MARCO English Native Stream | 49,507 |
132
+ | **`hi`** | Hindi | Devanagari (`Deva`) | `hi-IN` | ⚑ **Active In-Memory Index** | MS MARCO-XI (`hin`) Parquet Stream | 49,509 |
133
+ | **`mr`** | Marathi | Devanagari (`Deva`) | `mr-IN` | ⚑ **Active In-Memory Index** | MS MARCO-XI (`mar`) Parquet Stream | 49,529 |
134
+ | **`as`** | Assamese | Bengali/Assamese (`Beng`) | `as-IN` | πŸ”Œ Zero-Code Hot-Swappable | MS MARCO-XI (`asm`) Parquet Stream | 49,550 |
135
+ | **`bn`** | Bengali | Bengali (`Beng`) | `bn-IN` | πŸ”Œ Zero-Code Hot-Swappable | MS MARCO-XI (`ben`) Parquet Stream | 49,531 |
136
+ | **`gu`** | Gujarati | Gujarati (`Gujr`) | `gu-IN` | πŸ”Œ Zero-Code Hot-Swappable | MS MARCO-XI (`guj`) Parquet Stream | 49,550 |
137
+ | **`kn`** | Kannada | Kannada (`Knda`) | `kn-IN` | πŸ”Œ Zero-Code Hot-Swappable | MS MARCO-XI (`kan`) Parquet Stream | 49,545 |
138
+ | **`ml`** | Malayalam | Malayalam (`Mlym`) | `ml-IN` | πŸ”Œ Zero-Code Hot-Swappable | MS MARCO-XI (`mal`) Parquet Stream | 49,542 |
139
+ | **`ne`** | Nepali | Devanagari (`Deva`) | `ne-NP` | πŸ”Œ Zero-Code Hot-Swappable | MS MARCO-XI (`nep`) Parquet Stream | 49,520 |
140
+ | **`or`** | Odia | Odia (`Orya`) | `od-IN` | πŸ”Œ Zero-Code Hot-Swappable | MS MARCO-XI (`ori`) Parquet Stream | 49,560 |
141
+ | **`pa`** | Punjabi | Gurmukhi (`Guru`) | `pa-IN` | πŸ”Œ Zero-Code Hot-Swappable | MS MARCO-XI (`pan`) Parquet Stream | 49,534 |
142
+ | **`sa`** | Sanskrit | Devanagari (`Deva`) | `sa-IN` | πŸ”Œ Zero-Code Hot-Swappable | MS MARCO-XI (`san`) Parquet Stream | 49,633 |
143
+ | **`ta`** | Tamil | Tamil (`Taml`) | `ta-IN` | πŸ”Œ Zero-Code Hot-Swappable | MS MARCO-XI (`tam`) Parquet Stream | 49,581 |
144
+ | **`te`** | Telugu | Telugu (`Telu`) | `te-IN` | πŸ”Œ Zero-Code Hot-Swappable | MS MARCO-XI (`tel`) Parquet Stream | 49,604 |
145
+ | **`ur`** | Urdu | Perso-Arabic (`Arab`) | `ur-IN` | πŸ”Œ Zero-Code Hot-Swappable | MS MARCO-XI (`urd`) Parquet Stream | 49,576 |
146
 
147
  > [!NOTE]
148
+ > **Active Memory Allocation**: **148,545 Native Passage Vectors** + **309 LongDoc Chunks** = **148,854 Active Vectors** in FAISS HNSW graph memory. **Total Available Federated Corpus**: **~743,000 Deduplicated Passages**.
149
 
150
  ---
151