Text Classification
LiteRT
LiteRT
mediapipe
on-device
android
ios
web
python
flutter
nlp
language-identification
multilingual
fp32
Instructions to use litert-community/MediaPipe-LanguageDetector with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT
How to use litert-community/MediaPipe-LanguageDetector with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Commit ·
5811297
0
Parent(s):
Initial upload of MediaPipe-LanguageDetector from MediaPipe Solutions
Browse files- .gitattributes +38 -0
- LanguageDetector Model Card.pdf +3 -0
- README.md +130 -0
- labels.txt +111 -0
- language_detector.tflite +3 -0
.gitattributes
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
+
*.litertlm filter=lfs diff=lfs merge=lfs -text
|
| 12 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 13 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
| 14 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 15 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 16 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 17 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 18 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 19 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 20 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 21 |
+
*.pdf filter=lfs diff=lfs merge=lfs -text
|
| 22 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 23 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 24 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 25 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 26 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 27 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 28 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 29 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 30 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 31 |
+
*.task filter=lfs diff=lfs merge=lfs -text
|
| 32 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 33 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 34 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 35 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
LanguageDetector Model Card.pdf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b383b7d189279949e1cc27aa04d09d967989eacd784cf91188a19c8811b69152
|
| 3 |
+
size 250203
|
README.md
ADDED
|
@@ -0,0 +1,130 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: litert
|
| 4 |
+
pipeline_tag: text-classification
|
| 5 |
+
tags:
|
| 6 |
+
- mediapipe
|
| 7 |
+
- litert
|
| 8 |
+
- tflite
|
| 9 |
+
- text-classification
|
| 10 |
+
- on-device
|
| 11 |
+
- android
|
| 12 |
+
- ios
|
| 13 |
+
- web
|
| 14 |
+
- python
|
| 15 |
+
- flutter
|
| 16 |
+
- nlp
|
| 17 |
+
- language-identification
|
| 18 |
+
- multilingual
|
| 19 |
+
- fp32
|
| 20 |
+
---
|
| 21 |
+
|
| 22 |
+
# MediaPipe Language Detector
|
| 23 |
+
|
| 24 |
+
Ultra-lightweight UTF-8 text language identification model (`language_detector.tflite`, 315 KB, FP32) recognizing **110 languages** via ISO 639-1 / BCP-47 script codes (`af` through `zu`, plus `unknown`) in the [MediaPipe Language Detector (`LanguageDetector`)](https://developers.google.com/edge/mediapipe/solutions/text/language_detector) task.
|
| 25 |
+
|
| 26 |
+
<div style="border: 1px solid rgba(59, 130, 246, 0.28); background-color: rgba(59, 130, 246, 0.06); border-radius: 8px; padding: 12px 16px; margin: 16px 0;">
|
| 27 |
+
<p style="margin: 0 0 6px 0;"><strong>Documentation:</strong> <a href="https://developers.google.com/edge/mediapipe/solutions/text/language_detector">MediaPipe Language Detector Docs</a></p>
|
| 28 |
+
<p style="margin: 0 0 6px 0;"><strong>Samples:</strong> <a href="https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/language_detector/android">Android</a> · <a href="https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/language_detector/ios">iOS</a> · <a href="https://colab.research.google.com/github/google-ai-edge/mediapipe-samples/blob/main/examples/language_detector/python/%5BMediaPipe_Python_Tasks%5D_Language_Detector.ipynb">Python Colab</a> · <a href="https://github.com/google-ai-edge/mediapipe-samples-web/blob/main/src/tasks/language-detector.ts">Web</a> · <a href="https://github.com/google/flutter-mediapipe/tree/main/samples/text/language_detector">Flutter</a></p>
|
| 29 |
+
<p style="margin: 0;"><strong>Try live:</strong> <a href="https://google-ai-edge.github.io/mediapipe-samples-web/#/text/language_detector">Web Demo</a></p>
|
| 30 |
+
</div>
|
| 31 |
+
|
| 32 |
+
Embedding-based neural classification network optimized with feature hashing and compact embedding lookup tables. Takes a single UTF-8 string (`Placeholder` `[1]`) and outputs a 111-way probability distribution (`pod/Softmax` `[1, 111]`), bundled with [`labels.txt`](./labels.txt) (`unknown` + 110 ISO 639-1 / `-Latn` transliterated language codes).
|
| 33 |
+
|
| 34 |
+
## Files
|
| 35 |
+
|
| 36 |
+
| File | Precision | Size | SHA-256 |
|
| 37 |
+
|---|---|---:|---|
|
| 38 |
+
| [`language_detector.tflite`](https://huggingface.co/litert-community/MediaPipe-LanguageDetector/blob/main/language_detector.tflite) | `float32` | 315.3 KB (315,294 B) | `7db4f23dfe1ad8966b050b419a865da451143fd43eb6b606a256aadeeb1e5417` |
|
| 39 |
+
|
| 40 |
+
### Companion Files in Repo Root
|
| 41 |
+
|
| 42 |
+
| File | Role | Size | SHA-256 |
|
| 43 |
+
|---|---|---:|---|
|
| 44 |
+
| [`LanguageDetector Model Card.pdf`](./LanguageDetector%20Model%20Card.pdf) | Upstream Google model card (PDF) | 250.2 KB (250,203 B) | `b383b7d189279949e1cc27aa04d09d967989eacd784cf91188a19c8811b69152` |
|
| 45 |
+
| [`labels.txt`](./labels.txt) | Label map | 0.4 KB (376 B) | `938a527f1ed52b8a10cfb5491ac3229e60baae5befbbc7d3a60863917d0a0841` |
|
| 46 |
+
|
| 47 |
+
### Download
|
| 48 |
+
|
| 49 |
+
```bash
|
| 50 |
+
pip install -U huggingface_hub
|
| 51 |
+
hf download litert-community/MediaPipe-LanguageDetector language_detector.tflite --local-dir .
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
## Inputs, Outputs, and Model Specifications
|
| 55 |
+
|
| 56 |
+
### `language_detector.tflite`
|
| 57 |
+
|
| 58 |
+
- Embedded `TFLITE_METADATA`: name `LangID` · version `1` · author `MediaPipe` · packed associated files: `labels.txt`
|
| 59 |
+
|
| 60 |
+
| Direction | Tensor | Shape | Type | Notes |
|
| 61 |
+
|---|---|---|---|---|
|
| 62 |
+
| Input | `Placeholder` | `[1]` | `string` | Placeholder |
|
| 63 |
+
| Output | `pod/Softmax` | `[1, 111]` | `float32` | score — Prediction scores for each language. — Labels: `labels.txt` |
|
| 64 |
+
|
| 65 |
+
## Use with MediaPipe Tasks
|
| 66 |
+
|
| 67 |
+
### Python Quickstart
|
| 68 |
+
|
| 69 |
+
```bash
|
| 70 |
+
pip install -U mediapipe huggingface_hub
|
| 71 |
+
```
|
| 72 |
+
|
| 73 |
+
```python
|
| 74 |
+
import mediapipe as mp
|
| 75 |
+
from huggingface_hub import hf_hub_download
|
| 76 |
+
|
| 77 |
+
model_path = hf_hub_download(repo_id="litert-community/MediaPipe-LanguageDetector", filename="language_detector.tflite")
|
| 78 |
+
|
| 79 |
+
BaseOptions = mp.tasks.BaseOptions
|
| 80 |
+
LanguageDetector = mp.tasks.text.LanguageDetector
|
| 81 |
+
LanguageDetectorOptions = mp.tasks.text.LanguageDetectorOptions
|
| 82 |
+
|
| 83 |
+
options = LanguageDetectorOptions(
|
| 84 |
+
base_options=BaseOptions(model_asset_path=model_path),
|
| 85 |
+
score_threshold=0.5,
|
| 86 |
+
)
|
| 87 |
+
with LanguageDetector.create_from_options(options) as detector:
|
| 88 |
+
result = detector.detect("Bonjour tout le monde, comment allez-vous ?")
|
| 89 |
+
for det in result.detections:
|
| 90 |
+
print(f"{det.language_code}: {det.probability:.3f}")
|
| 91 |
+
```
|
| 92 |
+
|
| 93 |
+
### Cross-Platform Guides & Samples
|
| 94 |
+
|
| 95 |
+
- **Task overview & options:** [Language Detector documentation](https://developers.google.com/edge/mediapipe/solutions/text/language_detector)
|
| 96 |
+
- **Platform setup guides:** [Python](https://developers.google.com/edge/mediapipe/solutions/setup_python) · [Web / JavaScript](https://developers.google.com/edge/mediapipe/solutions/setup_web) · [Android (Kotlin / Java)](https://developers.google.com/edge/mediapipe/solutions/setup_android) · [iOS (Swift)](https://developers.google.com/edge/mediapipe/solutions/setup_ios) · [Flutter (Dart)](https://developers.google.com/edge/mediapipe/solutions/setup_flutter)
|
| 97 |
+
- **Runnable sample apps:** [Web Demo](https://google-ai-edge.github.io/mediapipe-samples-web/#/text/language_detector) · [Web Source](https://github.com/google-ai-edge/mediapipe-samples-web/blob/main/src/tasks/language-detector.ts) · [Android](https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/language_detector/android) · [iOS](https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/language_detector/ios) · [Python Notebook](https://colab.research.google.com/github/google-ai-edge/mediapipe-samples/blob/main/examples/language_detector/python/%5BMediaPipe_Python_Tasks%5D_Language_Detector.ipynb) · [Flutter](https://github.com/google/flutter-mediapipe/tree/main/samples/text/language_detector)
|
| 98 |
+
|
| 99 |
+
## Latency Benchmarks
|
| 100 |
+
|
| 101 |
+
End-to-end MediaPipe Task pipeline average latency reported by the [Language Detector documentation](https://developers.google.com/edge/mediapipe/solutions/text/language_detector#benchmarks):
|
| 102 |
+
|
| 103 |
+
| Model Variant | Device | CPU Latency | GPU Latency |
|
| 104 |
+
|---|---|---:|---:|
|
| 105 |
+
| Language Detector | Pixel 6 | 0.31 ms | — |
|
| 106 |
+
|
| 107 |
+
Additionally, the bundled PDF model card reports **90 µs** average latency on Pixel 2 (350-token text) with **F1 = 0.9781** (Precision 0.9893, Recall 0.9673) on Creative Commons v2.5 `en.wikinews.org` multilingual samples.
|
| 108 |
+
|
| 109 |
+
## Intended Use, Limitations, and Fairness
|
| 110 |
+
|
| 111 |
+
Summarized from the bundled [`LanguageDetector Model Card.pdf`](./LanguageDetector%20Model%20Card.pdf) (Google Research Learn2Compress & MediaPipe Teams, May 10, 2023):
|
| 112 |
+
|
| 113 |
+
- **Intended applications:** Predicting the language of input text (110 supported languages) as preprocessing for Smart Reply, OCR, and downstream NLP pipelines.
|
| 114 |
+
- **Training data & bias:** Trained on PII-stripped translated anonymized web data covering languages across all 17 UN geographic subregions; has more training data for popular languages such as English and French.
|
| 115 |
+
- **Out-of-scope uses:**
|
| 116 |
+
- Not appropriate for **human life-critical decisions**, including financial, contractual, legal, or medical decisions.
|
| 117 |
+
- Predicting languages outside the 110 preset codes in `labels.txt`.
|
| 118 |
+
- Mixed multilingual strings (no guarantee that all languages or even the dominant language will be ranked first).
|
| 119 |
+
|
| 120 |
+
## Provenance
|
| 121 |
+
|
| 122 |
+
Every model binary in this repository is byte-identical (unmodified FlatBuffer weights and embedded `TFLITE_METADATA` bytes) to the official artifact served by Google from `storage.googleapis.com/mediapipe-models`.
|
| 123 |
+
|
| 124 |
+
| Repo File | Upstream URL | GCS Generation | Upstream `Last-Modified` | Pinned Bucket Version |
|
| 125 |
+
|---|---|---|---|---|
|
| 126 |
+
| `language_detector.tflite` | [`language_detector.tflite`](https://storage.googleapis.com/mediapipe-models/language_detector/language_detector/float32/1/language_detector.tflite) | `1682624733160468` | Thu, 27 Apr 2023 19:45:33 GMT | — |
|
| 127 |
+
|
| 128 |
+
## License and Citation
|
| 129 |
+
|
| 130 |
+
- License: `apache-2.0` — Stated as Apache License, Version 2.0 in the bundled upstream Google model-card PDF.
|
labels.txt
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
unknown
|
| 2 |
+
af
|
| 3 |
+
am
|
| 4 |
+
ar
|
| 5 |
+
ar-Latn
|
| 6 |
+
az
|
| 7 |
+
be
|
| 8 |
+
bg
|
| 9 |
+
bg-Latn
|
| 10 |
+
bn
|
| 11 |
+
bs
|
| 12 |
+
ca
|
| 13 |
+
ceb
|
| 14 |
+
co
|
| 15 |
+
cs
|
| 16 |
+
cy
|
| 17 |
+
da
|
| 18 |
+
de
|
| 19 |
+
el
|
| 20 |
+
el-Latn
|
| 21 |
+
en
|
| 22 |
+
eo
|
| 23 |
+
es
|
| 24 |
+
et
|
| 25 |
+
eu
|
| 26 |
+
fa
|
| 27 |
+
fi
|
| 28 |
+
fil
|
| 29 |
+
fr
|
| 30 |
+
fy
|
| 31 |
+
ga
|
| 32 |
+
gd
|
| 33 |
+
gl
|
| 34 |
+
gu
|
| 35 |
+
ha
|
| 36 |
+
haw
|
| 37 |
+
hi
|
| 38 |
+
hi-Latn
|
| 39 |
+
hmn
|
| 40 |
+
hr
|
| 41 |
+
ht
|
| 42 |
+
hu
|
| 43 |
+
hy
|
| 44 |
+
id
|
| 45 |
+
ig
|
| 46 |
+
is
|
| 47 |
+
it
|
| 48 |
+
iw
|
| 49 |
+
ja
|
| 50 |
+
ja-Latn
|
| 51 |
+
jv
|
| 52 |
+
ka
|
| 53 |
+
kk
|
| 54 |
+
km
|
| 55 |
+
kn
|
| 56 |
+
ko
|
| 57 |
+
ku
|
| 58 |
+
ky
|
| 59 |
+
la
|
| 60 |
+
lb
|
| 61 |
+
lo
|
| 62 |
+
lt
|
| 63 |
+
lv
|
| 64 |
+
mg
|
| 65 |
+
mi
|
| 66 |
+
mk
|
| 67 |
+
ml
|
| 68 |
+
mn
|
| 69 |
+
mr
|
| 70 |
+
ms
|
| 71 |
+
mt
|
| 72 |
+
my
|
| 73 |
+
ne
|
| 74 |
+
nl
|
| 75 |
+
no
|
| 76 |
+
ny
|
| 77 |
+
pa
|
| 78 |
+
pl
|
| 79 |
+
ps
|
| 80 |
+
pt
|
| 81 |
+
ro
|
| 82 |
+
ru
|
| 83 |
+
ru-Latn
|
| 84 |
+
sd
|
| 85 |
+
si
|
| 86 |
+
sk
|
| 87 |
+
sl
|
| 88 |
+
sm
|
| 89 |
+
sn
|
| 90 |
+
so
|
| 91 |
+
sq
|
| 92 |
+
sr
|
| 93 |
+
st
|
| 94 |
+
su
|
| 95 |
+
sv
|
| 96 |
+
sw
|
| 97 |
+
ta
|
| 98 |
+
te
|
| 99 |
+
tg
|
| 100 |
+
th
|
| 101 |
+
tr
|
| 102 |
+
uk
|
| 103 |
+
ur
|
| 104 |
+
uz
|
| 105 |
+
vi
|
| 106 |
+
xh
|
| 107 |
+
yi
|
| 108 |
+
yo
|
| 109 |
+
zh
|
| 110 |
+
zh-Latn
|
| 111 |
+
zu
|
language_detector.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7db4f23dfe1ad8966b050b419a865da451143fd43eb6b606a256aadeeb1e5417
|
| 3 |
+
size 315294
|