| --- |
| title: LIBE Language Identification |
| emoji: 🌍 |
| colorFrom: blue |
| colorTo: green |
| sdk: gradio |
| sdk_version: 6.28.0 |
| python_version: '3.10' |
| app_file: app.py |
| pinned: false |
| --- |
| |
| # LIBE · Language identification |
|
|
| Sentence-transformer retrieval of language descriptions using cosine similarity. |
|
|
| ## Set up the Space |
|
|
| 1. Upload `app.py`, `requirements.txt`, and this `README.md` to the root of your Gradio Space. If your existing README has a licence or other metadata, preserve it. This template does not choose a licence. |
| 2. Add your actual `data/WorldsLangs.tsv` and `data/all_stats.pkl` files. The TSV must have language ID and description as its first two columns. The pickle must contain `glotlid-corpus` as in your original script. Only load your own trusted pickle. These files are not included in this package. On a public Space, uploaded files are public; publish only material you can redistribute. |
| 3. Upload the complete trained SentenceTransformer model folder to a separate Hugging Face **model repository**. Include its configuration, tokenizer, modules.json, and module subfolders as saved by SentenceTransformer, not just the weights. Preserve the checkpoint used for your existing local inference. Do not substitute the base LaBSE model. |
| 4. Under Space **Settings → Variables and secrets**, create a variable `MODEL_ID` with the model repository ID, for example `YOUR_ACCOUNT/LIBE`. Optionally set `MODEL_REVISION` to a model commit ID for reproducibility. |
| 5. If the model repository is private, add an `HF_TOKEN` secret with read access to that repository. Never put the token in code. Access to a public demo still lets visitors query the model even when its weights are private. |
| 6. Select CPU hardware if available under your account's plan. The Space installs dependencies, downloads the model, and encodes descriptions during startup. This repeats after a restart; a storage bucket is not needed. |
| 7. Once running, compare several inputs with your original local demo. The top language and similarity scores should agree within floating-point tolerance when model revision, data, and dependencies match. |
|
|
| ## Behaviour |
|
|
| The model and language description embeddings load once per process. Queries are queued and handled one at a time on CPU. Results show the closest language, its description, and up to five candidates. Similarity scores are not probabilities. There is no calibrated unknown-language threshold. A 5,000-character UI limit controls request size; the model's own token limit still applies. |
|
|
| The app does not deliberately store submitted texts or enable Gradio flagging. Hosting infrastructure may have its own logging policies. |
|
|
| ## Dependency note |
|
|
| Gradio is pinned in the Space metadata. Sentence Transformers has a compatibility range because the training environment was not provided. For the most faithful reproduction, use the versions from the environment where your original script works, including its Transformers and PyTorch versions, after checking Python compatibility. End-to-end inference has not been validated without the checkpoint and data. |
|
|
| ## Run locally |
|
|
| Install dependencies in a virtual environment: |
|
|
| ```sh |
| pip install gradio==5.49.1 -r requirements.txt |
| ``` |
|
|
| Then set `MODEL_ID` to either your model repository ID or an absolute local checkpoint directory and run `python app.py`. The data files must be in the `data/` folder beside `app.py`. |
