README.md

metadata

title: LIBE Language Identification
emoji: 🌍
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 6.28.0
python_version: '3.10'
app_file: app.py
pinned: false

LIBE · Language identification

Sentence-transformer retrieval of language descriptions using cosine similarity.

Set up the Space

1. Upload app.py, requirements.txt, and this README.md to the root of your Gradio Space. If your existing README has a licence or other metadata, preserve it. This template does not choose a licence.
2. Add your actual data/WorldsLangs.tsv and data/all_stats.pkl files. The TSV must have language ID and description as its first two columns. The pickle must contain glotlid-corpus as in your original script. Only load your own trusted pickle. These files are not included in this package. On a public Space, uploaded files are public; publish only material you can redistribute.
3. Upload the complete trained SentenceTransformer model folder to a separate Hugging Face model repository. Include its configuration, tokenizer, modules.json, and module subfolders as saved by SentenceTransformer, not just the weights. Preserve the checkpoint used for your existing local inference. Do not substitute the base LaBSE model.
4. Under Space Settings → Variables and secrets, create a variable MODEL_ID with the model repository ID, for example YOUR_ACCOUNT/LIBE. Optionally set MODEL_REVISION to a model commit ID for reproducibility.
5. If the model repository is private, add an HF_TOKEN secret with read access to that repository. Never put the token in code. Access to a public demo still lets visitors query the model even when its weights are private.
6. Select CPU hardware if available under your account's plan. The Space installs dependencies, downloads the model, and encodes descriptions during startup. This repeats after a restart; a storage bucket is not needed.
7. Once running, compare several inputs with your original local demo. The top language and similarity scores should agree within floating-point tolerance when model revision, data, and dependencies match.

Behaviour

The model and language description embeddings load once per process. Queries are queued and handled one at a time on CPU. Results show the closest language, its description, and up to five candidates. Similarity scores are not probabilities. There is no calibrated unknown-language threshold. A 5,000-character UI limit controls request size; the model's own token limit still applies.

The app does not deliberately store submitted texts or enable Gradio flagging. Hosting infrastructure may have its own logging policies.

Dependency note

Gradio is pinned in the Space metadata. Sentence Transformers has a compatibility range because the training environment was not provided. For the most faithful reproduction, use the versions from the environment where your original script works, including its Transformers and PyTorch versions, after checking Python compatibility. End-to-end inference has not been validated without the checkpoint and data.

Run locally

Install dependencies in a virtual environment:

pip install gradio==5.49.1 -r requirements.txt

Then set MODEL_ID to either your model repository ID or an absolute local checkpoint directory and run python app.py. The data files must be in the data/ folder beside app.py.