| --- |
| license: mit |
| --- |
| |
| # Language Detection |
|
|
| A lightweight language detection tool that uses character-level n-gram features and logistic regression to identify the language of a given text. |
|
|
| Supported languages out of the box: English, French, German, Turkish. |
|
|
| Model repository: https://huggingface.co/Isa0/language-detection/ |
|
|
| ## Installation |
|
|
| Requires Python 3.11 or higher. Install dependencies with [uv](https://github.com/astral-sh/uv): |
|
|
| ```bash |
| uv sync |
| ``` |
|
|
| ## Usage |
|
|
| ### Train |
|
|
| Train the model on the datasets in the `datasets/` directory: |
|
|
| ```bash |
| uv run main.py --train |
| ``` |
|
|
| You can point it to a different directory with `--dir`: |
|
|
| ```bash |
| uv run main.py --train --dir path/to/datasets |
| ``` |
|
|
| Each `.txt` file in the directory should contain one sentence per line. The filename (without extension) is used as the language label. |
|
|
| ### Detect |
|
|
| Detect the language of a text string: |
|
|
| ```bash |
| uv run main.py --detect "Bonjour, comment allez-vous?" |
| ``` |
|
|
| Output includes the predicted language and a confidence score. |
|
|
| ## Adding Languages |
|
|
| Add a new `.txt` file to the `datasets/` directory named after the language (e.g. `spanish.txt`), with one sentence per line, then retrain. |
|
|
| ## How It Works |
|
|
| Text is converted into character-level n-gram counts (1 to 3 characters), which capture language-specific patterns like accents, letter combinations, and suffixes. A logistic regression classifier is trained on these features and saved to disk for reuse. |
|
|
