| --- |
| license: cc-by-4.0 |
| language: |
| - pa |
| tags: |
| - gurmukhi |
| - panjabi |
| - punjabi |
| - aipanjab |
| --- |
| |
| # model-loader: AI Panjab models in one line |
|
|
| Load any [AI Panjab](https://huggingface.co/aipanjab) model and run it on your own machine, in two lines of Python. |
|
|
| ```bash |
| pip install git+https://huggingface.co/aipanjab/model-loader |
| ``` |
|
|
| ```python |
| from aipanjab import load |
| |
| load("digit-recognition").predict("digit.png") # {'੨': 0.99, '੩': 0.01, '੭': 0.0} |
| load("alphabet-recognition").predict("letter.png") # {'ਕ': 0.98, ...} |
| load("spell-checker-base").check("ਮੈਂ ਸਕੂਲ ਜਾਂਦਾ ਹਾ").corrected_text # 'ਮੈਂ ਸਕੂਲ ਜਾਂਦਾ ਹਾਂ' |
| ``` |
|
|
| `load()` downloads only the model you ask for, the first time (it is cached after that, and works offline), and runs |
| it on your CPU. This package holds no model: every model and its code live in their own repo, so `load()` always |
| gets the current one. Several models work side by side in one program. |
|
|
| ## Models |
|
|
| | `load(...)` | Use | Returns | |
| | :-- | :-- | :-- | |
| | `"digit-recognition"` | handwritten Gurmukhi digits ੦–੯ | `.predict(image, top_k=3)` → `{label: probability}` | |
| | `"alphabet-recognition"` | handwritten Gurmukhi letters and signs (57) | `.predict(image, top_k=3)` → `{label: probability}` | |
| | `"spell-checker-base"` | Panjabi spelling correction | `.check(text)` → `.corrected_text`, `.corrections`, `.to_dict()` | |
|
|
| Every model also has `.train(...)` and `.save(path)` (below). |
|
|
| `image` is a file path, an open file or a PIL image. Each model's card on Hugging Face has the details. |
|
|
| - A version of a model: `load("digit-recognition", revision="v1")` (a tag, branch or commit). |
| - A private model: log in first with `hf auth login`. |
| - A folder: `load("path/to/folder")` loads a model you saved, or a downloaded repo. |
| - On an NVIDIA GPU: install `onnxruntime-gpu` (instead of `onnxruntime`) and models use the GPU by themselves; |
| `load(..., device="cpu")` or `device="cuda"` chooses. Without a GPU everything runs on the CPU. |
|
|
| ## Train |
|
|
| Every model can be trained further on your own data, on your machine. Install the training extra once: |
|
|
| ```bash |
| pip install "aipanjab-loader[train] @ git+https://huggingface.co/aipanjab/model-loader" |
| ``` |
|
|
| ```python |
| from aipanjab import load |
| |
| # handwriting: a folder with one folder per label (my-digits/੨/1.png, ...), a Hugging Face dataset id, |
| # or (image, label) pairs; without data, the model's own dataset |
| digits = load("digit-recognition").train(data="my-digits", epochs=10) |
| digits.predict("digit.png") |
| |
| # spelling: a word list (.txt, one correctly spelled word per line, most common first) or a list of words |
| checker = load("spell-checker-base").train(data="my_words.txt", epochs=2) |
| checker.check("ਮੈਂ ਸਕੂਲ ਜਾਂਦਾ ਹਾ").corrected_text |
| |
| checker.save("my-spell-checker") # later: load("my-spell-checker") |
| ``` |
|
|
| `train()` continues from the model you loaded and returns the new one, ready to use. Options are keywords |
| (`epochs`, `batch_size`, `lr`, ...; the spell checker also `steps`, `gold`), `workdir="runs"` keeps the new model |
| there (default: a temporary folder). A saved folder is laid out like the model's repo, so it can be pushed to Hugging |
| Face as it is. Training uses a GPU when there is one (NVIDIA, or Apple silicon); `device="cpu"`, `"cuda"` or `"mps"` |
| chooses. The spell checker takes a word list of any |
| size: with 1,000 words or more, 1% are held out to test the model as it trains; with fewer, all of them train. |
|
|
| The first `load()` in a program also fetches this repo's small `config.json`, which is how Hugging Face counts the |
| loader's downloads; offline it is skipped. |
|
|
| ## Pin a version |
|
|
| To keep a program on one version of this package: `pip install git+https://huggingface.co/aipanjab/model-loader@v0.1.0`. |
|
|
| By [AI Panjab](https://aipanjab.com). Licence: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). |
|
|
