Instructions to use ArthaLabs/panini-tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ArthaLabs/panini-tokenizer with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("ArthaLabs/panini-tokenizer", device_map="auto") - Notebooks
- Google Colab
- Kaggle
| """ | |
| Panini Tokenizer | |
| Morphology-aware Sanskrit tokenizer with Sandhi Expansion. | |
| """ | |
| from .analyzer import VidyutAnalyzer, MorphParse | |
| from .splitter import SamasaSplitter, CompoundSplit | |
| from .sandhi_engine import SandhiEngine | |
| from .tokenizer import PaniniTokenizerV3, create_tokenizer | |
| __all__ = [ | |
| "VidyutAnalyzer", | |
| "MorphParse", | |
| "SamasaSplitter", | |
| "CompoundSplit", | |
| "SandhiEngine", | |
| "PaniniTokenizerV3", | |
| "create_tokenizer", | |
| ] | |
| __version__ = "1.5.0" | |