| --- |
| title: Chunking |
| emoji: 🐠 |
| colorFrom: pink |
| colorTo: pink |
| sdk: gradio |
| sdk_version: 4.0.2 |
| app_file: app.py |
| pinned: false |
| license: apache-2.0 |
| --- |
| |
| Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference |
|
|
| #LLMlocal's Text Tokenization Tool |
|
|
| ##Introduction |
|
|
| LLMlocal's Text Tokenization Tool is a user-friendly application designed to tokenize large bodies of text using various methods. This tool is built using Gradio, a Python library that allows the easy creation of web interfaces for machine learning models. |
|
|
| ##Features |
| Tokenization Method Selection: Users can select different text tokenization methods. The current version supports the RecursiveCharacterTextSplitter method. |
| Customizable Parameters: Users have the flexibility to set parameters like chunk size, chunk overlap, and the number of chunks to display. |
| Interactive Interface: The tool features an intuitive interface with dropdowns, textboxes, and number inputs for easy interaction. |
| Installation |
| Before using the tool, ensure you have Python and the necessary packages installed. You can install the required packages using the following command: |
|
|
| bash' |
| pip install gradio pandas langchain' |
|
|
| ##Usage |
| To use the tool, follow these simple steps: |
|
|
| Launch the Tool: Run the provided Python script. This will launch the Gradio interface in your default web browser. |
| Select a Tokenization Method: Choose the desired method from the dropdown menu. |
| Enter Text: Type or paste the text you want to tokenize in the textbox. |
| Set Parameters: Adjust the chunk size, chunk overlap, and the number of chunks as per your requirements. |
| View Results: The tokenized text will be displayed in a table format with details such as chunk number, text chunk, character count, and token count. |
| Contributing |
| Feedback and contributions to enhance the tool are always welcome. Please feel free to raise issues or submit pull requests on the repository. |
|
|
| ##License |
| This tool is open-source and available under apache-2.0 . |
|
|
| Acknowledgments |
| Thanks to the developers of Gradio, Pandas, and Langchain for providing the libraries that made this tool possible. |
|
|
