Spaces:
Runtime error
Runtime error
Download core/text_processor.py from utkarsh134/FastAPIExperience2: direct link, hf CLI and curl.
- Browser
- Download file 392 Bytes
-
https://huggingface.co/spaces/utkarsh134/FastAPIExperience2/resolve/fa57cb04110d5b855c905dbb8ec96563a2529559/core/text_processor.py
- Command line
-
hf download hf://spaces/utkarsh134/FastAPIExperience2@fa57cb04110d5b855c905dbb8ec96563a2529559/core/text_processor.py
-
curl -L -o text_processor.py https://huggingface.co/spaces/utkarsh134/FastAPIExperience2/resolve/fa57cb04110d5b855c905dbb8ec96563a2529559/core/text_processor.py
392 Bytes
| from langchain.text_splitter import RecursiveCharacterTextSplitter | |
| def chunk_text(text, chunk_size=1000, chunk_overlap=200): | |
| text_splitter = RecursiveCharacterTextSplitter( | |
| chunk_size=chunk_size, | |
| chunk_overlap=chunk_overlap, | |
| length_function=len, | |
| is_separator_regex=False, | |
| ) | |
| chunks = text_splitter.split_text(text) | |
| return chunks | |