nonllm-chatbot / src /preprocessing /text_cleaning.py
Hreeem's picture
Deploy to Hugging Face (clean history)
2750f74
Raw History Blame Contribute Delete
218 Bytes
import re
def clean_text(text: str) -> str:
if not isinstance(text, str):
return ''
text = text.lower().strip()
text = re.sub(r'\s+', ' ', text)
return re.sub(r"[^\w\s\?\!\.\,\'\-]", '', text)