File size: 735 Bytes
60d3f8f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | """
Rules that we have set for preprocessing Tagalog and Waray:
[1] Remove from dataset:
. (period)
, (comma)
! (exclamation)
? (question)
; (semicolon)
: (colon)
" (double quote)
– (en dash)
— (em dash)
( ) [ ] { } (paranthesis, brackets, curly braces)
[2] Keep on the dataset:
' (single quote)
’ (apostrophe)
- (hyphen)
"""
import re
def preprocess_text(text: str) -> str:
"""
Preprocess text to match training data rules.
Remove: . , ! ? ; : " – — ( ) [ ] { }
Keep: ' (apostrophe) - (hyphen)
"""
# Characters to remove
chars_to_remove = r"[\.\,\!\?\;\:\"\–\—\(\)\[\]\{\}]"
# Remove the specified characters
text = re.sub(chars_to_remove, "", text)
return text.strip()
|