Expanded Section 9 of the technical report with comprehensive POS
tagging error analysis on UDD-1: confusion pairs, error-prone tokens,
OOV analysis, sentence position effects, and root cause discussion.
Also added datasets/ to .gitignore.
Add external dictionary features for VLSP 2013 word segmentation
External Viet74K dictionary (63K multi-syllable entries) improves word F1
from 97.36% to 97.62% (+0.26%), narrowing gap to SOTA from 0.70% to 0.44%.
Includes configurable dictionary source, error analysis, and annotation
consistency analysis of 4+ syllable compounds.
Add VLSP 2013 word segmentation, feature ablation study, and Hydra config
- Add VLSP 2013 WTK dataset (75K train, 2K test sentences)
- Train word segmentation on VLSP 2013: 97.17% word F1
- Run systematic feature ablation study (13 experiments):
- Bigrams most critical (-2.86% when removed)
- Left context second most important (-1.21%)
- Type features provide zero benefit (+0.01% when removed)
- Minimal 12-template model achieves 97.01% (99.8% of full)
- Refactor training script from Click to Hydra configuration
- Add feature group toggles for ablation via config overrides
- Move scripts/ to src/, add Hydra conf/ structure
- Add 16 research paper extracts to references/
- Add research/ literature survey (41 papers, SOTA comparison)
- Update technical report with ablation results and VLSP 2013 eval
- Update handler.py to support both pycrfsuite and underthesea-core formats
- Add word segmentation training and prediction scripts
- Add training configurations (configs/pos_tagger.yaml, configs/word_segmentation.yaml)
- Update training scripts with multi-trainer support
- Update CLAUDE.md with folder structure and word segmentation docs
- Add scripts/train.py, evaluate.py, predict.py for reproducibility
- Add CLAUDE.md with development guidance
- Add .gitignore for Python and generated files
- Add .claude/skills/paper-review for technical report reviews
- Add TECHNICAL_REPORT_REVIEW.md with review findings
- Update TECHNICAL_REPORT.md to reflect new code structure
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
