Fetching metadata from the HF Docker repository...
May 19 17:33 docs(chatbot): sync evaluation report to Wave 5, add 4 new intents to docs Kiriten892 +1: Claude Sonnet 4.6 - docs/evaluation_report.md: full rewrite from Wave 2/3 → Wave 5
(Accuracy 90.97%, Macro F1 89.02%, all 17 intents, n=1251)
- docs/upgrade_proposals.md: update baseline to Wave 5, mark items 1-7
done, flag PhoBERT gate as triggered (0.8902 < 0.93)
- docs/Chatbot.md: 16 → 17 intents; add show_map, destination_info,
nearby_attractions, travel_tips to intent table
- chatbot-ml-service/docs/evaluation_report.md: fix stale NER coverage
summary (85→102 samples), LLM section (Qwen→SeaLLMs), artefacts list,
time_preference row, PhoBERT gate note
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> 17:21 fix(chatbot): add test coverage for all 17 intents; fix dataset quality - Stratified 15% split from train for 4 previously uncovered intents
(show_map 68, destination_info 63, nearby_attractions 26, travel_tips 25)
- Replace all 176 nearby_attractions samples (were Reddit posts) with
99 genuine nearby-POI user queries (Vietnamese + English)
- Remove ~120 noise samples from destination_info (travel alerts, airport
queries, Reddit trip reports)
- Add 90 distinctive VI+EN destination_info samples to improve recall
- Results: Accuracy 90.97%, Macro F1 89.02% on all 17 intents (no F1 below 0.77)
- Update evaluation_report.md with Wave 5 results 17:01 docs(chatbot): add ML evaluation notebook with Wave 4 results and weight sensitivity analysis Kiriten892 +1: Claude Sonnet 4.6 - Add wanderlust_ml_evaluation.ipynb: full ML workflow (EDA, training, evaluation, weight analysis)
- Add sensitivity_analysis.py: Monte Carlo ±30% perturbation showing budget tier invariance
- Update evaluation_report.md: Wave 4 results (intent acc 93.05%, macro F1 94.00% on 13 classes, NER 0.920)
- Document 4 intents with 0 test samples and their impact on 17-class macro F1 (0.76)
- Include +Tour academic reference (Esper et al., Computer Networks 2025) for itinerary optimization
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Apr 18 11:42 feat(ML): add show_map intent + context-aware map routing - Added 80 training samples for show_map intent (VI + EN, including
'from saved plan' phrasings like 'dựa vào kế hoạch đã lưu thì bật map')
- Added show_map handler in routes.py: falls back to context destination_id
when NER finds no location (covers 'from saved plan' use case)
- Added show_map to additional_rc map trigger list so map card is rendered
- Added show_map response templates to responses_vi.json + responses_en.json
- Added show_map quick replies to ResponseGenerator
- Retrained classifier: overall accuracy 94.2%, show_map P/R/F1=1.00 09:37 Wave 3: expand NER fixture 85->102, time_preference F1 0.40->0.98, people/duration fixes - NER fixture: 85 -> 102 samples (+12 time_preference + 5 people edge cases)
- time_preference F1: 0.40 -> 0.98 (was 0.00 in Wave 1)
* Add 'buoi sang som' -> early_morning keyword (highest priority)
* VI compound scrubbing: 'an sang' (breakfast), 'cho dem' (night market VI)
* EN compound scrubbing: night market/tour/life/show/club/bus
* Fix 4 gold annotation gaps (buoi toi, hoang hon, buoi chieu samples)
- duration: add EN natural-language week patterns (a week, one/two/three weeks)
- people: add VI vo chong, co N tre em; EN group of N, N of us, solo word boundary
- location: extend FP blocklist (temple, shrine, sunrise, batu, anh, market, beach)
* Location FP: 24 -> 7 (also fixed dual location+destination annotation in fixture)
- NER micro F1: 0.912 (85 samples) -> 0.920 (102 samples)
- evaluation_report.md: add Wave 3 section with full metrics table 09:06 Wave 2: compare_destinations data + NER fixes + evaluation update Intent:
- Add 120 compare_destinations samples (80 VI + 40 EN) to intent_train_enhanced.json
- Re-split: train=5102, val=1094, test=1094 (seed=42)
- Retrain: macro F1 0.8661 -> 0.9412 on 1094 held-out samples
- compare_destinations: F1 0.00 -> 0.92
NER model.py:
- origin: HIGH_CONF end-of-sentence 'tu X' pattern + LOW_CONF gate when 'tu' present
- hotel_type: suppress 'accommodation' in budget-allocation context (tiet kiem, chi nhieu)
- hotel_type: suppress EN 'budget' when monetary amount present
- budget: add 'ruoi' (3 trieu ruoi = 3.5M VND) pattern
- budget_priority: broaden VI patterns (tiet kiem tien an, chi nhieu hon cho, uu tien)
- cuisine: skip cuisine keyword when dietary negation phrase overlaps
- transport: add 'mrt'/'gan mrt' to VI keywords
- time_preference: suppress short-keyword matches in duration context (2 dem, 2 nights)
- _match_keyword_dict: consumed-text approach prevents substring double-match
- FALSE_POSITIVE_BLOCKLIST: add 'massa'/'massage'
evaluate.py:
- Add … 08:36 eval(chatbot): Wave 1 retrain + held-out evaluation results - Produced intent_test.json / intent_val.json via split_dataset.py
(stratified 15/15/70 split, seed=42, 7170 total samples, 12 intents)
- Retrained IntentClassifier with FeatureUnion pipeline:
Held-out accuracy=0.9377, macro F1=0.8661
greeting precision: 0.59 -> 0.72 (OOD rejector working)
food<->activity confusion: 31 -> 9 errors (word TF-IDF working)
- NER re-evaluated after regex fixes:
Micro F1: 0.903 -> 0.923
hotel_type F1: 0.40 -> 0.80 (priority-based selection)
dietary F1: 0.88 -> 0.94 (broadened alias)
- Updated evaluation_report.md with all Wave 1 results
- Updated eval_intent.json and eval_ner.json 
