- H1: validate_citations regex now matches References:/Reference/Sources (Primary)
- H2: _strip_sql_comments rewritten as state machine, preserves --//**/ inside strings
- H3: SQL citation count fixed: 1 source per query, not 1 per row
- M1: Context budget now accounts for web context size before local allocation
- M2: max_context_chars default aligned to 12000 (was 0 in code vs 12000 in docs)
10 agents dispatched: 5 audit + 3 fix + 2 cross-check. 175/175 tests passing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When phrase-level fuzzy fails on the original column (e.g. Country),
now tries phrase-level LIKE on other TEXT columns in the same table
before falling back to word-level fuzzy. This fixes South Korea queries
returning 98 rows (both Koreas via LIKE '%Korea%') instead of 49 rows
(just South Korea via Country_OLD LIKE '%South Korea%').
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: add word-level fuzzy SQL matching for country name variants
Country names like Korea, Republic of do not contain South Korea as a
substring, so phrase-level LIKE fails. Word-level fuzzy extracts significant
content words (dropping directional/geopolitical qualifiers) and matches
individually: WHERE Country LIKE %Korea%.
Three-step strategy: exact -> phrase fuzzy -> word-level fuzzy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: SQL fuzzy matching fallback for country names with alternate forms
Exact match (=) tried first; if 0 rows, auto-retries with LIKE '%...%'.
Match type (exact/fuzzy) transparently reported in UI source summaries.
Fixes queries like "South Korea" matching "South Korea (Republic of Korea)".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: strip trailing semicolons from LLM-generated SQL and add SQL-route fallback
Two bugs prevented SQL retrieval from working:
1. LLMs append trailing semicolons to SQL (~90%% of the time).
_validate_sql() rejected these as potential injection attacks.
Fix: rstrip(';') before validation (internal semicolons still blocked).
2. When route=sql and the LLM SQL query failed, the keyword-based
SQL fallback never triggered (condition excluded route=sql).
Fix: expanded fallback to build fresh keyword query for route=sql.
Verified end-to-end: China PTS scores now returns 49 rows (was 0).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
HIGH fixes:
- H1: Gemini blocked-response detection in verifier (safety filter short-circuit)
- H2: Config load crash guard in app_web.py (st.stop instead of traceback)
- H3: Brace-escape user text in kb_meta.py prompts (prevent KeyError)
- H4: Web UI clarification state tracking (pending_clarification_question)
MEDIUM fixes:
- M1: Preserve leading zeros as TEXT in sql_ingest type inference
- M2: Deferred clearing in ingest.py prevents data loss on total failure
- M3: Strip markdown fences from LLM-generated SQL in sql_retriever
- M4: max_iterations=0 guard in verifier (skip verification, do not refuse)
- M5: Add clarification QA to conversation history for pronoun resolution
- M6: Remove hardcoded knowledge_base/ prefix in retriever source paths
Also adds CLAUDE.md, docs/plans/, full test suite (162 tests, 17 files),
and updates WORKLOG.md with audit results.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: 7 bugs from multi-agent debug review of SQL layer
- Column name deduplication after sanitization (Score-A/Score_A collision)
- _safe_str() preserves None in Stata/SPSS/R data loaders
- Reject inf/Infinity in column type inference
- Guard against n=1 in _get_sample_values (ZeroDivisionError)
- NULL values render as "N/A" in SQL context (not "None")
- Defensive .get() in build_schema_summary for malformed schema
- Backtick-quoted table names in _extract_table_from_query regex
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: add LLM-powered column descriptions with codebook detection
During SQL ingestion, each table columns are now described by the LLM:
1. Search same directory for codebook/documentation files (PDF, DOCX, TXT, MD)
2. If codebook found, pass its text + column metadata to LLM for descriptions
3. If no codebook, LLM infers column meanings from names, types, samples
4. Descriptions stored in sql_schemas.json alongside existing column metadata
Column descriptions are always visible to the chatbot:
- build_schema_summary() includes descriptions in the QU prompt
- format_sql_results_as_context() includes descriptions with SQL results
- Graceful degradation: LLM failure is non-fatal
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: fallback to KB meta overview when vector search returns no results
When all chunks exceed the distance threshold, the retriever now fetches
the KB meta overview chunk directly as a fallback. This fixes meta-questions
like "do you have any dataset?" returning empty results when the embedding
similarity didn't match closely enough.
Also adds a system prompt instruction to use the KB overview to describe
what IS available when it's the only local source.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: add SQL routing, fallback, and context merging to retriever
- Add _escape_braces() to all prompt builders to prevent str.format() crash
when user documents contain {placeholder} patterns
- Fix app_web.py display_query fallback to preserve clarification context
- Add empty-response guards to all 3 LLM providers (openai, anthropic, gemini)
- Replace all remaining direct dict access with .get() + defaults
- Add 2 regression tests for curly-brace format injection
- Expand query engine tests with display_query coverage
- 62 tests pass across 13 test files
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: retrieve all relevant chunks using distance threshold
Instead of a fixed top_k count, retrieval now uses a distance
threshold (max_distance=0.55). Queries up to 50 candidates from
ChromaDB, then keeps only chunks with cosine distance below the
threshold. This means:
- Broad queries with lots of relevant docs get many chunks
- Narrow queries with few matches get only the relevant ones
- Irrelevant chunks are filtered out regardless of top_k
New config field: retrieval.max_distance (default 0.55)
top_k now acts as the candidate pool cap (default 50)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: increase top_k to 10 and remove last [N] from verification prompt
Two fixes:
1. Increased default top_k from 5 to 10. With top_k=5, short queries
like "China in 2024?" missed the relevant page (e.g., page 410)
because generic intro pages ranked higher. More chunks = better
chance of retrieving the right content.
2. Removed the last literal [N] from the verification prompt template
(checklist item 1) to prevent any residual [N] in output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Citation-verified research assistant with 6-layer anti-hallucination stack,
15 file format readers, 3 LLM providers, and dual CLI/Web interfaces.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
