LIANJie-Jason Claude Opus 4.6 commited on
Commit
31d8af3
Β·
1 Parent(s): 1eac8bc

docs: consolidate bug logs into single BUGLOG.md with Round 18 audit results

Browse files

Merged DEVLOG.md, PRODUCT_EVALUATION_ROUND15.md, and PRODUCT_EVALUATION_ROUND17.md
into one deduplicated bug log. Added Round 18 final audit findings (4 new MEDIUM,
10 new LOW, 0 CRITICAL/HIGH). Master open bug table, fix history, test coverage
gaps, and design evaluation all in one file.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Files changed (1) hide show
  1. docs/BUGLOG.md +169 -0
docs/BUGLOG.md ADDED
@@ -0,0 +1,169 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Bug Log β€” RAG Research Chatbot Template
2
+
3
+ **Consolidated tracking file for all bugs, evaluations, and test coverage gaps.**
4
+ **Last updated:** Round 18 (2026-03-07)
5
+
6
+ ---
7
+
8
+ ## Summary
9
+
10
+ | Metric | Value |
11
+ |--------|-------|
12
+ | Audit rounds completed | 18 (70+ agent dispatches) |
13
+ | Total bugs found (all rounds) | ~270+ |
14
+ | Total bugs fixed | ~210 |
15
+ | Open bugs | 0 CRITICAL, 0 HIGH, 14 MEDIUM, 28 LOW |
16
+ | Tests | 175 passing, 17 files |
17
+ | Design score | 6.6/10 (Round 17), 7/10 (Round 15) |
18
+
19
+ ---
20
+
21
+ ## Design Evaluation (Round 17)
22
+
23
+ | Dimension | Score | Key Issue |
24
+ |-----------|-------|-----------|
25
+ | Architecture Quality | 7/10 | Deferred imports hide coupling, duplicated path resolution (12+ sites) |
26
+ | Anti-Hallucination | 8/10 | Layer 0 airtight; verification prompt lacks user query |
27
+ | Query Understanding | 7/10 | Dual-query design strong; no input sanitization |
28
+ | Retrieval Strategy | 7/10 | Multi-step SQL fallback thorough; no hybrid ranking for "both" route |
29
+ | User Experience | 6/10 | No streaming (30-60s blank screen); CLI/Web feature parity gap |
30
+ | Production Readiness | 5/10 | No logging, no retry/backoff, no config validation, no timeouts |
31
+ | Design Gaps | 6/10 | No cost tracking, no incremental ingestion, no conversation persistence |
32
+ | **OVERALL** | **6.6/10** | Strong prototype, needs production hardening |
33
+
34
+ ### Top 10 Design Improvements
35
+
36
+ 1. Cache LLM clients (singleton per provider+key) β€” saves 500ms-2s per query
37
+ 2. Add structured logging (replace 40+ `print()` statements)
38
+ 3. Incremental ingestion (hash-based skip)
39
+ 4. Fix Layer 4 to check per-source overlap (not all context combined)
40
+ 5. Structured JSON output for verification calls
41
+ 6. Extract path resolution utility (12+ duplicate sites)
42
+ 7. Live `list_models()` via provider APIs
43
+ 8. API key validation in setup wizard
44
+ 9. Chat export/persistence
45
+ 10. Pagination for `_collect_all_data()` in kb_meta.py
46
+
47
+ ---
48
+
49
+ ## Open Bugs β€” MEDIUM (14)
50
+
51
+ Deduplicated across all rounds. Ordered by impact.
52
+
53
+ | ID | File:Line | Description | Found |
54
+ |----|-----------|-------------|-------|
55
+ | M-01 | `prompts.py:349` | **Verification prompt delimiter injection** β€” no sanitization for `--- END CONTEXT ---` in document chunks; attacker text could bypass verifier | R18 |
56
+ | M-02 | `verifier.py:452` | **Verification loop no-break on LLM failure** β€” `except: continue` burns all iterations when LLM consistently fails; should break after 2 consecutive failures | R17 |
57
+ | M-03 | `gemini.py:29` | **Gemini safety filter returns ""** β€” sentinel check at verifier.py:387-389 is dead code; user gets generic error, not safety-specific message | R15 |
58
+ | M-04 | `sql_retriever.py:81` | **Semicolon in quoted strings rejected** β€” `";" in stripped` checks raw SQL, not `_strip_quoted()`; blocks valid queries like `WHERE col = 'a;b'` | R15 |
59
+ | M-05 | `sql_retriever.py:13` | **WITH (CTE) blocked** β€” in dangerous keywords list; also blocked by multi-SELECT check on line 92; double-blocked, both overly aggressive | R17 |
60
+ | M-06 | `sql_retriever.py:92` | **Subquery block too aggressive** β€” `SELECT` count > 1 rejects `WHERE EXISTS(...)`, scalar subqueries, `CASE WHEN` with subselects | R17 |
61
+ | M-07 | `sql_retriever.py:89` | **sqlite_master check on raw SQL** β€” uses `stripped` instead of `unquoted`; false-rejects `WHERE col = 'sqlite_master'` as data value | R13 |
62
+ | M-08 | `retriever.py:271` | **_try_alternate_columns regex not anchored to WHERE** β€” can match expressions in SELECT, FROM, or other clauses | R17 |
63
+ | M-09 | `retriever.py:315` | **_try_alternate_columns sanitization order** β€” `replace("'","''")` then regex strip undoes the escaping; "Cote d'Ivoire" becomes "Cote dIvoire" | R17 |
64
+ | M-10 | `ingest.py:263` | **Partial-success ingestion data loss** β€” if first file clears old data but subsequent files fail, previous ingestion lost (partially mitigated by deferred clear) | R15 |
65
+ | M-11 | `sql_ingest.py:57` | **Type inference inconsistency** β€” `_infer_column_type` uses `int(s)` (rejects "3.0") but insertion uses `int(float(s))` (accepts "3.0") | R17 |
66
+ | M-12 | `app_web.py:165+190` | **Clarification doubles context in QU prompt** β€” clarification Q&A appears in both `combined` query string AND conversation history | R17 |
67
+ | M-13 | `app_web.py:59,104,109` | **Sidebar mutates shared cfg dict** β€” direct in-place mutation of session state config; fragile but currently works due to Streamlit rerun model | R18 |
68
+ | M-14 | `setup.py:198-200` | **.env overwritten on decline** β€” user declines config.yaml overwrite but .env is still written; can blank existing API key | R15 |
69
+
70
+ ---
71
+
72
+ ## Open Bugs β€” LOW (28)
73
+
74
+ | ID | File:Line | Description | Found |
75
+ |----|-----------|-------------|-------|
76
+ | L-01 | `llm/*.py` | No retry/backoff on LLM calls (only semantic_scholar has retry) | R13 |
77
+ | L-02 | `llm/*.py` | No request timeouts on LLM API calls | R13 |
78
+ | L-03 | `llm/*.py` | New LLM client on every `generate()` call (no connection reuse) | R13 |
79
+ | L-04 | `config_loader.py` | No config validation β€” `max_distance: "banana"` fails at ChromaDB | R15 |
80
+ | L-05 | `config_loader.py:61` | `get_api_key` doesn't strip whitespace | R15 |
81
+ | L-06 | `config_loader.py:42` | Non-dict section values (e.g. `paths: 42`) cause AttributeError | R18 |
82
+ | L-07 | `ingest.py` | No embedding model tracking β€” switching after ingestion corrupts search | R15 |
83
+ | L-08 | `ingest.py:226` | Overlap cap allows exactly 80% (should be < 80%) | R17 |
84
+ | L-09 | `ingest.py:255` | MD5 12-char prefix collision risk at ~25K files | R15 |
85
+ | L-10 | `prompts.py:85-141` | Verification prompt lacks user query β€” can't check relevance | R17 |
86
+ | L-11 | `prompts.py:312` | Dash delimiters (`---`) not sanitized (only `===` is) | R18 |
87
+ | L-12 | `verifier.py:473` | Correction prompt interpolates raw `query` without sanitization | R18 |
88
+ | L-13 | `sql_retriever.py:27-68` | `_strip_sql_comments` doesn't handle double-quoted identifiers | R13 |
89
+ | L-14 | `sql_retriever.py:23` | `_strip_quoted` misses escaped double quotes (`""` SQLite escaping) | R17 |
90
+ | L-15 | `sql_retriever.py:258` | Dead escape code in word-level `make_fuzzy_query` | R17 |
91
+ | L-16 | `sql_retriever.py:204` | Data values resembling chunk IDs not escaped | R18 |
92
+ | L-17 | `sql_retriever.py:54` | Unterminated block comment silently swallows rest of query | R18 |
93
+ | L-18 | `sql_retriever.py:239` | `make_fuzzy_query` doesn't match dotted column names (`t.col`) | R18 |
94
+ | L-19 | `readers/pdf.py:14` | ImportError swallowed silently (applies to all readers) | R15 |
95
+ | L-20 | `kb_meta.py:262` | Full collection loaded into memory (scalability at 100K+ chunks) | R18 |
96
+ | L-21 | `kb_meta.py:313` | Sample chunks may not be chunk_index=0 (no sort guarantee) | R17 |
97
+ | L-22 | `app_cli.py:380` | `effective_user_msg` no-op ternary (both branches identical) | R17 |
98
+ | L-23 | `app_web.py:159-170` | New questions treated as clarification answers (no cancel mechanism) | R15 |
99
+ | L-24 | `app_web.py` | No source/chunk viewer in web UI (CLI has `/sources`) | R17 |
100
+ | L-25 | `llm/openai.py:36` | Empty `choices` indistinguishable from content filter | R18 |
101
+ | L-26 | `setup.py:101` | Backslash unescape order wrong in `generate_env` | R13 |
102
+ | L-27 | `setup.py:140` | Invalid provider choice silently defaults to "openai" | R18 |
103
+ | L-28 | `readers/csv_tab.py:12` | Encoding inconsistency (`utf-8-sig`) vs `text.py` (`utf-8`) | R18 |
104
+
105
+ ---
106
+
107
+ ## Test Coverage Gaps
108
+
109
+ | Priority | Function | File | Risk |
110
+ |----------|----------|------|------|
111
+ | HIGH | `validate_citations()` | `verifier.py:152-236` | Core Layer 4.5 β€” complex regex, zero tests |
112
+ | HIGH | Verification correction loop (iteration > 1) | `verifier.py:422-485` | No test exercises correction path |
113
+ | HIGH | `app_web.py` (entire file) | `app_web.py` | Streamlit UI has zero test coverage |
114
+ | MEDIUM | `make_fuzzy_query()` | `sql_retriever.py:189-279` | SQL retry chain critical path |
115
+ | MEDIUM | `compute_similarity_flags()` | `verifier.py:101-149` | Layer 4 semantic checking |
116
+ | MEDIUM | `load_kb_meta_brief()` | `kb_meta.py:434-445` | Fallback logic untested |
117
+ | MEDIUM | `_try_alternate_columns()` | `retriever.py:252-325` | SQL alt-column search |
118
+ | LOW | `_build_fallback_sql_query()` | `retriever.py:190-249` | Only tested indirectly |
119
+ | LOW | `generate_env()` key merging | `setup.py:83-102` | Merge with existing .env untested |
120
+ | LOW | Unknown command handling | `app_cli.py` | No test for `/foobar` |
121
+
122
+ ### Misleading Tests
123
+
124
+ | Test | Issue |
125
+ |------|-------|
126
+ | `test_list_models_fallback` (test_llm.py:21) | Tests static list, not actual fallback β€” passes with broken code |
127
+ | `test_handle_command_unknown` (test_app_cli.py:14) | Tests "not a command" path, not unknown command like `/foobar` |
128
+ | `test_verify_and_respond` mock patches (test_verifier.py) | Undifferentiated mocks β€” fragile, relies on side effects |
129
+
130
+ ---
131
+
132
+ ## Fix History (by round)
133
+
134
+ | Round | Date | Bugs Fixed | Key Changes |
135
+ |-------|------|------------|-------------|
136
+ | R1 | 2026-03-05 | 3 | Config field names, run command, dead code |
137
+ | R2 | 2026-03-05 | 8 | Hardcoded paths, JSON parse crash, `[N]` placeholder, top_k |
138
+ | R3 | 2026-03-05 | 10 | SQL injection protection (SELECT-only, keyword blocklist, read-only conn) |
139
+ | R4 | 2026-03-05 | 4 | QU→retriever→verifier pipeline threading |
140
+ | R5 | 2026-03-05 | 6 | SQL ingest column dedup, markdown fences, brace escaping |
141
+ | R6 | 2026-03-06 | 4 | KB meta overview enrichment, QU prompt KB awareness |
142
+ | R7 | 2026-03-06 | 19 | UNION block, embedding crash, chunk ID collision, reasoning models |
143
+ | R8 | 2026-03-06 | 15 | Comment stripping execution, nested comments, 12 dangerous keywords |
144
+ | R9 | 2026-03-06 | 11 | Anti-hallucination: correction prompt, Layer 4.5, 10-point checklist |
145
+ | R10 | 2026-03-06 | 3 | SQL fuzzy matching for country names |
146
+ | R11 | 2026-03-06 | 16 | QU max_tokens, Gemini error leaks, fuzzy query sanitization |
147
+ | R12 | 2026-03-06 | 33 | Pre-release final audit β€” 174/174 tests. **Released to GitHub.** |
148
+ | R13 | 2026-03-07 | 28 | Post-release: verification bypass, SQL ESCAPE, Gemini safety, readers |
149
+ | R14 | 2026-03-07 | 29 | All R13 T1+T2 bugs + regression fix (double-backslash ESCAPE) |
150
+ | R15 | 2026-03-07 | 2 | sql_count systematic failure, reader try/except |
151
+ | R16 | 2026-03-07 | 4 | Context budget (`max_context_chars`), `top_k` 50β†’20 |
152
+ | R17 | 2026-03-07 | 5 | validate_citations regex, _strip_sql_comments state machine, web budget |
153
+ | R18 | 2026-03-07 | 0 | Final audit only β€” 0 CRITICAL, 0 HIGH found. 4 new MEDIUM, 10 new LOW. |
154
+ | **Total** | | **~210** | |
155
+
156
+ ---
157
+
158
+ ## Accepted Trade-offs
159
+
160
+ These are known limitations that are by-design or too low-risk to fix:
161
+
162
+ | ID | Description | Rationale |
163
+ |----|-------------|-----------|
164
+ | K1 | SQL validation false positives for data containing "UNION"/"SELECT" | Conservative security β€” read-only conn is the real guard |
165
+ | K2 | `WITH` (CTE) and subqueries blocked | Same β€” overly restrictive but safe direction |
166
+ | K3 | No SQL execution timeout for Cartesian products | Mitigated by progress handler (10s abort) |
167
+ | K4 | `list_models()` returns hardcoded lists, not live API | Avoids API key validation complexity at setup time |
168
+ | K5 | No streaming β€” 30-60s blank screen during verification | Would require significant refactor of verification loop |
169
+ | K6 | Path resolution duplicated 12+ times | Works correctly; utility extraction is cleanup, not bugfix |