LIANJie-Jason Claude Opus 4.6 commited on
Commit
d4090f3
·
1 Parent(s): 31d8af3

chore: remove docs/ from tracking (local-only, gitignored)

Browse files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Files changed (1) hide show
  1. docs/BUGLOG.md +0 -169
docs/BUGLOG.md DELETED
@@ -1,169 +0,0 @@
1
- # Bug Log — RAG Research Chatbot Template
2
-
3
- **Consolidated tracking file for all bugs, evaluations, and test coverage gaps.**
4
- **Last updated:** Round 18 (2026-03-07)
5
-
6
- ---
7
-
8
- ## Summary
9
-
10
- | Metric | Value |
11
- |--------|-------|
12
- | Audit rounds completed | 18 (70+ agent dispatches) |
13
- | Total bugs found (all rounds) | ~270+ |
14
- | Total bugs fixed | ~210 |
15
- | Open bugs | 0 CRITICAL, 0 HIGH, 14 MEDIUM, 28 LOW |
16
- | Tests | 175 passing, 17 files |
17
- | Design score | 6.6/10 (Round 17), 7/10 (Round 15) |
18
-
19
- ---
20
-
21
- ## Design Evaluation (Round 17)
22
-
23
- | Dimension | Score | Key Issue |
24
- |-----------|-------|-----------|
25
- | Architecture Quality | 7/10 | Deferred imports hide coupling, duplicated path resolution (12+ sites) |
26
- | Anti-Hallucination | 8/10 | Layer 0 airtight; verification prompt lacks user query |
27
- | Query Understanding | 7/10 | Dual-query design strong; no input sanitization |
28
- | Retrieval Strategy | 7/10 | Multi-step SQL fallback thorough; no hybrid ranking for "both" route |
29
- | User Experience | 6/10 | No streaming (30-60s blank screen); CLI/Web feature parity gap |
30
- | Production Readiness | 5/10 | No logging, no retry/backoff, no config validation, no timeouts |
31
- | Design Gaps | 6/10 | No cost tracking, no incremental ingestion, no conversation persistence |
32
- | **OVERALL** | **6.6/10** | Strong prototype, needs production hardening |
33
-
34
- ### Top 10 Design Improvements
35
-
36
- 1. Cache LLM clients (singleton per provider+key) — saves 500ms-2s per query
37
- 2. Add structured logging (replace 40+ `print()` statements)
38
- 3. Incremental ingestion (hash-based skip)
39
- 4. Fix Layer 4 to check per-source overlap (not all context combined)
40
- 5. Structured JSON output for verification calls
41
- 6. Extract path resolution utility (12+ duplicate sites)
42
- 7. Live `list_models()` via provider APIs
43
- 8. API key validation in setup wizard
44
- 9. Chat export/persistence
45
- 10. Pagination for `_collect_all_data()` in kb_meta.py
46
-
47
- ---
48
-
49
- ## Open Bugs — MEDIUM (14)
50
-
51
- Deduplicated across all rounds. Ordered by impact.
52
-
53
- | ID | File:Line | Description | Found |
54
- |----|-----------|-------------|-------|
55
- | M-01 | `prompts.py:349` | **Verification prompt delimiter injection** — no sanitization for `--- END CONTEXT ---` in document chunks; attacker text could bypass verifier | R18 |
56
- | M-02 | `verifier.py:452` | **Verification loop no-break on LLM failure** — `except: continue` burns all iterations when LLM consistently fails; should break after 2 consecutive failures | R17 |
57
- | M-03 | `gemini.py:29` | **Gemini safety filter returns ""** — sentinel check at verifier.py:387-389 is dead code; user gets generic error, not safety-specific message | R15 |
58
- | M-04 | `sql_retriever.py:81` | **Semicolon in quoted strings rejected** — `";" in stripped` checks raw SQL, not `_strip_quoted()`; blocks valid queries like `WHERE col = 'a;b'` | R15 |
59
- | M-05 | `sql_retriever.py:13` | **WITH (CTE) blocked** — in dangerous keywords list; also blocked by multi-SELECT check on line 92; double-blocked, both overly aggressive | R17 |
60
- | M-06 | `sql_retriever.py:92` | **Subquery block too aggressive** — `SELECT` count > 1 rejects `WHERE EXISTS(...)`, scalar subqueries, `CASE WHEN` with subselects | R17 |
61
- | M-07 | `sql_retriever.py:89` | **sqlite_master check on raw SQL** — uses `stripped` instead of `unquoted`; false-rejects `WHERE col = 'sqlite_master'` as data value | R13 |
62
- | M-08 | `retriever.py:271` | **_try_alternate_columns regex not anchored to WHERE** — can match expressions in SELECT, FROM, or other clauses | R17 |
63
- | M-09 | `retriever.py:315` | **_try_alternate_columns sanitization order** — `replace("'","''")` then regex strip undoes the escaping; "Cote d'Ivoire" becomes "Cote dIvoire" | R17 |
64
- | M-10 | `ingest.py:263` | **Partial-success ingestion data loss** — if first file clears old data but subsequent files fail, previous ingestion lost (partially mitigated by deferred clear) | R15 |
65
- | M-11 | `sql_ingest.py:57` | **Type inference inconsistency** — `_infer_column_type` uses `int(s)` (rejects "3.0") but insertion uses `int(float(s))` (accepts "3.0") | R17 |
66
- | M-12 | `app_web.py:165+190` | **Clarification doubles context in QU prompt** — clarification Q&A appears in both `combined` query string AND conversation history | R17 |
67
- | M-13 | `app_web.py:59,104,109` | **Sidebar mutates shared cfg dict** — direct in-place mutation of session state config; fragile but currently works due to Streamlit rerun model | R18 |
68
- | M-14 | `setup.py:198-200` | **.env overwritten on decline** — user declines config.yaml overwrite but .env is still written; can blank existing API key | R15 |
69
-
70
- ---
71
-
72
- ## Open Bugs — LOW (28)
73
-
74
- | ID | File:Line | Description | Found |
75
- |----|-----------|-------------|-------|
76
- | L-01 | `llm/*.py` | No retry/backoff on LLM calls (only semantic_scholar has retry) | R13 |
77
- | L-02 | `llm/*.py` | No request timeouts on LLM API calls | R13 |
78
- | L-03 | `llm/*.py` | New LLM client on every `generate()` call (no connection reuse) | R13 |
79
- | L-04 | `config_loader.py` | No config validation — `max_distance: "banana"` fails at ChromaDB | R15 |
80
- | L-05 | `config_loader.py:61` | `get_api_key` doesn't strip whitespace | R15 |
81
- | L-06 | `config_loader.py:42` | Non-dict section values (e.g. `paths: 42`) cause AttributeError | R18 |
82
- | L-07 | `ingest.py` | No embedding model tracking — switching after ingestion corrupts search | R15 |
83
- | L-08 | `ingest.py:226` | Overlap cap allows exactly 80% (should be < 80%) | R17 |
84
- | L-09 | `ingest.py:255` | MD5 12-char prefix collision risk at ~25K files | R15 |
85
- | L-10 | `prompts.py:85-141` | Verification prompt lacks user query — can't check relevance | R17 |
86
- | L-11 | `prompts.py:312` | Dash delimiters (`---`) not sanitized (only `===` is) | R18 |
87
- | L-12 | `verifier.py:473` | Correction prompt interpolates raw `query` without sanitization | R18 |
88
- | L-13 | `sql_retriever.py:27-68` | `_strip_sql_comments` doesn't handle double-quoted identifiers | R13 |
89
- | L-14 | `sql_retriever.py:23` | `_strip_quoted` misses escaped double quotes (`""` SQLite escaping) | R17 |
90
- | L-15 | `sql_retriever.py:258` | Dead escape code in word-level `make_fuzzy_query` | R17 |
91
- | L-16 | `sql_retriever.py:204` | Data values resembling chunk IDs not escaped | R18 |
92
- | L-17 | `sql_retriever.py:54` | Unterminated block comment silently swallows rest of query | R18 |
93
- | L-18 | `sql_retriever.py:239` | `make_fuzzy_query` doesn't match dotted column names (`t.col`) | R18 |
94
- | L-19 | `readers/pdf.py:14` | ImportError swallowed silently (applies to all readers) | R15 |
95
- | L-20 | `kb_meta.py:262` | Full collection loaded into memory (scalability at 100K+ chunks) | R18 |
96
- | L-21 | `kb_meta.py:313` | Sample chunks may not be chunk_index=0 (no sort guarantee) | R17 |
97
- | L-22 | `app_cli.py:380` | `effective_user_msg` no-op ternary (both branches identical) | R17 |
98
- | L-23 | `app_web.py:159-170` | New questions treated as clarification answers (no cancel mechanism) | R15 |
99
- | L-24 | `app_web.py` | No source/chunk viewer in web UI (CLI has `/sources`) | R17 |
100
- | L-25 | `llm/openai.py:36` | Empty `choices` indistinguishable from content filter | R18 |
101
- | L-26 | `setup.py:101` | Backslash unescape order wrong in `generate_env` | R13 |
102
- | L-27 | `setup.py:140` | Invalid provider choice silently defaults to "openai" | R18 |
103
- | L-28 | `readers/csv_tab.py:12` | Encoding inconsistency (`utf-8-sig`) vs `text.py` (`utf-8`) | R18 |
104
-
105
- ---
106
-
107
- ## Test Coverage Gaps
108
-
109
- | Priority | Function | File | Risk |
110
- |----------|----------|------|------|
111
- | HIGH | `validate_citations()` | `verifier.py:152-236` | Core Layer 4.5 — complex regex, zero tests |
112
- | HIGH | Verification correction loop (iteration > 1) | `verifier.py:422-485` | No test exercises correction path |
113
- | HIGH | `app_web.py` (entire file) | `app_web.py` | Streamlit UI has zero test coverage |
114
- | MEDIUM | `make_fuzzy_query()` | `sql_retriever.py:189-279` | SQL retry chain critical path |
115
- | MEDIUM | `compute_similarity_flags()` | `verifier.py:101-149` | Layer 4 semantic checking |
116
- | MEDIUM | `load_kb_meta_brief()` | `kb_meta.py:434-445` | Fallback logic untested |
117
- | MEDIUM | `_try_alternate_columns()` | `retriever.py:252-325` | SQL alt-column search |
118
- | LOW | `_build_fallback_sql_query()` | `retriever.py:190-249` | Only tested indirectly |
119
- | LOW | `generate_env()` key merging | `setup.py:83-102` | Merge with existing .env untested |
120
- | LOW | Unknown command handling | `app_cli.py` | No test for `/foobar` |
121
-
122
- ### Misleading Tests
123
-
124
- | Test | Issue |
125
- |------|-------|
126
- | `test_list_models_fallback` (test_llm.py:21) | Tests static list, not actual fallback — passes with broken code |
127
- | `test_handle_command_unknown` (test_app_cli.py:14) | Tests "not a command" path, not unknown command like `/foobar` |
128
- | `test_verify_and_respond` mock patches (test_verifier.py) | Undifferentiated mocks — fragile, relies on side effects |
129
-
130
- ---
131
-
132
- ## Fix History (by round)
133
-
134
- | Round | Date | Bugs Fixed | Key Changes |
135
- |-------|------|------------|-------------|
136
- | R1 | 2026-03-05 | 3 | Config field names, run command, dead code |
137
- | R2 | 2026-03-05 | 8 | Hardcoded paths, JSON parse crash, `[N]` placeholder, top_k |
138
- | R3 | 2026-03-05 | 10 | SQL injection protection (SELECT-only, keyword blocklist, read-only conn) |
139
- | R4 | 2026-03-05 | 4 | QU→retriever→verifier pipeline threading |
140
- | R5 | 2026-03-05 | 6 | SQL ingest column dedup, markdown fences, brace escaping |
141
- | R6 | 2026-03-06 | 4 | KB meta overview enrichment, QU prompt KB awareness |
142
- | R7 | 2026-03-06 | 19 | UNION block, embedding crash, chunk ID collision, reasoning models |
143
- | R8 | 2026-03-06 | 15 | Comment stripping execution, nested comments, 12 dangerous keywords |
144
- | R9 | 2026-03-06 | 11 | Anti-hallucination: correction prompt, Layer 4.5, 10-point checklist |
145
- | R10 | 2026-03-06 | 3 | SQL fuzzy matching for country names |
146
- | R11 | 2026-03-06 | 16 | QU max_tokens, Gemini error leaks, fuzzy query sanitization |
147
- | R12 | 2026-03-06 | 33 | Pre-release final audit — 174/174 tests. **Released to GitHub.** |
148
- | R13 | 2026-03-07 | 28 | Post-release: verification bypass, SQL ESCAPE, Gemini safety, readers |
149
- | R14 | 2026-03-07 | 29 | All R13 T1+T2 bugs + regression fix (double-backslash ESCAPE) |
150
- | R15 | 2026-03-07 | 2 | sql_count systematic failure, reader try/except |
151
- | R16 | 2026-03-07 | 4 | Context budget (`max_context_chars`), `top_k` 50→20 |
152
- | R17 | 2026-03-07 | 5 | validate_citations regex, _strip_sql_comments state machine, web budget |
153
- | R18 | 2026-03-07 | 0 | Final audit only — 0 CRITICAL, 0 HIGH found. 4 new MEDIUM, 10 new LOW. |
154
- | **Total** | | **~210** | |
155
-
156
- ---
157
-
158
- ## Accepted Trade-offs
159
-
160
- These are known limitations that are by-design or too low-risk to fix:
161
-
162
- | ID | Description | Rationale |
163
- |----|-------------|-----------|
164
- | K1 | SQL validation false positives for data containing "UNION"/"SELECT" | Conservative security — read-only conn is the real guard |
165
- | K2 | `WITH` (CTE) and subqueries blocked | Same — overly restrictive but safe direction |
166
- | K3 | No SQL execution timeout for Cartesian products | Mitigated by progress handler (10s abort) |
167
- | K4 | `list_models()` returns hardcoded lists, not live API | Avoids API key validation complexity at setup time |
168
- | K5 | No streaming — 30-60s blank screen during verification | Would require significant refactor of verification loop |
169
- | K6 | Path resolution duplicated 12+ times | Works correctly; utility extraction is cleanup, not bugfix |