Spaces:
Sleeping
Sleeping
Commit ·
d4090f3
1
Parent(s): 31d8af3
chore: remove docs/ from tracking (local-only, gitignored)
Browse filesCo-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- docs/BUGLOG.md +0 -169
docs/BUGLOG.md
DELETED
|
@@ -1,169 +0,0 @@
|
|
| 1 |
-
# Bug Log — RAG Research Chatbot Template
|
| 2 |
-
|
| 3 |
-
**Consolidated tracking file for all bugs, evaluations, and test coverage gaps.**
|
| 4 |
-
**Last updated:** Round 18 (2026-03-07)
|
| 5 |
-
|
| 6 |
-
---
|
| 7 |
-
|
| 8 |
-
## Summary
|
| 9 |
-
|
| 10 |
-
| Metric | Value |
|
| 11 |
-
|--------|-------|
|
| 12 |
-
| Audit rounds completed | 18 (70+ agent dispatches) |
|
| 13 |
-
| Total bugs found (all rounds) | ~270+ |
|
| 14 |
-
| Total bugs fixed | ~210 |
|
| 15 |
-
| Open bugs | 0 CRITICAL, 0 HIGH, 14 MEDIUM, 28 LOW |
|
| 16 |
-
| Tests | 175 passing, 17 files |
|
| 17 |
-
| Design score | 6.6/10 (Round 17), 7/10 (Round 15) |
|
| 18 |
-
|
| 19 |
-
---
|
| 20 |
-
|
| 21 |
-
## Design Evaluation (Round 17)
|
| 22 |
-
|
| 23 |
-
| Dimension | Score | Key Issue |
|
| 24 |
-
|-----------|-------|-----------|
|
| 25 |
-
| Architecture Quality | 7/10 | Deferred imports hide coupling, duplicated path resolution (12+ sites) |
|
| 26 |
-
| Anti-Hallucination | 8/10 | Layer 0 airtight; verification prompt lacks user query |
|
| 27 |
-
| Query Understanding | 7/10 | Dual-query design strong; no input sanitization |
|
| 28 |
-
| Retrieval Strategy | 7/10 | Multi-step SQL fallback thorough; no hybrid ranking for "both" route |
|
| 29 |
-
| User Experience | 6/10 | No streaming (30-60s blank screen); CLI/Web feature parity gap |
|
| 30 |
-
| Production Readiness | 5/10 | No logging, no retry/backoff, no config validation, no timeouts |
|
| 31 |
-
| Design Gaps | 6/10 | No cost tracking, no incremental ingestion, no conversation persistence |
|
| 32 |
-
| **OVERALL** | **6.6/10** | Strong prototype, needs production hardening |
|
| 33 |
-
|
| 34 |
-
### Top 10 Design Improvements
|
| 35 |
-
|
| 36 |
-
1. Cache LLM clients (singleton per provider+key) — saves 500ms-2s per query
|
| 37 |
-
2. Add structured logging (replace 40+ `print()` statements)
|
| 38 |
-
3. Incremental ingestion (hash-based skip)
|
| 39 |
-
4. Fix Layer 4 to check per-source overlap (not all context combined)
|
| 40 |
-
5. Structured JSON output for verification calls
|
| 41 |
-
6. Extract path resolution utility (12+ duplicate sites)
|
| 42 |
-
7. Live `list_models()` via provider APIs
|
| 43 |
-
8. API key validation in setup wizard
|
| 44 |
-
9. Chat export/persistence
|
| 45 |
-
10. Pagination for `_collect_all_data()` in kb_meta.py
|
| 46 |
-
|
| 47 |
-
---
|
| 48 |
-
|
| 49 |
-
## Open Bugs — MEDIUM (14)
|
| 50 |
-
|
| 51 |
-
Deduplicated across all rounds. Ordered by impact.
|
| 52 |
-
|
| 53 |
-
| ID | File:Line | Description | Found |
|
| 54 |
-
|----|-----------|-------------|-------|
|
| 55 |
-
| M-01 | `prompts.py:349` | **Verification prompt delimiter injection** — no sanitization for `--- END CONTEXT ---` in document chunks; attacker text could bypass verifier | R18 |
|
| 56 |
-
| M-02 | `verifier.py:452` | **Verification loop no-break on LLM failure** — `except: continue` burns all iterations when LLM consistently fails; should break after 2 consecutive failures | R17 |
|
| 57 |
-
| M-03 | `gemini.py:29` | **Gemini safety filter returns ""** — sentinel check at verifier.py:387-389 is dead code; user gets generic error, not safety-specific message | R15 |
|
| 58 |
-
| M-04 | `sql_retriever.py:81` | **Semicolon in quoted strings rejected** — `";" in stripped` checks raw SQL, not `_strip_quoted()`; blocks valid queries like `WHERE col = 'a;b'` | R15 |
|
| 59 |
-
| M-05 | `sql_retriever.py:13` | **WITH (CTE) blocked** — in dangerous keywords list; also blocked by multi-SELECT check on line 92; double-blocked, both overly aggressive | R17 |
|
| 60 |
-
| M-06 | `sql_retriever.py:92` | **Subquery block too aggressive** — `SELECT` count > 1 rejects `WHERE EXISTS(...)`, scalar subqueries, `CASE WHEN` with subselects | R17 |
|
| 61 |
-
| M-07 | `sql_retriever.py:89` | **sqlite_master check on raw SQL** — uses `stripped` instead of `unquoted`; false-rejects `WHERE col = 'sqlite_master'` as data value | R13 |
|
| 62 |
-
| M-08 | `retriever.py:271` | **_try_alternate_columns regex not anchored to WHERE** — can match expressions in SELECT, FROM, or other clauses | R17 |
|
| 63 |
-
| M-09 | `retriever.py:315` | **_try_alternate_columns sanitization order** — `replace("'","''")` then regex strip undoes the escaping; "Cote d'Ivoire" becomes "Cote dIvoire" | R17 |
|
| 64 |
-
| M-10 | `ingest.py:263` | **Partial-success ingestion data loss** — if first file clears old data but subsequent files fail, previous ingestion lost (partially mitigated by deferred clear) | R15 |
|
| 65 |
-
| M-11 | `sql_ingest.py:57` | **Type inference inconsistency** — `_infer_column_type` uses `int(s)` (rejects "3.0") but insertion uses `int(float(s))` (accepts "3.0") | R17 |
|
| 66 |
-
| M-12 | `app_web.py:165+190` | **Clarification doubles context in QU prompt** — clarification Q&A appears in both `combined` query string AND conversation history | R17 |
|
| 67 |
-
| M-13 | `app_web.py:59,104,109` | **Sidebar mutates shared cfg dict** — direct in-place mutation of session state config; fragile but currently works due to Streamlit rerun model | R18 |
|
| 68 |
-
| M-14 | `setup.py:198-200` | **.env overwritten on decline** — user declines config.yaml overwrite but .env is still written; can blank existing API key | R15 |
|
| 69 |
-
|
| 70 |
-
---
|
| 71 |
-
|
| 72 |
-
## Open Bugs — LOW (28)
|
| 73 |
-
|
| 74 |
-
| ID | File:Line | Description | Found |
|
| 75 |
-
|----|-----------|-------------|-------|
|
| 76 |
-
| L-01 | `llm/*.py` | No retry/backoff on LLM calls (only semantic_scholar has retry) | R13 |
|
| 77 |
-
| L-02 | `llm/*.py` | No request timeouts on LLM API calls | R13 |
|
| 78 |
-
| L-03 | `llm/*.py` | New LLM client on every `generate()` call (no connection reuse) | R13 |
|
| 79 |
-
| L-04 | `config_loader.py` | No config validation — `max_distance: "banana"` fails at ChromaDB | R15 |
|
| 80 |
-
| L-05 | `config_loader.py:61` | `get_api_key` doesn't strip whitespace | R15 |
|
| 81 |
-
| L-06 | `config_loader.py:42` | Non-dict section values (e.g. `paths: 42`) cause AttributeError | R18 |
|
| 82 |
-
| L-07 | `ingest.py` | No embedding model tracking — switching after ingestion corrupts search | R15 |
|
| 83 |
-
| L-08 | `ingest.py:226` | Overlap cap allows exactly 80% (should be < 80%) | R17 |
|
| 84 |
-
| L-09 | `ingest.py:255` | MD5 12-char prefix collision risk at ~25K files | R15 |
|
| 85 |
-
| L-10 | `prompts.py:85-141` | Verification prompt lacks user query — can't check relevance | R17 |
|
| 86 |
-
| L-11 | `prompts.py:312` | Dash delimiters (`---`) not sanitized (only `===` is) | R18 |
|
| 87 |
-
| L-12 | `verifier.py:473` | Correction prompt interpolates raw `query` without sanitization | R18 |
|
| 88 |
-
| L-13 | `sql_retriever.py:27-68` | `_strip_sql_comments` doesn't handle double-quoted identifiers | R13 |
|
| 89 |
-
| L-14 | `sql_retriever.py:23` | `_strip_quoted` misses escaped double quotes (`""` SQLite escaping) | R17 |
|
| 90 |
-
| L-15 | `sql_retriever.py:258` | Dead escape code in word-level `make_fuzzy_query` | R17 |
|
| 91 |
-
| L-16 | `sql_retriever.py:204` | Data values resembling chunk IDs not escaped | R18 |
|
| 92 |
-
| L-17 | `sql_retriever.py:54` | Unterminated block comment silently swallows rest of query | R18 |
|
| 93 |
-
| L-18 | `sql_retriever.py:239` | `make_fuzzy_query` doesn't match dotted column names (`t.col`) | R18 |
|
| 94 |
-
| L-19 | `readers/pdf.py:14` | ImportError swallowed silently (applies to all readers) | R15 |
|
| 95 |
-
| L-20 | `kb_meta.py:262` | Full collection loaded into memory (scalability at 100K+ chunks) | R18 |
|
| 96 |
-
| L-21 | `kb_meta.py:313` | Sample chunks may not be chunk_index=0 (no sort guarantee) | R17 |
|
| 97 |
-
| L-22 | `app_cli.py:380` | `effective_user_msg` no-op ternary (both branches identical) | R17 |
|
| 98 |
-
| L-23 | `app_web.py:159-170` | New questions treated as clarification answers (no cancel mechanism) | R15 |
|
| 99 |
-
| L-24 | `app_web.py` | No source/chunk viewer in web UI (CLI has `/sources`) | R17 |
|
| 100 |
-
| L-25 | `llm/openai.py:36` | Empty `choices` indistinguishable from content filter | R18 |
|
| 101 |
-
| L-26 | `setup.py:101` | Backslash unescape order wrong in `generate_env` | R13 |
|
| 102 |
-
| L-27 | `setup.py:140` | Invalid provider choice silently defaults to "openai" | R18 |
|
| 103 |
-
| L-28 | `readers/csv_tab.py:12` | Encoding inconsistency (`utf-8-sig`) vs `text.py` (`utf-8`) | R18 |
|
| 104 |
-
|
| 105 |
-
---
|
| 106 |
-
|
| 107 |
-
## Test Coverage Gaps
|
| 108 |
-
|
| 109 |
-
| Priority | Function | File | Risk |
|
| 110 |
-
|----------|----------|------|------|
|
| 111 |
-
| HIGH | `validate_citations()` | `verifier.py:152-236` | Core Layer 4.5 — complex regex, zero tests |
|
| 112 |
-
| HIGH | Verification correction loop (iteration > 1) | `verifier.py:422-485` | No test exercises correction path |
|
| 113 |
-
| HIGH | `app_web.py` (entire file) | `app_web.py` | Streamlit UI has zero test coverage |
|
| 114 |
-
| MEDIUM | `make_fuzzy_query()` | `sql_retriever.py:189-279` | SQL retry chain critical path |
|
| 115 |
-
| MEDIUM | `compute_similarity_flags()` | `verifier.py:101-149` | Layer 4 semantic checking |
|
| 116 |
-
| MEDIUM | `load_kb_meta_brief()` | `kb_meta.py:434-445` | Fallback logic untested |
|
| 117 |
-
| MEDIUM | `_try_alternate_columns()` | `retriever.py:252-325` | SQL alt-column search |
|
| 118 |
-
| LOW | `_build_fallback_sql_query()` | `retriever.py:190-249` | Only tested indirectly |
|
| 119 |
-
| LOW | `generate_env()` key merging | `setup.py:83-102` | Merge with existing .env untested |
|
| 120 |
-
| LOW | Unknown command handling | `app_cli.py` | No test for `/foobar` |
|
| 121 |
-
|
| 122 |
-
### Misleading Tests
|
| 123 |
-
|
| 124 |
-
| Test | Issue |
|
| 125 |
-
|------|-------|
|
| 126 |
-
| `test_list_models_fallback` (test_llm.py:21) | Tests static list, not actual fallback — passes with broken code |
|
| 127 |
-
| `test_handle_command_unknown` (test_app_cli.py:14) | Tests "not a command" path, not unknown command like `/foobar` |
|
| 128 |
-
| `test_verify_and_respond` mock patches (test_verifier.py) | Undifferentiated mocks — fragile, relies on side effects |
|
| 129 |
-
|
| 130 |
-
---
|
| 131 |
-
|
| 132 |
-
## Fix History (by round)
|
| 133 |
-
|
| 134 |
-
| Round | Date | Bugs Fixed | Key Changes |
|
| 135 |
-
|-------|------|------------|-------------|
|
| 136 |
-
| R1 | 2026-03-05 | 3 | Config field names, run command, dead code |
|
| 137 |
-
| R2 | 2026-03-05 | 8 | Hardcoded paths, JSON parse crash, `[N]` placeholder, top_k |
|
| 138 |
-
| R3 | 2026-03-05 | 10 | SQL injection protection (SELECT-only, keyword blocklist, read-only conn) |
|
| 139 |
-
| R4 | 2026-03-05 | 4 | QU→retriever→verifier pipeline threading |
|
| 140 |
-
| R5 | 2026-03-05 | 6 | SQL ingest column dedup, markdown fences, brace escaping |
|
| 141 |
-
| R6 | 2026-03-06 | 4 | KB meta overview enrichment, QU prompt KB awareness |
|
| 142 |
-
| R7 | 2026-03-06 | 19 | UNION block, embedding crash, chunk ID collision, reasoning models |
|
| 143 |
-
| R8 | 2026-03-06 | 15 | Comment stripping execution, nested comments, 12 dangerous keywords |
|
| 144 |
-
| R9 | 2026-03-06 | 11 | Anti-hallucination: correction prompt, Layer 4.5, 10-point checklist |
|
| 145 |
-
| R10 | 2026-03-06 | 3 | SQL fuzzy matching for country names |
|
| 146 |
-
| R11 | 2026-03-06 | 16 | QU max_tokens, Gemini error leaks, fuzzy query sanitization |
|
| 147 |
-
| R12 | 2026-03-06 | 33 | Pre-release final audit — 174/174 tests. **Released to GitHub.** |
|
| 148 |
-
| R13 | 2026-03-07 | 28 | Post-release: verification bypass, SQL ESCAPE, Gemini safety, readers |
|
| 149 |
-
| R14 | 2026-03-07 | 29 | All R13 T1+T2 bugs + regression fix (double-backslash ESCAPE) |
|
| 150 |
-
| R15 | 2026-03-07 | 2 | sql_count systematic failure, reader try/except |
|
| 151 |
-
| R16 | 2026-03-07 | 4 | Context budget (`max_context_chars`), `top_k` 50→20 |
|
| 152 |
-
| R17 | 2026-03-07 | 5 | validate_citations regex, _strip_sql_comments state machine, web budget |
|
| 153 |
-
| R18 | 2026-03-07 | 0 | Final audit only — 0 CRITICAL, 0 HIGH found. 4 new MEDIUM, 10 new LOW. |
|
| 154 |
-
| **Total** | | **~210** | |
|
| 155 |
-
|
| 156 |
-
---
|
| 157 |
-
|
| 158 |
-
## Accepted Trade-offs
|
| 159 |
-
|
| 160 |
-
These are known limitations that are by-design or too low-risk to fix:
|
| 161 |
-
|
| 162 |
-
| ID | Description | Rationale |
|
| 163 |
-
|----|-------------|-----------|
|
| 164 |
-
| K1 | SQL validation false positives for data containing "UNION"/"SELECT" | Conservative security — read-only conn is the real guard |
|
| 165 |
-
| K2 | `WITH` (CTE) and subqueries blocked | Same — overly restrictive but safe direction |
|
| 166 |
-
| K3 | No SQL execution timeout for Cartesian products | Mitigated by progress handler (10s abort) |
|
| 167 |
-
| K4 | `list_models()` returns hardcoded lists, not live API | Avoids API key validation complexity at setup time |
|
| 168 |
-
| K5 | No streaming — 30-60s blank screen during verification | Would require significant refactor of verification loop |
|
| 169 |
-
| K6 | Path resolution duplicated 12+ times | Works correctly; utility extraction is cleanup, not bugfix |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|