LIANJie-Jason Claude Opus 4.6 commited on
Commit
1eac8bc
·
1 Parent(s): f9f7706

fix: verification prompt now allows SQL data summarization across rows

Browse files

The verifier was rejecting valid responses for SQL-heavy queries because
check #2 demanded an EXACT data point match for each claim. When the LLM
summarizes across 49 SQL rows (e.g., "scores ranged from 3 to 5"), no
single row contains that statement, causing repeated verification failure
and strict-mode refusal.

Updated verification checklist:
- Check #2: SQL rows may be legitimately summarized; verify underlying
data points exist across rows, not that one row matches verbatim
- Check #10: SQL citations may cover an entire dataset/table

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Files changed (1) hide show
  1. src/prompts.py +11 -2
src/prompts.py CHANGED
@@ -106,7 +106,14 @@ problems.
106
  Run through this 10-point verification checklist and report results:
107
 
108
  1. Does every factual claim have a numbered citation endnote (e.g. [1], [2])?
109
- 2. For each cited claim, can you find the EXACT sentence or data point in the context that supports it? If you cannot point to specific supporting text, flag it as an error.
 
 
 
 
 
 
 
110
  3. Are there any claims that appear fabricated or hallucinated?
111
  4. Are direct quotes accurately reproduced from the context?
112
  5. Does the References section exist and list all cited sources?
@@ -114,7 +121,9 @@ Run through this 10-point verification checklist and report results:
114
  7. Are any phrase-level flags confirmed as hallucinations?
115
  8. Do similarity flags indicate unsupported semantic drift?
116
  9. Is the response length proportional to the available evidence?
117
- 10. Do the citation numbers match the correct sources? (e.g., [1] must refer to the first source listed in References, and that source must actually support the claim)
 
 
118
 
119
  Return your analysis as JSON in the following format:
120
 
 
106
  Run through this 10-point verification checklist and report results:
107
 
108
  1. Does every factual claim have a numbered citation endnote (e.g. [1], [2])?
109
+ 2. For each cited claim, can you find supporting evidence in the context?
110
+ - For prose/document sources: look for the specific sentence or passage.
111
+ - For SQL/tabular data ([CHUNK-SQL-NNN] rows): the response may
112
+ legitimately SUMMARIZE across multiple rows (e.g., "scores ranged
113
+ from 3 to 5"). Verify that the individual data points exist in the
114
+ SQL rows, even if no single row contains the exact summary statement.
115
+ Aggregation, averaging, and range descriptions across SQL rows are
116
+ VALID as long as the underlying numbers are present in the data.
117
  3. Are there any claims that appear fabricated or hallucinated?
118
  4. Are direct quotes accurately reproduced from the context?
119
  5. Does the References section exist and list all cited sources?
 
121
  7. Are any phrase-level flags confirmed as hallucinations?
122
  8. Do similarity flags indicate unsupported semantic drift?
123
  9. Is the response length proportional to the available evidence?
124
+ 10. Do the citation numbers match the sources? Each citation should refer to
125
+ a source in the References section that actually supports the claim.
126
+ For SQL data, a single citation may cover an entire dataset/table.
127
 
128
  Return your analysis as JSON in the following format:
129