Spaces:
Paused
Paused
Commit ·
1eac8bc
1
Parent(s): f9f7706
fix: verification prompt now allows SQL data summarization across rows
Browse filesThe verifier was rejecting valid responses for SQL-heavy queries because
check #2 demanded an EXACT data point match for each claim. When the LLM
summarizes across 49 SQL rows (e.g., "scores ranged from 3 to 5"), no
single row contains that statement, causing repeated verification failure
and strict-mode refusal.
Updated verification checklist:
- Check #2: SQL rows may be legitimately summarized; verify underlying
data points exist across rows, not that one row matches verbatim
- Check #10: SQL citations may cover an entire dataset/table
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- src/prompts.py +11 -2
src/prompts.py
CHANGED
|
@@ -106,7 +106,14 @@ problems.
|
|
| 106 |
Run through this 10-point verification checklist and report results:
|
| 107 |
|
| 108 |
1. Does every factual claim have a numbered citation endnote (e.g. [1], [2])?
|
| 109 |
-
2. For each cited claim, can you find
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 110 |
3. Are there any claims that appear fabricated or hallucinated?
|
| 111 |
4. Are direct quotes accurately reproduced from the context?
|
| 112 |
5. Does the References section exist and list all cited sources?
|
|
@@ -114,7 +121,9 @@ Run through this 10-point verification checklist and report results:
|
|
| 114 |
7. Are any phrase-level flags confirmed as hallucinations?
|
| 115 |
8. Do similarity flags indicate unsupported semantic drift?
|
| 116 |
9. Is the response length proportional to the available evidence?
|
| 117 |
-
10. Do the citation numbers match the
|
|
|
|
|
|
|
| 118 |
|
| 119 |
Return your analysis as JSON in the following format:
|
| 120 |
|
|
|
|
| 106 |
Run through this 10-point verification checklist and report results:
|
| 107 |
|
| 108 |
1. Does every factual claim have a numbered citation endnote (e.g. [1], [2])?
|
| 109 |
+
2. For each cited claim, can you find supporting evidence in the context?
|
| 110 |
+
- For prose/document sources: look for the specific sentence or passage.
|
| 111 |
+
- For SQL/tabular data ([CHUNK-SQL-NNN] rows): the response may
|
| 112 |
+
legitimately SUMMARIZE across multiple rows (e.g., "scores ranged
|
| 113 |
+
from 3 to 5"). Verify that the individual data points exist in the
|
| 114 |
+
SQL rows, even if no single row contains the exact summary statement.
|
| 115 |
+
Aggregation, averaging, and range descriptions across SQL rows are
|
| 116 |
+
VALID as long as the underlying numbers are present in the data.
|
| 117 |
3. Are there any claims that appear fabricated or hallucinated?
|
| 118 |
4. Are direct quotes accurately reproduced from the context?
|
| 119 |
5. Does the References section exist and list all cited sources?
|
|
|
|
| 121 |
7. Are any phrase-level flags confirmed as hallucinations?
|
| 122 |
8. Do similarity flags indicate unsupported semantic drift?
|
| 123 |
9. Is the response length proportional to the available evidence?
|
| 124 |
+
10. Do the citation numbers match the sources? Each citation should refer to
|
| 125 |
+
a source in the References section that actually supports the claim.
|
| 126 |
+
For SQL data, a single citation may cover an entire dataset/table.
|
| 127 |
|
| 128 |
Return your analysis as JSON in the following format:
|
| 129 |
|