[NOTICKET] fix(knowledge): the diff baseline never reached the ingest path
Browse files`run_document` called `extract_and_persist` WITHOUT `active_glossary`, so the
glossary diff ran against an empty list and every entry came back
`diff_status="new"`. Only the re-extract path passed the approved set.
The consequence is the thing the review queue exists to make tractable: a
company approves terms from document 1, ingests document 2, and document 2's
terms all read as `new` even where they duplicate or contradict what an expert
already ruled on. Cross-document dedup silently did not fire on the path people
will actually use most.
Invisible in the end-to-end run because there were no approvals yet - all 88
entries showed `new`, which was correct then and would be wrong now.
Fixed at the single point that needs it rather than at the call site:
`extract_and_persist` now fetches `approved_glossary(scope_id)` itself when the
argument is omitted, so no caller can forget it again, and the redundant fetch
in `_run_reextract` is gone. An explicit `[]` still means "diff against
nothing", which is what the tests use.
2 regression tests: the baseline is fetched when the argument is omitted, and an
explicit empty list does NOT trigger a fetch.
Verification: ruff clean on touched paths; `import main` OK; suite 572 passed /
7 skipped (570 + 2 new local tests).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@@ -206,9 +206,21 @@ class KnowledgePipeline:
|
|
| 206 |
Also the entry point for re-extracting a stored artifact, which needs no
|
| 207 |
parsing at all - prompt retuning is the common case and must not re-pay
|
| 208 |
the parser for a byte-identical result.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 209 |
"""
|
| 210 |
from src.knowledge_extraction.adapter import parsed_doc_from_artifact
|
| 211 |
|
|
|
|
|
|
|
|
|
|
| 212 |
doc = parsed_doc_from_artifact(artifact)
|
| 213 |
loop = asyncio.get_running_loop()
|
| 214 |
|
|
@@ -242,7 +254,7 @@ class KnowledgePipeline:
|
|
| 242 |
clustered,
|
| 243 |
filtered,
|
| 244 |
extractor,
|
| 245 |
-
active_glossary=active_glossary,
|
| 246 |
branches=BRANCHES,
|
| 247 |
),
|
| 248 |
)
|
|
|
|
| 206 |
Also the entry point for re-extracting a stored artifact, which needs no
|
| 207 |
parsing at all - prompt retuning is the common case and must not re-pay
|
| 208 |
the parser for a byte-identical result.
|
| 209 |
+
|
| 210 |
+
`active_glossary` defaults to **the tenant's approved entries,
|
| 211 |
+
fetched here**. Passing it was previously the caller's job and the
|
| 212 |
+
ingest path forgot, so every entry came back `diff_status="new"`: the
|
| 213 |
+
diff ran against an empty list, and a second document never showed as
|
| 214 |
+
duplicating or contradicting what an expert had already approved - which
|
| 215 |
+
is exactly the cross-document dedup that makes the queue finishable.
|
| 216 |
+
Fetching it at the single point that needs it means no caller can omit
|
| 217 |
+
it again. Pass an explicit `[]` to diff against nothing.
|
| 218 |
"""
|
| 219 |
from src.knowledge_extraction.adapter import parsed_doc_from_artifact
|
| 220 |
|
| 221 |
+
if active_glossary is None:
|
| 222 |
+
active_glossary = await self._store.approved_glossary(scope_id)
|
| 223 |
+
|
| 224 |
doc = parsed_doc_from_artifact(artifact)
|
| 225 |
loop = asyncio.get_running_loop()
|
| 226 |
|
|
|
|
| 254 |
clustered,
|
| 255 |
filtered,
|
| 256 |
extractor,
|
| 257 |
+
active_glossary=active_glossary or None,
|
| 258 |
branches=BRANCHES,
|
| 259 |
),
|
| 260 |
)
|
|
@@ -279,18 +279,17 @@ class IngestService:
|
|
| 279 |
|
| 280 |
result = job.results[0]
|
| 281 |
try:
|
| 282 |
-
# The approved set is
|
| 283 |
-
#
|
| 284 |
-
# paths
|
| 285 |
-
#
|
| 286 |
-
#
|
| 287 |
-
|
| 288 |
n, run_id = await self._pipeline.extract_and_persist(
|
| 289 |
document.artifact,
|
| 290 |
document.id,
|
| 291 |
scope_id=job.scope_id,
|
| 292 |
user_id=job.user_id,
|
| 293 |
-
active_glossary=active or None,
|
| 294 |
)
|
| 295 |
result.run_id = run_id
|
| 296 |
result.n_entries = n
|
|
|
|
| 279 |
|
| 280 |
result = job.results[0]
|
| 281 |
try:
|
| 282 |
+
# The approved set is fetched inside `extract_and_persist` for
|
| 283 |
+
# The approved set is fetched inside `extract_and_persist` for
|
| 284 |
+
# BOTH paths - it used to be passed here and forgotten on the
|
| 285 |
+
# ingest path, which silently disabled cross-document diffing.
|
| 286 |
+
# This is the edge that closes the loop: an expert's approvals
|
| 287 |
+
# become the baseline the next run classifies against.
|
| 288 |
n, run_id = await self._pipeline.extract_and_persist(
|
| 289 |
document.artifact,
|
| 290 |
document.id,
|
| 291 |
scope_id=job.scope_id,
|
| 292 |
user_id=job.user_id,
|
|
|
|
| 293 |
)
|
| 294 |
result.run_id = run_id
|
| 295 |
result.n_entries = n
|