Rifqi Hafizuddin Claude Opus 5 commited on
Commit
65f84bf
·
1 Parent(s): 840b74d

[NOTICKET] fix(knowledge): the diff baseline never reached the ingest path

Browse files

`run_document` called `extract_and_persist` WITHOUT `active_glossary`, so the
glossary diff ran against an empty list and every entry came back
`diff_status="new"`. Only the re-extract path passed the approved set.

The consequence is the thing the review queue exists to make tractable: a
company approves terms from document 1, ingests document 2, and document 2's
terms all read as `new` even where they duplicate or contradict what an expert
already ruled on. Cross-document dedup silently did not fire on the path people
will actually use most.

Invisible in the end-to-end run because there were no approvals yet - all 88
entries showed `new`, which was correct then and would be wrong now.

Fixed at the single point that needs it rather than at the call site:
`extract_and_persist` now fetches `approved_glossary(scope_id)` itself when the
argument is omitted, so no caller can forget it again, and the redundant fetch
in `_run_reextract` is gone. An explicit `[]` still means "diff against
nothing", which is what the tests use.

2 regression tests: the baseline is fetched when the argument is omitted, and an
explicit empty list does NOT trigger a fetch.

Verification: ruff clean on touched paths; `import main` OK; suite 572 passed /
7 skipped (570 + 2 new local tests).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

src/knowledge_ingest/pipeline.py CHANGED
@@ -206,9 +206,21 @@ class KnowledgePipeline:
206
  Also the entry point for re-extracting a stored artifact, which needs no
207
  parsing at all - prompt retuning is the common case and must not re-pay
208
  the parser for a byte-identical result.
 
 
 
 
 
 
 
 
 
209
  """
210
  from src.knowledge_extraction.adapter import parsed_doc_from_artifact
211
 
 
 
 
212
  doc = parsed_doc_from_artifact(artifact)
213
  loop = asyncio.get_running_loop()
214
 
@@ -242,7 +254,7 @@ class KnowledgePipeline:
242
  clustered,
243
  filtered,
244
  extractor,
245
- active_glossary=active_glossary,
246
  branches=BRANCHES,
247
  ),
248
  )
 
206
  Also the entry point for re-extracting a stored artifact, which needs no
207
  parsing at all - prompt retuning is the common case and must not re-pay
208
  the parser for a byte-identical result.
209
+
210
+ `active_glossary` defaults to **the tenant's approved entries,
211
+ fetched here**. Passing it was previously the caller's job and the
212
+ ingest path forgot, so every entry came back `diff_status="new"`: the
213
+ diff ran against an empty list, and a second document never showed as
214
+ duplicating or contradicting what an expert had already approved - which
215
+ is exactly the cross-document dedup that makes the queue finishable.
216
+ Fetching it at the single point that needs it means no caller can omit
217
+ it again. Pass an explicit `[]` to diff against nothing.
218
  """
219
  from src.knowledge_extraction.adapter import parsed_doc_from_artifact
220
 
221
+ if active_glossary is None:
222
+ active_glossary = await self._store.approved_glossary(scope_id)
223
+
224
  doc = parsed_doc_from_artifact(artifact)
225
  loop = asyncio.get_running_loop()
226
 
 
254
  clustered,
255
  filtered,
256
  extractor,
257
+ active_glossary=active_glossary or None,
258
  branches=BRANCHES,
259
  ),
260
  )
src/knowledge_ingest/service.py CHANGED
@@ -279,18 +279,17 @@ class IngestService:
279
 
280
  result = job.results[0]
281
  try:
282
- # The approved set is passed explicitly rather than read from
283
- # wherever the last run wrote, so the duplicate and conflicting
284
- # paths are actually reachable. This is the edge that closes the
285
- # loop: an expert's approvals become the baseline this run
286
- # classifies its candidates against.
287
- active = await self._knowledge.approved_glossary(job.scope_id)
288
  n, run_id = await self._pipeline.extract_and_persist(
289
  document.artifact,
290
  document.id,
291
  scope_id=job.scope_id,
292
  user_id=job.user_id,
293
- active_glossary=active or None,
294
  )
295
  result.run_id = run_id
296
  result.n_entries = n
 
279
 
280
  result = job.results[0]
281
  try:
282
+ # The approved set is fetched inside `extract_and_persist` for
283
+ # The approved set is fetched inside `extract_and_persist` for
284
+ # BOTH paths - it used to be passed here and forgotten on the
285
+ # ingest path, which silently disabled cross-document diffing.
286
+ # This is the edge that closes the loop: an expert's approvals
287
+ # become the baseline the next run classifies against.
288
  n, run_id = await self._pipeline.extract_and_persist(
289
  document.artifact,
290
  document.id,
291
  scope_id=job.scope_id,
292
  user_id=job.user_id,
 
293
  )
294
  result.run_id = run_id
295
  result.n_entries = n