Spaces:
Sleeping
Sleeping
Commit History
-
rename package doc_agent -> docfield, distribution -> docfield-extract
The last identifier still carrying the old framing. The project is not an agent and the README says so; the import path said otherwise on every line of every module. Moved with git mv so history follows all 26 files, and kept as a package directory rather than flattened -- removing it would break the src layout the Space entry point depends on. pyproject needs [tool.uv.build-backend] module-name = "docfield". The uv build backend derives the module name from the project name, so a distribution called docfield-extract would otherwise look for src/docfield_extract/ and fail to find the package. Distribution name and import name differ on purpose: docfield-extract matches the repo, docfield is what you type. DB_PATH default moves from ./data/agent.db to ./data/docfield.db. This orphans any existing local database -- a watcher run against the new default starts an empty store and the old file is simply ignored, not migrated. There was no .db file in this working tree, so nothing was act…b6beb2d -
eval: tick F12, correct FC-1's scope, document FC-3
F12 is done: all 44 quota-failed documents re-predicted, 0 errors, the 317 successful predictions verified byte-identical. FC-1 claimed to be "the single document that separates the two figures". That was true of the partial 217-document held-out slice it was written against, and stopped being true when the backfill completed the slice to 261 and surfaced a second false accept. Corrected, with the superseded claim named rather than quietly edited out. FC-3 records that second one, X51007846355. It matters less as a defect than as a boundary: the line items sum to 8.96, the subtotal is 8.96, tax is 0.00 and the total is 8.96. Every figure agrees with every other figure exactly, not within a tolerance. No cross-check over the extracted values can separate it from a correct document, because arithmetic checks detect internal inconsistency and a consistently-misread document is internally consistent. That is the ceiling of the approach, not a rule someone forgot to write. Catching it nee…462604d
-
eval: add --retry-errors to re-predict only the entries that failed
A quota outage left 44 of 361 cached SROIE documents with no extraction. Recovering them with --overwrite would have re-run all 361 and replaced the 317 successful predictions with fresh model output, invalidating every before/after rule comparison built on them -- S2 and the monetary tolerance change both depend on the predictions being frozen while only the rules vary. --retry-errors selects exactly the entries whose error field is set and skips every other cached id, so a successful prediction is never re-run, re-billed, or rewritten. Two guards, because both failure modes here spend quota silently: combining it with --overwrite raises rather than letting one win, and using it against an empty cache raises rather than falling through to a full predict. Note this leaves a cache that is mixed by construction: the backfilled 44 carry confidence and validation computed under current rules, the original 317 under the rules in force when they ran. That is only a disk-level inconsistency…fa6cf50 -
validation: monetary tolerance scales with rounded terms, not with value
H2 and H3 gate auto-acceptance, and their tolerance was max(0.02, 0.005 * amount) -- 0.5% of the document value. That is the same defect the README records fixing on the measurement side, still live on the decision side, where the consequence is a wrong total being written rather than a metric being off. At 500 it permitted a 2.50 discrepancy; at 100,000 it permitted 500. It is also mis-specified against its own stated intent. The old comment said large invoices should "tolerate the accumulated rounding of many line items", which is a fair thing to want, but rounding accumulates with the *number* of figures summed, not their magnitude. A 100,000 invoice with two lines has no more rounding in it than a 100 one. Replace the relative term with a count-based one: 0.02 plus half a cent per additional independently-rounded quantity. H2 compares three stated figures and takes the floor alone. H3 allows per line item summed. S4 allows per unit, since a unit price rounded to the cent carries …6eb59bb -
eval: correct FC-1's corpus claim, log FC-2, note cent-exact severity
FC-1 offered a gold-label ambiguity as "the likeliest explanation" for a wrong auto-accepted total. That hypothesis was then tested across the corpus and is not supported: of 317 usable documents only 3 disagree with gold, only one fits the pre-tax signature, and no document shows the reverse pattern. The reading now stands as an account of that one document and nothing more. A findings file that raises a hypothesis and never reports the test result is worse than one that never raised it. FC-2 records what the FC-1 investigation actually turned up, which is larger than FC-1. The 0.5% relative tolerance the README describes fixing on the measurement side is still live in H2/H3, where it gates acceptance. It is also mis-specified against its own stated intent: accumulated rounding scales with the number of line items, not with document value, so a 100,000 invoice with two lines receives 500 of tolerance. On this cache the term is the operative tolerance for 92.7% of documents and 5 of 8…dcab1e8 -
eval: document failure case FC-1 and log its open questions
X51005806696 was auto-accepted on the held-out slice with total 7.65 against a gold total of 7.20, clearing all four hard rules and all four soft rules. It is the single document separating 100% auto-accept precision on the tuning slice from 98.4% on held-out, so it deserves a written record rather than a line in a commit message. The first hypothesis about the mechanism was wrong, and the write-up says so. H2 was assumed to have cleared on the MONETARY_ABS_EPSILON boundary, since 7.20 + 0.43 = 7.63 against 7.65 is a residual of exactly 0.02 and the floor is 0.02. But money_close applies max(absolute, relative), and the relative term here is 0.005 * 7.65 = 0.03825. The relative term binds; the residual passed with 0.018 to spare. Two consequences: tightening the absolute epsilon would not have caught this document, and under an absolute-only rule it would in fact have failed -- by 4.6e-16, a float artifact, since the exact-decimal residual is precisely on the boundary while the IEEE 7…1b33ac9 -
eval: report documents that never reached the model, grouped by cause
44 of the 361 cached SROIE documents have no extraction at all: the predict run hit a Gemini monthly spend cap and returned 429 RESOURCE_EXHAUSTED. That is an operational fact about the run, not a property of the extractor, but the report gave a reader no way to see it -- it surfaced only as an unexplained recall gap (total recall 82.4% held-out against 99.1% on the documents that actually reached the model). The asymmetry is the trap: an empty record predicts nothing, so it counts as a miss in recall while leaving precision untouched. Summing infrastructure failures with model failures therefore understates recall specifically, and looks exactly like a model that got worse. Add a Document outcomes block stating how many documents reached the model, how many did not, and why, with causes classified into infrastructure, input, configuration, and model buckets. An unrecognised failure keeps its own message rather than being bucketed as "other", so a new failure mode is visible the firs…3907a18 -
eval: add tuning/held-out split selection to the score command
Expanding the SROIE cache from 100 to the full 361-document test split makes the distinction load-bearing: the original 100 are what the threshold was fitted on and what the S2 soft-rule mismatch was diagnosed from, so metrics over them are in-sample. Only the remaining 261 can support an independent claim. Reporting a combined number would quietly launder the contamination. Membership is by id, not by position. read_entries returns files in glob order, which has no guaranteed relationship to the order the dataset streamed them in, so "the first 100 cached" is not reliably the 100 that were tuned on -- and would silently reassign documents when the cache is regenerated on another machine or grows. eval/splits/sroie_tuning.json pins the exact ids, captured before the cache was expanded, since afterwards the original 100 are indistinguishable from the new ones. The manifest is committed even though eval/cache/ is git-ignored. The cache can be regenerated from the dataset; the record of…b8930b3 -
eval: score under current rules with --revalidate, and detect cache drift
The predict phase freezes both the confidence scalar and the validation report into every cache entry. The score phase replayed only route() over those frozen values, so any change to a soft rule in validation/rules.py or to the weights in routing/score.py was invisible to a re-scored report: the cache silently went stale against the codebase and still printed confident-looking numbers describing a rule set no longer in the code. Close it by recomputing from the cached predicted document, which predict already stores in full. Validation and scoring are pure functions of that document, so both recompute exactly, offline, for zero API calls. - eval/revalidate.py: reconstitute Document -> validate -> score, plus drift detection comparing recomputed against frozen values. - score --revalidate scores under current rules. Every report header now states which rule set produced its numbers, so cached-scalar and current-rule figures can never be confused six months from now. - Drift is …f1a9398 -
docs: rename to DocField Extract, reframe as key-value extraction pipeline
Repo renamed to docfield-extract. Three framing corrections, because the old README described a system this is not: - Drop "agent" language. There is no tool loop and no autonomous decision-making; the LLM output is data, never control flow. It is a modality router, one schema-constrained extraction call, and rule-based validation. The "Where's the agent?" section is gone. - Not a receipt parser. SROIE is the benchmark, not the scope; the schema is general business-document fields. - Position in the IDP taxonomy as key-value extraction, distinct from table extraction. line_items is populated opportunistically and carries no ground truth, so it is not a table capability. Lead with the evaluation harness rather than burying it, and state its limitations where a reader will hit them: only 4 fields labeled, tax and invoice_number unscored, line items unscored, 100% precision is over 18 values, the operating point was chosen on the slice it is reported on, confidence scoring i…14990f1
-
docs: correct stale money-comparison docstring in eval.normalize
values_match compares money cent-exact (round to 2dp, no relative tolerance); the docstring still said 'within the rounding tolerance'. Comment only.a1aecdd
-
phase 5: evaluation harness (SROIE)
Two-phase harness so inference runs exactly once and tuning is free: - predict phase (eval/predict.py): runs process_document over a dataset slice, saves PIL images to temp files, caches gold + predicted Document + confidence + validation report to eval/cache/ keyed by id. The only phase that hits the model; idempotent (cached ids skipped) so re-runs never re-bill. - score phase (eval/score.py + eval/metrics.py): loads the cache and computes per-field and per-critical-field precision/recall/F1, the auto-accept critical precision, and a threshold sweep (0.50-0.99) that replays the real route() over cached (confidence, report), honoring the hard-failure override. Fully offline. Datasets: SROIE (jsdnrs/ICDAR2019-SROIE test) wired end-to-end; CORD and an invoice-JSON set scaffolded as adapters but not wired (load() raises, absent from WIRED_DATASETS). Only dataset-labeled fields are scored. Comparison (eval/normalize.py) reuses the schema's number/date coercion, then compares…d2a6765