File size: 2,316 Bytes
3b3b2bd | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | # Evaluation interpretation
Read the measured-results section of the root model card and the raw JSON together. The known development suite, fresh authored declared-topic suite, outside-vocabulary controls, relational path suite, numerical checks and randomized graph oracle test different mechanisms. None is a public memory leaderboard result.
`reports/first_run_evidence_evaluation.json` and its implementation freeze preserve the initial post-implementation measurements. Later changes hardened validation, persistence contracts and cache behavior; the final rerun is a regression result, not a new blinded holdout. The weights and topic lexicon were not fitted after observing fresh test failures.
Files:
- `evidence_fixtures.json`: exact texts, supplied claim metadata, expected evidence and statuses.
- `evidence_evaluation.json`: every selection, status, answer and latency for all compared mechanisms.
- `evidence_stress.json`: randomized graph outputs, numerical parity, certificate tampering and raw scale timings.
- `pre_cache_evidence_stress.json`: earlier uncached graph measurement retained for provenance.
- `pytest.xml` / `pytest.log`: regression results.
- `http_smoke.json`: actual localhost process, authenticated API and restart evidence.
- `rc1_baseline/`: original 1.0 reports, explicitly historical.
The 60 declared-topic cases intentionally favor a test of lexical gap handling, not general query distribution. BM25 and the old hybrid do not have the new authored topic expansion. Linear utility is the proper control for whether serializing those utilities as neural weights itself improves quality. It ties the neural result. Path zero-score also ties: typed graph constraints, not neural intuition, provide correctness.
The cache scale test has one cold read and 39 warm reads, no interleaved mutations. The graph cache consumes additional Python memory; the checkpoint byte count does not include it. The cold reading cost remains important for write-heavy workloads.
The official LongMemEval runner is retrieval-only and requires a separately obtained non-oracle JSON file. It strips labels before retrieval and only uses gold session IDs for scoring afterward. No real benchmark data or host-model generation ran here. Assertions of frontier or SOTA performance are unsupported.
|